2024-09-11 17:14:46.494 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:14:46.497 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:14:46.497198816+08:00 2024-09-11 17:14:46.498 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:14:46.497+08:00 2024-09-11 17:14:46.499 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:14:46.500 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:14:46.499863008+08:00 2024-09-11 17:14:46.500 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:14:46.497+08:00 and status AVAILABLE 2024-09-11 17:14:46.500 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:14:46.500+08:00 2024-09-11 17:14:46.502 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:14:46.500+08:00 and status AVAILABLE 2024-09-11 17:14:46.504 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:46.509 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:14:46.510 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:46.512 INFO 2981023 --- [host-check-agent-status4,964d073e0bfffce5,2dd05d4e1792684c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:14:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:14:46.517 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:14:46.517 INFO 2981023 --- [host-check-agent-status4,964d073e0bfffce5,2dd05d4e1792684c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:14:46.517 INFO 2981023 --- [host-check-agent-status4,964d073e0bfffce5,2dd05d4e1792684c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:14:46.518 INFO 2981023 --- [host-check-agent-status4,964d073e0bfffce5,2dd05d4e1792684c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:14:46.537 INFO 2981023 --- [ocp-schedules-19,28729377afba161e,844898e7c11ada9c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:14:46.677 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:14:46.678 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:14:46.681 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:14:46.678390360+08:00 2024-09-11 17:14:46.681 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:14:46.683 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:14:46.681+08:00 2024-09-11 17:14:46.685 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:14:46.685 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:14:46.687 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:14:46.681+08:00 and status AVAILABLE 2024-09-11 17:14:46.786 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:14:46.800 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:14:46.808 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:14:46.950 INFO 2981023 --- [host-check-agent-status3,964d073e0bfffce5,e3f987762b7c4d43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:14:46.954 INFO 2981023 --- [host-check-agent-status3,964d073e0bfffce5,e3f987762b7c4d43] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:14:46.954 INFO 2981023 --- [host-check-agent-status3,964d073e0bfffce5,e3f987762b7c4d43] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:14:46.954 INFO 2981023 --- [host-check-agent-status3,964d073e0bfffce5,e3f987762b7c4d43] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:14:46.971 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:14:46.974 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:14:46.979 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:14:46.979 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:46.987 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:46.987 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:14:46.989 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:46.990 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:14:46.993 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:14:46.997 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:14:46.998 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:14:46.999 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.000 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:14:47.000 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:14:47.001 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.007 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:14:47.010 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:14:47.010 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:14:47.022 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.024 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.024 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:14:47.039 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:14:47.039 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:14:47.039 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:14:47.042 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:14:47.052 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:14:47.057 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:14:47.075 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.076 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.076 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:14:47.084 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:14:47.090 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.091 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.091 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:14:47.092 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:14:47.092 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.095 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:14:47.097 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:14:47.101 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:14:47.102 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:14:47.110 INFO 2981023 --- [ocp-async-3,e744b63972b3ab6e,bdf9c4ae66ff9266] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:47.115 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:14:47.116 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.117 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:14:47.117 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:14:47.118 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.118 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:14:47.121 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.121 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.121 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:14:47.121 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.122 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:14:47.128 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:14:47.129 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:14:46.339035+08:00 2024-09-11 17:14:47.131 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:14:47.134 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:14:47.134 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.135 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:14:47.136 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:14:47.136 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:14:47.138 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:14:47.139 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:14:47.142 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.146 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.146 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:14:47.146 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:14:47.147 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:14:47.153 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:14:47.159 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:14:47.159 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:14:47.159 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:14:47.166 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:14:47.172 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:14:47.181 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:14:47.183 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:14:47.187 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:14:47.187 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:14:47.188 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:14:47.196 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:14:47.197 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:14:47.198 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.199 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.199 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:14:47.200 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:14:47.206 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:14:47.208 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:14:47.211 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:14:47.212 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:14:47.212 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:14:47.214 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:14:47.224 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.224 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.225 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:14:47.225 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:14:47.225 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:14:47.226 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.226 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:14:47.228 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:14:47.230 INFO 2981023 --- [host-check-host-status1,6ed2e0f5a6396c78,505c157c485adfa4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:14:47.231 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:14:47.236 INFO 2981023 --- [host-check-host-status2,6ed2e0f5a6396c78,ac349565089ab01c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:14:47.237 INFO 2981023 --- [host-check-host-status3,6ed2e0f5a6396c78,a9e3437023b1ce87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:14:47.259 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:14:47.291 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:14:47.294 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.295 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.303 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:14:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,c8aa2d501a71e0b8,afce71eec56b17b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:47.314 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:14:47.315 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:14:47.315 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:14:47.319 INFO 2981023 --- [prometheus-minute-schedule-1,c8aa2d501a71e0b8,afce71eec56b17b3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:14:47.331 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:14:47.343 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:14:47.361 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:14:47.369 INFO 2981023 --- [ocp-schedules-8,2994132cd7af7e79,62e0cbec57a67111] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:14:47.399 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:14:47.410 INFO 2981023 --- [host-check-clockdiff2,078af027df141134,5fd2e71c63db7090] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046087 0 0 , error= 2024-09-11 17:14:47.411 INFO 2981023 --- [alarm-detect-15,f33ba2abf0d7eb60,db86ebe625a4fb5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643345, data is empty. 2024-09-11 17:14:47.420 INFO 2981023 --- [alarm-detect-15,f33ba2abf0d7eb60,db86ebe625a4fb5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643345, data is empty. 2024-09-11 17:14:47.426 INFO 2981023 --- [alarm-detect-15,f33ba2abf0d7eb60,db86ebe625a4fb5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643345, data is empty. 2024-09-11 17:14:47.436 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:14:47.437 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:47.439 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:47.445 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:14:47.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:14:47.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:14:47.469 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:14:47.479 INFO 2981023 --- [host-check-host-status4,6ed2e0f5a6396c78,6700b310fd882a17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:14:47.481 INFO 2981023 --- [ocp-schedules-4,5841680a06b1154e,1bf0ecc97a7b41c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:14:47.730 INFO 2981023 --- [ocp-async-4,89c2727f785a9914,52565d1842ce4f8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:14:49.110 INFO 2981023 --- [ocp-async-8,ef87bddf6bae9c39,3edfc0d45a3d2de0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:49.455 INFO 2981023 --- [ocp-schedules-23,5e359e78fe1d5a70,810f31dbcee14ec0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:14:49.455+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:14:49.458 INFO 2981023 --- [ocp-schedules-23,5e359e78fe1d5a70,810f31dbcee14ec0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:14:49.497 INFO 2981023 --- [ocp-schedules-23,5e359e78fe1d5a70,810f31dbcee14ec0] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 51 to SqlAggregateData 2024-09-11 17:14:49.498 INFO 2981023 --- [ocp-schedules-23,5e359e78fe1d5a70,810f31dbcee14ec0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:14:49.510 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:49.512 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:49.685 INFO 2981023 --- [ocp-schedules-4,f446c33648800ae1,19473d2c7ff8c43b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:14:49.686 INFO 2981023 --- [ocp-schedules-4,f446c33648800ae1,19473d2c7ff8c43b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:14:49.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.724 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.724 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.725 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.727 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.728 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.728 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:14:49.732 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.732 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.732 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.732 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.739 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.740 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.745 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.745 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.747 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.747 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:14:49.749 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.750 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.755 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.756 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:14:49.758 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.758 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.759 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:14:49.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1241D724-0-0] 2024-09-11 17:14:49.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:14:49.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.760 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.761 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.761 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.766 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.766 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.769 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.769 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.783 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.783 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.784 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.785 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.785 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.786 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.786 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.787 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1241D72E-0-0] 2024-09-11 17:14:49.788 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:14:49.788 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.788 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.789 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.789 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.790 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.792 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.792 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:14:49.792 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.793 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.796 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.796 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.797 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.798 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.798 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.798 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.799 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.799 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.800 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.800 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:14:49.801 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.802 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:14:49.803 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.803 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.803 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:14:49.804 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:14:49.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:14:49.806 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:14:49.807 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:14:49.815 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:49.818 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.818 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.823 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:49.825 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:14:49.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:14:49.828 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.828 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.831 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.832 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.835 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:14:49.835 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:14:49.835 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:14:49.835 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:14:49.836 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.836 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:14:49.839 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.841 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.843 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:14:49.844 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.844 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:14:49.844 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.844 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:49.845 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:14:49.851 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:14:49.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:14:49.853 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:14:49.856 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:14:49.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:14:49.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:14:49.857 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:14:49.857 INFO 2981023 --- [ocp-schedules-4,f446c33648800ae1,19473d2c7ff8c43b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:14:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:14:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:14:50.163 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.168 INFO 2981023 --- [ocp-schedules-12,c3c37fd1ef5ca1a4,7280d443298cb047] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:14:50.170 INFO 2981023 --- [ocp-schedules-21,7c6be46466fe6bac,6264d9ad62e79b96] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:14:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,53c5f12530e3e05d,b9c6a909e1bc8d85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:50.315 INFO 2981023 --- [prometheus-minute-schedule-1,53c5f12530e3e05d,b9c6a909e1bc8d85] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:14:51.110 INFO 2981023 --- [ocp-async-6,fa7a54c78f2ca956,b5d15864890e4da9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:51.479 INFO 2981023 --- [ocp-schedules-14,0b87dcf55cf8bcf2,61300e13df50ba25] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:14:51.480 INFO 2981023 --- [ocp-schedules-14,0b87dcf55cf8bcf2,61300e13df50ba25] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:14:51.480 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:14:51.480 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:14:51.481 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:14:51.481 INFO 2981023 --- [ocp-schedules-14,0b87dcf55cf8bcf2,61300e13df50ba25] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:14:51.546 INFO 2981023 --- [ocp-schedules-1,4e38c0db479d564b,ae883d0240cf2e7e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:14:51.630 INFO 2981023 --- [ocp-schedules-12,f062efd065cba489,aa436896fbd57475] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:14:51.647 WARN 2981023 --- [ocp-schedules-12,f062efd065cba489,aa436896fbd57475] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:14:52.223 INFO 2981023 --- [host-check-clockdiff3,078af027df141134,4698839c495fd1e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046092 1 1 , error= 2024-09-11 17:14:52.512 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:52.514 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:52.953 INFO 2981023 --- [ocp-schedules-8,cd2cbd297128d6f7,ad15d32ea7eddc97] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:14:52.953 INFO 2981023 --- [ocp-schedules-8,cd2cbd297128d6f7,ad15d32ea7eddc97] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:14:53.103 INFO 2981023 --- [ocp-schedules-12,71bbfe108ed694fb,d44453db9a314648] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:14:53.110 INFO 2981023 --- [ocp-async-2,5ca5f44ba560a162,006104bb5aee8769] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:53.112 INFO 2981023 --- [ocp-schedules-12,71bbfe108ed694fb,d44453db9a314648] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:14:53.113 INFO 2981023 --- [ocp-schedules-12,71bbfe108ed694fb,d44453db9a314648] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:14:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,7a64724a98c76032,1e473cc0a676d656] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,7a64724a98c76032,1e473cc0a676d656] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:14:54.104 INFO 2981023 --- [ocp-schedules-28,99ef3b92f9ae72b2,07eaa6c60db4d656] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:14:55.111 INFO 2981023 --- [ocp-async-1,b9baa55b907379a6,96160a52161058cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:55.131 INFO 2981023 --- [ocp-schedules-21,218d66244d094389,64e5962e23d0ebcd] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:14:55.131 INFO 2981023 --- [ocp-schedules-21,218d66244d094389,64e5962e23d0ebcd] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:14:55.147 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:14:55.148 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:14:55.148 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:14:55.149 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:14:55.150 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:14:55.151 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:14:55.164 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.165 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.165 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.165 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:14:55.166 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:14:55.167 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:14:55.169 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:14:55.169 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:14:55.169 INFO 2981023 --- [Collect all tenants major compactions1,218d66244d094389,d67306f7b9ca0140] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:14:55.169 INFO 2981023 --- [Collect all tenants major compactions3,218d66244d094389,5af453d0df686cbe] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:14:55.170 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:14:55.170 INFO 2981023 --- [Collect all tenants major compactions2,218d66244d094389,cd10a8c694ee6468] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:14:55.171 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.171 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.172 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:14:55.172 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:14:55.173 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions5,218d66244d094389,c572474a344f1065] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:14:55.174 INFO 2981023 --- [Collect all tenants major compactions4,218d66244d094389,e4b730862a5181e2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:14:55.176 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:14:55.176 INFO 2981023 --- [Collect all tenants major compactions6,218d66244d094389,55ea8eb40c986b13] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:14:55.177 INFO 2981023 --- [ocp-schedules-21,218d66244d094389,64e5962e23d0ebcd] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:14:55.514 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:55.515 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:55.585 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:14:55.586 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:14:55.586 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:14:55.586 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:14:55.587 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:14:55.587 INFO 2981023 --- [ocp-schedules-29,4da27ba930213901,86c87d5eaec678a8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:14:55.605 INFO 2981023 --- [ocp-schedules-28,7b8ef5b65f2806a8,005d16b026a4b16d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:14:55.612 INFO 2981023 --- [ocp-schedules-28,7b8ef5b65f2806a8,005d16b026a4b16d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:14:55.612 INFO 2981023 --- [ocp-schedules-28,7b8ef5b65f2806a8,005d16b026a4b16d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:14:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,e1026f94f8058551,654d0eb81e0b3efd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,e1026f94f8058551,654d0eb81e0b3efd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:14:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:14:56.350607+08:00, lastProcessTime=2024-09-11T17:14:46.339035+08:00 2024-09-11 17:14:56.500 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:14:56.350607+08:00 2024-09-11 17:14:57.111 INFO 2981023 --- [ocp-async-9,3210ee291f3c716f,54656650a8cf1910] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:57.504 INFO 2981023 --- [ocp-schedules-16,312c9f9b6e51fdee,9576f8efc7ff628b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:14:57.564 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56} 2024-09-11 17:14:57.566 WARN 2981023 --- [scanner-13446614-1,312c9f9b6e51fdee,f35d1816ea6c479b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:14:57.569 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkpr 2024-09-11 17:14:57.570 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.577 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.577 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.577 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.579 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.587 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.587 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.587 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.615 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:14:57.615 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.620 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.620 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.620 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.673 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:14:57.719 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697719000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.719Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.723 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:14:57.723 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.728 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.728 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.730 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.730 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.730 INFO 2981023 --- [ocp-async-5,a3af490ecca20ef8,18ffeb84efdbd9c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:14:57.735 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.735 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.736 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:14:57.736 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.719, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.736 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.737 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.737 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.737 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.738 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.738 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.738 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.738 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.739 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:14:57.741 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:10:57.209, end=2024-09-11T17:12:57.56},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:14:57.741 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:14:57.741 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:14:57.782 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697782000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.782Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.782 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697782000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.782Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.785 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697785000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.785Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.791 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697791000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.791Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.792 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697792000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.792Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.793 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697793000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.793Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.796 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 56 2024-09-11 17:14:57.796 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.802 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697802000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.802Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.803 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697803000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.803Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.807 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:14:57.807 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.810 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.810 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.811 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.811 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.816 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:14:57.816 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.819 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.819 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.822 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.822 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.824 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:14:57.824 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.824 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.824 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.827 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 227 2024-09-11 17:14:57.827 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.828 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.828 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.829 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.829 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.830 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.830 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.831 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.831 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.832 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.832 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.834 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.834 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.837 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.837 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.839 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.841 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.841 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.842 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.842 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.844 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.844 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.844 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.844 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.845 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.845 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.845 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.846 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.846 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.847 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.848 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.782, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.848 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.848 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.851 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.851 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.851 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.851 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.852 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.785, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.852 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.852 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.853 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.782, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.853 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.854 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.854 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.861 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.791, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.864 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.864 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.865 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.865 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.867 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.867 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.872 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.872 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.873 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.792, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.875 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.793, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.875 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.875 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.876 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.878 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.880 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697880000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.880Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.883 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697883000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.883Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.885 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.802, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.888 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.803, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.890 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697890000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.890Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.897 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.897 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.900 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:14:57.900 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.903 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.903 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.911 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697911000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.911Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.912 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.912 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.913 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.913 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.914 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.914 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.917 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.917 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.923 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.923 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.924 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.924 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.924 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697924000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=C0995E24A01F869BEB311E8C790E6C86, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.924Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.926 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697926000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.926Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.928 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.928 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.935 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.935 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.936 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.936 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.936 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.936 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.937 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.937 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959697937000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:14:57.937Z, endTime=2024-09-11T09:12:57.560Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:14:57.939 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.940 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.88, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.941 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.941 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.942 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.942 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.943 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.945 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.945 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.947 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.883, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.948 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.948 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.949 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:14:57.949 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:14:57.950 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.89, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.950 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.950 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.951 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.951 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.952 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.952 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.955 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.955 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.957 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.958 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.958 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.960 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.911, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.960 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:14:57.960 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:14:57.961 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.961 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.961 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.961 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.962 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.962 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.965 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.924, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=C0995E24A01F869BEB311E8C790E6C86, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.965 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.926, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.967 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:14:57.967 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726045977560000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:14:57.968 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:14:57.970 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:14:57.937, end=2024-09-11T17:12:57.56}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:14:57.971 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:14:58.175 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,ad45983778f83f93,17b76a97476314f9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=ad45983778f83f93, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:14:58.197 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,ad45983778f83f93,17b76a97476314f9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=ad45983778f83f93, duration=27 ms] 2024-09-11 17:14:58.515 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:14:58.515 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:14:59.111 INFO 2981023 --- [ocp-async-3,cedc6836e45155b9,c0e36afdd7c4db26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:14:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,14262f274542b7d5,4e4e425e65e3e9bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046100, interval=60 2024-09-11 17:14:59.315 INFO 2981023 --- [prometheus-minute-schedule-1,14262f274542b7d5,4e4e425e65e3e9bb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:00.456 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:10:01.0, nextRun=2024-09-11 17:15:00.0, currentDate=Wed Sep 11 17:15:00 HKT 2024 2024-09-11 17:15:00.456 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:10:01.0, nextRunTime=2024-09-11 17:15:00.0} 2024-09-11 17:15:00.460 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:15:00.460 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:15:00.460 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:15:00.460+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:15:00.472 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:10:01.0, nextRun=2024-09-11 17:15:00.0, currentDate=Wed Sep 11 17:15:00 HKT 2024 2024-09-11 17:15:00.472 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:10:01.0, nextRunTime=2024-09-11 17:15:00.0} 2024-09-11 17:15:00.485 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 17:15:00.491 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:15:00.492 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:15:00.492 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:15:00.492 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,e38d8e56d5ec6068] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T17:15:00.492+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:15:00.525 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,aec0913d1c45dc7a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048155, operation:EXECUTE, state: RUNNING 2024-09-11 17:15:00.535 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,aec0913d1c45dc7a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048156, operation:EXECUTE, state: RUNNING 2024-09-11 17:15:00.545 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048155, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048175, task_operation=execute, latest_execution_start_time=2024-09-11T17:15:00.518+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048155}, listMap={}}, executor=10.31.35.10 2024-09-11 17:15:00.554 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048156, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048176, task_operation=execute, latest_execution_start_time=2024-09-11T17:15:00.534+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048156}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 17:15:00.556 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:15:00.561 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.575 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:15:00.578 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:15:00.593 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:15:00.594 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:15:00.594 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:15:00.610 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:15:00.611 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.615 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:15:00.627 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,93566f0156bf2e20] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048156, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:15:00.687 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.702 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.722 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.735 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:15:00.736 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:15:00.740 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.823 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048176, status:SUCCESS 2024-09-11 17:15:00.826 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048176, status:SUCCESS 2024-09-11 17:15:00.859 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.912 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:15:00.931 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:00.946 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.010 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.020 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.047 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.054 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:15:01.054 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:15:01.060 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.115 INFO 2981023 --- [ocp-async-1,44177221a3fb30a8,fa4aeecef0ffd5c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:01.456 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.496 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:15:01.509 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.515 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:01.516 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:01.519 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.531 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.537 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.543 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.545 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:15:01.546 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.549 INFO 2981023 --- [ocp-schedules-7,830a0bebad6e5dbe,9e228a5b362139b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:01.566 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.575 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:15:01.586 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,ee0076e3ebd96ca0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048155, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:15:01.838 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048175, status:SUCCESS 2024-09-11 17:15:01.839 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048175, status:SUCCESS 2024-09-11 17:15:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,e6781b2a7b9825fd,a899b01b6e634728] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:02.317 INFO 2981023 --- [prometheus-minute-schedule-1,e6781b2a7b9825fd,a899b01b6e634728] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:03.111 INFO 2981023 --- [ocp-async-6,c677a03b13280396,8fa19eaf551947bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:04.516 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:04.518 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:05.111 INFO 2981023 --- [ocp-async-10,3579462b9c823148,4887d245ff753672] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,5ef39fffa7272850,7281d061c0d015bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,5ef39fffa7272850,7281d061c0d015bf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:06.351739+08:00, lastProcessTime=2024-09-11T17:14:56.350607+08:00 2024-09-11 17:15:06.494 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:06.351739+08:00 2024-09-11 17:15:06.985 INFO 2981023 --- [ocp-schedules-21,87fc260bc103b8ab,4efc8c5eb66ed096] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:15:06.996 INFO 2981023 --- [ocp-schedules-28,b9584813af8d6c83,95ed595e18200ded] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:15:06.996+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:15:06.997 INFO 2981023 --- [ocp-schedules-28,b9584813af8d6c83,95ed595e18200ded] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:15:06.998 INFO 2981023 --- [ocp-schedules-28,b9584813af8d6c83,95ed595e18200ded] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:15:06.999 INFO 2981023 --- [ocp-schedules-28,b9584813af8d6c83,95ed595e18200ded] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:15:07.111 INFO 2981023 --- [ocp-async-3,e6ad6fa7b719bb84,8fc1349d6e374129] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:07.518 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:07.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:07.725 INFO 2981023 --- [ocp-schedules-11,b5e469a1e8677010,eeae6a17f80b81bf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:07.731 INFO 2981023 --- [ocp-async-7,d426046d35e686c5,83ff9411592bb425] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:07.763 WARN 2981023 --- [scanner-13447013-1,b5e469a1e8677010,f97da4213289fcef] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:07.807 INFO 2981023 --- [alarm-detect-1,b5e469a1e8677010,5cfc9cd61859382b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643345, data is empty. 2024-09-11 17:15:07.815 INFO 2981023 --- [alarm-detect-1,b5e469a1e8677010,5cfc9cd61859382b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:07.821 INFO 2981023 --- [alarm-detect-1,b5e469a1e8677010,5cfc9cd61859382b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:07.838 INFO 2981023 --- [alarm-detect-6,b5e469a1e8677010,dc8fa961b7001f90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643345, data is empty. 2024-09-11 17:15:07.838 INFO 2981023 --- [alarm-detect-7,b5e469a1e8677010,986ee8e10dc1a7a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643345, data is empty. 2024-09-11 17:15:07.842 INFO 2981023 --- [alarm-detect-7,b5e469a1e8677010,986ee8e10dc1a7a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:07.845 INFO 2981023 --- [alarm-detect-6,b5e469a1e8677010,dc8fa961b7001f90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:07.847 INFO 2981023 --- [alarm-detect-7,b5e469a1e8677010,986ee8e10dc1a7a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:07.850 INFO 2981023 --- [alarm-detect-6,b5e469a1e8677010,dc8fa961b7001f90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:07.964 INFO 2981023 --- [alarm-detect-14,b5e469a1e8677010,eba7fddf796a6d6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643345, data is empty. 2024-09-11 17:15:07.970 INFO 2981023 --- [alarm-detect-14,b5e469a1e8677010,eba7fddf796a6d6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:07.975 INFO 2981023 --- [alarm-detect-14,b5e469a1e8677010,eba7fddf796a6d6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,18e782c41e3749fe,0554d29e7ae5eabc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,18e782c41e3749fe,0554d29e7ae5eabc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:09.112 INFO 2981023 --- [ocp-async-5,9102f5ef663f3075,b4c0e7e490767d71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:10.003 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:15:10.003 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:15:10.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:10.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:15:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:15:11.112 INFO 2981023 --- [ocp-async-4,6f4c25ede2095d66,fa2d41399b9c43f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,6acc61569a433d0b,df2c68c31b3a0f19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,6acc61569a433d0b,df2c68c31b3a0f19] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:11.556 INFO 2981023 --- [ocp-schedules-4,49680e8223521139,ff8f7e0bdd5d25a6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:13.112 INFO 2981023 --- [ocp-async-7,d46837ead8d239dc,8e9df53226ad02e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:13.116 INFO 2981023 --- [ocp-schedules-15,7da1a393caf0c46b,fb8f2deca976c0bd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:15:13.126 INFO 2981023 --- [ocp-schedules-15,7da1a393caf0c46b,fb8f2deca976c0bd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:13.126 INFO 2981023 --- [ocp-schedules-15,7da1a393caf0c46b,fb8f2deca976c0bd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:13.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:13.522 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:13.590 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046040, nodeCount=480, affectRows=480 2024-09-11 17:15:13.753 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046040, nodeCount=49, affectRows=49 2024-09-11 17:15:13.753 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046040, nodeCount=47, validCount=47 2024-09-11 17:15:13.771 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046040, nodeCount=1000, affectRows=1000 2024-09-11 17:15:13.774 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046040, nodeCount=999, validCount=999 2024-09-11 17:15:13.819 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046040, nodeCount=1000, affectRows=1000 2024-09-11 17:15:13.821 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046040, nodeCount=1000, validCount=1000 2024-09-11 17:15:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,0dd4f73ef56d4cb8,a9f808e3c04a0898] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,9d61113375506af2,750ca2e61e639ad7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:15:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,0dd4f73ef56d4cb8,a9f808e3c04a0898] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:15.112 INFO 2981023 --- [ocp-async-8,d5789919230a966d,5ad562eaac204375] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:15.365 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046040, nodeCount=480, validCount=480 2024-09-11 17:15:16.021 INFO 2981023 --- [ocp-schedules-19,a457f1f152d73781,870508e664abefbf] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:16.021 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:15:16.022 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:15:16.023 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:15:16.024 INFO 2981023 --- [ocp-schedules-29,3782f9f9f3992e70,32f1a055308ab715] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:16.024 INFO 2981023 --- [host-check-clockdiff1,3782f9f9f3992e70,ba0c7e17acf422ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:15:16.024 INFO 2981023 --- [host-check-clockdiff2,3782f9f9f3992e70,b2aaf7501296a874] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:15:16.024 INFO 2981023 --- [host-check-clockdiff3,3782f9f9f3992e70,3538d0d89e1d0a3b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:15:16.025 INFO 2981023 --- [host-check-clockdiff4,3782f9f9f3992e70,e9b60b30660f5095] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:15:16.034 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:15:16.052 INFO 2981023 --- [ocp-schedules-10,f606e9f8b52de1c0,69d4b03307872284] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:16.053 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:15:16.053 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:15:16.053 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:15:16.054 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:15:16.059 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:16.059 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:16.063 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:16.063 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:16.069 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:16.069 INFO 2981023 --- [host-check-clockdiff4,3782f9f9f3992e70,e9b60b30660f5095] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046116 0 0 , error= 2024-09-11 17:15:16.069 INFO 2981023 --- [host-check-clockdiff2,3782f9f9f3992e70,b2aaf7501296a874] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046116 0 0 , error= 2024-09-11 17:15:16.070 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:16.071 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:16.072 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:16.073 INFO 2981023 --- [host-check-clockdiff1,3782f9f9f3992e70,ba0c7e17acf422ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046116 0 0 , error= 2024-09-11 17:15:16.077 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:16.100 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:16.100 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:16.183 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:16.187 INFO 2981023 --- [ocp-schedules-23,3b266ceea9817fa9,4822856ea4180b6a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:15:16.188 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:16.189 INFO 2981023 --- [ocp-schedules-23,3b266ceea9817fa9,4822856ea4180b6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:16.190 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:16.190 INFO 2981023 --- [ocp-schedules-17,42ea7e83c24650f6,980771af966f9a99] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:15:16.190 INFO 2981023 --- [ocp-schedules-9,217995b3ea6cf3eb,9a318f817c23f9fc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:15:16.190 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:16.190 INFO 2981023 --- [ocp-schedules-6,fbd4b38e686ebef6,cb620410fd83ee32] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:15:16.209 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:15:16.263 INFO 2981023 --- [ocp-schedules-17,42ea7e83c24650f6,980771af966f9a99] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:16.264 INFO 2981023 --- [ocp-schedules-17,42ea7e83c24650f6,980771af966f9a99] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:15:16.277 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:16.281 INFO 2981023 --- [ocp-schedules-9,217995b3ea6cf3eb,9a318f817c23f9fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:16.390 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:15:16.392 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:15:16.394 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:15:16.394 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:15:16.394 INFO 2981023 --- [host-check-agent-status1,a457f1f152d73781,f52486d1f5db6b69] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:15:16.396 INFO 2981023 --- [ocp-schedules-13,4b631d937f127bfe,5f887bcb096bc944] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:15:16.396 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:15:16.395337687+08:00 2024-09-11 17:15:16.400 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:15:16.396+08:00 2024-09-11 17:15:16.456 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:15:16.396+08:00 and status AVAILABLE 2024-09-11 17:15:16.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:16.458293+08:00, lastProcessTime=2024-09-11T17:15:06.351739+08:00 2024-09-11 17:15:16.466 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:15:16.468 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:15:16.469 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:15:16.469 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:15:16.471 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:15:16.471 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:15:16.471334653+08:00 2024-09-11 17:15:16.472 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:15:16.472 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:15:16.472 INFO 2981023 --- [host-check-agent-status3,a457f1f152d73781,2a903b99087afc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:15:16.473 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:15:16.473 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:15:16.473 INFO 2981023 --- [host-check-agent-status2,a457f1f152d73781,bba5a159a808e134] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:15:16.474 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:15:16.473834171+08:00 2024-09-11 17:15:16.474 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:15:16.471+08:00 2024-09-11 17:15:16.474 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:15:16.475 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:15:16.474950060+08:00 2024-09-11 17:15:16.475 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:15:16.474+08:00 2024-09-11 17:15:16.476 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:15:16.475+08:00 2024-09-11 17:15:16.476 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:15:16.471+08:00 and status AVAILABLE 2024-09-11 17:15:16.476 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.477 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:16.477 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.478 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.478 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:15:16.478 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.479 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:16.480 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:15:16.480 INFO 2981023 --- [ocp-schedules-23,3b266ceea9817fa9,4822856ea4180b6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:16.482 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:15:16.474+08:00 and status AVAILABLE 2024-09-11 17:15:16.482 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:15:16.475+08:00 and status AVAILABLE 2024-09-11 17:15:16.486 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:15:16.492 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:15:16.494 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:15:16.494 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:15:16.494 INFO 2981023 --- [host-check-agent-status4,a457f1f152d73781,2bb853f6dcd2221c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:15:16.501 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:15:16.519 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:15:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.522 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:16.522 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:16.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:16.524 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:16.524 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:16.526 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:16.560 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:16.562 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:16.563 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:16.564 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:16.572 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:16.575 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.576 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.590 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:16.592 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:16.593 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:15:16.595 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:15:16.610 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:15:16.615 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:15:16.617 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:15:16.627 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:15:16.628 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:15:16.630 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:15:16.644 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:15:16.644 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:15:16.659 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:15:16.660 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:16.664 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.665 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.665 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.666 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:15:16.666 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.666 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:16.669 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:15:16.669 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:15:16.674 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:16.682 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:15:16.692 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:16.692 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:15:16.697 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:15:16.702 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:15:16.702 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:15:16.711 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:16.712 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:15:16.713 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:16.714 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:15:16.719 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.719 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.720 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:15:16.720 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.721 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.721 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:16.722 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:15:16.722 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:16.725 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:15:16.729 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:16.730 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:16.732 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:16.742 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:15:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.744 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:15:16.744 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:16.745 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:15:16.745 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:15:16.748 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:15:16.756 INFO 2981023 --- [host-check-host-status3,f606e9f8b52de1c0,41d03e8e6d30d99e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:15:16.757 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:15:16.758 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:15:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:15:16.761 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:15:16.768 INFO 2981023 --- [host-check-host-status4,f606e9f8b52de1c0,21bd4c823eeafe47] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:15:16.772 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:15:16.775 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:16.782 INFO 2981023 --- [host-check-host-status1,f606e9f8b52de1c0,64261b1d7cad1acd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:15:16.786 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.787 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.787 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:16.787 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:15:16.791 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:15:16.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:16.795 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:16.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:15:16.819 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:15:16.829 INFO 2981023 --- [host-check-host-status2,f606e9f8b52de1c0,ac1546fc16ed0516] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:15:16.835 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:15:16.842 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.843 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.843 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:15:16.843 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:15:16.849 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:16.458293+08:00 2024-09-11 17:15:16.851 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:16.852 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:16.852 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:15:16.854 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:15:16.854 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:15:16.858 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:15:16.858 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:15:16.859 INFO 2981023 --- [ocp-schedules-9,217995b3ea6cf3eb,9a318f817c23f9fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:17.112 INFO 2981023 --- [ocp-async-5,3125272fff6210a6,f30518e78a90e5e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,b2274e9f067bd7a6,29c7e49043641c99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,b2274e9f067bd7a6,29c7e49043641c99] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:17.731 INFO 2981023 --- [ocp-async-8,7bc69515a95ebe21,e977ba75d54dd8a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:18.015 INFO 2981023 --- [ocp-schedules-2,d5af3c002443a0d8,c6ce52bc98630b47] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:18.064 WARN 2981023 --- [scanner-13448830-1,d5af3c002443a0d8,b491fb4ec5a0a584] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:18.102 INFO 2981023 --- [alarm-detect-2,d5af3c002443a0d8,3ecd3615b227d52c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046085 ~ 1726046108 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726046100, value=1.751)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.751, values={1726046100=1.751}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:15:19.112 INFO 2981023 --- [ocp-async-7,20f03ca3ee2598aa,97b3a21f6bbd482e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:19.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:19.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,6c201668e92458d0,ab7af4988cd4ab9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,6c201668e92458d0,ab7af4988cd4ab9d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:21.112 INFO 2981023 --- [ocp-async-6,a156e49c08bce1a6,0c01767d6d7b800a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:21.438 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:15:21.438 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:15:21.442 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:15:21.442 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:15:21.442 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:15:21.442 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:15:21.444 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:15:21.444 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:15:21.444 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:15:21.444 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:15:21.472 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:15:21.476 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:15:21.481 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:15:21.527 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b0087ed0fb50a108,a0cd5809aaa7dff7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=b0087ed0fb50a108, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:15:21.556 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b0087ed0fb50a108,a0cd5809aaa7dff7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=b0087ed0fb50a108, duration=34 ms] 2024-09-11 17:15:21.558 INFO 2981023 --- [ocp-schedules-14,0650b8266b4e02c0,0d0635ea1c2f137f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:21.732 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:15:21.903 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:21.903 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:15:21.904 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:15:21.904 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:15:21.904 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:15:21.909 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:21.910 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:21.911 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:21.912 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:21.912 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:21.913 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:21.914 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:21.914 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:21.935 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:15:21.942 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:15:21.945 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:15:21.946 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:15:21.948 INFO 2981023 --- [ocp-schedules-8,eac3a2ec56e14644,814821bd8c80b262] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:15:22.002 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.557222096S 2024-09-11 17:15:22.002 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:15:22.010 INFO 2981023 --- [host-check-clockdiff3,3782f9f9f3992e70,3538d0d89e1d0a3b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046122 1 1 , error= 2024-09-11 17:15:22.052 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:15:22.052 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:15:22.070 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:15:22.107 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:15:22.112 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:15:22.192 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] o.s.boot.SpringApplication : Started application in 0.75 seconds (JVM running for 81922.065) 2024-09-11 17:15:22.192 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.750769236S 2024-09-11 17:15:22.337 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_agent_process_status, cached timestamp: 1726038916 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_agent_process_status 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_agent_process_status, cached timestamp: 1726038916 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_agent_process_status 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_agent_process_status, cached timestamp: 1726038916 2024-09-11 17:15:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_agent_process_status 2024-09-11 17:15:22.339 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: clock_diff_milliseconds, cached timestamp: 1726038916 2024-09-11 17:15:22.339 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = clock_diff_milliseconds 2024-09-11 17:15:22.344 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_available, cached timestamp: 1726038916 2024-09-11 17:15:22.344 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_available 2024-09-11 17:15:22.413 INFO 2981023 --- [ocp-schedules-23,17c3cf1dc173ffe1,2357e53597f9cd42] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:15:22.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:22.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:23.112 INFO 2981023 --- [ocp-async-5,dd719cf3b4ce5f0d,03cf26f3d5fcd65b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,918ce2a15b21c239,b81fc132f0aa6164] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,918ce2a15b21c239,b81fc132f0aa6164] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:24.106 INFO 2981023 --- [ocp-schedules-10,3d584cb67330a46e,0e133a1332224a78] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:15:25.112 INFO 2981023 --- [ocp-async-1,dd9b7117da28a5e6,ce18c50f419f4195] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:25.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:25.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,1390a8b144c703d9,9aacfac3c55310ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,1390a8b144c703d9,9aacfac3c55310ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:26.337700+08:00, lastProcessTime=2024-09-11T17:15:16.458293+08:00 2024-09-11 17:15:26.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:26.337700+08:00 2024-09-11 17:15:27.113 INFO 2981023 --- [ocp-async-9,9695dc98a3ae6f7f,d28c4110bbb06387] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:27.731 INFO 2981023 --- [ocp-async-7,11d2e98d8664b62e,9507751ff578e493] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:28.235 INFO 2981023 --- [ocp-schedules-27,4f43c1c9bb57f109,0f8fb5d514f64376] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:28.279 WARN 2981023 --- [scanner-13449070-1,4f43c1c9bb57f109,61a7483251a89ed9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:28.307 INFO 2981023 --- [alarm-detect-1,4f43c1c9bb57f109,f70ba9a700eb193d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:28.315 INFO 2981023 --- [alarm-detect-1,4f43c1c9bb57f109,f70ba9a700eb193d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:28.323 INFO 2981023 --- [alarm-detect-1,4f43c1c9bb57f109,f70ba9a700eb193d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:28.327 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,d487155fed4fa95e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:28.329 INFO 2981023 --- [alarm-detect-10,4f43c1c9bb57f109,b66362ff9c48a353] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:28.335 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,d487155fed4fa95e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:28.335 INFO 2981023 --- [alarm-detect-10,4f43c1c9bb57f109,b66362ff9c48a353] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:28.340 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,d487155fed4fa95e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:28.343 INFO 2981023 --- [alarm-detect-10,4f43c1c9bb57f109,b66362ff9c48a353] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:28.432 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,5ac9b49a5ea2d94d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:28.438 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,5ac9b49a5ea2d94d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:28.442 INFO 2981023 --- [alarm-detect-2,4f43c1c9bb57f109,5ac9b49a5ea2d94d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:28.527 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:28.527 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:29.113 INFO 2981023 --- [ocp-async-5,72afa57d1014171a,024c535c79a63ff4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,67e822a0efdb7132,880fab40bbec552d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,67e822a0efdb7132,880fab40bbec552d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:31.113 INFO 2981023 --- [ocp-async-4,53159b4d0b190545,b046f5fbf22ad2e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:31.527 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:31.528 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:31.561 INFO 2981023 --- [ocp-schedules-14,9f95cdb6e3dfb638,929e5eb10a8dad6e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,72925cfe6b90bf16,a7204fbee707f9cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:32.317 INFO 2981023 --- [prometheus-minute-schedule-1,72925cfe6b90bf16,a7204fbee707f9cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:33.113 INFO 2981023 --- [ocp-async-9,5c56f0404a7c0d11,4952b9864c3c7593] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:33.128 INFO 2981023 --- [ocp-schedules-29,275afa08fbfecd17,7b2ff8356c3844b7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:15:33.137 INFO 2981023 --- [ocp-schedules-29,275afa08fbfecd17,7b2ff8356c3844b7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:33.137 INFO 2981023 --- [ocp-schedules-29,275afa08fbfecd17,7b2ff8356c3844b7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:34.528 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:34.530 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:35.113 INFO 2981023 --- [ocp-async-2,5a05b5b5fd4a7a4a,91036f82aa993607] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,fe22cc47d61cb77b,cb4ce551574a9212] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,fe22cc47d61cb77b,cb4ce551574a9212] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:36.231 INFO 2981023 --- [ocp-schedules-29,6fb645ed10f25783,6a49a98fd4ac7f61] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:15:36.248 WARN 2981023 --- [ocp-schedules-29,6fb645ed10f25783,6a49a98fd4ac7f61] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:15:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:36.353266+08:00, lastProcessTime=2024-09-11T17:15:26.337700+08:00 2024-09-11 17:15:36.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:36.353266+08:00 2024-09-11 17:15:36.935 INFO 2981023 --- [ocp-schedules-5,e78c0eee4b30853c,4f0ce08d9bdc2226] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:15:37.113 INFO 2981023 --- [ocp-async-1,f6257201ef04e3f3,b4729f19f10e74e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:37.314 INFO 2981023 --- [prometheus-cache-monitor1,abedbdf99b3c455c,b0788db7c32d4349] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4529} 2024-09-11 17:15:37.315 INFO 2981023 --- [prometheus-cache-monitor1,abedbdf99b3c455c,b0788db7c32d4349] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:15:37.530 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:37.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:37.731 INFO 2981023 --- [ocp-async-5,f4caf58bcc0e1618,5f26ef35006af425] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,c4fd8d7d07d3ef3d,301e614abbdd8c49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,c4fd8d7d07d3ef3d,301e614abbdd8c49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:38.482 INFO 2981023 --- [ocp-schedules-9,9181966f8e336a31,52997034d750bc85] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:38.525 WARN 2981023 --- [scanner-13449348-1,9181966f8e336a31,cdb3bedad62cdc98] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:39.113 INFO 2981023 --- [ocp-async-9,bf615885256afc97,e9621a2db5aff5a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:40.003 INFO 2981023 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:15:40.003 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:15:40.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:40.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:41.113 INFO 2981023 --- [ocp-async-8,eca4bfb2301b3294,b8becedcca84bb69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,0c5097d0d0ffbcdd,559d5fcab312b82e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,0c5097d0d0ffbcdd,559d5fcab312b82e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:41.564 INFO 2981023 --- [ocp-schedules-7,48fe7bc138ed57df,ae4f5e25b2d9f47a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:43.113 INFO 2981023 --- [ocp-async-10,606f50285b4a797e,d2b5e108dc9e9638] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:43.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:43.533 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,03627dba0ba4c573,81ddf499139486e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,03627dba0ba4c573,81ddf499139486e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:45.101 INFO 2981023 --- [prometheus-second-schedule-1,2d1f333ec756bf51,0ed340125b2d971c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:15:45.114 INFO 2981023 --- [ocp-async-7,4be069b7b2a098ab,a55aa480391e0d33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:46.014 INFO 2981023 --- [ocp-schedules-21,8b11a26c35d22dd0,8f2e0211f62c7260] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:46.014 INFO 2981023 --- [host-check-clockdiff1,8b11a26c35d22dd0,5f157f559ff81197] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:15:46.014 INFO 2981023 --- [host-check-clockdiff2,8b11a26c35d22dd0,20ceb8fcb24cd7c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:15:46.015 INFO 2981023 --- [host-check-clockdiff4,8b11a26c35d22dd0,944540f6c1efb892] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:15:46.015 INFO 2981023 --- [host-check-clockdiff3,8b11a26c35d22dd0,2a137a9c594d7c5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:15:46.023 INFO 2981023 --- [ocp-schedules-6,59f055caaf67e8a5,dd1fd9471e63e53b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:46.028 INFO 2981023 --- [host-check-clockdiff1,8b11a26c35d22dd0,5f157f559ff81197] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046146 0 0 , error= 2024-09-11 17:15:46.029 INFO 2981023 --- [ocp-schedules-23,2895f7beb4b1873c,6dd19d607f832135] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:15:46.029 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:15:46.029 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:15:46.029 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:15:46.031 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:15:46.038 INFO 2981023 --- [host-check-clockdiff4,8b11a26c35d22dd0,944540f6c1efb892] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046146 0 0 , error= 2024-09-11 17:15:46.044 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:15:46.044 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:15:46.048 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:15:46.048 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:15:46.050 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:46.054 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:46.056 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:46.056 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:46.058 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:46.058 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:46.061 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:46.064 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:46.066 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:46.067 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:46.104 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:46.104 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:46.118 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:46.165 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:15:46.177 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:46.191 INFO 2981023 --- [ocp-schedules-5,4b53f84b25858971,4cb08de8039f215b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:15:46.193 INFO 2981023 --- [ocp-schedules-13,cf3e56d8799dc451,193f0be62ffebeaf] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:15:46.195 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:46.198 INFO 2981023 --- [ocp-schedules-10,83d9c234ffb0269b,4116e6d21ab88fb2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:15:46.198 INFO 2981023 --- [ocp-schedules-2,4c44d269f3d6dade,6ce9f7fcdcec8648] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:15:46.257 INFO 2981023 --- [ocp-schedules-10,83d9c234ffb0269b,4116e6d21ab88fb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:46.266 INFO 2981023 --- [ocp-schedules-13,cf3e56d8799dc451,193f0be62ffebeaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:46.266 INFO 2981023 --- [ocp-schedules-27,a1cc40a920c14eb3,438bb0aa2208cfd3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:15:46.271 INFO 2981023 --- [ocp-schedules-5,4b53f84b25858971,4cb08de8039f215b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:46.277 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:46.285 INFO 2981023 --- [ocp-schedules-1,f862140bdfd75496,b46618d3e8d79fff] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:15:46.286 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:15:46.304 INFO 2981023 --- [ocp-schedules-27,a1cc40a920c14eb3,438bb0aa2208cfd3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:15:46.307 INFO 2981023 --- [ocp-schedules-13,cf3e56d8799dc451,193f0be62ffebeaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:15:46.368 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:15:46.380 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:15:46.384 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:15:46.384 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:15:46.384 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:15:46.384 INFO 2981023 --- [host-check-agent-status4,2895f7beb4b1873c,004d42001a89f288] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:15:46.387 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:15:46.388 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:46.387955+08:00, lastProcessTime=2024-09-11T17:15:36.353266+08:00 2024-09-11 17:15:46.389 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:15:46.389049432+08:00 2024-09-11 17:15:46.390 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:15:46.389+08:00 2024-09-11 17:15:46.393 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:15:46.391210575+08:00 2024-09-11 17:15:46.393 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:15:46.393+08:00 2024-09-11 17:15:46.396 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:15:46.389+08:00 and status AVAILABLE 2024-09-11 17:15:46.398 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:15:46.393+08:00 and status AVAILABLE 2024-09-11 17:15:46.402 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.402 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:15:46.403 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:46.403 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:15:46.405 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:15:46.405 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:15:46.405 INFO 2981023 --- [host-check-agent-status2,2895f7beb4b1873c,6b3752e02c4fabd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:15:46.453 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:15:46.453 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:15:46.454 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:15:46.457 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:15:46.457 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:15:46.457 INFO 2981023 --- [host-check-agent-status1,2895f7beb4b1873c,e8c9b5f2ce851bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:15:46.458 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:15:46.455306545+08:00 2024-09-11 17:15:46.458 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.460 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:46.461 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:15:46.461 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:15:46.462 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:15:46.458+08:00 2024-09-11 17:15:46.463 INFO 2981023 --- [ocp-schedules-5,4b53f84b25858971,4cb08de8039f215b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:46.463 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:15:46.465 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:15:46.465 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:15:46.465 INFO 2981023 --- [host-check-agent-status3,2895f7beb4b1873c,4a415b932dfda6b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:15:46.466 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:15:46.458+08:00 and status AVAILABLE 2024-09-11 17:15:46.476 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.476 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:46.477 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:46.477 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.479 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:15:46.481 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:15:46.481460529+08:00 2024-09-11 17:15:46.482 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:15:46.483 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:15:46.481+08:00 2024-09-11 17:15:46.486 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:15:46.488 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:15:46.481+08:00 and status AVAILABLE 2024-09-11 17:15:46.493 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:15:46.496 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.497 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.501 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:15:46.503 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:15:46.503 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:15:46.504 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:15:46.511 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:46.515 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:46.515 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.515 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.520 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:46.520 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:46.533 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:46.544 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:15:46.556 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:46.565 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:46.570 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:15:46.575 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:46.582 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:15:46.583 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:15:46.595 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.596 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:15:46.597 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.597 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:46.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.607 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.608 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.608 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:15:46.610 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:46.611 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:15:46.621 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:15:46.621 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.624 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:15:46.625 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:15:46.626 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.639 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:15:46.671 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:15:46.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:46.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:15:46.684 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:46.685 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:15:46.705 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:15:46.706 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:15:46.708 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:15:46.708 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:15:46.708 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046041977000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:15:46.720 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:15:46.724 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.726 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.727 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:15:46.730 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:46.731 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:46.735 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.748 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.755 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.758 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.759 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:46.761 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:15:46.763 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:15:46.773 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.773 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:46.773 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:15:46.780 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:15:46.788 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:15:46.789 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.796 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:46.797 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:46.801 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:15:46.801 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:15:46.803 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:15:46.803 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.810 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:15:46.813 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.813 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.816 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:46.819 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:15:46.826 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:15:46.826 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.829 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:15:46.834 INFO 2981023 --- [ocp-schedules-11,895763297b294cc7,c2ed514a43f30e3a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4529} 2024-09-11 17:15:46.834 INFO 2981023 --- [ocp-schedules-11,895763297b294cc7,c2ed514a43f30e3a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:15:46.837 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:15:46.837 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:15:46.855 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:15:46.862 INFO 2981023 --- [host-check-host-status3,59f055caaf67e8a5,c3e4c2808862f69b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:15:46.863 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:15:46.867 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:15:46.883 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:46.883 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:15:46.904 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:15:46.905 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:15:46.928 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:15:46.930 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.931 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.931 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:15:46.931 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:46.932 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:46.933 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.950 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:15:46.958 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:46.959 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:46.960 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:15:46.965 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:15:46.971 INFO 2981023 --- [host-check-clockdiff2,8b11a26c35d22dd0,20ceb8fcb24cd7c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046146 0 0 , error= 2024-09-11 17:15:46.974 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:15:46.979 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:15:46.981 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:15:46.983 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:15:46.985 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:46.987 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:15:47.009 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:47.009 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:15:47.012 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:47.013 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:47.013 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:15:47.013 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:15:47.014 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:15:47.014 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:47.032 INFO 2981023 --- [host-check-host-status2,59f055caaf67e8a5,30817a50082c33ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:15:47.032 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:15:47.044 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:15:47.057 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:15:47.060 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,90dd321703bb24c9,6a1487ef76153afb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=90dd321703bb24c9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:15:47.073 INFO 2981023 --- [host-check-host-status1,59f055caaf67e8a5,0ce5c537575d57df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:15:47.073 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:47.075 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:15:47.078 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:47.082 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:15:47.095 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:15:47.109 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:15:47.109 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:15:47.114 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:15:47.114 INFO 2981023 --- [ocp-async-5,bb6da06dc9766248,109c578ee6e8ac18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:47.117 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:15:47.120 INFO 2981023 --- [host-check-host-status4,59f055caaf67e8a5,58e08796201149b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:15:47.126 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:47.131 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:15:47.134 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:47.135 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:15:47.135 INFO 2981023 --- [ocp-schedules-19,27f4561258511e49,87f8ab2a2094609d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:15:47.142 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:46.387955+08:00 2024-09-11 17:15:47.145 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,90dd321703bb24c9,6a1487ef76153afb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=90dd321703bb24c9, duration=94 ms] 2024-09-11 17:15:47.152 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:15:47.152 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:15:47.157 INFO 2981023 --- [ocp-schedules-10,83d9c234ffb0269b,4116e6d21ab88fb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,77aae23621b1cb32,b40a45ef2258f117] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,77aae23621b1cb32,b40a45ef2258f117] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:47.731 INFO 2981023 --- [ocp-async-6,32218adc5f2187cb,4a3161d3d698ac44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:48.662 INFO 2981023 --- [ocp-schedules-20,b07d38b39f3c255e,abada15bc49990d2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:48.687 WARN 2981023 --- [scanner-13449581-1,b07d38b39f3c255e,452b9d0b067c8e24] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:48.755 INFO 2981023 --- [alarm-detect-20,b07d38b39f3c255e,4873a6b10c03acab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:48.761 INFO 2981023 --- [alarm-detect-20,b07d38b39f3c255e,4873a6b10c03acab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:48.764 INFO 2981023 --- [alarm-detect-20,b07d38b39f3c255e,4873a6b10c03acab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:15:48.772 INFO 2981023 --- [alarm-detect-1,b07d38b39f3c255e,4ad209135ddd54f2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:48.776 INFO 2981023 --- [alarm-detect-16,b07d38b39f3c255e,68254d4f966adf8d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:48.776 INFO 2981023 --- [alarm-detect-1,b07d38b39f3c255e,4ad209135ddd54f2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:48.781 INFO 2981023 --- [alarm-detect-1,b07d38b39f3c255e,4ad209135ddd54f2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:15:48.781 INFO 2981023 --- [alarm-detect-16,b07d38b39f3c255e,68254d4f966adf8d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:48.786 INFO 2981023 --- [alarm-detect-16,b07d38b39f3c255e,68254d4f966adf8d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:15:48.891 INFO 2981023 --- [alarm-detect-5,b07d38b39f3c255e,f2a80a4916cbf5e3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:48.898 INFO 2981023 --- [alarm-detect-5,b07d38b39f3c255e,f2a80a4916cbf5e3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:48.911 INFO 2981023 --- [alarm-detect-5,b07d38b39f3c255e,f2a80a4916cbf5e3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:15:49.114 INFO 2981023 --- [ocp-async-10,7aaf7fc041e6b144,7dd16e6df10ca951] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:49.512 INFO 2981023 --- [ocp-schedules-26,5ede4ee78ea656ef,3b689d257e78d3d6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:15:49.512+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:15:49.515 INFO 2981023 --- [ocp-schedules-26,5ede4ee78ea656ef,3b689d257e78d3d6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:15:49.565 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:49.566 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:49.625 INFO 2981023 --- [ocp-schedules-26,5ede4ee78ea656ef,3b689d257e78d3d6] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 21 to SqlAggregateData 2024-09-11 17:15:49.626 INFO 2981023 --- [ocp-schedules-26,5ede4ee78ea656ef,3b689d257e78d3d6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:15:49.857 INFO 2981023 --- [ocp-schedules-28,03ad766688ad3e3c,9b1ac2ec12231685] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:15:49.859 INFO 2981023 --- [ocp-schedules-28,03ad766688ad3e3c,9b1ac2ec12231685] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:49.888 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.888 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.889 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.890 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.891 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.893 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.893 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:15:49.899 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.900 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.901 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.902 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.902 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.902 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:15:49.906 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.907 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:15:49.913 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.913 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.914 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.914 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:15:49.917 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.917 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.929 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.929 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:15:49.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE13B1D734-0-0] 2024-09-11 17:15:49.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:15:49.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.941 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.942 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.942 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.942 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.942 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.943 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.943 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.944 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.944 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:15:49.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.946 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.953 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:15:49.955 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.955 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:49.956 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:15:49.956 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE13B1D73E-0-0] 2024-09-11 17:15:49.956 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:15:49.958 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.959 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.959 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.960 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.961 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.962 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.963 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.963 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.964 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.965 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.966 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.967 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.967 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.968 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.968 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.968 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.969 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.969 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.970 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.970 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.970 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.971 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.971 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:15:49.971 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.972 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.972 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:15:49.972 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:15:49.973 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.973 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:15:49.974 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:15:49.974 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:15:49.976 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:49.977 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:15:49.978 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.979 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:49.987 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:49.990 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:49.990 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:15:49.992 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:49.996 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:15:49.996 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:15:49.996 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:15:49.997 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:49.999 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:15:49.999 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:50.000 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:50.001 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:15:50.003 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:50.004 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:15:50.006 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:50.007 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:15:50.008 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:50.008 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:50.013 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:50.014 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:15:50.014 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:15:50.014 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:15:50.016 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:50.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:15:50.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:50.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:15:50.029 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:15:50.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:15:50.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:15:50.030 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:15:50.030 INFO 2981023 --- [ocp-schedules-28,03ad766688ad3e3c,9b1ac2ec12231685] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:15:50.173 INFO 2981023 --- [ocp-schedules-5,39080cf6cc304398,a5968418ced2f417] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:15:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,ad054d2a893088b3,62540c0c0ba839a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,ad054d2a893088b3,62540c0c0ba839a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:51.114 INFO 2981023 --- [ocp-async-7,50a030ed9ca8ce50,2fb0ba9cc4f6ac44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:51.481 INFO 2981023 --- [ocp-schedules-24,fa38c281dbc65d86,63ead2309c95bbcb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:15:51.482 INFO 2981023 --- [ocp-schedules-24,fa38c281dbc65d86,63ead2309c95bbcb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:15:51.482 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:15:51.482 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:15:51.482 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:15:51.483 INFO 2981023 --- [ocp-schedules-24,fa38c281dbc65d86,63ead2309c95bbcb] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:15:51.566 INFO 2981023 --- [ocp-schedules-1,1ad9808188b8f829,2ddcd093752e6cfb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:15:51.673 INFO 2981023 --- [ocp-schedules-4,5413a2ad708b13d9,910b879b3cf0155f] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:15:51.690 WARN 2981023 --- [ocp-schedules-4,5413a2ad708b13d9,910b879b3cf0155f] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:15:51.977 INFO 2981023 --- [host-check-clockdiff3,8b11a26c35d22dd0,2a137a9c594d7c5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046151 1 1 , error= 2024-09-11 17:15:52.566 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:52.568 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:52.959 INFO 2981023 --- [ocp-schedules-16,c17a17bc02b7fa3f,de1a76b961cd4753] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:15:52.959 INFO 2981023 --- [ocp-schedules-16,c17a17bc02b7fa3f,de1a76b961cd4753] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:15:53.114 INFO 2981023 --- [ocp-async-6,9c0169099d197c9f,698957e72401714d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:53.139 INFO 2981023 --- [ocp-schedules-1,69862ee561fc92a9,de9dee7911429e2e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:15:53.148 INFO 2981023 --- [ocp-schedules-1,69862ee561fc92a9,de9dee7911429e2e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:53.148 INFO 2981023 --- [ocp-schedules-1,69862ee561fc92a9,de9dee7911429e2e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:15:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,fc72cc1b9a5ed7b3,c1f1218133237eb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,fc72cc1b9a5ed7b3,c1f1218133237eb8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:54.109 INFO 2981023 --- [ocp-schedules-6,30f388162f58c44b,740590ab504a99de] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:15:54.603 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:15:54.603 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:15:54.608 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:15:54.608 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:15:54.608 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:15:54.611 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:15:54.611 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:15:54.611 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:15:54.621 INFO 2981023 --- [ocp-schedules-10,cf7d24e5d87ae9bc,5111aa2c0ae03394] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:15:55.114 INFO 2981023 --- [ocp-async-1,d67ae20b0ed54087,185a8651ae9b3bf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:55.177 INFO 2981023 --- [ocp-schedules-13,34375a3a3346e528,00f26e816797ec29] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:15:55.177 INFO 2981023 --- [ocp-schedules-13,34375a3a3346e528,00f26e816797ec29] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:15:55.202 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:15:55.204 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:15:55.204 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:15:55.204 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:15:55.205 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:15:55.205 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:15:55.223 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.223 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.224 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:15:55.224 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.225 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:15:55.225 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.227 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:15:55.227 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:15:55.228 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:15:55.228 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.229 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:15:55.230 INFO 2981023 --- [Collect all tenants major compactions2,34375a3a3346e528,3078fab6e4fce5b1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions1,34375a3a3346e528,a9888cb3dd50c7ff] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions3,34375a3a3346e528,dfdcf24fd5c0bad6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:15:55.231 INFO 2981023 --- [Collect all tenants major compactions6,34375a3a3346e528,8fa246318bb68562] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:15:55.232 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:15:55.232 INFO 2981023 --- [Collect all tenants major compactions4,34375a3a3346e528,215a481dee108694] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:15:55.232 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:15:55.234 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:15:55.234 INFO 2981023 --- [Collect all tenants major compactions5,34375a3a3346e528,e2778a2e354f2305] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:15:55.234 INFO 2981023 --- [ocp-schedules-13,34375a3a3346e528,00f26e816797ec29] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:15:55.568 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:55.568 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:55.587 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:15:55.587 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:15:55.590 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:15:55.590 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:15:55.592 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:15:55.592 INFO 2981023 --- [ocp-schedules-12,5c24c8a3913ba065,bb88b0ff6294d57b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:15:55.612 INFO 2981023 --- [ocp-schedules-2,bcc59aa68e462541,c8e7e1defc3a78b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:15:55.622 INFO 2981023 --- [ocp-schedules-2,bcc59aa68e462541,c8e7e1defc3a78b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:15:55.622 INFO 2981023 --- [ocp-schedules-2,bcc59aa68e462541,c8e7e1defc3a78b1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:15:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,dfbff7b247fc26f6,047d2be0bec62b10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,dfbff7b247fc26f6,047d2be0bec62b10] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:15:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:15:56.352086+08:00, lastProcessTime=2024-09-11T17:15:46.387955+08:00 2024-09-11 17:15:56.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:15:56.352086+08:00 2024-09-11 17:15:57.114 INFO 2981023 --- [ocp-async-9,59a7946132c3f706,f4d90c2a572f8dc9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:57.731 INFO 2981023 --- [ocp-async-3,8d62c607f10f523e,b3bac20ce64c474e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:15:58.568 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:15:58.570 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:15:58.953 INFO 2981023 --- [ocp-schedules-14,9b098878e1cb1d61,effdd520ea4f576a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:15:59.000 WARN 2981023 --- [scanner-13449873-1,9b098878e1cb1d61,c454452ab3e5e9a5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:15:59.114 INFO 2981023 --- [ocp-async-6,178beb1f7880a3ec,51a6427e2a73a05d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:15:59.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,6b96245ec14fdf0b,f3e6c5a31dc454f5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=6b96245ec14fdf0b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:15:59.163 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,6b96245ec14fdf0b,f3e6c5a31dc454f5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=6b96245ec14fdf0b, duration=54 ms] 2024-09-11 17:15:59.297 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8d68a734cd10c5cb,7f9440d2e17d27e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=8d68a734cd10c5cb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:15:59.306 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8d68a734cd10c5cb,7f9440d2e17d27e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=8d68a734cd10c5cb, duration=12 ms] 2024-09-11 17:15:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,c68cd8ac3d9eaf62,2999d7e9326814c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046160, interval=60 2024-09-11 17:15:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,c68cd8ac3d9eaf62,2999d7e9326814c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:01.118 INFO 2981023 --- [ocp-async-1,a03422dc325d181b,24ca11fd0596a2be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:01.385 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,9b5180f72f6a3233,9d34801ce4e89476] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=9b5180f72f6a3233, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:01.400 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,9b5180f72f6a3233,9d34801ce4e89476] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=9b5180f72f6a3233, duration=19 ms] 2024-09-11 17:16:01.569 INFO 2981023 --- [ocp-schedules-4,4e807e7579a32221,1d03191796780cd5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:01.570 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:01.571 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:01.653 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,9dc595a6f572a3eb,9281e95b68a5d779] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=9dc595a6f572a3eb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:01.656 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,9dc595a6f572a3eb,9281e95b68a5d779] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=9dc595a6f572a3eb, duration=5 ms] 2024-09-11 17:16:01.987 INFO 2981023 --- [ocp-schedules-28,3bd698f21b8b4a12,5547e2dea04bd6df] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:16:01.988 INFO 2981023 --- [ocp-schedules-28,3bd698f21b8b4a12,5547e2dea04bd6df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:02.033 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:02.034 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:16:02.039 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:02.041 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:16:02.041 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000} 2024-09-11 17:16:02.047 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:16:02.050 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000221, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:16:02.050 INFO 2981023 --- [ocp-schedules-28,3bd698f21b8b4a12,5547e2dea04bd6df] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:16:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,46ed7042502c5919,083014143f8f3cd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,46ed7042502c5919,083014143f8f3cd8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:02.785 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 17:16:02.785 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 17:16:02.785 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 17:16:02.789 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000221, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:16:02.791 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 17:16:03.115 INFO 2981023 --- [ocp-async-3,a6a5a64b3b623be6,087570b6965ed2dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:03.185 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f94e313cc412084f,016e0aaefd9acc99] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=f94e313cc412084f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:03.195 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f94e313cc412084f,016e0aaefd9acc99] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=f94e313cc412084f, duration=13 ms] 2024-09-11 17:16:03.474 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b6f32bb9125eb4de,837619f7cf402b13] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b6f32bb9125eb4de, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:03.477 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b6f32bb9125eb4de,837619f7cf402b13] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b6f32bb9125eb4de, duration=6 ms] 2024-09-11 17:16:03.671 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000221 2024-09-11 17:16:04.571 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:04.572 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:05.090 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1958b13a940e4965,03b4d8fa8b5b904d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=1958b13a940e4965, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:05.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1958b13a940e4965,03b4d8fa8b5b904d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=1958b13a940e4965, duration=13 ms] 2024-09-11 17:16:05.115 INFO 2981023 --- [ocp-async-5,cc1f7ef2dcb73695,ba72b068026a04b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,2a0dad5996b34f52,7eb06736489e5ee6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,2a0dad5996b34f52,7eb06736489e5ee6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:05.356 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,387a76dbae9aa39c,e19dca127fc982ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=387a76dbae9aa39c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:05.358 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,387a76dbae9aa39c,e19dca127fc982ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=387a76dbae9aa39c, duration=5 ms] 2024-09-11 17:16:05.983 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:05.984 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:16:05.984 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:05.985 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:16:05.985 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:16:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:16:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:16:06.009 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 17:16:06.009 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:16:06.009 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:16:06.012 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:06.012 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:16:06.013 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:06.014 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:16:06.062 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:13:41, start=2024-09-11T17:10:11, end=2024-09-11T17:12:41, updates=289, granule=120000ms, delay=210000ms 2024-09-11 17:16:06.068 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:13:41, target=2024-09-11T17:12:41, granule=120000ms, delay=60000ms) 2024-09-11 17:16:06.068 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:12:41, target=2024-09-11T17:00:51, granule=600000ms, delay=710000ms) 2024-09-11 17:16:06.069 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=84ms, period=120000ms 2024-09-11 17:16:06.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:06.351213+08:00, lastProcessTime=2024-09-11T17:15:56.352086+08:00 2024-09-11 17:16:06.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:06.351213+08:00 2024-09-11 17:16:06.999 INFO 2981023 --- [ocp-schedules-15,5a4e4ab8355bbf9f,1f10f03ab399c612] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:16:07.012 INFO 2981023 --- [ocp-schedules-12,38f27f792f91651e,05b84c3f6496853c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:16:07.012+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:16:07.013 INFO 2981023 --- [ocp-schedules-12,38f27f792f91651e,05b84c3f6496853c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:16:07.014 INFO 2981023 --- [ocp-schedules-12,38f27f792f91651e,05b84c3f6496853c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:16:07.015 INFO 2981023 --- [ocp-schedules-12,38f27f792f91651e,05b84c3f6496853c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:16:07.018 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,df2f96bde29266c0,8f6b741dd2a17040] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=df2f96bde29266c0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:07.028 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,df2f96bde29266c0,8f6b741dd2a17040] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=df2f96bde29266c0, duration=12 ms] 2024-09-11 17:16:07.115 INFO 2981023 --- [ocp-async-6,7543de848749d975,8cef745c4d766a22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:07.253 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7fdc63d372916954,954ce951f6c26ba3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=7fdc63d372916954, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:07.256 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7fdc63d372916954,954ce951f6c26ba3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=7fdc63d372916954, duration=5 ms] 2024-09-11 17:16:07.572 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:07.572 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:07.731 INFO 2981023 --- [ocp-async-4,6f8f07db81b92a1d,5a0f95bc6f52ba0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,78862a733afa9b6b,891a592d01e48c16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,78862a733afa9b6b,891a592d01e48c16] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:08.864 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8869de07392740aa,7121c64c6708f601] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=8869de07392740aa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:08.874 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8869de07392740aa,7121c64c6708f601] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=8869de07392740aa, duration=13 ms] 2024-09-11 17:16:09.115 INFO 2981023 --- [ocp-async-3,8c513a147a477bba,fa40382d192fbc3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:09.178 INFO 2981023 --- [ocp-schedules-23,9185b9c9e3c2d9c3,a98e05e2c2f01772] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:09.223 WARN 2981023 --- [scanner-13450121-1,9185b9c9e3c2d9c3,b21b0eb32a943679] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:09.253 INFO 2981023 --- [alarm-detect-6,9185b9c9e3c2d9c3,8a563ea85bf8e22a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643396, data is empty. 2024-09-11 17:16:09.256 INFO 2981023 --- [alarm-detect-6,9185b9c9e3c2d9c3,8a563ea85bf8e22a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:09.261 INFO 2981023 --- [alarm-detect-6,9185b9c9e3c2d9c3,8a563ea85bf8e22a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:09.279 INFO 2981023 --- [alarm-detect-9,9185b9c9e3c2d9c3,536ed9a5bb7a06a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643396, data is empty. 2024-09-11 17:16:09.281 INFO 2981023 --- [alarm-detect-11,9185b9c9e3c2d9c3,e928497359296b1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643396, data is empty. 2024-09-11 17:16:09.282 INFO 2981023 --- [alarm-detect-9,9185b9c9e3c2d9c3,536ed9a5bb7a06a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:09.285 INFO 2981023 --- [alarm-detect-11,9185b9c9e3c2d9c3,e928497359296b1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:09.287 INFO 2981023 --- [alarm-detect-9,9185b9c9e3c2d9c3,536ed9a5bb7a06a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:09.289 INFO 2981023 --- [alarm-detect-11,9185b9c9e3c2d9c3,e928497359296b1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:09.317 INFO 2981023 --- [alarm-detect-19,9185b9c9e3c2d9c3,9f416d7fa465c44b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046130 ~ 1726046159 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046145, value=2.962)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.962, values={1726046145=2.962}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:09.398 INFO 2981023 --- [alarm-detect-13,9185b9c9e3c2d9c3,d1a532c246c326cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643396, data is empty. 2024-09-11 17:16:09.402 INFO 2981023 --- [alarm-detect-13,9185b9c9e3c2d9c3,d1a532c246c326cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:09.406 INFO 2981023 --- [alarm-detect-13,9185b9c9e3c2d9c3,d1a532c246c326cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:10.002 INFO 2981023 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:16:10.003 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:16:10.458 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d7767ee493acc082,a965211a71c6b440] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=d7767ee493acc082, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:10.469 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d7767ee493acc082,a965211a71c6b440] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=d7767ee493acc082, duration=13 ms] 2024-09-11 17:16:10.572 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:10.573 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:10.689 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,5e38289a6645ac48,f7ee7a61693200ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=5e38289a6645ac48, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:10.692 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,5e38289a6645ac48,f7ee7a61693200ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=5e38289a6645ac48, duration=6 ms] 2024-09-11 17:16:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:16:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:16:11.115 INFO 2981023 --- [ocp-async-2,33b81b297ead258f,8f74d3f0cc7afb56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,520e53623b57b7e9,477d2e2c5222c8f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,520e53623b57b7e9,477d2e2c5222c8f2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:11.572 INFO 2981023 --- [ocp-schedules-25,fca5b0729e803d6c,3eb90cfd2c81afde] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:12.158 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,41ac541771c14512,abfad251b962fb19] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=41ac541771c14512, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:12.169 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,41ac541771c14512,abfad251b962fb19] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=41ac541771c14512, duration=13 ms] 2024-09-11 17:16:12.488 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,b3eea57453d714de,102d7976716c0304] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b3eea57453d714de, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:12.490 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,b3eea57453d714de,102d7976716c0304] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b3eea57453d714de, duration=5 ms] 2024-09-11 17:16:13.115 INFO 2981023 --- [ocp-async-6,56430d06a7608523,ac0f93c51eaa584d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:13.151 INFO 2981023 --- [ocp-schedules-25,404a1d483ea580b3,b7fdfe0f5cd2d4cc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:16:13.159 INFO 2981023 --- [ocp-schedules-25,404a1d483ea580b3,b7fdfe0f5cd2d4cc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:13.159 INFO 2981023 --- [ocp-schedules-25,404a1d483ea580b3,b7fdfe0f5cd2d4cc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:13.573 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:13.574 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:13.830 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046100, nodeCount=1000, affectRows=1000 2024-09-11 17:16:13.833 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046100, nodeCount=999, validCount=999 2024-09-11 17:16:13.853 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046100, nodeCount=48, affectRows=48 2024-09-11 17:16:13.853 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046100, nodeCount=46, validCount=46 2024-09-11 17:16:13.858 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046100, nodeCount=1000, affectRows=1000 2024-09-11 17:16:13.860 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046100, nodeCount=1000, validCount=1000 2024-09-11 17:16:13.992 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7869047f98083008,b341b532dd128d35] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=7869047f98083008, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:14.003 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7869047f98083008,b341b532dd128d35] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=7869047f98083008, duration=14 ms] 2024-09-11 17:16:14.258 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,eeccbcac1a310fc3,8acda2b0ad39bfca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=eeccbcac1a310fc3, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:14.260 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,eeccbcac1a310fc3,8acda2b0ad39bfca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=eeccbcac1a310fc3, duration=5 ms] 2024-09-11 17:16:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,dde5e05d090b7694,6d77c20baf414082] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,35f2cc3a1400d10e,8a4fa70a2971c698] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:16:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,dde5e05d090b7694,6d77c20baf414082] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:14.904 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046100, nodeCount=480, affectRows=480 2024-09-11 17:16:15.115 INFO 2981023 --- [ocp-async-9,36a23d8e25765a11,5a7ec70d6d0bf7d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:15.753 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2fb36e2d960ff2d2,96f6d5326c4d96e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=2fb36e2d960ff2d2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:15.763 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2fb36e2d960ff2d2,96f6d5326c4d96e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=2fb36e2d960ff2d2, duration=13 ms] 2024-09-11 17:16:16.014 INFO 2981023 --- [ocp-schedules-26,97a3a4b122e58884,41b32df0ac0278ec] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:16.015 INFO 2981023 --- [host-check-clockdiff1,97a3a4b122e58884,0f4e605b477458aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:16:16.015 INFO 2981023 --- [host-check-clockdiff2,97a3a4b122e58884,2d92282dd026c4a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:16:16.015 INFO 2981023 --- [host-check-clockdiff3,97a3a4b122e58884,6954326b7b1a25cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:16:16.017 INFO 2981023 --- [host-check-clockdiff4,97a3a4b122e58884,e0ff3cdbe2f73466] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:16:16.026 INFO 2981023 --- [ocp-schedules-29,04403b2295dad6ae,db33ebf95892af3b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:16.028 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:16:16.028 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:16:16.028 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:16:16.030 INFO 2981023 --- [host-check-clockdiff4,97a3a4b122e58884,e0ff3cdbe2f73466] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046176 0 0 , error= 2024-09-11 17:16:16.030 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:16:16.031 INFO 2981023 --- [ocp-schedules-30,a1c3c4b4640e2522,c7d07eb597b7ffe8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:16.031 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:16:16.034 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:16:16.039 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:16:16.039 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:16:16.065 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:16.071 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:16.073 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:16.073 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:16.073 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:16.074 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:16.077 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:16.077 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:16.079 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:16.080 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:16.081 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:16.104 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:16.106 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:16.167 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:16.195 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:16.196 INFO 2981023 --- [ocp-schedules-23,f95d022a22efa56a,05f30de67ff1fa2a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:16:16.197 INFO 2981023 --- [ocp-schedules-6,308cd64dc3384e77,aa21275fda41cc57] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:16:16.197 INFO 2981023 --- [ocp-schedules-12,dfaf1b355b56f768,fe91c8be21dfbc33] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:16:16.197 INFO 2981023 --- [ocp-schedules-2,89cdbffce203ec8c,57157a1872de8e5e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:16:16.282 INFO 2981023 --- [ocp-schedules-12,dfaf1b355b56f768,fe91c8be21dfbc33] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:16.284 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:16.372 INFO 2981023 --- [ocp-schedules-23,f95d022a22efa56a,05f30de67ff1fa2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:16.374 INFO 2981023 --- [ocp-schedules-6,308cd64dc3384e77,aa21275fda41cc57] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:16.374 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:16:16.380 INFO 2981023 --- [ocp-schedules-6,308cd64dc3384e77,aa21275fda41cc57] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:16:16.392 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:16:16.399 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:16:16.399 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:16:16.399 INFO 2981023 --- [host-check-agent-status4,a1c3c4b4640e2522,9f9822cdf2711ae8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:16:16.401 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:16.395298+08:00, lastProcessTime=2024-09-11T17:16:06.351213+08:00 2024-09-11 17:16:16.405 INFO 2981023 --- [ocp-schedules-19,ecc7aadab6fb871d,8c52a747b53102ca] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:16:16.406 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:16:16.409 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.409 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:16.409 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:16:16.409297808+08:00 2024-09-11 17:16:16.410 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:16:16.409+08:00 2024-09-11 17:16:16.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:16:16.413 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:16:16.409+08:00 and status AVAILABLE 2024-09-11 17:16:16.413 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:16:16.415 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.415 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:16.416 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:16:16.416 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:16:16.417 INFO 2981023 --- [ocp-schedules-23,f95d022a22efa56a,05f30de67ff1fa2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:16.418 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:16:16.418338333+08:00 2024-09-11 17:16:16.419 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:16:16.418+08:00 2024-09-11 17:16:16.420 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:16:16.420 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:16:16.420 INFO 2981023 --- [host-check-agent-status2,a1c3c4b4640e2522,f45b40a9d0480d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:16:16.420 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:16:16.423 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:16:16.423 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:16:16.423 INFO 2981023 --- [host-check-agent-status1,a1c3c4b4640e2522,0da2d37f9fbb1057] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:16:16.454 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:16:16.455 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:16:16.418+08:00 and status AVAILABLE 2024-09-11 17:16:16.459 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:16:16.460 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:16:16.460 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:16:16.460642994+08:00 2024-09-11 17:16:16.462 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:16:16.462 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:16:16.462 INFO 2981023 --- [host-check-agent-status3,a1c3c4b4640e2522,9fdb7dabe27997c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:16:16.466 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:16:16.460+08:00 2024-09-11 17:16:16.468 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:16:16.460+08:00 and status AVAILABLE 2024-09-11 17:16:16.470 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:16:16.472 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.472 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.474 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:16:16.470788989+08:00 2024-09-11 17:16:16.476 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:16:16.477 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:16:16.474+08:00 2024-09-11 17:16:16.480 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:16:16.474+08:00 and status AVAILABLE 2024-09-11 17:16:16.483 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:16:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.484 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.485 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:16.485 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.485 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:16.486 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:16.487 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:16.488 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:16:16.495 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:16.496 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:16.507 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:16.509 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:16.512 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:16.518 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:16.535 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:16.539 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:16:16.543 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:16:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.559 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:16:16.559 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.560 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:16:16.562 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:16:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:16:16.569 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:16:16.570 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:16:16.573 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:16:16.574 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:16.576 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:16.587 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:16:16.598 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:16:16.601 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.605 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:16:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.606 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:16.608 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:16:16.609 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:16:16.609 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:16:16.621 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:16:16.630 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:16:16.635 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:16:16.639 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:16:16.653 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:16.660 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4f0f0feac2bea7d8,8226ea9c7b34bc60] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=4f0f0feac2bea7d8, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.666 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:16:16.666 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:16:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.666 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4f0f0feac2bea7d8,8226ea9c7b34bc60] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=4f0f0feac2bea7d8, duration=21 ms] 2024-09-11 17:16:16.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:16:16.670 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:16:16.673 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:16.675 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:16:16.679 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:16.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:16.682 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:16:16.689 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:16.693 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:16:16.694 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:16.694 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:16:16.701 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:16.704 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:16:16.704 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:16:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:16:16.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:16:16.710 INFO 2981023 --- [host-check-host-status3,04403b2295dad6ae,fed0c8b5789b5480] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:16:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:16:16.721 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:16:16.724 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:16:16.733 INFO 2981023 --- [host-check-host-status4,04403b2295dad6ae,19c5b01042e43c17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:16:16.733 INFO 2981023 --- [host-check-host-status2,04403b2295dad6ae,44e4b2c17ed8a22d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:16:16.735 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:16.755 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:16.760 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:16:16.769 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.769 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.770 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:16:16.775 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:16:16.778 INFO 2981023 --- [host-check-host-status1,04403b2295dad6ae,2cbc19f87e348f36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:16:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:16.781 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:16.785 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:16.395298+08:00 2024-09-11 17:16:16.799 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:16:16.801 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:16:16.805 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.805 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.806 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:16:16.811 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:16:16.811 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:16:16.830 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:16:16.835 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:16.836 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:16.836 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:16:16.841 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:16:16.841 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:16:16.845 INFO 2981023 --- [ocp-schedules-12,dfaf1b355b56f768,fe91c8be21dfbc33] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:17.032 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046100, nodeCount=480, validCount=480 2024-09-11 17:16:17.115 INFO 2981023 --- [ocp-async-5,28cbb53ce783108f,fcd20e6d39ca4831] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,77c05cf8464f3c1b,b33e52c728c384a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:17.318 INFO 2981023 --- [prometheus-minute-schedule-1,77c05cf8464f3c1b,b33e52c728c384a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:17.731 INFO 2981023 --- [ocp-async-3,775b59c41dea89d5,22402f10187829da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:17.810 INFO 2981023 --- [host-check-clockdiff2,97a3a4b122e58884,2d92282dd026c4a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046177 0 0 , error= 2024-09-11 17:16:18.375 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,736f74268e2be753,a996b6754f99ebdb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=736f74268e2be753, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:18.398 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,736f74268e2be753,a996b6754f99ebdb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=736f74268e2be753, duration=28 ms] 2024-09-11 17:16:18.670 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b0847a1eb4a2fe13,d9ebaf5950f537de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b0847a1eb4a2fe13, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:18.676 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b0847a1eb4a2fe13,d9ebaf5950f537de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b0847a1eb4a2fe13, duration=9 ms] 2024-09-11 17:16:19.115 INFO 2981023 --- [ocp-async-1,272995077aa42b5a,3675d4989a520f5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:19.448 INFO 2981023 --- [ocp-schedules-12,92f9ebfdef77a83f,b3c36114608811bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:19.508 WARN 2981023 --- [scanner-13451968-1,92f9ebfdef77a83f,26e0e3a7d490683c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:19.577 INFO 2981023 --- [alarm-detect-17,92f9ebfdef77a83f,0964e75d002bb351] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046145 ~ 1726046169 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046160, value=2.325)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.325, values={1726046160=2.325}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:19.577 INFO 2981023 --- [alarm-detect-17,92f9ebfdef77a83f,0964e75d002bb351] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046145 ~ 1726046169 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726046160, value=1.706)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.706, values={1726046160=1.706}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:19.577 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:19.580 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:19.722 INFO 2981023 --- [host-check-clockdiff1,97a3a4b122e58884,0f4e605b477458aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046179 0 0 , error= 2024-09-11 17:16:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,aefd9552f7b21502,2d5c6e421f6d70a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,aefd9552f7b21502,2d5c6e421f6d70a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:20.407 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a08c653e1a00a207,e88478209781858b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=a08c653e1a00a207, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:20.417 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a08c653e1a00a207,e88478209781858b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=a08c653e1a00a207, duration=12 ms] 2024-09-11 17:16:21.115 INFO 2981023 --- [ocp-async-8,d5b1de8868f67fa9,8313a92f41a9d26c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:21.576 INFO 2981023 --- [ocp-schedules-22,78cdfc9fa8e114c1,764491b4eea8c299] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:21.951 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:21.951 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:16:21.952 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:16:21.952 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:16:21.952 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:16:21.956 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:21.956 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:21.957 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:21.960 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:21.961 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:21.961 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:21.961 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:21.964 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:21.987 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:16:21.987 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:16:21.990 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:16:21.992 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:16:21.993 INFO 2981023 --- [ocp-schedules-27,26c102d9b9d19b93,e5a21bda3defa7d9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:16:22.013 INFO 2981023 --- [host-check-clockdiff3,97a3a4b122e58884,6954326b7b1a25cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046182 1 1 , error= 2024-09-11 17:16:22.166 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2c9a7de9eb04ef8d,8c4a4303490ad7e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=2c9a7de9eb04ef8d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:22.176 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2c9a7de9eb04ef8d,8c4a4303490ad7e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=2c9a7de9eb04ef8d, duration=13 ms] 2024-09-11 17:16:22.439 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,eb9f35ce4547f4f5,ea5fe17cc9f81847] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=eb9f35ce4547f4f5, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:22.441 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,eb9f35ce4547f4f5,ea5fe17cc9f81847] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=eb9f35ce4547f4f5, duration=5 ms] 2024-09-11 17:16:22.580 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:22.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:23.116 INFO 2981023 --- [ocp-async-7,0403c7a249e78315,9cd8ef8890e779e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,a76dd42a49bb3c0a,1b9ea3407b026a87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,a76dd42a49bb3c0a,1b9ea3407b026a87] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:24.112 INFO 2981023 --- [ocp-schedules-1,6d8238d1e8ed60be,7e2c40181c273c71] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:16:24.459 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7c89cb34151f7855,730061f16e44e262] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=7c89cb34151f7855, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:24.469 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7c89cb34151f7855,730061f16e44e262] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=7c89cb34151f7855, duration=13 ms] 2024-09-11 17:16:24.846 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8728fbc85b543b9c,0a38446264e883e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=8728fbc85b543b9c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:24.848 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8728fbc85b543b9c,0a38446264e883e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=8728fbc85b543b9c, duration=5 ms] 2024-09-11 17:16:25.116 INFO 2981023 --- [ocp-async-6,cf450cfd721af1a6,cdb74c1a0fa88858] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:25.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:25.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,c379fd6990e8d000,c40605c5754a14f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,c379fd6990e8d000,c40605c5754a14f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:26.358 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:26.357214+08:00, lastProcessTime=2024-09-11T17:16:16.395298+08:00 2024-09-11 17:16:26.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:26.357214+08:00 2024-09-11 17:16:26.584 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e3465d0c138117da,ca7a2dd8a33797fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=e3465d0c138117da, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:26.595 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e3465d0c138117da,ca7a2dd8a33797fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=e3465d0c138117da, duration=14 ms] 2024-09-11 17:16:26.897 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,988ac4830c337be6,b44c94d5242fa7e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=988ac4830c337be6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:26.899 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,988ac4830c337be6,b44c94d5242fa7e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=988ac4830c337be6, duration=5 ms] 2024-09-11 17:16:27.116 INFO 2981023 --- [ocp-async-10,a142e5da68887622,78a4c89f11f57ae4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:27.732 INFO 2981023 --- [ocp-async-1,f2de69171ded8a24,8799fedc099e345d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:28.501 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,fbe326dcab22d239,5e5cd0e0ce407b58] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=fbe326dcab22d239, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:28.511 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,fbe326dcab22d239,5e5cd0e0ce407b58] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=fbe326dcab22d239, duration=13 ms] 2024-09-11 17:16:28.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:28.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:28.773 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71ca981096374ef3,a6e70af85c48ca32] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=71ca981096374ef3, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:28.775 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71ca981096374ef3,a6e70af85c48ca32] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=71ca981096374ef3, duration=5 ms] 2024-09-11 17:16:29.116 INFO 2981023 --- [ocp-async-7,877fe94451f8f2e2,90b62d861c8c50da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,57a16983e237ab1e,4580d6256d82e585] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,57a16983e237ab1e,4580d6256d82e585] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:29.649 INFO 2981023 --- [ocp-schedules-12,80db34e1913a17cb,d5855b5b25cb067d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:29.695 WARN 2981023 --- [scanner-13452161-1,80db34e1913a17cb,5eb3653f21e1003d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:29.723 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,84c23978729fd1e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:29.728 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,84c23978729fd1e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:29.733 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,84c23978729fd1e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:29.741 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,b86202092d525dcb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:29.745 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,b86202092d525dcb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:29.748 INFO 2981023 --- [alarm-detect-20,80db34e1913a17cb,d9006f5aa5d93046] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:29.752 INFO 2981023 --- [alarm-detect-20,80db34e1913a17cb,d9006f5aa5d93046] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:29.753 INFO 2981023 --- [alarm-detect-15,80db34e1913a17cb,b86202092d525dcb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:29.757 INFO 2981023 --- [alarm-detect-20,80db34e1913a17cb,d9006f5aa5d93046] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:29.865 INFO 2981023 --- [alarm-detect-6,80db34e1913a17cb,e2f7015979b1fffb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:29.868 INFO 2981023 --- [alarm-detect-6,80db34e1913a17cb,e2f7015979b1fffb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:29.872 INFO 2981023 --- [alarm-detect-6,80db34e1913a17cb,e2f7015979b1fffb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:30.374 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ed6caf3145304b18,571813d45091b771] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=ed6caf3145304b18, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:30.383 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ed6caf3145304b18,571813d45091b771] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=ed6caf3145304b18, duration=12 ms] 2024-09-11 17:16:30.632 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,44c8f5bf0fcd3360,e1950a02ef0caefa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=44c8f5bf0fcd3360, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:30.634 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,44c8f5bf0fcd3360,e1950a02ef0caefa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=44c8f5bf0fcd3360, duration=5 ms] 2024-09-11 17:16:31.116 INFO 2981023 --- [ocp-async-4,e61b797bc83c0f83,305f091f6dbb4b47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:31.579 INFO 2981023 --- [ocp-schedules-2,8dee53b25a0f8f72,6801e76f08b4ede4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:31.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:31.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:32.272 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,eaa6993608def31c,53e1ef5d82392344] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=eaa6993608def31c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:32.282 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,eaa6993608def31c,53e1ef5d82392344] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=eaa6993608def31c, duration=13 ms] 2024-09-11 17:16:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,6a386013c3d07ed7,e4c48f14c75f91ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,6a386013c3d07ed7,e4c48f14c75f91ea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:33.116 INFO 2981023 --- [ocp-async-9,34c83986622075f4,db3ee251abde94fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:33.162 INFO 2981023 --- [ocp-schedules-3,43b600ed1f126f32,c5757a0347457c24] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:16:33.171 INFO 2981023 --- [ocp-schedules-3,43b600ed1f126f32,c5757a0347457c24] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:33.171 INFO 2981023 --- [ocp-schedules-3,43b600ed1f126f32,c5757a0347457c24] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:34.065 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,359d1d3c7eafca68,a12743a6a51e98e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=359d1d3c7eafca68, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:34.075 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,359d1d3c7eafca68,a12743a6a51e98e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=359d1d3c7eafca68, duration=12 ms] 2024-09-11 17:16:34.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:34.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:34.826 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f4fcfd21b8822592,5927530ea6d1bc0c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=f4fcfd21b8822592, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:34.829 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f4fcfd21b8822592,5927530ea6d1bc0c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=f4fcfd21b8822592, duration=5 ms] 2024-09-11 17:16:35.116 INFO 2981023 --- [ocp-async-3,33848c9b09204228,4ce6e12d7505bc6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,913555b8553d1a38,dd92cde6213ca73e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,913555b8553d1a38,dd92cde6213ca73e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:36.337234+08:00, lastProcessTime=2024-09-11T17:16:26.357214+08:00 2024-09-11 17:16:36.399 INFO 2981023 --- [ocp-schedules-22,1d586c71cecd02dd,ec070cbf5e2cb3b7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:16:36.422 WARN 2981023 --- [ocp-schedules-22,1d586c71cecd02dd,ec070cbf5e2cb3b7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:16:36.475 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:36.337234+08:00 2024-09-11 17:16:36.877 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,f86ae446c7e44dc9,d32ee2f05186a1f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=f86ae446c7e44dc9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:36.887 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,f86ae446c7e44dc9,d32ee2f05186a1f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=f86ae446c7e44dc9, duration=12 ms] 2024-09-11 17:16:37.116 INFO 2981023 --- [ocp-async-2,a042c66defe1a295,d8a808efe82a0e89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:37.151 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,7f2473a089a5b8da,fd1d2d80dbbe3a78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=7f2473a089a5b8da, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:37.154 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,7f2473a089a5b8da,fd1d2d80dbbe3a78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=7f2473a089a5b8da, duration=5 ms] 2024-09-11 17:16:37.314 INFO 2981023 --- [prometheus-cache-monitor1,cb53c58c702f4ab2,f623ccba0e0bac2c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2050,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4529} 2024-09-11 17:16:37.315 INFO 2981023 --- [prometheus-cache-monitor1,cb53c58c702f4ab2,f623ccba0e0bac2c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:16:37.410 INFO 2981023 --- [ocp-schedules-15,2984d6730d6f4343,35c495e097b9b226] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:16:37.584 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:37.584 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:37.732 INFO 2981023 --- [ocp-async-7,f00987b33179e3dd,03bb30f0cb674cb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,819774dd24830b63,2de6e27b378a34cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,819774dd24830b63,2de6e27b378a34cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:38.813 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aa020f15c7cf5a6e,fa97aa8130471ac6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=aa020f15c7cf5a6e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:38.824 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aa020f15c7cf5a6e,fa97aa8130471ac6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=aa020f15c7cf5a6e, duration=14 ms] 2024-09-11 17:16:39.116 INFO 2981023 --- [ocp-async-1,4425d65e4b9dc2a8,bd75bab613336f61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:39.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,aa88213dc94d63e0,0b241ea0b8f4e2b2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=aa88213dc94d63e0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:39.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,aa88213dc94d63e0,0b241ea0b8f4e2b2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=aa88213dc94d63e0, duration=5 ms] 2024-09-11 17:16:39.911 INFO 2981023 --- [ocp-schedules-23,cc7a395370811e36,d9d0f224ddbbc2ea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:39.954 WARN 2981023 --- [scanner-13452477-1,cc7a395370811e36,8a89ff65584710c9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:40.008 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:16:40.009 INFO 2981023 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:16:40.011 INFO 2981023 --- [alarm-detect-7,cc7a395370811e36,dd10561af3cb052e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046160 ~ 1726046189 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046175, value=1.955)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.955, values={1726046175=1.955}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:40.011 INFO 2981023 --- [alarm-detect-7,cc7a395370811e36,dd10561af3cb052e] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-3] 2024-09-11 17:16:40.011 INFO 2981023 --- [alarm-detect-7,cc7a395370811e36,dd10561af3cb052e] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-2, alarm_template_id=0:host=10.31.35.10-3, alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.6-2, alarm_template_id=0:host=10.31.35.5-2, alarm_template_id=0:host=10.31.35.7-3, alarm_template_id=0:host=10.31.35.7-2, alarm_template_id=0:host=10.31.35.6-3] 2024-09-11 17:16:40.584 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:40.585 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:40.802 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f3f9afcb8e4c02a5,551730e6a1fcbcc7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=f3f9afcb8e4c02a5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:40.812 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f3f9afcb8e4c02a5,551730e6a1fcbcc7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=f3f9afcb8e4c02a5, duration=13 ms] 2024-09-11 17:16:41.084 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,13c75876056d5860,c1d8cc941e980016] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=13c75876056d5860, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:41.086 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,13c75876056d5860,c1d8cc941e980016] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=13c75876056d5860, duration=5 ms] 2024-09-11 17:16:41.116 INFO 2981023 --- [ocp-async-8,d817802a7bdf8139,59f452b9135699f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,78e284f89fc17cc4,b80038ca34e7ccda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,78e284f89fc17cc4,b80038ca34e7ccda] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:41.582 INFO 2981023 --- [ocp-schedules-28,04176ff13a0fd9ab,7b30cf0541f7da8d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:42.804 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d8d33c2be30f6765,1e06257e573f67b4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=d8d33c2be30f6765, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:42.824 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d8d33c2be30f6765,1e06257e573f67b4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=d8d33c2be30f6765, duration=23 ms] 2024-09-11 17:16:43.117 INFO 2981023 --- [ocp-async-6,1177f4ff89aa017f,8325e7a3cda9f5d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:43.146 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,585f18eed84e6c3e,ae3d8985957db465] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=585f18eed84e6c3e, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:43.150 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,585f18eed84e6c3e,ae3d8985957db465] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=585f18eed84e6c3e, duration=6 ms] 2024-09-11 17:16:43.585 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:43.585 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,d1785da55b503961,292a6cfeca2299d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,d1785da55b503961,292a6cfeca2299d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:45.101 INFO 2981023 --- [prometheus-second-schedule-1,32f0909e4016f533,75ee45a07b67cd10] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:16:45.117 INFO 2981023 --- [ocp-async-4,c473cd7d576e7359,4ee2aca56caf3a97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:45.559 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,feac42dacbac723d,2822b512a300d220] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=feac42dacbac723d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:45.569 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,feac42dacbac723d,2822b512a300d220] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=feac42dacbac723d, duration=13 ms] 2024-09-11 17:16:45.822 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,614434d87186d8ea,38a0110309c9e953] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=614434d87186d8ea, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:45.825 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,614434d87186d8ea,38a0110309c9e953] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=614434d87186d8ea, duration=6 ms] 2024-09-11 17:16:46.015 INFO 2981023 --- [ocp-schedules-12,fc31e50b7280b1a6,8b0c7c7bc56ec92e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:46.016 INFO 2981023 --- [host-check-clockdiff1,fc31e50b7280b1a6,14cad3233dc2e63a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:16:46.016 INFO 2981023 --- [host-check-clockdiff2,fc31e50b7280b1a6,4211805bc0cdf6d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:16:46.016 INFO 2981023 --- [host-check-clockdiff3,fc31e50b7280b1a6,96f3dfa91eb4bc0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:16:46.018 INFO 2981023 --- [host-check-clockdiff4,fc31e50b7280b1a6,c966db7f1f91556a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:16:46.022 INFO 2981023 --- [ocp-schedules-22,cfd8cfc5a1e185ce,8fd0744bb55edceb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:46.026 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:16:46.034 INFO 2981023 --- [host-check-clockdiff4,fc31e50b7280b1a6,c966db7f1f91556a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046206 0 0 , error= 2024-09-11 17:16:46.034 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:16:46.034 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:16:46.035 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:16:46.038 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:46.038 INFO 2981023 --- [ocp-schedules-14,c15d1ac1a5fb60c5,ee5fa4112a1d0e02] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:16:46.038 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:16:46.038 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:16:46.040 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:46.041 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:16:46.042 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:46.043 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:46.044 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:46.044 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:46.045 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:46.046 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:46.046 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:16:46.047 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:46.048 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:46.049 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:46.052 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:46.052 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:46.054 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:46.058 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:46.063 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:46.065 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:16:46.067 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:16:46.067592407+08:00 2024-09-11 17:16:46.070 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:16:46.067+08:00 2024-09-11 17:16:46.071 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:16:46.072 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:16:46.067+08:00 and status AVAILABLE 2024-09-11 17:16:46.076 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:16:46.076 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:16:46.077 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:16:46.077 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:16:46.077 INFO 2981023 --- [host-check-agent-status3,c15d1ac1a5fb60c5,5004bc8d73794ade] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:16:46.078 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:16:46.078 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:16:46.078026314+08:00 2024-09-11 17:16:46.079 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:16:46.078+08:00 2024-09-11 17:16:46.081 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:16:46.078+08:00 and status AVAILABLE 2024-09-11 17:16:46.083 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:16:46.084 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:46.085 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:46.086 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:16:46.086 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:16:46.086 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:16:46.086 INFO 2981023 --- [host-check-agent-status1,c15d1ac1a5fb60c5,cdd58b8c89158d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:16:46.092 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:16:46.080860564+08:00 2024-09-11 17:16:46.093 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:16:46.092+08:00 2024-09-11 17:16:46.093 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:46.094 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:16:46.095 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:46.102 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:16:46.092+08:00 and status AVAILABLE 2024-09-11 17:16:46.106 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:16:46.106257528+08:00 2024-09-11 17:16:46.107 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:16:46.106+08:00 2024-09-11 17:16:46.109 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:16:46.106+08:00 and status AVAILABLE 2024-09-11 17:16:46.115 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:46.117 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:46.126 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:16:46.128 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:16:46.128 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:16:46.128 INFO 2981023 --- [host-check-agent-status2,c15d1ac1a5fb60c5,fe1824e835acba8b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:16:46.132 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:16:46.138 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:16:46.139 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:16:46.148 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:16:46.150 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:16:46.166 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:16:46.169 INFO 2981023 --- [ocp-schedules-24,a214188a82b59a2f,74810e6c10e945e1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:16:46.171 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:16:46.176 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:46.183 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:16:46.183 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:16:46.183 INFO 2981023 --- [host-check-agent-status4,c15d1ac1a5fb60c5,0a66f00533ff77e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:16:46.184 INFO 2981023 --- [ocp-schedules-24,a214188a82b59a2f,74810e6c10e945e1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:16:46.187 INFO 2981023 --- [ocp-schedules-20,ce9e9108d5c1ad82,53f1b2ed1c4eee3c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:16:46.187 INFO 2981023 --- [ocp-schedules-3,d44799973594492f,80521b45313a1094] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:16:46.188 INFO 2981023 --- [ocp-schedules-10,28e30da3ded50e97,37c2de5f423f8d11] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:16:46.189 INFO 2981023 --- [ocp-schedules-3,d44799973594492f,80521b45313a1094] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:46.190 INFO 2981023 --- [ocp-schedules-20,ce9e9108d5c1ad82,53f1b2ed1c4eee3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:46.191 INFO 2981023 --- [ocp-schedules-10,28e30da3ded50e97,37c2de5f423f8d11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:46.194 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:16:46.195 INFO 2981023 --- [ocp-schedules-6,ba810cd98d4e07d5,f1eb3606d11bad04] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:16:46.195 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:16:46.197 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:16:46.198 INFO 2981023 --- [ocp-schedules-3,d44799973594492f,80521b45313a1094] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:16:46.199 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:16:46.209 INFO 2981023 --- [ocp-schedules-11,c0b0cfe06356444f,fb984ab10f360b14] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:16:46.211 INFO 2981023 --- [ocp-schedules-11,c0b0cfe06356444f,fb984ab10f360b14] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:46.216 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:16:46.216 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:16:46.226 INFO 2981023 --- [ocp-schedules-26,2110fe6ffc8a4c54,3dcbc5b207cd991b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:16:46.226 INFO 2981023 --- [ocp-schedules-26,2110fe6ffc8a4c54,3dcbc5b207cd991b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:16:46.228 INFO 2981023 --- [ocp-schedules-13,eacf4010d64a33e1,d1fdffbac5fb3b8d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:16:46.238 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:16:46.257 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:16:46.279 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:46.282 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.283 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:46.284 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.284 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:46.285 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:16:46.288 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:16:46.288 INFO 2981023 --- [ocp-schedules-20,ce9e9108d5c1ad82,53f1b2ed1c4eee3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:46.289 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.289 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:16:46.291 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.291 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.294 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:16:46.305 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:16:46.305 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,14dcb2710686bab4,a38b4af499a6797c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=14dcb2710686bab4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:16:46.305 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,14dcb2710686bab4,a38b4af499a6797c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=14dcb2710686bab4, duration=0 ms] 2024-09-11 17:16:46.307 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.307 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.318 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.319 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:46.319 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.321 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.321 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:16:46.321 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.322 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:16:46.328 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:16:46.328 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:16:46.332 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,96de2e2f83f1b4d0,d2f4945b935181ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=96de2e2f83f1b4d0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:16:46.332 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,96de2e2f83f1b4d0,d2f4945b935181ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=96de2e2f83f1b4d0, duration=1 ms] 2024-09-11 17:16:46.337 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.338 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:16:46.338 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.340 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:16:46.345 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.348 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:46.347806+08:00, lastProcessTime=2024-09-11T17:16:36.337234+08:00 2024-09-11 17:16:46.348 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:16:46.350 INFO 2981023 --- [ocp-schedules-11,c0b0cfe06356444f,fb984ab10f360b14] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:16:46.352 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:16:46.352 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.352 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:16:46.352 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.353 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.355 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.355 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.367 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.373 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:16:46.373 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:16:46.379 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:46.379 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:16:46.383 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:16:46.384 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:46.388 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:16:46.388 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:16:46.409 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.412 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.412 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.413 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:16:46.413 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.417 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:16:46.419 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:16:46.420 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:16:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:16:46.431 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:16:46.437 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:46.443 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:16:46.444 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:16:46.444 INFO 2981023 --- [host-check-host-status1,cfd8cfc5a1e185ce,3e7115c61644c4c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:16:46.448 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:46.449 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.450 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.452 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.453 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.453 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.458 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.459 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.460 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:46.461 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:16:46.465 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:16:46.466 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:16:46.468 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:16:46.478 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:16:46.480 INFO 2981023 --- [host-check-host-status2,cfd8cfc5a1e185ce,ff6a3e06c2d4035c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:16:46.480 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:16:46.481 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:16:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:16:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.488 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:16:46.491 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:16:46.498 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:46.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:16:46.514 INFO 2981023 --- [pool-10-thread-6,0857941388d70e85,1f055cfb73fddd13] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:16:46.524 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:16:46.530 INFO 2981023 --- [pool-10-thread-5,0857941388d70e85,dafa118382b43b61] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:16:46.530 INFO 2981023 --- [pool-10-thread-5,0857941388d70e85,dafa118382b43b61] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:16:46.531 INFO 2981023 --- [pool-10-thread-6,0857941388d70e85,1f055cfb73fddd13] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:16:46.531 INFO 2981023 --- [pool-10-thread-6,0857941388d70e85,1f055cfb73fddd13] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:16:46.541 INFO 2981023 --- [pool-10-thread-7,0857941388d70e85,5fd44c6bf5620d50] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:16:46.541 INFO 2981023 --- [pool-10-thread-8,0857941388d70e85,e063a5202097be7b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:16:46.542 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.543 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:16:46.544 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.545 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:16:46.545 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:16:46.546 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.549 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:16:46.553 INFO 2981023 --- [host-check-host-status4,cfd8cfc5a1e185ce,30b95359abd42d12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:16:46.553 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:16:46.555 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:16:46.555 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:16:46.557 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:16:46.557 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:16:46.565 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:16:46.571 INFO 2981023 --- [ForkJoinPool-6-worker-2,0857941388d70e85,628095018c8aacc0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:16:46.571 INFO 2981023 --- [ForkJoinPool-6-worker-2,0857941388d70e85,628095018c8aacc0] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:16:46.585 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:46.587 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.589 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:46.589 INFO 2981023 --- [host-check-host-status3,cfd8cfc5a1e185ce,c2547d875f38767c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:16:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:16:46.608 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.609 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.609 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:16:46.610 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:16:46.610 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.610 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.611 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.611 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.612 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:16:46.612 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.612 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:16:46.612 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.613 WARN 2981023 --- [ForkJoinPool-6-worker-2,0857941388d70e85,628095018c8aacc0] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:16:46.613 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.613 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.614 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:16:46.614 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:16:46.616 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.617 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:16:46.619 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:16:46.619 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:16:46.633 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:16:46.645 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.648 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:16:46.648 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:16:46.652 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.652 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:16:46.654 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.655 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:16:46, 2024-09-11 17:16:46] 2024-09-11 17:16:46.655 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:16:46.655 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.655 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.656 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:16:46.656 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:16:46.656 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:16:46.656 INFO 2981023 --- [ForkJoinPool-6-worker-4,0857941388d70e85,724eb56dd5c5ffe3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:16:46.657 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:16:46.657 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.658 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:16:46.659 INFO 2981023 --- [ForkJoinPool-6-worker-11,0857941388d70e85,62f2b8c17f0841df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:16:46.667 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:16:46.669 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.669 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:16:46.670 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.671 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:16:46.672 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.672 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:16:46.673 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:16:46.675 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.676 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:16:46.678 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:16:46.680 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:16:46.680 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:16:46.681 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.682 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:16:46.682 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.683 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.684 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.684 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.685 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.685 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.685 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:16:46.689 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:16:46.690 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.690 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:16:46.690 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:16:46.691 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:16:46, 2024-09-11 17:16:46] 2024-09-11 17:16:46.701 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:16:46.702 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:16:46.703 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.704 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:16:46, 2024-09-11 17:16:46] 2024-09-11 17:16:46.705 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:16:46.705 INFO 2981023 --- [ForkJoinPool-6-worker-9,0857941388d70e85,0a457f3a24a862d1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:16:46.706 INFO 2981023 --- [pool-10-thread-8,0857941388d70e85,e063a5202097be7b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:16:46.707 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.709 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:16:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:16:46.714 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.719 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.719 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:16:46.723 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:16:46.728 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:16:46.729 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:16:46.731 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.732 INFO 2981023 --- [ForkJoinPool-6-worker-7,0857941388d70e85,03d900681bad1802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:16:46.733 INFO 2981023 --- [pool-10-thread-7,0857941388d70e85,5fd44c6bf5620d50] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:16:46.734 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.735 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.735 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:16:46.736 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:16:46.736 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.753 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:16:46.764 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:46.765 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:46.765 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:16:46.772 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:16:46.772 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:16:46.773 INFO 2981023 --- [ocp-schedules-10,28e30da3ded50e97,37c2de5f423f8d11] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:46.789 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:16:46.806 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:16:46.818 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:16:46.821 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=2, resultsCount=2 2024-09-11 17:16:46.823 INFO 2981023 --- [ocp-schedules-15,68f40a99137a2d62,0c2dd96f567a385d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:16:46.829 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:46.347806+08:00 2024-09-11 17:16:46.858 INFO 2981023 --- [pool-10-thread-5,0857941388d70e85,dafa118382b43b61] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:16:46.860 INFO 2981023 --- [pool-10-thread-5,0857941388d70e85,dafa118382b43b61] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:16:47.117 INFO 2981023 --- [ocp-async-5,16af094555db13bf,f75c5f55a1f124a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,9e5da42541590711,023679812bc61dc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:47.318 INFO 2981023 --- [prometheus-minute-schedule-1,9e5da42541590711,023679812bc61dc5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:47.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,626664bd9897bcc0,141355fdad33067a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=626664bd9897bcc0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:47.494 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,626664bd9897bcc0,141355fdad33067a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=626664bd9897bcc0, duration=27 ms] 2024-09-11 17:16:47.732 INFO 2981023 --- [ocp-async-3,e83a5dcbe7708102,395b04e7a816e0ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:47.754 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aa8ca79bc11f97e3,6a51f6f076d1bc81] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=aa8ca79bc11f97e3, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:47.760 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aa8ca79bc11f97e3,6a51f6f076d1bc81] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=aa8ca79bc11f97e3, duration=9 ms] 2024-09-11 17:16:47.807 INFO 2981023 --- [host-check-clockdiff2,fc31e50b7280b1a6,4211805bc0cdf6d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046207 0 0 , error= 2024-09-11 17:16:49.117 INFO 2981023 --- [ocp-async-9,9862bc2bb732800d,810dc954605dfae9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:49.433 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,88d10bda729cb70e,503e568a55a0b3d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=88d10bda729cb70e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:49.443 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,88d10bda729cb70e,503e568a55a0b3d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=88d10bda729cb70e, duration=12 ms] 2024-09-11 17:16:49.589 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:49.591 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:49.648 INFO 2981023 --- [ocp-schedules-17,541b22b9d1de1560,f121ac69e7670a57] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:16:49.648+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:16:49.650 INFO 2981023 --- [ocp-schedules-17,541b22b9d1de1560,f121ac69e7670a57] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:16:49.695 INFO 2981023 --- [ocp-schedules-17,541b22b9d1de1560,f121ac69e7670a57] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 48 to SqlAggregateData 2024-09-11 17:16:49.697 INFO 2981023 --- [ocp-schedules-17,541b22b9d1de1560,f121ac69e7670a57] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:16:49.727 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ac1c8cf17022d373,279ea37e12629f22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=ac1c8cf17022d373, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:49.729 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ac1c8cf17022d373,279ea37e12629f22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=ac1c8cf17022d373, duration=5 ms] 2024-09-11 17:16:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:16:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:16:50.030 INFO 2981023 --- [ocp-schedules-13,887fba4558129367,38617b104231ea30] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:16:50.032 INFO 2981023 --- [ocp-schedules-13,887fba4558129367,38617b104231ea30] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:50.063 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.064 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.066 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.067 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.068 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.068 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.069 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:16:50.071 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.072 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.072 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.073 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.073 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.073 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.078 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.078 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.083 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.083 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:16:50.087 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.087 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.093 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.094 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:16:50.094 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.094 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.099 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE14D1D73E-0-0] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.100 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.100 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.115 INFO 2981023 --- [ocp-schedules-21,6d269dda202190be,3bc2637c5e024f87] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:16:50.121 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.121 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.125 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.125 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.126 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.127 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.127 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.127 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.128 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.128 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:16:50.128 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE14D1D748-0-0] 2024-09-11 17:16:50.128 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:16:50.128 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.129 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.129 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.129 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:16:50.132 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.133 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:16:50.135 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.136 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.137 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.139 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.140 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.140 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.141 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.141 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.142 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.142 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.143 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.143 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.144 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.144 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.145 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.145 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.145 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:16:50.146 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.146 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.147 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.147 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.147 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:16:50.148 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.148 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.148 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:16:50.149 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:16:50.150 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:16:50.152 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:16:50.154 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:16:50.156 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:50.158 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.158 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.170 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:50.170 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:16:50.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.172 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.173 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:16:50.173 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.173 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.178 INFO 2981023 --- [ocp-schedules-3,b65fbb8c95eaf294,9d603b6a04a7904a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:16:50.178 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.180 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:16:50.180 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:16:50.180 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:16:50.180 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:16:50.181 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.181 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:16:50.184 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.187 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.187 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:16:50.188 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.188 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:16:50.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:50.190 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:16:50.196 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.198 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:16:50.198 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:16:50.198 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:16:50.200 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:16:50.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:16:50.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:16:50.202 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:16:50.213 WARN 2981023 --- [scanner-13452658-1,6d269dda202190be,827a47ebaad58651] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:16:50.233 INFO 2981023 --- [ocp-schedules-13,887fba4558129367,38617b104231ea30] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:16:50.236 INFO 2981023 --- [alarm-detect-13,6d269dda202190be,ab701e5cc09b075d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643417, data is empty. 2024-09-11 17:16:50.242 INFO 2981023 --- [host-check-clockdiff1,fc31e50b7280b1a6,14cad3233dc2e63a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046210 0 0 , error= 2024-09-11 17:16:50.246 INFO 2981023 --- [alarm-detect-13,6d269dda202190be,ab701e5cc09b075d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:16:50.251 INFO 2981023 --- [alarm-detect-13,6d269dda202190be,ab701e5cc09b075d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:16:50.267 INFO 2981023 --- [alarm-detect-19,6d269dda202190be,f41b31264b7cc987] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643417, data is empty. 2024-09-11 17:16:50.269 INFO 2981023 --- [alarm-detect-2,6d269dda202190be,d742d9a0cc2bd59e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643417, data is empty. 2024-09-11 17:16:50.270 INFO 2981023 --- [alarm-detect-19,6d269dda202190be,f41b31264b7cc987] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:16:50.274 INFO 2981023 --- [alarm-detect-19,6d269dda202190be,f41b31264b7cc987] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:16:50.274 INFO 2981023 --- [alarm-detect-2,6d269dda202190be,d742d9a0cc2bd59e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:16:50.279 INFO 2981023 --- [alarm-detect-2,6d269dda202190be,d742d9a0cc2bd59e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:16:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,f1c5e66be3575c1b,5fa19965e398e753] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:50.318 INFO 2981023 --- [alarm-detect-13,6d269dda202190be,49787abe6320d3fb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046175 ~ 1726046200 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046190, value=1.522)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.522, values={1726046190=1.522}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:16:50.324 INFO 2981023 --- [prometheus-minute-schedule-1,f1c5e66be3575c1b,5fa19965e398e753] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:50.406 INFO 2981023 --- [alarm-detect-17,6d269dda202190be,5126f9803c79255c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643417, data is empty. 2024-09-11 17:16:50.409 INFO 2981023 --- [alarm-detect-17,6d269dda202190be,5126f9803c79255c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:16:50.415 INFO 2981023 --- [alarm-detect-17,6d269dda202190be,5126f9803c79255c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:16:51.117 INFO 2981023 --- [ocp-async-4,6a0f4916c1bd2429,4e66a9811cd351f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:51.357 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fe8271684fdf20d2,8d4ce0dfb56965d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=fe8271684fdf20d2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:51.367 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fe8271684fdf20d2,8d4ce0dfb56965d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=fe8271684fdf20d2, duration=13 ms] 2024-09-11 17:16:51.483 INFO 2981023 --- [ocp-schedules-14,1a2e721be4f27b9a,8af9340fdf254198] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:16:51.484 INFO 2981023 --- [ocp-schedules-14,1a2e721be4f27b9a,8af9340fdf254198] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:16:51.484 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:16:51.484 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:16:51.484 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:16:51.484 INFO 2981023 --- [ocp-schedules-14,1a2e721be4f27b9a,8af9340fdf254198] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:16:51.585 INFO 2981023 --- [ocp-schedules-2,68205cccabf48dc1,2b7b28f5c414c1d8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:16:51.639 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f72934174d5b95a6,b8583125c6709711] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=f72934174d5b95a6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:51.642 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f72934174d5b95a6,b8583125c6709711] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=f72934174d5b95a6, duration=6 ms] 2024-09-11 17:16:51.716 INFO 2981023 --- [ocp-schedules-9,24d57373fa03ef3e,0265e2848f2e6ccd] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:16:51.734 WARN 2981023 --- [ocp-schedules-9,24d57373fa03ef3e,0265e2848f2e6ccd] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:16:52.013 INFO 2981023 --- [host-check-clockdiff3,fc31e50b7280b1a6,96f3dfa91eb4bc0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046212 1 1 , error= 2024-09-11 17:16:52.591 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:52.593 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:52.966 INFO 2981023 --- [ocp-schedules-5,7da632c4b1fb2702,7b6c7f95e00e738a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:16:52.966 INFO 2981023 --- [ocp-schedules-5,7da632c4b1fb2702,7b6c7f95e00e738a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:16:53.117 INFO 2981023 --- [ocp-async-3,9f1fab4fe9488399,5c6c8ab354ad461c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:53.173 INFO 2981023 --- [ocp-schedules-9,193bfca0ee2b28d7,b6e4184c1fd39a36] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:16:53.183 INFO 2981023 --- [ocp-schedules-9,193bfca0ee2b28d7,b6e4184c1fd39a36] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:53.183 INFO 2981023 --- [ocp-schedules-9,193bfca0ee2b28d7,b6e4184c1fd39a36] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:16:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,b6d6e593eee1f665,5ad5e08d736255ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,b6d6e593eee1f665,5ad5e08d736255ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:53.956 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b2162631831c248c,54c42d6247a9c41e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=b2162631831c248c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:53.966 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b2162631831c248c,54c42d6247a9c41e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=b2162631831c248c, duration=13 ms] 2024-09-11 17:16:54.115 INFO 2981023 --- [ocp-schedules-3,d9030317d66ef784,c5b806a324296387] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:16:54.231 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8ded399ce418e402,11ca4e266fd6f6a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=8ded399ce418e402, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:54.234 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8ded399ce418e402,11ca4e266fd6f6a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=8ded399ce418e402, duration=5 ms] 2024-09-11 17:16:55.117 INFO 2981023 --- [ocp-async-2,4cc874407fa16fc3,fdde66d37e1981af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:55.234 INFO 2981023 --- [ocp-schedules-16,4ed210ea4653ceff,383d1803fcb1e717] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:16:55.234 INFO 2981023 --- [ocp-schedules-16,4ed210ea4653ceff,383d1803fcb1e717] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:16:55.267 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:16:55.267 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:16:55.273 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:16:55.273 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:16:55.274 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:16:55.275 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:16:55.287 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.288 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.288 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:16:55.289 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.290 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:16:55.290 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.290 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:16:55.291 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions3,4ed210ea4653ceff,bb28370323175034] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:16:55.292 INFO 2981023 --- [Collect all tenants major compactions1,4ed210ea4653ceff,99359e91562c4986] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:16:55.293 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:16:55.293 INFO 2981023 --- [Collect all tenants major compactions2,4ed210ea4653ceff,a47682ac71e5a969] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:16:55.293 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:16:55.294 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:16:55.294 INFO 2981023 --- [Collect all tenants major compactions5,4ed210ea4653ceff,9b50cccd75346d09] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:16:55.295 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:16:55.295 INFO 2981023 --- [Collect all tenants major compactions4,4ed210ea4653ceff,1d9c09f86eb889b9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:16:55.296 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:16:55.297 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:16:55.300 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:16:55.300 INFO 2981023 --- [Collect all tenants major compactions6,4ed210ea4653ceff,fe4fcc09147a373e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:16:55.301 INFO 2981023 --- [ocp-schedules-16,4ed210ea4653ceff,383d1803fcb1e717] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:16:55.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:55.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:16:55.595 INFO 2981023 --- [ocp-schedules-11,74943a1bc9929cb8,b377f132110b5756] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:16:55.622 INFO 2981023 --- [ocp-schedules-3,88db8fcc0b4d8641,eb8d87e7188977cd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:16:55.628 INFO 2981023 --- [ocp-schedules-3,88db8fcc0b4d8641,eb8d87e7188977cd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:16:55.628 INFO 2981023 --- [ocp-schedules-3,88db8fcc0b4d8641,eb8d87e7188977cd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:16:55.881 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ae11b2f84b826e4b,c01db995d989c6ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=ae11b2f84b826e4b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:55.891 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ae11b2f84b826e4b,c01db995d989c6ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=ae11b2f84b826e4b, duration=12 ms] 2024-09-11 17:16:56.143 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7d92ac71a59b51e2,546375dcdbfbce5f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=7d92ac71a59b51e2, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:56.146 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7d92ac71a59b51e2,546375dcdbfbce5f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=7d92ac71a59b51e2, duration=6 ms] 2024-09-11 17:16:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,b6bf4b274aeca0f4,71336672210c771f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,b6bf4b274aeca0f4,71336672210c771f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:16:56.352747+08:00, lastProcessTime=2024-09-11T17:16:46.347806+08:00 2024-09-11 17:16:56.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:16:56.352747+08:00 2024-09-11 17:16:57.117 INFO 2981023 --- [ocp-async-4,365d5ae979266b03,a1e0196f7b47c373] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:57.732 INFO 2981023 --- [ocp-async-9,70dd10c4ed597182,b13c3b3aaf06523a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:16:57.767 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,283b4a0d66cb397b,36574465af9e7b07] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=283b4a0d66cb397b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:57.777 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,283b4a0d66cb397b,36574465af9e7b07] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=283b4a0d66cb397b, duration=12 ms] 2024-09-11 17:16:57.991 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987} 2024-09-11 17:16:57.994 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obCluste 2024-09-11 17:16:57.994 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:57.998 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:57.998 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:57.998 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.028 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,007d693802ee1d8a,c483e293d7b705d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=007d693802ee1d8a, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:58.037 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818037000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.037Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.042 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818042000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.042Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.042 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,007d693802ee1d8a,c483e293d7b705d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=007d693802ee1d8a, duration=19 ms] 2024-09-11 17:16:58.042 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818042000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.042Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.058 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818058000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.058Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.058 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818058000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.058Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.060 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818060000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.060Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.061 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818061000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.061Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.062 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818062000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.062Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.066 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.066 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.066 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.066 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.095 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.095 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.095 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.095 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.105 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:16:58.105 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.108 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:16:58.108 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 57 2024-09-11 17:16:58.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.113 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 228 2024-09-11 17:16:58.113 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.118 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.118 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.121 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.121 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.121 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.121 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.122 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.122 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.123 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.123 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.125 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.125 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.126 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.126 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.127 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.128 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.128 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.128 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.130 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.130 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.131 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.131 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.132 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.132 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.134 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.134 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.136 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.137 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.139 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.140 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.140 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.140 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.141 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.142 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.042, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.143 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.143 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.145 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.145 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.146 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.058, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.147 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.147 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.148 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.148 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.150 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.042, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.150 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.037, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.151 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.151 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.152 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.154 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.156 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.06, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=F5AF8EF09E9E3C6FE7EB79D8EC048E5A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.160 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.160 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.162 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.162 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.162 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.165 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.058, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.165 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.168 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.062, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.172 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.061, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.176 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818176000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.176Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.183 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818183000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.183Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.186 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818186000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.186Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.187 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:16:58.187 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.194 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818194000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.194Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.195 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818195000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.195Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.199 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.199 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.205 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.205 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.206 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.206 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.208 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818208000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.208Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.208 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:16:58.208 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.209 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.209 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.211 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818211000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.211Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.220 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.220 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.221 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.221 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.223 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.223 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.224 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.224 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.225 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.225 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.226 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.226 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.227 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:16:58.227 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.228 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.228 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.229 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.229 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.230 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.231 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.231 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.236 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.176, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.236 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.236 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.237 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.237 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.238 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.238 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.238 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.239 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.239 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.240 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.240 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.240 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.243 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.243 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.243 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.243 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.244 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.186, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.245 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.183, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.246 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.246 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.246 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.246 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.247 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.249 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.249 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.251 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.194, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.251 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.253 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.253 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.254 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.255 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.255 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.208, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.257 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:16:58.257 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.211, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.260 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.195, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.260 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.262 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.262 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.262 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.272 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:16:58.285 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959818285000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:16:58.285Z, endTime=2024-09-11T09:14:57.987Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:16:58.287 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:16:58.287 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:16:58.292 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.292 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.293 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:16:58.293 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:16:58.298 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:16:58.298 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046097987000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:16:58.299 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:16:58.299 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:16:58.285, end=2024-09-11T17:14:57.987}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:16:58.299 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.300 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.300 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.300 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.302 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.302 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.302 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.302 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.303 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.307 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.307 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.307 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.317 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:16:58.318 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:16:58.318 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:57.56, end=2024-09-11T17:14:57.987},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:16:58.318 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:16:58.318 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:16:58.319 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:16:58.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:16:58.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:16:59.117 INFO 2981023 --- [ocp-async-7,8bd334a9a8efd27c,7e422ce890283615] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:16:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,d4d04353bf1021fa,e0e58b2286443dfa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046220, interval=60 2024-09-11 17:16:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,d4d04353bf1021fa,e0e58b2286443dfa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:16:59.663 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1175129c0a51688c,d3928665ea48dde0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=1175129c0a51688c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:16:59.673 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1175129c0a51688c,d3928665ea48dde0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=1175129c0a51688c, duration=13 ms] 2024-09-11 17:16:59.931 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,73fb8dc43839c37c,3ae463fb5260eb88] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=73fb8dc43839c37c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:16:59.933 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,73fb8dc43839c37c,3ae463fb5260eb88] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=73fb8dc43839c37c, duration=5 ms] 2024-09-11 17:17:00.471 INFO 2981023 --- [ocp-schedules-19,1adac90f75234eca,303140d73babf562] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:00.506 WARN 2981023 --- [scanner-13452972-1,1adac90f75234eca,a160c0ca67b0b106] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:01.117 INFO 2981023 --- [ocp-async-10,9b38f4c1854fa14c,c54d89a0e6fc09ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:01.588 INFO 2981023 --- [ocp-schedules-13,80395de33e5f9020,a0305bd51e5c7f9d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:01.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:01.595 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,d467b9caf89901a0,4b5fa0e08d6c8b6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,d467b9caf89901a0,4b5fa0e08d6c8b6e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:02.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,99e8c6a008def68e,6ccb818f602d2c22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=99e8c6a008def68e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:02.367 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,99e8c6a008def68e,6ccb818f602d2c22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=99e8c6a008def68e, duration=27 ms] 2024-09-11 17:17:02.624 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,467cbec0531a7b16,be82fb48f18a98d5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=467cbec0531a7b16, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:02.631 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,467cbec0531a7b16,be82fb48f18a98d5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=467cbec0531a7b16, duration=11 ms] 2024-09-11 17:17:03.118 INFO 2981023 --- [ocp-async-1,6e6dfe1ab1e6c9ef,edbd306b8775e4fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:04.279 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aad1086294ca5b54,8af013d03aab0111] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=aad1086294ca5b54, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:04.289 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,aad1086294ca5b54,8af013d03aab0111] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=aad1086294ca5b54, duration=13 ms] 2024-09-11 17:17:04.588 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,cfcb7b271a772ca7,3b8efbc0e2148021] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=cfcb7b271a772ca7, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:04.591 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,cfcb7b271a772ca7,3b8efbc0e2148021] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=cfcb7b271a772ca7, duration=6 ms] 2024-09-11 17:17:04.595 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:04.597 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:05.118 INFO 2981023 --- [ocp-async-8,93b7520f5f340e3a,d799df9d12abe7e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,7ab0a6e8f2d48dca,2b9dab0f2bcdf632] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,7ab0a6e8f2d48dca,2b9dab0f2bcdf632] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:06.352434+08:00, lastProcessTime=2024-09-11T17:16:56.352747+08:00 2024-09-11 17:17:06.406 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,75b363cebf9819e1,eb9284478831c1bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=75b363cebf9819e1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:06.418 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,75b363cebf9819e1,eb9284478831c1bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=75b363cebf9819e1, duration=15 ms] 2024-09-11 17:17:06.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:06.352434+08:00 2024-09-11 17:17:06.713 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b98bb1731d2908aa,6b1ae9e3a16d00c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b98bb1731d2908aa, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:06.716 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b98bb1731d2908aa,6b1ae9e3a16d00c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b98bb1731d2908aa, duration=6 ms] 2024-09-11 17:17:07.035 INFO 2981023 --- [ocp-schedules-20,b8a63ec5a2035c99,ec663e1b9702671f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:17:07.037 INFO 2981023 --- [ocp-schedules-7,4c71096f1b23c9b8,33ffce3b4220d19f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:17:07.037+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:17:07.037 INFO 2981023 --- [ocp-schedules-7,4c71096f1b23c9b8,33ffce3b4220d19f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:17:07.039 INFO 2981023 --- [ocp-schedules-7,4c71096f1b23c9b8,33ffce3b4220d19f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:17:07.040 INFO 2981023 --- [ocp-schedules-7,4c71096f1b23c9b8,33ffce3b4220d19f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:17:07.118 INFO 2981023 --- [ocp-async-10,61c21a6f3f0f8312,a671379fcbbf1f32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:07.597 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:07.599 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:07.732 INFO 2981023 --- [ocp-async-7,c7593e05ffb89a59,f3074435ccd7571c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,cade69d59dc33ad1,cf28443f5bbf950a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:08.318 INFO 2981023 --- [prometheus-minute-schedule-1,cade69d59dc33ad1,cf28443f5bbf950a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:08.333 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,63910a9921c951e2,69df9fe9ecbeea02] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=63910a9921c951e2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:08.358 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,63910a9921c951e2,69df9fe9ecbeea02] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=63910a9921c951e2, duration=33 ms] 2024-09-11 17:17:08.580 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b8322757ea16c7ae,89bf7c7f2c9228e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=b8322757ea16c7ae, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:08.586 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b8322757ea16c7ae,89bf7c7f2c9228e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=b8322757ea16c7ae, duration=9 ms] 2024-09-11 17:17:09.118 INFO 2981023 --- [ocp-async-1,8a3a68b2a6751454,9667fe088cef936b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:17:10.003 INFO 2981023 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:17:10.599 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:10.600 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:10.662 INFO 2981023 --- [ocp-schedules-3,66dd1bf89458cb0e,965477a8df432d5e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:10.690 WARN 2981023 --- [scanner-13453195-1,66dd1bf89458cb0e,785d3043345724f8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:10.747 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,51aeece1e33d5d98] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:10.754 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,51aeece1e33d5d98] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:10.758 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,51aeece1e33d5d98] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:10.777 INFO 2981023 --- [alarm-detect-7,66dd1bf89458cb0e,49831816aff26898] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:10.780 INFO 2981023 --- [alarm-detect-8,66dd1bf89458cb0e,dc43b4f3b381429a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:10.783 INFO 2981023 --- [alarm-detect-7,66dd1bf89458cb0e,49831816aff26898] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:10.787 INFO 2981023 --- [alarm-detect-8,66dd1bf89458cb0e,dc43b4f3b381429a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:10.793 INFO 2981023 --- [alarm-detect-7,66dd1bf89458cb0e,49831816aff26898] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:10.793 INFO 2981023 --- [alarm-detect-8,66dd1bf89458cb0e,dc43b4f3b381429a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:10.900 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,eaa6f20acbdd51ad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:10.903 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,eaa6f20acbdd51ad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:10.908 INFO 2981023 --- [alarm-detect-14,66dd1bf89458cb0e,eaa6f20acbdd51ad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:10.926 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:17:10.926 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:17:11.118 INFO 2981023 --- [ocp-async-8,da6bde86881fdb95,2885804db4914453] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,08a9855683c975b9,959015304d89a2de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,08a9855683c975b9,959015304d89a2de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:11.381 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7577cfe1f4aea94f,9a216dae1d44712c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=7577cfe1f4aea94f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:11.391 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7577cfe1f4aea94f,9a216dae1d44712c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=7577cfe1f4aea94f, duration=13 ms] 2024-09-11 17:17:11.591 INFO 2981023 --- [ocp-schedules-22,678ea62bb00a5b6e,f14c0e08ccbec206] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:11.653 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a82eb491380553cd,4260ee8a3e26e53e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=a82eb491380553cd, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:11.657 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a82eb491380553cd,4260ee8a3e26e53e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=a82eb491380553cd, duration=7 ms] 2024-09-11 17:17:12.082 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046160, nodeCount=401, affectRows=401 2024-09-11 17:17:12.207 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:17:12.225 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:12.228 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:12.303 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:12.304 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:12.320 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:12.320 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:12.333 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:12.334 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:12.346 INFO 2981023 --- [ocp-schedules-27,f809fb4796ad2d44,d7af5a6fd5297d46] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=5 2024-09-11 17:17:13.118 INFO 2981023 --- [ocp-async-6,6a06383b95e9346c,effc77ba5f4f49db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:13.186 INFO 2981023 --- [ocp-schedules-21,f3e6c2916e943493,145798f4c3b32f50] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:17:13.194 INFO 2981023 --- [ocp-schedules-21,f3e6c2916e943493,145798f4c3b32f50] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:13.194 INFO 2981023 --- [ocp-schedules-21,f3e6c2916e943493,145798f4c3b32f50] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:13.600 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:13.601 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:13.659 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05f389073fc35e80,2caad21f0331941f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=05f389073fc35e80, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:13.669 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05f389073fc35e80,2caad21f0331941f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=05f389073fc35e80, duration=13 ms] 2024-09-11 17:17:13.905 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046160, nodeCount=51, affectRows=51 2024-09-11 17:17:13.905 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046160, nodeCount=49, validCount=49 2024-09-11 17:17:13.942 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,84aec8587d66d9c9,cb6693859b523d4f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=84aec8587d66d9c9, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:17:13.945 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,84aec8587d66d9c9,cb6693859b523d4f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=84aec8587d66d9c9, duration=6 ms] 2024-09-11 17:17:14.044 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046160, nodeCount=1000, affectRows=1000 2024-09-11 17:17:14.045 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046160, nodeCount=1000, affectRows=1000 2024-09-11 17:17:14.047 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046160, nodeCount=1000, validCount=1000 2024-09-11 17:17:14.047 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046160, nodeCount=999, validCount=999 2024-09-11 17:17:14.112 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046160, nodeCount=401, validCount=401 2024-09-11 17:17:14.130 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046160, nodeCount=79, affectRows=79 2024-09-11 17:17:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,857e378482c5c87e,aa5661223de4635b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,675e4a6be4dbdccd,fb457a83449cf23f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:17:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,857e378482c5c87e,aa5661223de4635b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:14.443 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046160, nodeCount=79, validCount=79 2024-09-11 17:17:15.118 INFO 2981023 --- [ocp-async-4,5c1b00403ffa0fd0,15cc9dc0ae52a2cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:16.020 INFO 2981023 --- [ocp-schedules-20,81f60903753e0e98,3a8b838c38b77f13] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:16.021 INFO 2981023 --- [host-check-clockdiff1,81f60903753e0e98,ba226117c4ae5366] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:17:16.021 INFO 2981023 --- [ocp-schedules-7,3194b8fd1e664d04,0659e50fac637a9f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:16.022 INFO 2981023 --- [host-check-clockdiff3,81f60903753e0e98,38b30efe0bd21175] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:17:16.022 INFO 2981023 --- [host-check-clockdiff4,81f60903753e0e98,51016309c92fd6d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:17:16.023 INFO 2981023 --- [host-check-clockdiff2,81f60903753e0e98,21f6ffaf178f9764] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:17:16.025 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:17:16.025 INFO 2981023 --- [ocp-schedules-9,b8e9079e82222c81,7fc408979742b901] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:16.028 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:17:16.028 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:17:16.029 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:17:16.029 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:17:16.044 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:17:16.044 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:17:16.046 INFO 2981023 --- [host-check-clockdiff4,81f60903753e0e98,51016309c92fd6d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046236 0 0 , error= 2024-09-11 17:17:16.046 INFO 2981023 --- [host-check-clockdiff2,81f60903753e0e98,21f6ffaf178f9764] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046236 0 0 , error= 2024-09-11 17:17:16.046 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:17:16.063 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:16.067 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:16.070 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:16.073 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:16.073 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:16.074 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:16.075 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:16.075 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:16.077 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:16.078 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:16.078 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:16.079 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:16.084 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:16.085 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:16.086 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:16.090 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:17:16.092 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:17:16.092863307+08:00 2024-09-11 17:17:16.094 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:17:16.093+08:00 2024-09-11 17:17:16.095 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:17:16.093+08:00 and status AVAILABLE 2024-09-11 17:17:16.101 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:16.109 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:17:16.124 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:17:16.126 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:17:16.125664444+08:00 2024-09-11 17:17:16.138 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:17:16.126+08:00 2024-09-11 17:17:16.138 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:17:16.141 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:17:16.141 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:17:16.141 INFO 2981023 --- [host-check-agent-status1,3194b8fd1e664d04,960dbb71749b5813] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:17:16.170 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:17:16.126+08:00 and status AVAILABLE 2024-09-11 17:17:16.175 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:17:16.175 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:17:16.175589881+08:00 2024-09-11 17:17:16.176 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:17:16.175+08:00 2024-09-11 17:17:16.179 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:17:16.175+08:00 and status AVAILABLE 2024-09-11 17:17:16.184 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:17:16.184 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:17:16.186 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:17:16.186491274+08:00 2024-09-11 17:17:16.187 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:17:16.187 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:17:16.187 INFO 2981023 --- [host-check-agent-status2,3194b8fd1e664d04,4adb596d897aa1cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:17:16.187 INFO 2981023 --- [ocp-schedules-27,75635fcc16330ab6,fcc08868a75e3a92] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:17:16.191 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:17:16.194 INFO 2981023 --- [ocp-schedules-22,ce1ba4df73f1b615,17837a14a526ac2e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:17:16.194 INFO 2981023 --- [ocp-schedules-18,649655b1323da95b,7764a6566b5a2b08] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:17:16.194 INFO 2981023 --- [ocp-schedules-28,1739e27b161aba3b,66e1c19582f0d18a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:17:16.195 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:17:16.186+08:00 2024-09-11 17:17:16.197 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:17:16.197 INFO 2981023 --- [ocp-schedules-18,649655b1323da95b,7764a6566b5a2b08] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:16.200 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:17:16.200 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:17:16.200 INFO 2981023 --- [host-check-agent-status3,3194b8fd1e664d04,b79f3d307f1a89bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:17:16.202 INFO 2981023 --- [ocp-schedules-27,75635fcc16330ab6,fcc08868a75e3a92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:16.202 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:17:16.203 INFO 2981023 --- [ocp-schedules-22,ce1ba4df73f1b615,17837a14a526ac2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:16.204 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:16.206 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:17:16.186+08:00 and status AVAILABLE 2024-09-11 17:17:16.206 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:16.212 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:16.213 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:16.214 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:16.220 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:16.222 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:17:16.239 INFO 2981023 --- [ocp-schedules-29,1aa0956fdebed3bf,1e136f378ef64007] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:17:16.242 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:17:16.246 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:17:16.246 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:17:16.246 INFO 2981023 --- [host-check-agent-status4,3194b8fd1e664d04,22f88d8b6faf28f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:17:16.247 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:16.254 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:16.256 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:16.257 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:16.267 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:16.274 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:17:16.280 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:17:16.287 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:16.293 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.294 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.294 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:17:16.302 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:17:16.302 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:17:16.327 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:17:16.328 INFO 2981023 --- [ocp-schedules-22,ce1ba4df73f1b615,17837a14a526ac2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:17:16.339 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.343 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.343 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:16.344 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:17:16.349 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.350 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.353 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.353 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.354 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:17:16.356 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.356 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:16.357 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:17:16.371 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:17:16.371 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:17:16.371 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.372 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:16.373 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:17:16.381 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:16.382 INFO 2981023 --- [ocp-schedules-27,75635fcc16330ab6,fcc08868a75e3a92] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:16.382 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:16.381779+08:00, lastProcessTime=2024-09-11T17:17:06.352434+08:00 2024-09-11 17:17:16.388 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:17:16.389 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:17:16.397 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:16.406 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:17:16.413 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:16.422 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:17:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.425 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:17:16.425 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:17:16.425 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.425 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:16.426 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.426 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:17:16.427 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:17:16.427 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.429 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:17:16.430 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:17:16.434 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:17:16.439 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:17:16.450 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:17:16.454 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:17:16.458 INFO 2981023 --- [host-check-host-status3,b8e9079e82222c81,e198ef2908f8493d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:17:16.467 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:17:16.479 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:16.480 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:16.480 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:17:16.490 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.491 INFO 2981023 --- [host-check-host-status1,b8e9079e82222c81,7e1156458226b227] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:17:16.491 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.491 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:16.492 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:17:16.493 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:16.502 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:17:16.502 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.502 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.503 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:17:16.505 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:16.506 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:16.507 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:17:16.518 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:17:16.520 INFO 2981023 --- [host-check-host-status2,b8e9079e82222c81,58d1eb6f7fef21f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:17:16.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:17:16.527 INFO 2981023 --- [host-check-host-status4,b8e9079e82222c81,8f037dd6b190ab06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:17:16.537 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.538 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.539 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:16.539 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:17:16.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:17:16.544 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:17:16.552 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:16.552 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:17:16.565 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.573 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:16.573 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:17:16.576 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:17:16.580 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:16.581 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:16.593 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:17:16.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:17:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:17:16.601 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:16.601 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:17:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:17:16.626 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:17:16.634 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:16.635 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:16.635 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:17:16.636 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:16.381779+08:00 2024-09-11 17:17:16.641 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:17:16.641 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:17:16.641 INFO 2981023 --- [ocp-schedules-18,649655b1323da95b,7764a6566b5a2b08] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:17.118 INFO 2981023 --- [ocp-async-3,308e03bf103a6360,e026f155850d3056] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,5bb503ffdcaf98fb,058a79706bc1c08e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,5bb503ffdcaf98fb,058a79706bc1c08e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:17.732 INFO 2981023 --- [ocp-async-1,10b900b5aa8c4991,7226cb5faf1a3c3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:17.814 INFO 2981023 --- [host-check-clockdiff1,81f60903753e0e98,ba226117c4ae5366] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046237 0 0 , error= 2024-09-11 17:17:18.562 INFO 2981023 --- [host-check-clockdiff3,81f60903753e0e98,38b30efe0bd21175] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046238 1 1 , error= 2024-09-11 17:17:19.118 INFO 2981023 --- [ocp-async-10,99db165f6c879ab7,345b953e6ff37092] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:19.601 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:19.602 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,f74cebbb6b35a1d1,49c6ab034e9f19ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,f74cebbb6b35a1d1,49c6ab034e9f19ca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:20.968 INFO 2981023 --- [ocp-schedules-30,6703530af43bac2e,6a7b609d880175a7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:20.983 WARN 2981023 --- [scanner-13455029-1,6703530af43bac2e,170f8e39d0bbee3e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:21.119 INFO 2981023 --- [ocp-async-4,5b20b1809514a6d9,65575de0a0508153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:21.594 INFO 2981023 --- [ocp-schedules-9,eef3b71a6f30d850,3fa81643f9248e08] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:21.997 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:21.997 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:17:21.997 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:17:21.997 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:17:21.997 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:17:22.001 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:22.002 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:22.002 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:22.002 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:22.003 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:22.003 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:22.003 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:22.003 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:22.020 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:17:22.020 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:17:22.023 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:17:22.023 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:17:22.025 INFO 2981023 --- [ocp-schedules-15,14735d4b6d0cef2d,d4948783d0a82de2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:17:22.602 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:22.603 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:23.119 INFO 2981023 --- [ocp-async-3,41e68bc67ed95365,0765172904f42d0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,6fcc42d47260d144,cee28043c5a92a49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,6fcc42d47260d144,cee28043c5a92a49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:24.118 INFO 2981023 --- [ocp-schedules-8,7ecf35265b5775b8,71d1fe7d33fac2b0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:17:25.119 INFO 2981023 --- [ocp-async-7,27a6cd559c6ab4fc,09ab56aff5370e13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:25.603 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:25.604 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,2b2bb08275b0cea7,d78beef64dc28e89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,2b2bb08275b0cea7,d78beef64dc28e89] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:26.336983+08:00, lastProcessTime=2024-09-11T17:17:16.381779+08:00 2024-09-11 17:17:26.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:26.336983+08:00 2024-09-11 17:17:27.119 INFO 2981023 --- [ocp-async-2,bbc3e052399d04e3,54a02eb54fb7962a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:27.732 INFO 2981023 --- [ocp-async-10,01b4fc6256fb7adb,3c912c93f97f23c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:28.604 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:28.606 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:29.119 INFO 2981023 --- [ocp-async-5,5c82fdfd392be87a,f8b00c8579f21297] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,9c0b9b7f74726d1b,10e2b8c22e6cac36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,9c0b9b7f74726d1b,10e2b8c22e6cac36] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:31.119 INFO 2981023 --- [ocp-async-6,1b4273a9f03bd975,fc8a12fbccb3a120] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:31.153 INFO 2981023 --- [ocp-schedules-14,4118350254a53b73,a298453cb4c692af] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:31.297 WARN 2981023 --- [scanner-13455218-1,4118350254a53b73,264ba6c51695cb31] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:31.330 INFO 2981023 --- [alarm-detect-6,4118350254a53b73,7daf2617c486db1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:31.334 INFO 2981023 --- [alarm-detect-6,4118350254a53b73,7daf2617c486db1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:31.338 INFO 2981023 --- [alarm-detect-6,4118350254a53b73,7daf2617c486db1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:31.352 INFO 2981023 --- [alarm-detect-19,4118350254a53b73,41c281164087a7ee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:31.352 INFO 2981023 --- [alarm-detect-5,4118350254a53b73,25923f75d777bd25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:31.356 INFO 2981023 --- [alarm-detect-19,4118350254a53b73,41c281164087a7ee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:31.356 INFO 2981023 --- [alarm-detect-5,4118350254a53b73,25923f75d777bd25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:31.360 INFO 2981023 --- [alarm-detect-19,4118350254a53b73,41c281164087a7ee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:31.362 INFO 2981023 --- [alarm-detect-5,4118350254a53b73,25923f75d777bd25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:31.488 INFO 2981023 --- [alarm-detect-15,4118350254a53b73,accee0a53cfe3182] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:31.494 INFO 2981023 --- [alarm-detect-15,4118350254a53b73,accee0a53cfe3182] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:31.499 INFO 2981023 --- [alarm-detect-15,4118350254a53b73,accee0a53cfe3182] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:31.597 INFO 2981023 --- [ocp-schedules-16,264f629d3713f0d0,883c46038de1467c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:31.606 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:31.606 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,21d737339da66273,dfd9362ed093e377] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,21d737339da66273,dfd9362ed093e377] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:33.119 INFO 2981023 --- [ocp-async-9,cb22c4c36395b439,6c4081dd13fee4c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:33.197 INFO 2981023 --- [ocp-schedules-1,3fa8d79c57cf7a06,3ffb55c6b17c2132] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:17:33.205 INFO 2981023 --- [ocp-schedules-1,3fa8d79c57cf7a06,3ffb55c6b17c2132] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:33.205 INFO 2981023 --- [ocp-schedules-1,3fa8d79c57cf7a06,3ffb55c6b17c2132] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:34.606 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:34.607 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:35.119 INFO 2981023 --- [ocp-async-3,9ba163d91beb5b98,1d75da8a2cac9b44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,30ba455be8bdfc1a,e55d5df915bd855b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,30ba455be8bdfc1a,e55d5df915bd855b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:36.165 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7e7407bac282880a,78aba385f6527084] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=7e7407bac282880a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:36.187 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7e7407bac282880a,78aba385f6527084] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=7e7407bac282880a, duration=26 ms] 2024-09-11 17:17:36.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:36.351389+08:00, lastProcessTime=2024-09-11T17:17:26.336983+08:00 2024-09-11 17:17:36.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:36.351389+08:00 2024-09-11 17:17:36.568 INFO 2981023 --- [ocp-schedules-1,a6bebb5d1cf49350,df7d2f990aa7c350] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:17:36.585 WARN 2981023 --- [ocp-schedules-1,a6bebb5d1cf49350,df7d2f990aa7c350] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:17:37.119 INFO 2981023 --- [ocp-async-7,aca5bfd6932b8d60,09036cdf630c880e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:37.314 INFO 2981023 --- [prometheus-cache-monitor1,41ad6dbe6bae398c,70852fc49bc076d9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:17:37.315 INFO 2981023 --- [prometheus-cache-monitor1,41ad6dbe6bae398c,70852fc49bc076d9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:17:37.607 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:37.608 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:37.732 INFO 2981023 --- [ocp-async-5,87ab6aed07cae8a4,777c79331cc80591] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:37.873 INFO 2981023 --- [ocp-schedules-25,2acf5ab31b7fa168,5b5d58fb5ae1c81b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:17:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,ac0e9c76b897f4f9,4cff8b90fa8e138a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:38.315 INFO 2981023 --- [prometheus-minute-schedule-1,ac0e9c76b897f4f9,4cff8b90fa8e138a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:39.119 INFO 2981023 --- [ocp-async-2,83d4e745e040e079,56667f8758ec1ebf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:40.003 INFO 2981023 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:17:40.004 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:17:40.608 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:40.608 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:41.120 INFO 2981023 --- [ocp-async-4,a0d91c89a02114a6,f652f9b2329fcdee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,30728fa6dfb59c02,cd6396b9fac965d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,30728fa6dfb59c02,cd6396b9fac965d6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:41.532 INFO 2981023 --- [ocp-schedules-4,3c31328c3704768e,1a76f90be1031a9d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:41.572 WARN 2981023 --- [scanner-13455559-1,3c31328c3704768e,c1151fe745055fde] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:41.599 INFO 2981023 --- [ocp-schedules-14,9c30c19b706cbc19,115a854b14499094] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:43.120 INFO 2981023 --- [ocp-async-5,57f38c11191503f4,b9e36457fa6534c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:43.609 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:43.609 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,3798cc2ccfce1062,edfe862c0bd17cc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,3798cc2ccfce1062,edfe862c0bd17cc1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:45.101 INFO 2981023 --- [prometheus-second-schedule-1,fbbe531bc2cd626e,53e739d19e8f3331] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:17:45.120 INFO 2981023 --- [ocp-async-10,6e52a05ff0a04588,8570815192544a82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:46.020 INFO 2981023 --- [ocp-schedules-2,1ccad5e156179f1f,8ce2b54e3a1095c2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:46.020 INFO 2981023 --- [host-check-clockdiff1,1ccad5e156179f1f,84f501aa0028d35e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:17:46.021 INFO 2981023 --- [host-check-clockdiff2,1ccad5e156179f1f,5c67e56ee1a994e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:17:46.021 INFO 2981023 --- [host-check-clockdiff3,1ccad5e156179f1f,6b8d02504c1674b2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:17:46.022 INFO 2981023 --- [host-check-clockdiff4,1ccad5e156179f1f,103a218ebafbf298] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:17:46.026 INFO 2981023 --- [ocp-schedules-18,c16404f3aefd5ece,e0e5d75c86373176] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:46.031 INFO 2981023 --- [ocp-schedules-27,f279e4160a89dfce,54281717a1d36f42] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:17:46.031 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:17:46.031 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:17:46.031 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:17:46.033 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:17:46.034 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:17:46.034 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:17:46.034 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:17:46.037 INFO 2981023 --- [host-check-clockdiff4,1ccad5e156179f1f,103a218ebafbf298] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046266 0 0 , error= 2024-09-11 17:17:46.038 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:17:46.041 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:46.042 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:46.042 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:46.043 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:46.044 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:46.046 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:46.046 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:46.047 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:46.047 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:46.049 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:46.049 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:46.050 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:46.050 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:46.050 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:46.050 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:46.051 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:46.089 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:17:46.089 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:17:46.090 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:17:46.090 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:17:46.092 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:17:46.091875122+08:00 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status4,f279e4160a89dfce,2ce4e7c5e4cde366] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:17:46.093 INFO 2981023 --- [host-check-agent-status1,f279e4160a89dfce,f17c00dd91180486] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:17:46.094 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:17:46.094 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:17:46.095 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:17:46.095 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:17:46.095 INFO 2981023 --- [host-check-agent-status3,f279e4160a89dfce,90923ac5f177bcb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:17:46.096 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:17:46.096143242+08:00 2024-09-11 17:17:46.096 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:17:46.096+08:00 2024-09-11 17:17:46.096 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:17:46.097046845+08:00 2024-09-11 17:17:46.097 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:17:46.096+08:00 2024-09-11 17:17:46.098 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:17:46.096+08:00 and status AVAILABLE 2024-09-11 17:17:46.099 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:17:46.096+08:00 and status AVAILABLE 2024-09-11 17:17:46.102 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:17:46.092+08:00 2024-09-11 17:17:46.104 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:17:46.092+08:00 and status AVAILABLE 2024-09-11 17:17:46.105 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:17:46.105 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:17:46.107 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:17:46.107 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:17:46.107 INFO 2981023 --- [host-check-agent-status2,f279e4160a89dfce,c20b5d6e9148e60c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:17:46.117 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:17:46.117 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:17:46.117729124+08:00 2024-09-11 17:17:46.118 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:17:46.117+08:00 2024-09-11 17:17:46.120 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:17:46.117+08:00 and status AVAILABLE 2024-09-11 17:17:46.124 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:17:46.131 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:17:46.132 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:17:46.132 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:17:46.133 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:17:46.134 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:17:46.136 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:17:46.145 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:46.147 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:46.148 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:46.150 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:46.162 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:46.164 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:17:46.165 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:46.169 INFO 2981023 --- [ocp-schedules-28,6b680598c89dc853,ed3ad5f9399df552] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:17:46.177 INFO 2981023 --- [ocp-schedules-28,6b680598c89dc853,ed3ad5f9399df552] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:17:46.177 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:17:46.189 INFO 2981023 --- [ocp-schedules-6,00e116a63453d62c,83cb45b2d246956e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:17:46.189 INFO 2981023 --- [ocp-schedules-21,1df19799afde1634,dd5c03a1c2f12ba6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:17:46.189 INFO 2981023 --- [ocp-schedules-15,2997528274fc906c,2381b741c9db2728] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:17:46.189 INFO 2981023 --- [ocp-schedules-10,5ba648f694f94b20,62b178d2fdc6a3b6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:17:46.191 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:17:46.198 INFO 2981023 --- [ocp-schedules-15,2997528274fc906c,2381b741c9db2728] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:46.200 INFO 2981023 --- [ocp-schedules-21,1df19799afde1634,dd5c03a1c2f12ba6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:46.202 INFO 2981023 --- [ocp-schedules-6,00e116a63453d62c,83cb45b2d246956e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:46.202 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:17:46.217 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:17:46.228 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:17:46.232 INFO 2981023 --- [ocp-schedules-21,1df19799afde1634,dd5c03a1c2f12ba6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:17:46.243 INFO 2981023 --- [ocp-schedules-3,edbf5d9cee14e7e8,7f72e4007f56234a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:17:46.256 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:17:46.285 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:46.288 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:17:46.291 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.292 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:46.293 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:17:46.294 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.295 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:46.295 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:46.296 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:17:46.297 INFO 2981023 --- [ocp-schedules-6,00e116a63453d62c,83cb45b2d246956e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:46.309 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:46.309 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:17:46.310 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.310 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.311 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.311 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.316 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:17:46.319 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:17:46.324 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:17:46.331 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:17:46.338 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:17:46.344 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:17:46.344 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:46.342869+08:00, lastProcessTime=2024-09-11T17:17:36.351389+08:00 2024-09-11 17:17:46.349 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.349 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:17:46.349 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.356 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:17:46.360 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.360 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.364 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.365 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.376 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:46.380 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:46.393 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.394 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.394 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:46.394 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.395 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.395 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.398 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:17:46.401 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:17:46.409 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:17:46.411 INFO 2981023 --- [host-check-host-status3,c16404f3aefd5ece,c3d2adf87ecf1056] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:17:46.411 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:17:46.411 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046162041000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:17:46.413 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.418 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:17:46.418 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:17:46.419 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:17:46.424 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:17:46.430 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:17:46.432 INFO 2981023 --- [host-check-host-status2,c16404f3aefd5ece,334a5c320678800e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:17:46.454 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:46.458 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.459 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:17:46.484 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:17:46.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.489 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:17:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:17:46.495 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:17:46.499 INFO 2981023 --- [host-check-host-status1,c16404f3aefd5ece,dafc9a9560abb6ce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:17:46.499 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:17:46.509 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.509 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.510 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:17:46.510 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:17:46.511 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.511 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.512 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.512 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:46.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:17:46.518 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:17:46.532 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:46.536 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:17:46.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.539 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:17:46.542 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.543 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.544 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:46.546 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:17:46.548 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:17:46.552 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:17:46.558 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:17:46.561 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:17:46.569 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:17:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.575 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:17:46.575 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:17:46.576 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.576 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.577 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.577 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.578 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.578 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.578 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:17:46.579 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:17:46.581 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:17:46.581 INFO 2981023 --- [host-check-host-status4,c16404f3aefd5ece,9a9f899cdd9462bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:17:46.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:46.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:17:46.594 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:17:46.603 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:17:46.609 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:46.610 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:17:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:17:46.619 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:17:46.624 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:17:46.625 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:17:46.626 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:17:46.649 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.649 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:17:46.650 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.650 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:17:46.650 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:17:46.651 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.656 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:17:46.664 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:17:46.664 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:17:46.669 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:17:46.691 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:46.342869+08:00 2024-09-11 17:17:46.697 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:17:46.699 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:17:46.702 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:46.702 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:46.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:17:46.708 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:17:46.708 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:17:46.709 INFO 2981023 --- [ocp-schedules-15,2997528274fc906c,2381b741c9db2728] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:46.715 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:17:46.720 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:17:46.723 INFO 2981023 --- [ocp-schedules-5,020195d4fb16ec0c,88a69304b89523f3] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:17:46.834 INFO 2981023 --- [ocp-schedules-17,eca42ba3fd099579,d08644c954f09125] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:17:46.834 INFO 2981023 --- [ocp-schedules-17,eca42ba3fd099579,d08644c954f09125] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:17:46.967 INFO 2981023 --- [host-check-clockdiff2,1ccad5e156179f1f,5c67e56ee1a994e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046266 0 0 , error= 2024-09-11 17:17:47.120 INFO 2981023 --- [ocp-async-3,f1f1446916ab1d32,814d13435f934bc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,f72348386c4826c1,fe43774f3580293c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,f72348386c4826c1,fe43774f3580293c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:47.733 INFO 2981023 --- [ocp-async-1,cf588ebf9bbc2575,eceee7618cdfbd4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:49.120 INFO 2981023 --- [ocp-async-5,785fb426fa16d782,9c934a6b9ad352e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:49.610 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:49.612 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:49.718 INFO 2981023 --- [ocp-schedules-30,80e192e9667c5cb9,173bccffdc106f60] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:17:49.718+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:17:49.721 INFO 2981023 --- [host-check-clockdiff1,1ccad5e156179f1f,84f501aa0028d35e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046269 0 0 , error= 2024-09-11 17:17:49.750 INFO 2981023 --- [ocp-schedules-30,80e192e9667c5cb9,173bccffdc106f60] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:17:49.898 INFO 2981023 --- [ocp-schedules-30,80e192e9667c5cb9,173bccffdc106f60] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 47 to SqlAggregateData 2024-09-11 17:17:49.900 INFO 2981023 --- [ocp-schedules-30,80e192e9667c5cb9,173bccffdc106f60] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:17:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:17:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:17:50.171 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.175 INFO 2981023 --- [ocp-schedules-12,a8215d1b25a7a76e,feb906b4ee66d4d2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:17:50.180 INFO 2981023 --- [ocp-schedules-3,11920b0116e0cb1d,95a2f6b5d2ef090f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:17:50.233 INFO 2981023 --- [ocp-schedules-11,cbf9a3862ef3ea62,39d38c45e3491944] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:17:50.235 INFO 2981023 --- [ocp-schedules-11,cbf9a3862ef3ea62,39d38c45e3491944] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:50.268 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.268 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.270 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.271 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.272 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.272 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.273 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:17:50.276 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.276 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.277 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.277 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.277 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.277 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.284 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.285 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.287 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.288 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.290 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.290 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.291 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.292 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:17:50.299 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.299 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.299 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1591D753-0-0] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.300 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:17:50.303 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.304 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.307 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.307 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.312 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.312 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.313 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.313 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,0474537e59577610,e60e3104a712ed07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:50.313 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.314 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.314 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.315 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.315 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:17:50.317 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,0474537e59577610,e60e3104a712ed07] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:50.318 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.327 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:17:50.328 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.329 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.329 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:17:50.330 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1591D75E-0-0] 2024-09-11 17:17:50.330 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:17:50.336 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.337 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.337 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.337 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.337 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.338 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.339 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.339 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.339 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.339 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.340 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.340 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.340 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.340 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.341 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.342 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.342 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.342 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.343 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.343 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:17:50.343 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.344 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.345 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:17:50.345 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.345 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:17:50.346 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:50.346 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:17:50.346 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.347 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:17:50.347 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.348 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:17:50.348 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.349 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:17:50.356 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:50.358 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.358 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.359 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.362 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:17:50.362 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:17:50.362 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:17:50.366 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:17:50.367 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.368 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:17:50.368 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.370 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:17:50.370 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.370 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.373 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:17:50.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:17:50.379 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:17:50.380 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:50.381 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:17:50.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:17:50.386 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:17:50.386 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:17:50.386 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:17:50.386 INFO 2981023 --- [ocp-schedules-11,cbf9a3862ef3ea62,39d38c45e3491944] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:17:51.120 INFO 2981023 --- [ocp-async-2,be7caa5e04cf2096,7b081aee2f119c2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:51.485 INFO 2981023 --- [ocp-schedules-21,e8cfa6ccc7863d30,499bcbe7e955fc37] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:17:51.486 INFO 2981023 --- [ocp-schedules-21,e8cfa6ccc7863d30,499bcbe7e955fc37] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:17:51.486 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:17:51.486 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:17:51.486 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:17:51.487 INFO 2981023 --- [ocp-schedules-21,e8cfa6ccc7863d30,499bcbe7e955fc37] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:17:51.605 INFO 2981023 --- [ocp-schedules-3,4c609d679c02d162,91cbea0dc6adc06b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:17:51.723 INFO 2981023 --- [ocp-schedules-14,c7ade940b2150d38,d49976fac83c7fa2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:17:51.765 INFO 2981023 --- [ocp-schedules-11,8ecd85690c8f8280,6fbd2aee34ae8b36] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:17:51.798 WARN 2981023 --- [ocp-schedules-11,8ecd85690c8f8280,6fbd2aee34ae8b36] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:17:51.805 WARN 2981023 --- [scanner-13455803-1,c7ade940b2150d38,be9e46b8130478d9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:17:51.836 INFO 2981023 --- [alarm-detect-15,c7ade940b2150d38,e001ff19c7684aae] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643465, data is empty. 2024-09-11 17:17:51.841 INFO 2981023 --- [alarm-detect-15,c7ade940b2150d38,e001ff19c7684aae] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:17:51.847 INFO 2981023 --- [alarm-detect-15,c7ade940b2150d38,e001ff19c7684aae] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:17:51.848 INFO 2981023 --- [alarm-detect-4,c7ade940b2150d38,f6944ad8c4a765a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643465, data is empty. 2024-09-11 17:17:51.851 INFO 2981023 --- [alarm-detect-4,c7ade940b2150d38,f6944ad8c4a765a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:17:51.853 INFO 2981023 --- [alarm-detect-1,c7ade940b2150d38,4a1bb4caa7e3aad4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643465, data is empty. 2024-09-11 17:17:51.857 INFO 2981023 --- [alarm-detect-4,c7ade940b2150d38,f6944ad8c4a765a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:17:51.866 INFO 2981023 --- [alarm-detect-1,c7ade940b2150d38,4a1bb4caa7e3aad4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:17:51.870 INFO 2981023 --- [alarm-detect-1,c7ade940b2150d38,4a1bb4caa7e3aad4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:17:51.993 INFO 2981023 --- [alarm-detect-10,c7ade940b2150d38,a39a2b4f1c343101] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643465, data is empty. 2024-09-11 17:17:52.001 INFO 2981023 --- [alarm-detect-10,c7ade940b2150d38,a39a2b4f1c343101] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:17:52.014 INFO 2981023 --- [alarm-detect-10,c7ade940b2150d38,a39a2b4f1c343101] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:17:52.032 INFO 2981023 --- [host-check-clockdiff3,1ccad5e156179f1f,6b8d02504c1674b2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046272 1 1 , error= 2024-09-11 17:17:52.612 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:52.614 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:52.973 INFO 2981023 --- [ocp-schedules-22,af1dad9c566fbee7,9162b480d3ccab9b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:17:52.973 INFO 2981023 --- [ocp-schedules-22,af1dad9c566fbee7,9162b480d3ccab9b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:17:53.120 INFO 2981023 --- [ocp-async-1,ad49f1404de663cc,178c7b44921eb038] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:53.208 INFO 2981023 --- [ocp-schedules-27,270ae652bc987696,7a173a25039ec216] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:17:53.216 INFO 2981023 --- [ocp-schedules-27,270ae652bc987696,7a173a25039ec216] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:53.216 INFO 2981023 --- [ocp-schedules-27,270ae652bc987696,7a173a25039ec216] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:17:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,c7f08d1674612198,fe75f9e2c2c690cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,c7f08d1674612198,fe75f9e2c2c690cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:54.121 INFO 2981023 --- [ocp-schedules-26,fd4c173664afa72c,3f9800be5bc1b90a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:17:55.120 INFO 2981023 --- [ocp-async-8,2fff02c28114517d,e1211b87a465db0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:55.301 INFO 2981023 --- [ocp-schedules-16,de362979466917ee,f9d6acf7357a4bfe] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:17:55.301 INFO 2981023 --- [ocp-schedules-16,de362979466917ee,f9d6acf7357a4bfe] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:17:55.333 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:17:55.335 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:17:55.335 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:17:55.338 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:17:55.341 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:17:55.342 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:17:55.359 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.359 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.360 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:17:55.360 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:17:55.361 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.361 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.361 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.362 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:17:55.362 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:17:55.363 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions4,de362979466917ee,9fef51ce4461a278] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions2,de362979466917ee,0047126d2e6e2cf9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:17:55.364 INFO 2981023 --- [Collect all tenants major compactions1,de362979466917ee,88f249d8f8f692e9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:17:55.365 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:17:55.365 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:17:55.366 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:17:55.366 INFO 2981023 --- [Collect all tenants major compactions5,de362979466917ee,67319a4b480b5629] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:17:55.366 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:17:55.366 INFO 2981023 --- [Collect all tenants major compactions3,de362979466917ee,ea679002ccaa8921] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:17:55.367 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:17:55.367 INFO 2981023 --- [Collect all tenants major compactions6,de362979466917ee,f3c0b5514d27b929] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:17:55.368 INFO 2981023 --- [ocp-schedules-16,de362979466917ee,f9d6acf7357a4bfe] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:17:55.596 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:17:55.596 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:17:55.597 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:17:55.597 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:17:55.597 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:17:55.597 INFO 2981023 --- [ocp-schedules-30,5a8c24fb7ddd9d60,cf82ac794d962adc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:17:55.614 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:55.615 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:55.628 INFO 2981023 --- [ocp-schedules-1,22ee4dba498b480c,5b24709ff66d7e1e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:17:55.634 INFO 2981023 --- [ocp-schedules-1,22ee4dba498b480c,5b24709ff66d7e1e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:17:55.634 INFO 2981023 --- [ocp-schedules-1,22ee4dba498b480c,5b24709ff66d7e1e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:17:55.745 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,ec65929c4a66107e,71b7973f94cce0ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=ec65929c4a66107e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:17:55.769 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,ec65929c4a66107e,71b7973f94cce0ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=ec65929c4a66107e, duration=29 ms] 2024-09-11 17:17:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,fb2d9ebfc20fced2,30d1effd426cfe7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,fb2d9ebfc20fced2,30d1effd426cfe7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:17:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:17:56.351700+08:00, lastProcessTime=2024-09-11T17:17:46.342869+08:00 2024-09-11 17:17:56.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:17:56.351700+08:00 2024-09-11 17:17:57.120 INFO 2981023 --- [ocp-async-2,4b4013a630a6ef0b,944e23edd167f0fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:57.733 INFO 2981023 --- [ocp-async-9,96d22fdb3937ca5f,a4656f16e4cf6ab7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:17:58.009 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.012 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obC 2024-09-11 17:17:58.015 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.018 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.020 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.022 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.026 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.029 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.032 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.035 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.037 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.038 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.038 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.039 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.049 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.053 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.057 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.061 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.062 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.063 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.064 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.064 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.065 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.066 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.067 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.067 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:11:57.916, end=2024-09-11T17:16:58.004} 2024-09-11 17:17:58.067 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:17:58.615 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:17:58.615 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:17:59.121 INFO 2981023 --- [ocp-async-1,165a2d09f25b945d,cf5c35fde174232d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:17:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,599216fca57684e7,3303d34e6c976ca6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046280, interval=60 2024-09-11 17:17:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,599216fca57684e7,3303d34e6c976ca6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:01.125 INFO 2981023 --- [ocp-async-5,2bd2e59a96bc54af,695a8002ff7b18b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:01.608 INFO 2981023 --- [ocp-schedules-8,088efcd09884418a,d2ece8543af8cf12] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:18:01.615 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:01.616 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:01.927 INFO 2981023 --- [ocp-schedules-27,2fbb998ca7aec2be,4a8163dda62d2973] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:01.927 INFO 2981023 --- [ocp-schedules-27,2fbb998ca7aec2be,4a8163dda62d2973] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:18:02.048 INFO 2981023 --- [ocp-schedules-16,9fc2cd38d89e8136,472ad1785aeaa3fc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:18:02.051 INFO 2981023 --- [ocp-schedules-12,17b0fd469ad8e548,6c11e80c03e7577c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:18:02.054 INFO 2981023 --- [ocp-schedules-12,17b0fd469ad8e548,6c11e80c03e7577c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:02.095 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:02.095 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:18:02.101 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:02.105 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:18:02.106 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000} 2024-09-11 17:18:02.108 WARN 2981023 --- [scanner-13456110-1,9fc2cd38d89e8136,39530c56b2f9b56a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:18:02.114 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:18:02.123 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000222, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:18:02.156 INFO 2981023 --- [ocp-schedules-12,17b0fd469ad8e548,6c11e80c03e7577c] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:18:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,6ec184fb5165277d,45e412fbe08f13a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,6ec184fb5165277d,45e412fbe08f13a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:02.907 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 17:18:02.907 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 17:18:02.907 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 17:18:02.913 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000222, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:18:02.913 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 17:18:03.121 INFO 2981023 --- [ocp-async-2,1d7729124fc634af,272c5c7d449bfe14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:03.686 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000222 2024-09-11 17:18:04.616 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:04.617 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:05.121 INFO 2981023 --- [ocp-async-7,f09c5778ccd14936,e5a76ef1c951044e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f9f8f5dde61d0559,0d53612ae18be74d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,f9f8f5dde61d0559,0d53612ae18be74d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:18:05.981 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:18:05.986 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:18:05.991 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 17:18:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:18:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:18:05.994 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:05.994 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:18:05.995 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:05.995 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:18:06.044 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:15:41, start=2024-09-11T17:12:41, end=2024-09-11T17:15:11, updates=284, granule=120000ms, delay=180000ms 2024-09-11 17:18:06.051 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:15:41, target=2024-09-11T17:15:11, granule=120000ms, delay=30000ms) 2024-09-11 17:18:06.155 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T17:15:11, start=2024-09-11T17:00:51, end=2024-09-11T17:10:51, updates=387, granule=600000ms, delay=860000ms 2024-09-11 17:18:06.160 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:15:11, target=2024-09-11T17:10:51, granule=600000ms, delay=260000ms) 2024-09-11 17:18:06.160 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=193ms, period=120000ms 2024-09-11 17:18:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:18:06.337390+08:00, lastProcessTime=2024-09-11T17:17:56.351700+08:00 2024-09-11 17:18:06.468 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:18:06.337390+08:00 2024-09-11 17:18:07.051 INFO 2981023 --- [ocp-schedules-4,dd463a4516c53d12,a71837149ba3fd80] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:18:07.055 INFO 2981023 --- [ocp-schedules-20,73079e0b1df3a351,e00ddfb3aa032bc6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:18:07.055+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:18:07.056 INFO 2981023 --- [ocp-schedules-20,73079e0b1df3a351,e00ddfb3aa032bc6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:18:07.057 INFO 2981023 --- [ocp-schedules-20,73079e0b1df3a351,e00ddfb3aa032bc6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:18:07.057 INFO 2981023 --- [ocp-schedules-20,73079e0b1df3a351,e00ddfb3aa032bc6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:18:07.121 INFO 2981023 --- [ocp-async-6,a82a4d6f6c79ee88,0e86fe82e0709851] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:07.617 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:07.618 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:07.733 INFO 2981023 --- [ocp-async-1,7936a91297d2b5cb,8d19f1d3a485eeb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:18:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,75dcb84542e1490e,eaeb519adb2e1bea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:08.317 INFO 2981023 --- [prometheus-minute-schedule-1,75dcb84542e1490e,eaeb519adb2e1bea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:09.121 INFO 2981023 --- [ocp-async-10,ae79f1533fbb7074,8dae044b2be3c907] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:18:10.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:18:10.618 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:10.619 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:18:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:18:11.121 INFO 2981023 --- [ocp-async-7,c053708ce3203238,4f6f4326981375d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,6bf430de7abbd0b9,cc7e5fe3c46d65ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:11.320 INFO 2981023 --- [prometheus-minute-schedule-1,6bf430de7abbd0b9,cc7e5fe3c46d65ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:11.611 INFO 2981023 --- [ocp-schedules-30,305ce07f2165dc8c,fa80d2be45ceac59] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:18:12.287 INFO 2981023 --- [ocp-schedules-23,2588921b27719736,22beeae0622dd389] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:18:12.336 WARN 2981023 --- [scanner-13457785-1,2588921b27719736,42582f5860c7520b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:18:12.366 INFO 2981023 --- [alarm-detect-17,2588921b27719736,afb78f76fc7a0b69] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:12.370 INFO 2981023 --- [alarm-detect-17,2588921b27719736,afb78f76fc7a0b69] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:12.379 INFO 2981023 --- [alarm-detect-17,2588921b27719736,afb78f76fc7a0b69] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:12.402 INFO 2981023 --- [alarm-detect-1,2588921b27719736,b524cdcf3aad30dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:12.406 INFO 2981023 --- [alarm-detect-1,2588921b27719736,b524cdcf3aad30dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:12.409 INFO 2981023 --- [alarm-detect-6,2588921b27719736,6e79da15c16028d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:12.410 INFO 2981023 --- [alarm-detect-1,2588921b27719736,b524cdcf3aad30dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:12.414 INFO 2981023 --- [alarm-detect-6,2588921b27719736,6e79da15c16028d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:12.419 INFO 2981023 --- [alarm-detect-6,2588921b27719736,6e79da15c16028d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:12.541 INFO 2981023 --- [alarm-detect-11,2588921b27719736,3056f51df7272edb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:12.547 INFO 2981023 --- [alarm-detect-11,2588921b27719736,3056f51df7272edb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:12.550 INFO 2981023 --- [alarm-detect-11,2588921b27719736,3056f51df7272edb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:13.121 INFO 2981023 --- [ocp-async-9,7c7e63898d65d44c,743d4ee23f88760b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:13.219 INFO 2981023 --- [ocp-schedules-2,1091ad4cb7584a4e,74faad01f690e71c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:18:13.230 INFO 2981023 --- [ocp-schedules-2,1091ad4cb7584a4e,74faad01f690e71c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:18:13.230 INFO 2981023 --- [ocp-schedules-2,1091ad4cb7584a4e,74faad01f690e71c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:18:13.619 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:13.620 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:14.006 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046220, nodeCount=1000, affectRows=1000 2024-09-11 17:18:14.008 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046220, nodeCount=999, validCount=999 2024-09-11 17:18:14.055 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046220, nodeCount=21, affectRows=21 2024-09-11 17:18:14.055 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046220, nodeCount=21, validCount=21 2024-09-11 17:18:14.058 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046220, nodeCount=27, affectRows=27 2024-09-11 17:18:14.058 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046220, nodeCount=25, validCount=25 2024-09-11 17:18:14.087 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046220, nodeCount=1000, affectRows=1000 2024-09-11 17:18:14.090 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046220, nodeCount=1000, validCount=1000 2024-09-11 17:18:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,c941a60aff938a50,878852697f3e50cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,191aaa14a4522df3,bad9f94e935d7529] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:18:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,c941a60aff938a50,878852697f3e50cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:14.546 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046220, nodeCount=480, affectRows=480 2024-09-11 17:18:15.122 INFO 2981023 --- [ocp-async-5,b62068295eab72c3,dd6aae2060f3ce6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:16.020 INFO 2981023 --- [ocp-schedules-16,2678d6748d520ae2,34108a77642471db] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:16.021 INFO 2981023 --- [ocp-schedules-25,f1ac4286ce7cd150,1a917601b8de24b5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:16.021 INFO 2981023 --- [host-check-clockdiff2,2678d6748d520ae2,eece4053cac91780] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:18:16.021 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:18:16.022 INFO 2981023 --- [host-check-clockdiff3,2678d6748d520ae2,4378e6534f5316ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:18:16.022 INFO 2981023 --- [host-check-clockdiff4,2678d6748d520ae2,5084b5dd42f55a7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:18:16.023 INFO 2981023 --- [host-check-clockdiff1,2678d6748d520ae2,f2bf02e28d3756e5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:18:16.023 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:18:16.026 INFO 2981023 --- [ocp-schedules-12,d4ee40a8278dee3d,aa236d7143bdb3bf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:16.026 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:18:16.027 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:18:16.027 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:18:16.027 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:18:16.029 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:18:16.029 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:18:16.046 INFO 2981023 --- [host-check-clockdiff4,2678d6748d520ae2,5084b5dd42f55a7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046296 0 0 , error= 2024-09-11 17:18:16.066 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:16.067 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:16.070 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:16.070 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:16.071 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:16.071 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:16.071 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:16.075 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:16.076 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:16.076 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:16.076 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:16.077 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:16.090 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:16.192 INFO 2981023 --- [ocp-schedules-1,efc45cf8d3ce86db,b5f5fdcde18bb56e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:18:16.192 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:16.201 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:16.201 INFO 2981023 --- [ocp-schedules-9,f688bd01bbbd0e85,295f2c3d3f526203] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:18:16.201 INFO 2981023 --- [ocp-schedules-21,15cc8a29adc970ef,380e7a66185c8141] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:18:16.201 INFO 2981023 --- [ocp-schedules-4,c77334bc8bdb2302,6cf87a95fbf6b50f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:18:16.269 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:16.288 INFO 2981023 --- [ocp-schedules-21,15cc8a29adc970ef,380e7a66185c8141] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:16.290 INFO 2981023 --- [ocp-schedules-1,efc45cf8d3ce86db,b5f5fdcde18bb56e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:16.290 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:18:16.296 INFO 2981023 --- [ocp-schedules-9,f688bd01bbbd0e85,295f2c3d3f526203] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:16.298 INFO 2981023 --- [ocp-schedules-9,f688bd01bbbd0e85,295f2c3d3f526203] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:18:16.456 INFO 2981023 --- [ocp-schedules-26,10e281746ebbc948,c47db187ec5ed70d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:18:16.473 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:18:16.472768+08:00, lastProcessTime=2024-09-11T17:18:06.337390+08:00 2024-09-11 17:18:16.475 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:18:16.480 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:18:16.481 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:18:16.481 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:18:16.481 INFO 2981023 --- [host-check-agent-status3,f1ac4286ce7cd150,895258e6a9294041] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:18:16.482 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:18:16.483 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:18:16.485 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:18:16.484160726+08:00 2024-09-11 17:18:16.485 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:18:16.485063365+08:00 2024-09-11 17:18:16.488 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:18:16.488 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:18:16.485+08:00 2024-09-11 17:18:16.489 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:18:16.490 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.490 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:18:16.485+08:00 and status AVAILABLE 2024-09-11 17:18:16.491 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:18:16.490803532+08:00 2024-09-11 17:18:16.491 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:18:16.490966705+08:00 2024-09-11 17:18:16.491 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:16.491 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:18:16.485+08:00 2024-09-11 17:18:16.492 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:18:16.492 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:18:16.492 INFO 2981023 --- [host-check-agent-status4,f1ac4286ce7cd150,cac95f11e1510d75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:18:16.493 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:18:16.494 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:18:16.485+08:00 and status AVAILABLE 2024-09-11 17:18:16.495 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:18:16.491+08:00 2024-09-11 17:18:16.495 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:18:16.495 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:18:16.495 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:18:16.495 INFO 2981023 --- [host-check-agent-status1,f1ac4286ce7cd150,12dbdf11c374f0c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:18:16.498 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:18:16.491+08:00 2024-09-11 17:18:16.499 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:18:16.491+08:00 and status AVAILABLE 2024-09-11 17:18:16.500 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:18:16.491+08:00 and status AVAILABLE 2024-09-11 17:18:16.501 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.501 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.501 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:16.502 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:16.502 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:18:16.502 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:18:16.503 INFO 2981023 --- [ocp-schedules-1,efc45cf8d3ce86db,b5f5fdcde18bb56e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:18:16.503 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.504 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:16.507 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:18:16.507 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:18:16.507 INFO 2981023 --- [host-check-agent-status2,f1ac4286ce7cd150,51d4b7444460b603] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:18:16.508 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:18:16.512 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:18:16.513 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:18:16.513 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:18:16.570 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:16.570 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:16.570 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:16.571 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:16.571 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:16.573 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:16.574 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:16.575 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046220, nodeCount=480, validCount=480 2024-09-11 17:18:16.578 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:16.584 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.585 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.594 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:18:16.615 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:16.615 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:16.620 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:16.621 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:16.622 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:16.630 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:18:16.640 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.641 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:18:16.641 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.642 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:16.642 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:18:16.643 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.644 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:18:16.645 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:18:16.646 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:18:16.646 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:18:16.660 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:18:16.661 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:18:16.664 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:18:16.667 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:18:16.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:16.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:18:16.681 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:18:16.681 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:18:16.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:16.698 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:16.699 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:16.699 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:16.700 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:16.717 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.718 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.718 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:16.719 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:18:16.721 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:18:16.724 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:18:16.725 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:18:16.729 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:18:16.730 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:16.730 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:18:16.732 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:18:16.732 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:16.734 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:18:16.735 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:18:16.755 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:18:16.759 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:18:16.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:18:16.773 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:18:16.777 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:18:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:16.779 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:18:16.780 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.782 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:18:16.783 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:18:16.784 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:18:16.787 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:18:16.793 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:18:16.795 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:16.795 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:18:16.808 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:18:16.812 INFO 2981023 --- [host-check-host-status3,d4ee40a8278dee3d,ed284fc19dfa805d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:18:16.817 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:18:16.818 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:16.818 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:18:16.821 INFO 2981023 --- [host-check-host-status1,d4ee40a8278dee3d,c58524ea59478cc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:18:16.833 INFO 2981023 --- [host-check-host-status4,d4ee40a8278dee3d,d2809d134eae5a30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:18:16.842 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.842 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.842 INFO 2981023 --- [host-check-host-status2,d4ee40a8278dee3d,28eb1db83504f076] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:18:16.843 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:16.844 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:18:16.849 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:18:16.856 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:16.857 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:16.863 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:18:16.873 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.874 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.875 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:18:16.876 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:18:16.882 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:18:16.882 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:18:16.885 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:18:16.472768+08:00 2024-09-11 17:18:16.909 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:18:16.915 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:16.916 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:16.919 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:18:16.924 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:18:16.924 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:18:16.925 INFO 2981023 --- [ocp-schedules-21,15cc8a29adc970ef,380e7a66185c8141] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:18:16.970 INFO 2981023 --- [host-check-clockdiff2,2678d6748d520ae2,eece4053cac91780] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046296 0 0 , error= 2024-09-11 17:18:16.982 INFO 2981023 --- [host-check-clockdiff1,2678d6748d520ae2,f2bf02e28d3756e5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046296 0 0 , error= 2024-09-11 17:18:17.122 INFO 2981023 --- [ocp-async-4,40bfd23a25cd6c06,798fbe70335694f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,54aacaffb947a234,39f1299396e4dfef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,54aacaffb947a234,39f1299396e4dfef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:17.733 INFO 2981023 --- [ocp-async-10,c92d4b4e67d4e7b6,672d029614173b10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:18:19.122 INFO 2981023 --- [ocp-async-6,bfc296766390ef3d,578a841afed71932] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:19.621 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:19.621 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,296c61028b93a215,5de6fbd8e9f2e75e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,296c61028b93a215,5de6fbd8e9f2e75e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:21.122 INFO 2981023 --- [ocp-async-5,abb811e6aceb3d61,993913754094b9b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:21.318 INFO 2981023 --- [ocp-schedules-25,95fb38f60b89c653,a94386b9fe0b765e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 17:18:21.318 INFO 2981023 --- [ocp-schedules-25,95fb38f60b89c653,a94386b9fe0b765e] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 17:18:21.333 INFO 2981023 --- [pool-446-thread-2,95fb38f60b89c653,c5389b78c510d685] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:18:21.333 INFO 2981023 --- [pool-446-thread-3,95fb38f60b89c653,e048baf315db34b3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:18:21.334 INFO 2981023 --- [pool-446-thread-1,95fb38f60b89c653,22c07668484a8c8f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:18:21.352 INFO 2981023 --- [pool-446-thread-1,95fb38f60b89c653,22c07668484a8c8f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:21.353 INFO 2981023 --- [pool-446-thread-2,95fb38f60b89c653,c5389b78c510d685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:21.354 INFO 2981023 --- [pool-446-thread-1,95fb38f60b89c653,22c07668484a8c8f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046001000000, 1726046301000000] 2024-09-11 17:18:21.354 INFO 2981023 --- [pool-446-thread-3,95fb38f60b89c653,e048baf315db34b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:21.356 INFO 2981023 --- [pool-446-thread-2,95fb38f60b89c653,c5389b78c510d685] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046001000000, 1726046301000000] 2024-09-11 17:18:21.356 INFO 2981023 --- [pool-446-thread-3,95fb38f60b89c653,e048baf315db34b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046001000000, 1726046301000000] 2024-09-11 17:18:21.360 INFO 2981023 --- [pool-446-thread-3,95fb38f60b89c653,e048baf315db34b3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:18:21.427 INFO 2981023 --- [pool-446-thread-1,95fb38f60b89c653,22c07668484a8c8f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:18:21.497 INFO 2981023 --- [pool-446-thread-2,95fb38f60b89c653,c5389b78c510d685] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:18:21.498 INFO 2981023 --- [ocp-schedules-25,95fb38f60b89c653,a94386b9fe0b765e] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 17:18:21.614 INFO 2981023 --- [ocp-schedules-18,5fa9fa9221f79827,56902fd787de0ea2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:18:21.877 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,6958ea1a131c0888,1fcb0306e39aad4f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=6958ea1a131c0888, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:18:21.903 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,6958ea1a131c0888,1fcb0306e39aad4f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=6958ea1a131c0888, duration=32 ms] 2024-09-11 17:18:22.015 INFO 2981023 --- [host-check-clockdiff3,2678d6748d520ae2,4378e6534f5316ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046302 1 1 , error= 2024-09-11 17:18:22.029 INFO 2981023 --- [ocp-schedules-14,a724ed2dfc80dc78,b170f49246907ed0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:22.029 INFO 2981023 --- [ocp-schedules-14,a724ed2dfc80dc78,b170f49246907ed0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:18:22.029 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:18:22.029 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:18:22.030 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:18:22.034 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:22.034 INFO 2981023 --- [ocp-schedules-14,a724ed2dfc80dc78,b170f49246907ed0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:22.035 INFO 2981023 --- [ocp-schedules-14,a724ed2dfc80dc78,b170f49246907ed0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:22.036 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:22.037 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:22.037 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:22.038 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:22.039 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:22.056 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:18:22.058 INFO 2981023 --- [ocp-schedules-14,a724ed2dfc80dc78,b170f49246907ed0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:18:22.059 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:18:22.060 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:18:22.062 INFO 2981023 --- [ocp-schedules-14,a724ed2dfc80dc78,b170f49246907ed0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:18:22.413 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:18:22.413 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:18:22.416 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:18:22.416 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:18:22.416 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:18:22.416 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:18:22.418 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:18:22.418 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:18:22.418 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:18:22.418 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:18:22.440 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:18:22.444 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:18:22.444 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:18:22.444 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:18:22.444 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:18:22.444 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:18:22.448 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:18:22.608 INFO 2981023 --- [ocp-schedules-27,3e2cf8cd4611ff58,aaf859081a41ad10] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:18:22.622 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:22.623 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:22.728 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:18:22.773 WARN 2981023 --- [scanner-13458115-1,3e2cf8cd4611ff58,c1b6f965736d32df] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:18:22.925 INFO 2981023 --- [alarm-detect-18,3e2cf8cd4611ff58,52200848dc7151c4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046265 ~ 1726046292 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726046280, value=1.695)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.695, values={1726046280=1.695}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:18:23.097 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.678860771S 2024-09-11 17:18:23.097 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:18:23.122 INFO 2981023 --- [ocp-async-4,7ac476b64ab34b2d,89a962cd674b0cb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:23.163 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:18:23.163 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:18:23.188 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:18:23.230 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:18:23.230 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:18:23.230 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:18:23.230 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:18:23.230 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:18:23.230 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:18:23.235 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:18:23.312 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] o.s.boot.SpringApplication : Started application in 0.896 seconds (JVM running for 82103.185) 2024-09-11 17:18:23.313 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.896978552S 2024-09-11 17:18:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,f453e698e28ca06d,85ad3c75e3ced766] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,f453e698e28ca06d,85ad3c75e3ced766] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:23.515 INFO 2981023 --- [ocp-schedules-2,761a919d9900e4b6,9b3da638a50a7d5f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:18:24.124 INFO 2981023 --- [ocp-schedules-29,378feb7735a8c664,6dcdf1b16e784e61] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:18:25.122 INFO 2981023 --- [ocp-async-9,39151c05aa70b1d3,a910057835980df5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:25.623 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:25.623 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:26.126 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.130 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId= 2024-09-11 17:18:26.134 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.134 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:18:26.136 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.139 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.146 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.150 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.150 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:18:26.153 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.156 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.160 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.161 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.161 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:18:26.162 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.162 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.163 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.168 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.200 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 31 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.242 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 12 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.258 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 172 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.273 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.275 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.276 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.277 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.277 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.278 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.279 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.279 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.280 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.281 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.281 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:12:25.832, end=2024-09-11T17:17:26.122} 2024-09-11 17:18:26.281 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 4 dubious sqls 2024-09-11 17:18:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,ea66fa7d3b9380fc,c1ebef720f1a1b4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:26.315 INFO 2981023 --- [prometheus-minute-schedule-1,ea66fa7d3b9380fc,c1ebef720f1a1b4e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:18:26.336628+08:00, lastProcessTime=2024-09-11T17:18:16.472768+08:00 2024-09-11 17:18:26.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:18:26.336628+08:00 2024-09-11 17:18:27.122 INFO 2981023 --- [ocp-async-2,cac1115f3ef52022,51b6d4e15d851e99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:27.733 INFO 2981023 --- [ocp-async-6,9c732de199a87f4c,4a2006ac4058f559] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:18:28.623 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:28.624 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:29.122 INFO 2981023 --- [ocp-async-3,b0c1366e7f8598f5,3f2acdfca93718e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,4a62bd883e440420,c8d0b3a43ae22aa3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,4a62bd883e440420,c8d0b3a43ae22aa3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:31.124 INFO 2981023 --- [ocp-async-7,51f99fc8b9008fbb,5d706a074514888e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:31.616 INFO 2981023 --- [ocp-schedules-3,48aa42e35c584b67,003782bd7b78d1d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:18:31.624 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:31.633 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,f6ed99a4cfcf7c4a,1ed1a744916d88ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,f6ed99a4cfcf7c4a,1ed1a744916d88ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:33.122 INFO 2981023 --- [ocp-async-6,853add530a593f2d,a6b511780e6e3b21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:33.214 INFO 2981023 --- [ocp-schedules-22,4200034036c3e206,aace2cc6a73e824f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:18:33.234 INFO 2981023 --- [ocp-schedules-30,e947e3dbb095117d,7dbd9a2051b37ad4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:18:33.247 INFO 2981023 --- [ocp-schedules-30,e947e3dbb095117d,7dbd9a2051b37ad4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:18:33.247 INFO 2981023 --- [ocp-schedules-30,e947e3dbb095117d,7dbd9a2051b37ad4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:18:33.251 WARN 2981023 --- [scanner-13458366-1,4200034036c3e206,49681ea8b2283fab] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:18:33.285 INFO 2981023 --- [alarm-detect-17,4200034036c3e206,8b0f217188d946c1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:33.289 INFO 2981023 --- [alarm-detect-17,4200034036c3e206,8b0f217188d946c1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:33.293 INFO 2981023 --- [alarm-detect-17,4200034036c3e206,8b0f217188d946c1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:33.316 INFO 2981023 --- [alarm-detect-15,4200034036c3e206,95307c6c552b6c1a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:33.316 INFO 2981023 --- [alarm-detect-4,4200034036c3e206,3236409b3a0ab298] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:33.321 INFO 2981023 --- [alarm-detect-4,4200034036c3e206,3236409b3a0ab298] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:33.321 INFO 2981023 --- [alarm-detect-15,4200034036c3e206,95307c6c552b6c1a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:33.327 INFO 2981023 --- [alarm-detect-15,4200034036c3e206,95307c6c552b6c1a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:33.328 INFO 2981023 --- [alarm-detect-4,4200034036c3e206,3236409b3a0ab298] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:33.419 INFO 2981023 --- [alarm-detect-2,4200034036c3e206,fd27146302b3b0db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:33.422 INFO 2981023 --- [alarm-detect-2,4200034036c3e206,fd27146302b3b0db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:33.426 INFO 2981023 --- [alarm-detect-2,4200034036c3e206,fd27146302b3b0db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:34.633 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:34.635 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:35.123 INFO 2981023 --- [ocp-async-4,afa2ab2399bf3a5e,4a476c47136ad5f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,c395c66e959f38b7,c40ac1e1e4628322] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:35.317 INFO 2981023 --- [prometheus-minute-schedule-1,c395c66e959f38b7,c40ac1e1e4628322] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:18:36.352017+08:00, lastProcessTime=2024-09-11T17:18:26.336628+08:00 2024-09-11 17:18:36.478 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:18:36.352017+08:00 2024-09-11 17:18:36.729 INFO 2981023 --- [ocp-schedules-1,2157f1267c1e7ed3,7d88ae1e8fafccd5] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:18:36.748 WARN 2981023 --- [ocp-schedules-1,2157f1267c1e7ed3,7d88ae1e8fafccd5] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:18:37.123 INFO 2981023 --- [ocp-async-9,daa7754270baf66a,ad0439ab46167f10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:37.314 INFO 2981023 --- [prometheus-cache-monitor1,c5d01014c73988e9,dc309a84cf3625b6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:18:37.315 INFO 2981023 --- [prometheus-cache-monitor1,c5d01014c73988e9,dc309a84cf3625b6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:18:37.635 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:37.636 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:37.733 INFO 2981023 --- [ocp-async-4,f92d9aea46113e06,46ada4db8d34651f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:18:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,d5b85d06590968a5,8d4370a0a899e613] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,d5b85d06590968a5,8d4370a0a899e613] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:38.352 INFO 2981023 --- [ocp-schedules-1,479319e542e383f9,3968ac5af6eec327] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:18:39.123 INFO 2981023 --- [ocp-async-2,96dad896c4da836f,1eccae174cccbe5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:40.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:18:40.004 INFO 2981023 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:18:40.261 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ab500f24aa70d57e,7d7a1071c24b6947] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=ab500f24aa70d57e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:18:40.284 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ab500f24aa70d57e,7d7a1071c24b6947] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=ab500f24aa70d57e, duration=29 ms] 2024-09-11 17:18:40.636 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:40.637 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:41.123 INFO 2981023 --- [ocp-async-5,fc1d6ad0dd7a70fe,148b6421e309cad0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,20809bfe1d195867,ea68f7da99cdbddd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,20809bfe1d195867,ea68f7da99cdbddd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:41.619 INFO 2981023 --- [ocp-schedules-27,6f3e0ab296a51eb4,b48176de3ef1af06] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:18:43.123 INFO 2981023 --- [ocp-async-4,a6ffb0f08868cb82,e93d54f3136b11e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:43.474 INFO 2981023 --- [ocp-schedules-1,475410d11c5c2fb5,717d6e7de25d2407] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:18:43.523 WARN 2981023 --- [scanner-13458671-1,475410d11c5c2fb5,e7c3fb12b19d846e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:18:43.637 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:43.639 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,c9f52ae939ecd797,6f0fd7c081a677ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,c9f52ae939ecd797,6f0fd7c081a677ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:45.101 INFO 2981023 --- [prometheus-second-schedule-1,90e35718219f9d32,71cac9dabc022804] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:18:45.123 INFO 2981023 --- [ocp-async-8,06c912b62d176776,1119f787e6c31530] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:46.021 INFO 2981023 --- [ocp-schedules-28,899a87828c358f0d,35a46db2135b5011] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:46.021 INFO 2981023 --- [host-check-clockdiff1,899a87828c358f0d,4011aea8be46b4a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:18:46.021 INFO 2981023 --- [host-check-clockdiff2,899a87828c358f0d,2bbbe16dc44bc833] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:18:46.021 INFO 2981023 --- [host-check-clockdiff3,899a87828c358f0d,04edae9faee94fda] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:18:46.022 INFO 2981023 --- [host-check-clockdiff4,899a87828c358f0d,677271feb51126cd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:18:46.024 INFO 2981023 --- [ocp-schedules-18,c9001555f103421e,4029c6a7f743b820] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:46.026 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:18:46.026 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:18:46.026 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:18:46.027 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:18:46.041 INFO 2981023 --- [host-check-clockdiff4,899a87828c358f0d,677271feb51126cd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046326 0 0 , error= 2024-09-11 17:18:46.058 INFO 2981023 --- [ocp-schedules-27,556226b4f01e947f,74d0da4870e39e31] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:18:46.058 INFO 2981023 --- [host-check-agent-status1,556226b4f01e947f,09384c91a9d8ede8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:18:46.060 INFO 2981023 --- [host-check-agent-status2,556226b4f01e947f,c1e0e3aadaef55d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:18:46.069 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:46.073 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:46.084 INFO 2981023 --- [host-check-agent-status3,556226b4f01e947f,cf18f28464a8bd3d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:18:46.087 INFO 2981023 --- [host-check-agent-status1,556226b4f01e947f,09384c91a9d8ede8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:46.088 INFO 2981023 --- [host-check-agent-status1,556226b4f01e947f,09384c91a9d8ede8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:46.098 INFO 2981023 --- [host-check-agent-status4,556226b4f01e947f,044106ac507c4e2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:18:46.166 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:18:46.177 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:46.180 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:46.187 INFO 2981023 --- [host-check-agent-status2,556226b4f01e947f,c1e0e3aadaef55d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:46.187 INFO 2981023 --- [ocp-schedules-7,3ef2dcb03e01b7a1,e168ee8f4b00e42b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:18:46.187 INFO 2981023 --- [ocp-schedules-3,529e7544a62cda35,8db722219f7cd801] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:18:46.192 INFO 2981023 --- [ocp-schedules-25,5498479ba7d1db39,b705cceb29c5cfae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:18:46.197 INFO 2981023 --- [ocp-schedules-5,57e353645898fa13,e02f2e147cd07915] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:18:46.197 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:46.199 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:46.201 INFO 2981023 --- [ocp-schedules-25,5498479ba7d1db39,b705cceb29c5cfae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:46.204 INFO 2981023 --- [ocp-schedules-2,e1f0d04f56a6aa5a,c26cf0f5930574e8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:18:46.205 INFO 2981023 --- [host-check-agent-status2,556226b4f01e947f,c1e0e3aadaef55d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:46.206 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:46.213 INFO 2981023 --- [host-check-agent-status4,556226b4f01e947f,044106ac507c4e2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:46.214 INFO 2981023 --- [ocp-schedules-7,3ef2dcb03e01b7a1,e168ee8f4b00e42b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:46.216 INFO 2981023 --- [ocp-schedules-3,529e7544a62cda35,8db722219f7cd801] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:46.216 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:18:46.217 INFO 2981023 --- [ocp-schedules-3,529e7544a62cda35,8db722219f7cd801] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:18:46.260 INFO 2981023 --- [host-check-agent-status4,556226b4f01e947f,044106ac507c4e2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:46.261 INFO 2981023 --- [ocp-schedules-11,d546c0989010f0b4,d7ccd80265281532] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:18:46.262 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:46.274 INFO 2981023 --- [ocp-schedules-2,e1f0d04f56a6aa5a,c26cf0f5930574e8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:18:46.275 INFO 2981023 --- [host-check-agent-status3,556226b4f01e947f,cf18f28464a8bd3d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:46.290 INFO 2981023 --- [host-check-agent-status3,556226b4f01e947f,cf18f28464a8bd3d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:46.308 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:18:46.356 INFO 2981023 --- [host-check-agent-status1,556226b4f01e947f,09384c91a9d8ede8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:18:46.357 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:18:46.360 INFO 2981023 --- [host-check-agent-status1,556226b4f01e947f,09384c91a9d8ede8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:18:46.360 INFO 2981023 --- [host-check-agent-status1,556226b4f01e947f,09384c91a9d8ede8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:18:46.360 INFO 2981023 --- [host-check-agent-status1,556226b4f01e947f,09384c91a9d8ede8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:18:46.360 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:18:46.360292204+08:00 2024-09-11 17:18:46.363 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:18:46.360+08:00 2024-09-11 17:18:46.371 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:18:46.360+08:00 and status AVAILABLE 2024-09-11 17:18:46.375 INFO 2981023 --- [host-check-agent-status3,556226b4f01e947f,cf18f28464a8bd3d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:18:46.376 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:18:46.376 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:18:46.377 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:18:46.377030493+08:00 2024-09-11 17:18:46.379 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:18:46.377+08:00 2024-09-11 17:18:46.379 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:18:46.379429575+08:00 2024-09-11 17:18:46.381 INFO 2981023 --- [host-check-agent-status3,556226b4f01e947f,cf18f28464a8bd3d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:18:46.381 INFO 2981023 --- [host-check-agent-status3,556226b4f01e947f,cf18f28464a8bd3d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:18:46.381 INFO 2981023 --- [host-check-agent-status3,556226b4f01e947f,cf18f28464a8bd3d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:18:46.381 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:18:46.379+08:00 2024-09-11 17:18:46.381 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:18:46.377+08:00 and status AVAILABLE 2024-09-11 17:18:46.383 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.384 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:18:46.379+08:00 and status AVAILABLE 2024-09-11 17:18:46.384 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:46.385 INFO 2981023 --- [host-check-agent-status4,556226b4f01e947f,044106ac507c4e2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:18:46.385 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:18:46.385 INFO 2981023 --- [host-check-agent-status2,556226b4f01e947f,c1e0e3aadaef55d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:18:46.388 INFO 2981023 --- [host-check-agent-status2,556226b4f01e947f,c1e0e3aadaef55d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:18:46.388 INFO 2981023 --- [host-check-agent-status2,556226b4f01e947f,c1e0e3aadaef55d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:18:46.388 INFO 2981023 --- [host-check-agent-status2,556226b4f01e947f,c1e0e3aadaef55d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:18:46.393 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.393 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:46.398 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.399 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:46.400 INFO 2981023 --- [host-check-agent-status4,556226b4f01e947f,044106ac507c4e2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:18:46.400 INFO 2981023 --- [host-check-agent-status4,556226b4f01e947f,044106ac507c4e2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:18:46.400 INFO 2981023 --- [host-check-agent-status4,556226b4f01e947f,044106ac507c4e2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:18:46.403 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:18:46.403 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:18:46.403645378+08:00 2024-09-11 17:18:46.404 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:18:46.403+08:00 2024-09-11 17:18:46.406 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:18:46.405157+08:00, lastProcessTime=2024-09-11T17:18:36.352017+08:00 2024-09-11 17:18:46.407 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:18:46.403+08:00 and status AVAILABLE 2024-09-11 17:18:46.409 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:18:46.412 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:18:46.413 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:18:46.457 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:18:46.474 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.474 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:46.475 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:18:46.475 INFO 2981023 --- [ocp-schedules-7,3ef2dcb03e01b7a1,e168ee8f4b00e42b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:18:46.479 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:18:46.480 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:46.480 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:46.480 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.480 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:18:46.481 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:46.481 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:46.481 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:46.481 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:18:46.482 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:18:46.493 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.494 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:18:46.495 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:18:46.495 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.495 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.499 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:18:46.508 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:46.510 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:18:46.521 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:46.532 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:18:46.551 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:18:46.554 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:46.564 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:18:46.574 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:18:46.583 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:18:46.589 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:18:46.589 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:18:46.589 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:18:46.593 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.594 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.595 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:18:46.595 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:18:46.595 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.598 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:18:46.613 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:18:46.624 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:18:46.639 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:46.657 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:46.666 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:18:46.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.678 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.678 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:46.679 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:18:46.679 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:18:46.681 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.683 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.685 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:18:46.685 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:18:46.703 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:46.704 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.705 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.705 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:18:46.706 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:18:46.709 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.710 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:18:46.712 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:46.712 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:18:46.712 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.717 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:46.717 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:18:46.718 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:18:46.728 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:46.735 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:18:46.749 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:18:46.756 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:18:46.765 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:18:46.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.771 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.772 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:18:46.773 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:18:46.778 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:18:46.783 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:18:46.787 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:18:46.795 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:18:46.797 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:46.799 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.799 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.799 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:18:46.801 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:18:46.801 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.804 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:18:46.804 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:18:46.804 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:18:46.806 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:18:46.819 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.820 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.820 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:46.820 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:18:46.820 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:18:46.820 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:46.820 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:18:46.821 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:18:46.825 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.826 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.827 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:18:46.827 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:18:46.827 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.828 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:18:46.829 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:18:46.838 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:18:46.848 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:18:46.855 INFO 2981023 --- [host-check-host-status4,c9001555f103421e,82558fae58cd5c68] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:18:46.857 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:18:46.860 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:18:46.861 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:18:46.862 INFO 2981023 --- [host-check-host-status2,c9001555f103421e,3ad3886baf5af77a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:18:46.862 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:18:46.865 INFO 2981023 --- [host-check-host-status1,c9001555f103421e,23fccbae086d5b3e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:18:46.870 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.880 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.880 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:18:46.881 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:18:46.884 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:18:46.885 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:18:46.892 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:18:46.892 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:18:46.893 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:18:46.896 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:18:46.911 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:18:46.921 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:18:46.922 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.923 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.923 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:18:46.924 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:18:46.924 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.940 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:18:46.949 INFO 2981023 --- [host-check-host-status3,c9001555f103421e,c8f16ac0e709ef20] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:18:46.953 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.962 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.964 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:18:46.970 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:18:46.970 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:18:46.982 INFO 2981023 --- [host-check-clockdiff2,899a87828c358f0d,2bbbe16dc44bc833] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046326 0 0 , error= 2024-09-11 17:18:46.983 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:18:46.988 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:18:46.997 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:46.997 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:46.998 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:18:47.005 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:18:47.005 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:18:47.010 INFO 2981023 --- [ocp-schedules-25,5498479ba7d1db39,b705cceb29c5cfae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:18:47.012 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:18:46.405157+08:00 2024-09-11 17:18:47.016 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:18:47.023 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:18:47.026 INFO 2981023 --- [ocp-schedules-6,68954651d3d598ec,3687c6407ae054f7] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:18:47.123 INFO 2981023 --- [ocp-async-5,b0bc0adab336168e,fcbd6f5c33b6d16f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,428aab37c475ffe9,28ab87d60b86db4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,428aab37c475ffe9,28ab87d60b86db4a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:47.733 INFO 2981023 --- [ocp-async-3,7a346a8da57fb16f,27ca13e5ca982c6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:18:47.837 INFO 2981023 --- [host-check-clockdiff1,899a87828c358f0d,4011aea8be46b4a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046327 0 0 , error= 2024-09-11 17:18:49.123 INFO 2981023 --- [ocp-async-7,cf81d66f0e307f11,ba2ccc301887b16d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:49.657 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:49.659 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:18:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:18:49.921 INFO 2981023 --- [ocp-schedules-14,a399e83854c85b3f,08ba13726970f612] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:18:49.921+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:18:49.925 INFO 2981023 --- [ocp-schedules-14,a399e83854c85b3f,08ba13726970f612] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:18:50.026 INFO 2981023 --- [ocp-schedules-14,a399e83854c85b3f,08ba13726970f612] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 36 to SqlAggregateData 2024-09-11 17:18:50.027 INFO 2981023 --- [ocp-schedules-14,a399e83854c85b3f,08ba13726970f612] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:18:50.183 INFO 2981023 --- [ocp-schedules-29,b99ccd559300f0c0,52990fb8446cd562] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:18:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,f120390fd7bed483,888dae1a545c01d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,f120390fd7bed483,888dae1a545c01d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:50.387 INFO 2981023 --- [ocp-schedules-16,7434513ff0998b54,63cd6033a256172f] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:18:50.387 INFO 2981023 --- [ocp-schedules-16,7434513ff0998b54,63cd6033a256172f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:50.426 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:18:50.426 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.427 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:18:50.427 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.428 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:18:50.428 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.430 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:18:50.431 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:18:50.431 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:18:50.438 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.438 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.439 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.439 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.440 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.441 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.444 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.444 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.449 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.449 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:18:50.451 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.451 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.455 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.455 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.456 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.456 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:18:50.461 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.461 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:18:50.466 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.467 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.467 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:18:50.468 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1691D732-0-0] 2024-09-11 17:18:50.468 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:18:50.468 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.468 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:18:50.469 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.469 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.472 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.473 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:18:50.481 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.481 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:18:50.489 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.489 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:18:50.490 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.490 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:18:50.491 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.491 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.491 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:18:50.491 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:18:50.492 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.492 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.493 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:18:50.493 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:18:50.493 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.493 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.493 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.494 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:18:50.494 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:18:50.494 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.494 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:18:50.494 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:18:50.495 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:18:50.495 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:18:50.495 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1691D73C-0-0] 2024-09-11 17:18:50.495 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:18:50.495 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.496 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:18:50.496 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.497 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:18:50.497 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.498 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:18:50.499 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:18:50.502 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.502 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:18:50.503 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:18:50.503 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:18:50.503 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.504 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:18:50.504 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.505 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:18:50.505 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.506 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:18:50.507 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.507 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:18:50.508 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.508 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:18:50.509 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:18:50.511 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:18:50.516 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:50.518 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.518 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:18:50.518 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:50.520 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.520 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.523 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:18:50.524 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.525 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:18:50.525 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:18:50.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:18:50.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:18:50.532 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:18:50.533 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:18:50.534 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:18:50.535 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:18:50.535 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.535 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:18:50.535 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:18:50.535 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:18:50.535 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:18:50.536 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:50.536 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.536 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:18:50.541 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:18:50.542 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:18:50.542 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:18:50.542 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:18:50.545 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:18:50.546 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:18:50.546 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:18:50.546 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:18:50.546 INFO 2981023 --- [ocp-schedules-16,7434513ff0998b54,63cd6033a256172f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:18:51.123 INFO 2981023 --- [ocp-async-2,96080555b3ef96e7,dddc18d81d2dd2e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:51.487 INFO 2981023 --- [ocp-schedules-21,42d7422124b9daa4,bc72b05bf79174c8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:18:51.488 INFO 2981023 --- [ocp-schedules-21,42d7422124b9daa4,bc72b05bf79174c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:18:51.488 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:18:51.488 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:18:51.488 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:18:51.489 INFO 2981023 --- [ocp-schedules-21,42d7422124b9daa4,bc72b05bf79174c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:18:51.621 INFO 2981023 --- [ocp-schedules-11,cc296e4908bc789f,92c3209effde201c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:18:51.842 INFO 2981023 --- [ocp-schedules-12,a9c6b035068a69ad,cf88b24e9f88fd9f] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:18:51.860 WARN 2981023 --- [ocp-schedules-12,a9c6b035068a69ad,cf88b24e9f88fd9f] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:18:51.971 INFO 2981023 --- [host-check-clockdiff3,899a87828c358f0d,04edae9faee94fda] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046331 1 1 , error= 2024-09-11 17:18:52.659 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:52.661 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:52.979 INFO 2981023 --- [ocp-schedules-17,19df328a7b08f863,f265d62098cee498] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:18:52.979 INFO 2981023 --- [ocp-schedules-17,19df328a7b08f863,f265d62098cee498] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:18:53.124 INFO 2981023 --- [ocp-async-1,5bcd5c2c06587922,d97ae55932f1b28a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:53.251 INFO 2981023 --- [ocp-schedules-27,f008918b2638bb86,d2486f8b283975ed] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:18:53.261 INFO 2981023 --- [ocp-schedules-27,f008918b2638bb86,d2486f8b283975ed] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:18:53.261 INFO 2981023 --- [ocp-schedules-27,f008918b2638bb86,d2486f8b283975ed] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:18:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,e5fef7aa6a396b18,df113ed1242d97a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:53.318 INFO 2981023 --- [prometheus-minute-schedule-1,e5fef7aa6a396b18,df113ed1242d97a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:53.666 INFO 2981023 --- [ocp-schedules-26,ccae93f7de5d7f93,2d3db89a5b4743ea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:18:53.690 WARN 2981023 --- [scanner-13458873-1,ccae93f7de5d7f93,9fe1752250f63d32] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:18:53.734 INFO 2981023 --- [alarm-detect-4,ccae93f7de5d7f93,e1584407ac33a311] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643512, data is empty. 2024-09-11 17:18:53.740 INFO 2981023 --- [alarm-detect-4,ccae93f7de5d7f93,e1584407ac33a311] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643548, data is empty. 2024-09-11 17:18:53.744 INFO 2981023 --- [alarm-detect-4,ccae93f7de5d7f93,e1584407ac33a311] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643548, data is empty. 2024-09-11 17:18:53.761 INFO 2981023 --- [alarm-detect-3,ccae93f7de5d7f93,9f5cb78b3be9315d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643512, data is empty. 2024-09-11 17:18:53.761 INFO 2981023 --- [alarm-detect-12,ccae93f7de5d7f93,f46f3c433674c498] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643512, data is empty. 2024-09-11 17:18:53.764 INFO 2981023 --- [alarm-detect-12,ccae93f7de5d7f93,f46f3c433674c498] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643548, data is empty. 2024-09-11 17:18:53.764 INFO 2981023 --- [alarm-detect-3,ccae93f7de5d7f93,9f5cb78b3be9315d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643548, data is empty. 2024-09-11 17:18:53.767 INFO 2981023 --- [alarm-detect-12,ccae93f7de5d7f93,f46f3c433674c498] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643548, data is empty. 2024-09-11 17:18:53.769 INFO 2981023 --- [alarm-detect-3,ccae93f7de5d7f93,9f5cb78b3be9315d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643548, data is empty. 2024-09-11 17:18:53.881 INFO 2981023 --- [alarm-detect-3,ccae93f7de5d7f93,01c015a884b9f9d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643512, data is empty. 2024-09-11 17:18:53.885 INFO 2981023 --- [alarm-detect-3,ccae93f7de5d7f93,01c015a884b9f9d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643548, data is empty. 2024-09-11 17:18:53.889 INFO 2981023 --- [alarm-detect-3,ccae93f7de5d7f93,01c015a884b9f9d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643548, data is empty. 2024-09-11 17:18:54.127 INFO 2981023 --- [ocp-schedules-6,038e663b0a42ab71,7a271c146f777d22] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:18:54.461 INFO 2981023 --- [ocp-schedules-20,415d7cb6c4142400,c3c0cc99629279bb] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T17:18:54.461+08:00, tenantSize=6 2024-09-11 17:18:54.473 INFO 2981023 --- [ocp-schedules-20,415d7cb6c4142400,c3c0cc99629279bb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726046334461000 2024-09-11 17:18:54.473 INFO 2981023 --- [ocp-schedules-20,415d7cb6c4142400,c3c0cc99629279bb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726046034430000 2024-09-11 17:18:54.503 INFO 2981023 --- [ocp-schedules-20,415d7cb6c4142400,c3c0cc99629279bb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 17:18:54.621 INFO 2981023 --- [ocp-schedules-14,4a45e55f3a888d3f,11ace9f7daff5f12] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:18:54.621 INFO 2981023 --- [ocp-schedules-14,4a45e55f3a888d3f,11ace9f7daff5f12] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:18:54.626 INFO 2981023 --- [ocp-schedules-14,4a45e55f3a888d3f,11ace9f7daff5f12] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:18:54.626 INFO 2981023 --- [ocp-schedules-14,4a45e55f3a888d3f,11ace9f7daff5f12] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:18:54.626 INFO 2981023 --- [ocp-schedules-14,4a45e55f3a888d3f,11ace9f7daff5f12] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:18:54.636 INFO 2981023 --- [ocp-schedules-14,4a45e55f3a888d3f,11ace9f7daff5f12] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:18:54.636 INFO 2981023 --- [ocp-schedules-14,4a45e55f3a888d3f,11ace9f7daff5f12] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:18:54.636 INFO 2981023 --- [ocp-schedules-14,4a45e55f3a888d3f,11ace9f7daff5f12] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:18:54.647 INFO 2981023 --- [ocp-schedules-14,4a45e55f3a888d3f,11ace9f7daff5f12] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:18:55.124 INFO 2981023 --- [ocp-async-9,b5565b1d07216afa,0fc6b73364ce6cd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:55.368 INFO 2981023 --- [ocp-schedules-4,d00e6cd922f4c051,b58ab082d4670d8a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:18:55.368 INFO 2981023 --- [ocp-schedules-4,d00e6cd922f4c051,b58ab082d4670d8a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:18:55.384 INFO 2981023 --- [Collect all tenants major compactions6,d00e6cd922f4c051,dfae7bceade42c39] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:18:55.386 INFO 2981023 --- [Collect all tenants major compactions5,d00e6cd922f4c051,3ad1342d709dab57] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:18:55.386 INFO 2981023 --- [Collect all tenants major compactions4,d00e6cd922f4c051,adc9b5e9aa63f2e1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:18:55.386 INFO 2981023 --- [Collect all tenants major compactions2,d00e6cd922f4c051,e292cc267ba22ea3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:18:55.390 INFO 2981023 --- [Collect all tenants major compactions1,d00e6cd922f4c051,d0f2521c4688b693] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:18:55.391 INFO 2981023 --- [Collect all tenants major compactions3,d00e6cd922f4c051,4761240747e266df] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:18:55.412 INFO 2981023 --- [Collect all tenants major compactions4,d00e6cd922f4c051,adc9b5e9aa63f2e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:55.413 INFO 2981023 --- [Collect all tenants major compactions4,d00e6cd922f4c051,adc9b5e9aa63f2e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:18:55.413 INFO 2981023 --- [Collect all tenants major compactions2,d00e6cd922f4c051,e292cc267ba22ea3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:55.414 INFO 2981023 --- [Collect all tenants major compactions5,d00e6cd922f4c051,3ad1342d709dab57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:55.414 INFO 2981023 --- [Collect all tenants major compactions1,d00e6cd922f4c051,d0f2521c4688b693] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:55.415 INFO 2981023 --- [Collect all tenants major compactions5,d00e6cd922f4c051,3ad1342d709dab57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:18:55.416 INFO 2981023 --- [Collect all tenants major compactions3,d00e6cd922f4c051,4761240747e266df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:55.416 INFO 2981023 --- [Collect all tenants major compactions6,d00e6cd922f4c051,dfae7bceade42c39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:18:55.417 INFO 2981023 --- [Collect all tenants major compactions2,d00e6cd922f4c051,e292cc267ba22ea3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:18:55.417 INFO 2981023 --- [Collect all tenants major compactions6,d00e6cd922f4c051,dfae7bceade42c39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:18:55.418 INFO 2981023 --- [Collect all tenants major compactions4,d00e6cd922f4c051,adc9b5e9aa63f2e1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:18:55.419 INFO 2981023 --- [Collect all tenants major compactions1,d00e6cd922f4c051,d0f2521c4688b693] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:18:55.419 INFO 2981023 --- [Collect all tenants major compactions3,d00e6cd922f4c051,4761240747e266df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:18:55.419 INFO 2981023 --- [Collect all tenants major compactions5,d00e6cd922f4c051,3ad1342d709dab57] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:18:55.420 INFO 2981023 --- [Collect all tenants major compactions6,d00e6cd922f4c051,dfae7bceade42c39] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:18:55.421 INFO 2981023 --- [Collect all tenants major compactions2,d00e6cd922f4c051,e292cc267ba22ea3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:18:55.421 INFO 2981023 --- [Collect all tenants major compactions4,d00e6cd922f4c051,adc9b5e9aa63f2e1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:18:55.421 INFO 2981023 --- [Collect all tenants major compactions3,d00e6cd922f4c051,4761240747e266df] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:18:55.421 INFO 2981023 --- [Collect all tenants major compactions3,d00e6cd922f4c051,4761240747e266df] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:18:55.421 INFO 2981023 --- [Collect all tenants major compactions1,d00e6cd922f4c051,d0f2521c4688b693] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:18:55.421 INFO 2981023 --- [Collect all tenants major compactions1,d00e6cd922f4c051,d0f2521c4688b693] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:18:55.424 INFO 2981023 --- [Collect all tenants major compactions2,d00e6cd922f4c051,e292cc267ba22ea3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:18:55.425 INFO 2981023 --- [Collect all tenants major compactions5,d00e6cd922f4c051,3ad1342d709dab57] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:18:55.426 INFO 2981023 --- [Collect all tenants major compactions6,d00e6cd922f4c051,dfae7bceade42c39] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:18:55.427 INFO 2981023 --- [ocp-schedules-4,d00e6cd922f4c051,b58ab082d4670d8a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:18:55.598 INFO 2981023 --- [ocp-schedules-22,01b9eeca75067b60,1b19839b1e33f861] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:18:55.598 INFO 2981023 --- [ocp-schedules-22,01b9eeca75067b60,1b19839b1e33f861] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:18:55.601 INFO 2981023 --- [ocp-schedules-22,01b9eeca75067b60,1b19839b1e33f861] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:18:55.601 INFO 2981023 --- [ocp-schedules-22,01b9eeca75067b60,1b19839b1e33f861] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:18:55.604 INFO 2981023 --- [ocp-schedules-22,01b9eeca75067b60,1b19839b1e33f861] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:18:55.604 INFO 2981023 --- [ocp-schedules-22,01b9eeca75067b60,1b19839b1e33f861] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:18:55.635 INFO 2981023 --- [ocp-schedules-2,cbb601451d19934e,c24ef0009c32b6fe] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:18:55.641 INFO 2981023 --- [ocp-schedules-2,cbb601451d19934e,c24ef0009c32b6fe] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:18:55.641 INFO 2981023 --- [ocp-schedules-2,cbb601451d19934e,c24ef0009c32b6fe] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:18:55.661 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:55.661 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,65bef1eec15c35e2,ad9ecfb2e9652fd1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,65bef1eec15c35e2,ad9ecfb2e9652fd1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:18:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:18:56.352602+08:00, lastProcessTime=2024-09-11T17:18:46.405157+08:00 2024-09-11 17:18:56.500 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:18:56.352602+08:00 2024-09-11 17:18:57.124 INFO 2981023 --- [ocp-async-8,fb900e68da2829a8,9c00f929a5495fdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:57.734 INFO 2981023 --- [ocp-async-6,b338969537a249fd,de3815c259f7b7b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:18:58.343 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:14:57.987, end=2024-09-11T17:16:58.339} 2024-09-11 17:18:58.345 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName 2024-09-11 17:18:58.346 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:18:58.346 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:14:57.987, end=2024-09-11T17:16:58.339},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:18:58.346 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:18:58.346 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:18:58.347 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:18:58.349 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:14:57.987, end=2024-09-11T17:16:58.339},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:18:58.349 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:18:58.349 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:18:58.379 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:18:58.380 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:18:58.382 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:14:57.987, end=2024-09-11T17:16:58.339},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:18:58.387 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:18:58.387 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:18:58.436 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938436000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.436Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.436 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938436000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.436Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.439 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938439000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.439Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.440 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938440000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.440Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.447 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938447000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.447Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.449 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938449000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.449Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.450 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938450000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.450Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.450 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938450000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.450Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.460 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:18:58.460 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.461 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.461 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.466 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 57 2024-09-11 17:18:58.466 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.469 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.469 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.469 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.469 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.470 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:18:58.470 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.477 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:18:58.477 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.479 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.479 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.479 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.479 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.480 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.480 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.485 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.485 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.487 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.487 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.499 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.499 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.504 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.504 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.505 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.505 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.507 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.507 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.508 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.508 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.508 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.508 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.509 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.509 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.510 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.510 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.510 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.510 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.514 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.520 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.520 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.521 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.521 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.522 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.524 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.524 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.524 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.525 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.525 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.525 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.525 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.436, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.526 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.526 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.527 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.528 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.528 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.528 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.528 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.529 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.530 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.530 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.45, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.532 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.532 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.532 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.535 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.44, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.536 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.436, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.537 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.439, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.542 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.447, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.543 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.543 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.546 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.547 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.547 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.550 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.555 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.449, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.559 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.45, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.564 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938564000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.564Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.569 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938569000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.569Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.570 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938570000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.570Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.578 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938578000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.578Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.578 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:18:58.578 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.581 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938581000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.581Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.618 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.618 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.618 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.618 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.620 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:18:58.620 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.620 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938620000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.620Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.623 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.623 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.634 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.634 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.637 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938637000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.637Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.639 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.639 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.640 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.640 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.640 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.640 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.641 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.641 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.644 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.644 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.646 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938646000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.646Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.652 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.652 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.653 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.653 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.654 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.654 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.655 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.655 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.656 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.656 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.657 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.657 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.661 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:18:58.663 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:18:58.666 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.666 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.667 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.667 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.668 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.668 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.669 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.672 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:18:58.672 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.673 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.673 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.673 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.674 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.674 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.675 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.675 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.675 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.675 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.676 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.569, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.676 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.677 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.677 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.677 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.678 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.680 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.680 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.682 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.578, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.684 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.564, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.684 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.57, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.685 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.581, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.686 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.686 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.688 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.688 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.688 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.688 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.689 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.690 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.690 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.691 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.693 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.62, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.695 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.637, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.697 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.697 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.698 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:18:58.701 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.646, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.702 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:18:58.702 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:14:57.987, end=2024-09-11T17:16:58.339},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:18:58.702 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:18:58.702 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:18:58.703 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:18:58.703 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:14:57.987, end=2024-09-11T17:16:58.339},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:18:58.704 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:18:58.704 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:18:58.704 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:18:58.706 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:14:57.987, end=2024-09-11T17:16:58.339},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:18:58.706 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:18:58.706 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:18:58.715 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:18:58.729 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725959938729000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:18:58.729Z, endTime=2024-09-11T09:16:58.339Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:18:58.731 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:18:58.731 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:18:58.735 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.735 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.737 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:18:58.737 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:18:58.741 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:18:58.741 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046218339000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:18:58.742 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:18:58.742 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:18:58.729, end=2024-09-11T17:16:58.339}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:18:58.742 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:18:59.124 INFO 2981023 --- [ocp-async-1,0e3dfed33bc05e92,bd67a7e016f7f202] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:18:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,38065ae23fb8b8b2,4e077e1c1e9beda1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046340, interval=60 2024-09-11 17:18:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,38065ae23fb8b8b2,4e077e1c1e9beda1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:01.124 INFO 2981023 --- [ocp-async-7,b0f0b3b5b268887b,56b647e6e3a1afe2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:01.624 INFO 2981023 --- [ocp-schedules-7,f71a7108b20995a0,3506ea6432bcf3da] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:19:01.664 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:01.664 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,f94f3a49a9521a30,e2332b8910c93cde] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,f94f3a49a9521a30,e2332b8910c93cde] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:03.124 INFO 2981023 --- [ocp-async-2,b32d81bfe32a4190,c1939e2e13118cd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:03.748 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,0a31a1461fddb7aa,bc0dc416d50a6314] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=0a31a1461fddb7aa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:19:03.769 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,0a31a1461fddb7aa,bc0dc416d50a6314] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=0a31a1461fddb7aa, duration=26 ms] 2024-09-11 17:19:03.930 INFO 2981023 --- [ocp-schedules-22,d429bb7f2490e1d9,da1638cd2b4a112f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:19:03.971 WARN 2981023 --- [scanner-13459205-1,d429bb7f2490e1d9,10d43aac7dc2c92d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:19:04.051 INFO 2981023 --- [alarm-detect-17,d429bb7f2490e1d9,b3227bcada75e928] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046310 ~ 1726046334 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046325, value=1.64)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.64, values={1726046325=1.64}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:19:04.664 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:04.665 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:05.124 INFO 2981023 --- [ocp-async-10,db3962eafeadadb5,9d4c394d48aaa64f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,76811b6b22dd48c3,006dea8ed6c5aef0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,76811b6b22dd48c3,006dea8ed6c5aef0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:19:06.336772+08:00, lastProcessTime=2024-09-11T17:18:56.352602+08:00 2024-09-11 17:19:06.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:19:06.336772+08:00 2024-09-11 17:19:07.068 INFO 2981023 --- [ocp-schedules-14,191a0a0b132300e5,56725893fc980c2d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:19:07.072 INFO 2981023 --- [ocp-schedules-10,19bfd8fb4c00b7e9,ca4007d20242640b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:19:07.072+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:19:07.073 INFO 2981023 --- [ocp-schedules-10,19bfd8fb4c00b7e9,ca4007d20242640b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:19:07.074 INFO 2981023 --- [ocp-schedules-10,19bfd8fb4c00b7e9,ca4007d20242640b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:19:07.074 INFO 2981023 --- [ocp-schedules-10,19bfd8fb4c00b7e9,ca4007d20242640b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:19:07.124 INFO 2981023 --- [ocp-async-4,e5f29dc71213f9c5,262833b8a7b1d893] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:07.665 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:07.666 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:07.734 INFO 2981023 --- [ocp-async-1,1510781ab86f4f48,c215adeae64d0863] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:19:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,9c731a8c74cba238,16bf00c49138b1e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:08.318 INFO 2981023 --- [prometheus-minute-schedule-1,9c731a8c74cba238,16bf00c49138b1e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:09.124 INFO 2981023 --- [ocp-async-8,6195a743f1d7abfb,a2cc8dc6d78851ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:10.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:19:10.003 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:19:10.666 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:10.667 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:19:10.920 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:19:11.124 INFO 2981023 --- [ocp-async-5,6de2c395d9c2de22,d5c9d31b2a7678c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,09c416485f9bc12d,6bc24f60942e7d53] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,09c416485f9bc12d,6bc24f60942e7d53] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:11.627 INFO 2981023 --- [ocp-schedules-4,71e9d514b39d159c,72eab30e38d835a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:19:13.125 INFO 2981023 --- [ocp-async-9,0a00bae3eea6eb8c,0ba9f6966c44e774] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:13.264 INFO 2981023 --- [ocp-schedules-11,c8ebe998c41aa52a,ba120cc7edb0c723] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:19:13.275 INFO 2981023 --- [ocp-schedules-11,c8ebe998c41aa52a,ba120cc7edb0c723] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:19:13.275 INFO 2981023 --- [ocp-schedules-11,c8ebe998c41aa52a,ba120cc7edb0c723] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:19:13.667 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:13.668 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:14.111 INFO 2981023 --- [ocp-schedules-12,4597d0eaff6a2172,f033e4728c0774bd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:19:14.126 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046280, nodeCount=48, affectRows=48 2024-09-11 17:19:14.126 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046280, nodeCount=46, validCount=46 2024-09-11 17:19:14.161 WARN 2981023 --- [scanner-13460941-1,4597d0eaff6a2172,8f0e3f0beff5b27f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:19:14.195 INFO 2981023 --- [alarm-detect-19,4597d0eaff6a2172,e813c8bfc5d0d1b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643548, data is empty. 2024-09-11 17:19:14.199 INFO 2981023 --- [alarm-detect-19,4597d0eaff6a2172,e813c8bfc5d0d1b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643548, data is empty. 2024-09-11 17:19:14.204 INFO 2981023 --- [alarm-detect-19,4597d0eaff6a2172,e813c8bfc5d0d1b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643548, data is empty. 2024-09-11 17:19:14.232 INFO 2981023 --- [alarm-detect-15,4597d0eaff6a2172,15af0a319e6261a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643548, data is empty. 2024-09-11 17:19:14.232 INFO 2981023 --- [alarm-detect-5,4597d0eaff6a2172,d9d91b7947849dde] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643548, data is empty. 2024-09-11 17:19:14.240 INFO 2981023 --- [alarm-detect-15,4597d0eaff6a2172,15af0a319e6261a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643548, data is empty. 2024-09-11 17:19:14.243 INFO 2981023 --- [alarm-detect-5,4597d0eaff6a2172,d9d91b7947849dde] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643548, data is empty. 2024-09-11 17:19:14.249 INFO 2981023 --- [alarm-detect-15,4597d0eaff6a2172,15af0a319e6261a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643548, data is empty. 2024-09-11 17:19:14.250 INFO 2981023 --- [alarm-detect-5,4597d0eaff6a2172,d9d91b7947849dde] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643548, data is empty. 2024-09-11 17:19:14.277 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046280, nodeCount=1000, affectRows=1000 2024-09-11 17:19:14.280 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046280, nodeCount=1000, validCount=1000 2024-09-11 17:19:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,63d19fe8a1c898e3,9fa41c5c68218cc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:19:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,abdd1a9f79c7ef11,5e3f4904e4c1f3a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:14.335 INFO 2981023 --- [prometheus-minute-schedule-1,abdd1a9f79c7ef11,5e3f4904e4c1f3a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:14.358 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046280, nodeCount=1000, affectRows=1000 2024-09-11 17:19:14.362 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046280, nodeCount=999, validCount=999 2024-09-11 17:19:14.368 INFO 2981023 --- [alarm-detect-11,4597d0eaff6a2172,0f7cd9db4abb5406] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643548, data is empty. 2024-09-11 17:19:14.372 INFO 2981023 --- [alarm-detect-11,4597d0eaff6a2172,0f7cd9db4abb5406] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643548, data is empty. 2024-09-11 17:19:14.378 INFO 2981023 --- [alarm-detect-11,4597d0eaff6a2172,0f7cd9db4abb5406] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643548, data is empty. 2024-09-11 17:19:15.125 INFO 2981023 --- [ocp-async-6,a2850294d4165a31,16f45d9b0d5cb251] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:15.462 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046280, nodeCount=480, affectRows=480 2024-09-11 17:19:16.027 INFO 2981023 --- [ocp-schedules-22,80f416d0562ba519,156c62447410130a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:19:16.027 INFO 2981023 --- [host-check-clockdiff1,80f416d0562ba519,44e456b3b3deeb91] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:19:16.027 INFO 2981023 --- [host-check-clockdiff3,80f416d0562ba519,e8860abdf34499d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:19:16.027 INFO 2981023 --- [host-check-clockdiff2,80f416d0562ba519,9544b586b15c855c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:19:16.030 INFO 2981023 --- [ocp-schedules-1,b24dc121324eba52,e3b5ebc10ad2eba0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:19:16.030 INFO 2981023 --- [host-check-clockdiff4,80f416d0562ba519,120cc70793c26357] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:19:16.030 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:19:16.031 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:19:16.031 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:19:16.032 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:19:16.034 INFO 2981023 --- [ocp-schedules-12,efe25fd926190ad9,0b4ca0220b951e90] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:19:16.057 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:19:16.058 INFO 2981023 --- [host-check-agent-status1,efe25fd926190ad9,740d17d5cc89983f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:19:16.062 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:19:16.066 INFO 2981023 --- [host-check-agent-status2,efe25fd926190ad9,29973b940702d4f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:19:16.066 INFO 2981023 --- [host-check-agent-status3,efe25fd926190ad9,945e4fd025419942] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:19:16.077 INFO 2981023 --- [host-check-agent-status4,efe25fd926190ad9,7c6afdbe0ff84445] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:19:16.089 INFO 2981023 --- [host-check-clockdiff4,80f416d0562ba519,120cc70793c26357] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046356 0 0 , error= 2024-09-11 17:19:16.092 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:19:16.094 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:19:16.095 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:19:16.099 INFO 2981023 --- [host-check-agent-status1,efe25fd926190ad9,740d17d5cc89983f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:19:16.112 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:19:16.113 INFO 2981023 --- [host-check-clockdiff2,80f416d0562ba519,9544b586b15c855c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046356 0 0 , error= 2024-09-11 17:19:16.119 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:19:16.120 INFO 2981023 --- [host-check-agent-status1,efe25fd926190ad9,740d17d5cc89983f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:19:16.172 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:19:16.173 INFO 2981023 --- [host-check-agent-status2,efe25fd926190ad9,29973b940702d4f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:19:16.186 INFO 2981023 --- [host-check-agent-status2,efe25fd926190ad9,29973b940702d4f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:19:16.193 INFO 2981023 --- [ocp-schedules-5,fccd6661d3400e78,85920d62c4c488c7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:19:16.199 INFO 2981023 --- [ocp-schedules-29,40253b7dbba32d49,2b84cf1b910f6932] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:19:16.199 INFO 2981023 --- [ocp-schedules-26,0bc6e448e99664f0,1438467922ad25b3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:19:16.200 INFO 2981023 --- [ocp-schedules-4,6240db495d4169e9,e242c9d806b110c8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:19:16.203 INFO 2981023 --- [host-check-agent-status4,efe25fd926190ad9,7c6afdbe0ff84445] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:19:16.203 INFO 2981023 --- [host-check-agent-status3,efe25fd926190ad9,945e4fd025419942] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:19:16.206 INFO 2981023 --- [ocp-schedules-29,40253b7dbba32d49,2b84cf1b910f6932] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:19:16.207 INFO 2981023 --- [ocp-schedules-5,fccd6661d3400e78,85920d62c4c488c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:19:16.207 INFO 2981023 --- [ocp-schedules-26,0bc6e448e99664f0,1438467922ad25b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:19:16.211 INFO 2981023 --- [host-check-agent-status4,efe25fd926190ad9,7c6afdbe0ff84445] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:19:16.211 INFO 2981023 --- [host-check-agent-status3,efe25fd926190ad9,945e4fd025419942] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:19:16.227 INFO 2981023 --- [ocp-schedules-24,39024348eab0120f,d79e2a5355641929] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:19:16.255 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:19:16.257 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:19:16.257007579+08:00 2024-09-11 17:19:16.267 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:19:16.257+08:00 2024-09-11 17:19:16.272 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:19:16.257+08:00 and status AVAILABLE 2024-09-11 17:19:16.289 INFO 2981023 --- [host-check-agent-status1,efe25fd926190ad9,740d17d5cc89983f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:19:16.289 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:19:16.292 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:19:16.292325197+08:00 2024-09-11 17:19:16.292 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:19:16.293 INFO 2981023 --- [host-check-agent-status1,efe25fd926190ad9,740d17d5cc89983f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:19:16.293 INFO 2981023 --- [host-check-agent-status1,efe25fd926190ad9,740d17d5cc89983f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:19:16.293 INFO 2981023 --- [host-check-agent-status1,efe25fd926190ad9,740d17d5cc89983f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:19:16.294 INFO 2981023 --- [ocp-schedules-29,40253b7dbba32d49,2b84cf1b910f6932] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:19:16.294 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:19:16.294 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:19:16.292+08:00 2024-09-11 17:19:16.298 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:19:16.297300457+08:00 2024-09-11 17:19:16.300 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:19:16.299+08:00 2024-09-11 17:19:16.303 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:19:16.292+08:00 and status AVAILABLE 2024-09-11 17:19:16.303 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:19:16.305 INFO 2981023 --- [host-check-agent-status2,efe25fd926190ad9,29973b940702d4f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:19:16.305 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:19:16.299+08:00 and status AVAILABLE 2024-09-11 17:19:16.310 INFO 2981023 --- [host-check-agent-status2,efe25fd926190ad9,29973b940702d4f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:19:16.310 INFO 2981023 --- [host-check-agent-status2,efe25fd926190ad9,29973b940702d4f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:19:16.310 INFO 2981023 --- [host-check-agent-status2,efe25fd926190ad9,29973b940702d4f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:19:16.329 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:19:16.329 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:19:16.338 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:19:16.340 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:19:16.340 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:19:16.341 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:19:16.341028294+08:00 2024-09-11 17:19:16.344 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:19:16.345 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:19:16.346 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:19:16.346 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:19:16.354 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:19:16.341+08:00 2024-09-11 17:19:16.357 INFO 2981023 --- [host-check-agent-status4,efe25fd926190ad9,7c6afdbe0ff84445] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:19:16.357 INFO 2981023 --- [host-check-agent-status3,efe25fd926190ad9,945e4fd025419942] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:19:16.359 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:19:16.341+08:00 and status AVAILABLE 2024-09-11 17:19:16.361 INFO 2981023 --- [host-check-agent-status3,efe25fd926190ad9,945e4fd025419942] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:19:16.361 INFO 2981023 --- [host-check-agent-status3,efe25fd926190ad9,945e4fd025419942] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:19:16.361 INFO 2981023 --- [host-check-agent-status3,efe25fd926190ad9,945e4fd025419942] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:19:16.362 INFO 2981023 --- [host-check-agent-status4,efe25fd926190ad9,7c6afdbe0ff84445] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:19:16.362 INFO 2981023 --- [host-check-agent-status4,efe25fd926190ad9,7c6afdbe0ff84445] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:19:16.362 INFO 2981023 --- [host-check-agent-status4,efe25fd926190ad9,7c6afdbe0ff84445] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:19:16.367 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:19:16.365461+08:00, lastProcessTime=2024-09-11T17:19:06.336772+08:00 2024-09-11 17:19:16.370 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:19:16.371 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.371 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:19:16.372 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:19:16.373 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.374 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:19:16.375 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.375 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:16.375 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:19:16.376 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.376 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:16.377 INFO 2981023 --- [ocp-schedules-5,fccd6661d3400e78,85920d62c4c488c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:19:16.387 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:19:16.387 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:19:16.389 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:19:16.397 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:19:16.399 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.399 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:16.401 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:19:16.410 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:19:16.418 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:19:16.430 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:19:16.430 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:19:16.439 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:19:16.441 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:16.442 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.443 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:19:16.443 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:19:16.455 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:19:16.475 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:19:16.477 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:19:16.479 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:16.481 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:19:16.481 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.481 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:16.482 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:19:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:19:16.487 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:19:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:19:16.491 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:19:16.491 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:19:16.499 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:19:16.500 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:19:16.501 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:19:16.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:19:16.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:19:16.521 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:19:16.526 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:19:16.528 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:19:16.536 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:19:16.540 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:19:16.540 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:19:16.541 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:16.541 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:19:16.544 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:19:16.556 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:16.556 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:19:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:19:16.561 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:19:16.562 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:19:16.562 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:19:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:19:16.569 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:19:16.569 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:16.569 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:19:16.570 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.570 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:19:16.570 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:16.571 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:19:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.574 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:19:16.575 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:19:16.582 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:19:16.591 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:19:16.593 INFO 2981023 --- [host-check-host-status3,b24dc121324eba52,18b32cde508a9509] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:19:16.594 INFO 2981023 --- [host-check-host-status1,b24dc121324eba52,bd0ec344fa335791] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:19:16.596 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:19:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:19:16.600 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:19:16.600 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:19:16.607 INFO 2981023 --- [host-check-host-status4,b24dc121324eba52,411a008d39ee4389] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:19:16.621 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:19:16.623 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:19:16.632 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:16.637 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.637 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:19:16.638 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:19:16.641 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:19:16.644 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:19:16.645 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:19:16.652 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:19:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:19:16.668 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:16.668 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:16.670 INFO 2981023 --- [host-check-host-status2,b24dc121324eba52,b24512c01e7ca40d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:19:16.679 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:19:16.686 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:16.687 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.687 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:19:16.693 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:19:16.694 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:19:16.694 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:19:16.699 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:16.699 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:16.700 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:19:16.700 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:19:16.365461+08:00 2024-09-11 17:19:16.705 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:19:16.705 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:19:16.705 INFO 2981023 --- [ocp-schedules-26,0bc6e448e99664f0,1438467922ad25b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:19:16.975 INFO 2981023 --- [host-check-clockdiff1,80f416d0562ba519,44e456b3b3deeb91] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046356 0 0 , error= 2024-09-11 17:19:17.125 INFO 2981023 --- [ocp-async-3,ad03c87fef0402ba,b6434964c449b79a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,253ccba1f6c95490,105b215aa8d2a365] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,253ccba1f6c95490,105b215aa8d2a365] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:17.610 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046280, nodeCount=480, validCount=480 2024-09-11 17:19:17.734 INFO 2981023 --- [ocp-async-10,7f3c08cbb757d63e,9bcfad4488840c24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:19:19.125 INFO 2981023 --- [ocp-async-4,5f8299a87e7b477f,c64002b5e8d4aab3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:19.668 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:19.670 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,386a8bfa8e0734ba,3c75784ed2b27649] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,386a8bfa8e0734ba,3c75784ed2b27649] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:21.125 INFO 2981023 --- [ocp-async-7,32aa64c1d736d2cd,c2821919c9a2a066] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:21.632 INFO 2981023 --- [ocp-schedules-20,0e05f7dadbadd985,ccb192f72773c312] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:19:22.061 INFO 2981023 --- [host-check-clockdiff3,80f416d0562ba519,e8860abdf34499d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046362 1 1 , error= 2024-09-11 17:19:22.066 INFO 2981023 --- [ocp-schedules-1,8e5f918561b984bd,94b9d1c60c8a1a97] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:19:22.066 INFO 2981023 --- [ocp-schedules-1,8e5f918561b984bd,94b9d1c60c8a1a97] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:19:22.066 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:19:22.066 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:19:22.066 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:19:22.070 INFO 2981023 --- [ocp-schedules-1,8e5f918561b984bd,94b9d1c60c8a1a97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:19:22.073 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:19:22.074 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:19:22.074 INFO 2981023 --- [ocp-schedules-1,8e5f918561b984bd,94b9d1c60c8a1a97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:19:22.075 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:19:22.077 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:19:22.078 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:19:22.080 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:19:22.104 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:19:22.105 INFO 2981023 --- [ocp-schedules-1,8e5f918561b984bd,94b9d1c60c8a1a97] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:19:22.107 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:19:22.107 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:19:22.109 INFO 2981023 --- [ocp-schedules-1,8e5f918561b984bd,94b9d1c60c8a1a97] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:19:22.670 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:22.671 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:23.125 INFO 2981023 --- [ocp-async-10,43121a6392698c36,83190d91edcedc8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,3ab380ee377ecff4,dc3c549b444c7b59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,3ab380ee377ecff4,dc3c549b444c7b59] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:24.130 INFO 2981023 --- [ocp-schedules-12,cdcd388bb1d59ac1,a9fe1909937f25b0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:19:24.417 INFO 2981023 --- [ocp-schedules-29,4ce93c40ed2b33c9,f9cb6eaa8e122243] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:19:24.463 WARN 2981023 --- [scanner-13461269-1,4ce93c40ed2b33c9,02ba1ce9a4ac93b0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:19:25.125 INFO 2981023 --- [ocp-async-1,6b8c7e4cf7db1706,8573751fa60bb1e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:25.671 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:25.672 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,18788f37928228cc,14462c16afadee2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,18788f37928228cc,14462c16afadee2c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:19:26.352605+08:00, lastProcessTime=2024-09-11T17:19:16.365461+08:00 2024-09-11 17:19:26.475 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:19:26.352605+08:00 2024-09-11 17:19:27.125 INFO 2981023 --- [ocp-async-2,120830f6eb8caa8d,7642d460dd7567a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:27.734 INFO 2981023 --- [ocp-async-9,376e5c94bfba7436,40dffd8e80384398] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:19:27.846 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,94b5688f3f5126d4,5fc7feb91e00273e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=94b5688f3f5126d4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:19:27.868 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,94b5688f3f5126d4,5fc7feb91e00273e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=94b5688f3f5126d4, duration=27 ms] 2024-09-11 17:19:28.672 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:28.672 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:29.125 INFO 2981023 --- [ocp-async-8,822b1622444bf0db,83fbf94061c73c28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,6f82e276c350d7b8,9fcfde12b5588129] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,6f82e276c350d7b8,9fcfde12b5588129] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:31.126 INFO 2981023 --- [ocp-async-1,9d15542d91c2f575,2384370b29b21eab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:31.634 INFO 2981023 --- [ocp-schedules-2,418a513d3f19276d,d7c3130b0c5a251b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:19:31.672 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:31.673 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,aff6b110fde780af,e573481d14487ac3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,aff6b110fde780af,e573481d14487ac3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:33.126 INFO 2981023 --- [ocp-async-9,47c5c6671f7049c0,78e6efa86ffef79b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:33.278 INFO 2981023 --- [ocp-schedules-16,0b63c2474f7542d9,42ee93ff40f3a8c0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:19:33.290 INFO 2981023 --- [ocp-schedules-16,0b63c2474f7542d9,42ee93ff40f3a8c0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:19:33.290 INFO 2981023 --- [ocp-schedules-16,0b63c2474f7542d9,42ee93ff40f3a8c0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:19:34.614 INFO 2981023 --- [ocp-schedules-24,acda551041787b61,9fd44ad7aec80905] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:19:34.660 WARN 2981023 --- [scanner-13461478-1,acda551041787b61,ae654cb10d62de38] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:19:34.673 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:34.674 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:34.683 INFO 2981023 --- [alarm-detect-6,acda551041787b61,f7459f3d7f0c996f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643548, data is empty. 2024-09-11 17:19:34.703 INFO 2981023 --- [alarm-detect-6,acda551041787b61,f7459f3d7f0c996f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643548, data is empty. 2024-09-11 17:19:34.713 INFO 2981023 --- [alarm-detect-6,acda551041787b61,f7459f3d7f0c996f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643548, data is empty. 2024-09-11 17:19:34.721 INFO 2981023 --- [alarm-detect-17,acda551041787b61,bbe5024a5808000c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643548, data is empty. 2024-09-11 17:19:34.723 INFO 2981023 --- [alarm-detect-11,acda551041787b61,371856f8abcda574] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643548, data is empty. 2024-09-11 17:19:34.727 INFO 2981023 --- [alarm-detect-11,acda551041787b61,371856f8abcda574] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643548, data is empty. 2024-09-11 17:19:34.727 INFO 2981023 --- [alarm-detect-17,acda551041787b61,bbe5024a5808000c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643548, data is empty. 2024-09-11 17:19:34.732 INFO 2981023 --- [alarm-detect-17,acda551041787b61,bbe5024a5808000c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643548, data is empty. 2024-09-11 17:19:34.733 INFO 2981023 --- [alarm-detect-11,acda551041787b61,371856f8abcda574] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643548, data is empty. 2024-09-11 17:19:34.828 INFO 2981023 --- [alarm-detect-10,acda551041787b61,da97aed308532cd8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643548, data is empty. 2024-09-11 17:19:34.832 INFO 2981023 --- [alarm-detect-10,acda551041787b61,da97aed308532cd8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643548, data is empty. 2024-09-11 17:19:34.835 INFO 2981023 --- [alarm-detect-10,acda551041787b61,da97aed308532cd8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643548, data is empty. 2024-09-11 17:19:35.126 INFO 2981023 --- [ocp-async-10,c256a8a107a83a07,fc13b15f32602a18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,3b0170081589f64e,9e8bf4cdaf943ff4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,3b0170081589f64e,9e8bf4cdaf943ff4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:19:36.337251+08:00, lastProcessTime=2024-09-11T17:19:26.352605+08:00 2024-09-11 17:19:36.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:19:36.337251+08:00 2024-09-11 17:19:36.886 INFO 2981023 --- [ocp-schedules-20,927cc055ade8fd8c,c80c9af83554b226] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:19:36.905 WARN 2981023 --- [ocp-schedules-20,927cc055ade8fd8c,c80c9af83554b226] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:19:37.126 INFO 2981023 --- [ocp-async-5,00753d383d944412,d77549d9354678de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:37.314 INFO 2981023 --- [prometheus-cache-monitor1,67a6ca50f5faf9f6,ac5f9e7a04f15712] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:19:37.315 INFO 2981023 --- [prometheus-cache-monitor1,67a6ca50f5faf9f6,ac5f9e7a04f15712] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:19:37.674 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:37.675 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:37.734 INFO 2981023 --- [ocp-async-8,cb2e22335822e60c,749b0dbd9f37314c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:19:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,c102e759961042e5,e13a3d6ee1e74532] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,c102e759961042e5,e13a3d6ee1e74532] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:38.796 INFO 2981023 --- [ocp-schedules-25,22620a789c5108ea,1483a4b8239cc147] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:19:39.126 INFO 2981023 --- [ocp-async-9,2c4f34f704c4b8b6,4716362d89df2682] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:40.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:19:40.003 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:19:40.675 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:40.675 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:41.126 INFO 2981023 --- [ocp-async-7,a76a4807c4d02cb0,e5b4785b18d46f23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,5aed2c1679032b0d,26b671fccdc5b757] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,5aed2c1679032b0d,26b671fccdc5b757] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:41.636 INFO 2981023 --- [ocp-schedules-10,b510d1fbe66c8081,87646fc37eacfab1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:19:43.126 INFO 2981023 --- [ocp-async-3,050f3c165ef84d36,941ca893abc458b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:43.675 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:43.676 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,710d7ac5f17139a5,ca382ecd97809266] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,710d7ac5f17139a5,ca382ecd97809266] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:44.873 INFO 2981023 --- [ocp-schedules-25,99fe5320c6c5855d,444dbdc85ec98c86] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:19:44.951 WARN 2981023 --- [scanner-13461811-1,99fe5320c6c5855d,0fba931b3c733884] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:19:45.101 INFO 2981023 --- [prometheus-second-schedule-1,014e0fa465b65b61,eb6a2f822778012c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:19:45.126 INFO 2981023 --- [ocp-async-2,6275ce18a8a7ca1d,f1a99a1831158e15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:46.022 INFO 2981023 --- [ocp-schedules-28,98ae85b1608aafe1,ddcffa3b9e3385e4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:19:46.022 INFO 2981023 --- [host-check-clockdiff1,98ae85b1608aafe1,523d04817a50531e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:19:46.022 INFO 2981023 --- [host-check-clockdiff2,98ae85b1608aafe1,8c55a036332c5c2a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:19:46.023 INFO 2981023 --- [host-check-clockdiff3,98ae85b1608aafe1,47baae3f464c958c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:19:46.023 INFO 2981023 --- [ocp-schedules-13,670a3904810087b4,70e02cbf8a09c23d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:19:46.025 INFO 2981023 --- [host-check-clockdiff4,98ae85b1608aafe1,be2390c0abfc7495] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:19:46.025 INFO 2981023 --- [ocp-schedules-26,e6a77e194dde516e,b21123d26c33615b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:19:46.027 INFO 2981023 --- [host-check-agent-status1,670a3904810087b4,54085f2a0f7146ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:19:46.044 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:19:46.044 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:19:46.044 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:19:46.044 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:19:46.051 INFO 2981023 --- [host-check-agent-status2,670a3904810087b4,1a3039c74db9b8a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:19:46.052 INFO 2981023 --- [host-check-agent-status3,670a3904810087b4,ef7bb0bb367383d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:19:46.052 INFO 2981023 --- [host-check-agent-status1,670a3904810087b4,54085f2a0f7146ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:19:46.053 INFO 2981023 --- [host-check-agent-status4,670a3904810087b4,09051c06ffd52eb3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:19:46.053 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:19:46.056 INFO 2981023 --- [host-check-agent-status2,670a3904810087b4,1a3039c74db9b8a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:19:46.057 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:19:46.057 INFO 2981023 --- [host-check-agent-status1,670a3904810087b4,54085f2a0f7146ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:19:46.062 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:19:46.062 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:19:46.063 INFO 2981023 --- [host-check-agent-status2,670a3904810087b4,1a3039c74db9b8a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:19:46.063 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:19:46.064 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:19:46.067 INFO 2981023 --- [host-check-clockdiff2,98ae85b1608aafe1,8c55a036332c5c2a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046386 0 0 , error= 2024-09-11 17:19:46.067 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:19:46.096 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:19:46.105 INFO 2981023 --- [host-check-agent-status3,670a3904810087b4,ef7bb0bb367383d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:19:46.111 INFO 2981023 --- [host-check-clockdiff4,98ae85b1608aafe1,be2390c0abfc7495] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046386 0 0 , error= 2024-09-11 17:19:46.171 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:19:46.176 INFO 2981023 --- [host-check-agent-status4,670a3904810087b4,09051c06ffd52eb3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:19:46.187 INFO 2981023 --- [ocp-schedules-15,7ef2f8742d0e018a,76d3e03e40736337] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:19:46.188 INFO 2981023 --- [host-check-agent-status3,670a3904810087b4,ef7bb0bb367383d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:19:46.191 INFO 2981023 --- [ocp-schedules-25,ff67bf96c66269c6,7ba1b8bcb4855008] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:19:46.192 INFO 2981023 --- [ocp-schedules-29,a0cffc02b97697f9,47cb481c29559ee4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:19:46.194 INFO 2981023 --- [ocp-schedules-22,195bce47c226e0d1,cddf1d5c6e47b50a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:19:46.198 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:19:46.198 INFO 2981023 --- [ocp-schedules-25,ff67bf96c66269c6,7ba1b8bcb4855008] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:19:46.200 INFO 2981023 --- [ocp-schedules-25,ff67bf96c66269c6,7ba1b8bcb4855008] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:19:46.202 INFO 2981023 --- [ocp-schedules-2,890e65656184d141,27e26718cb57756a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:19:46.203 INFO 2981023 --- [host-check-agent-status4,670a3904810087b4,09051c06ffd52eb3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:19:46.204 INFO 2981023 --- [ocp-schedules-29,a0cffc02b97697f9,47cb481c29559ee4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:19:46.207 INFO 2981023 --- [ocp-schedules-15,7ef2f8742d0e018a,76d3e03e40736337] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:19:46.274 INFO 2981023 --- [ocp-schedules-2,890e65656184d141,27e26718cb57756a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:19:46.285 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:19:46.329 INFO 2981023 --- [ocp-schedules-10,303edff5781f4dfd,92296364a40cb4da] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:19:46.334 INFO 2981023 --- [host-check-agent-status2,670a3904810087b4,1a3039c74db9b8a1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:19:46.338 INFO 2981023 --- [host-check-agent-status2,670a3904810087b4,1a3039c74db9b8a1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:19:46.338 INFO 2981023 --- [host-check-agent-status2,670a3904810087b4,1a3039c74db9b8a1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:19:46.338 INFO 2981023 --- [host-check-agent-status2,670a3904810087b4,1a3039c74db9b8a1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:19:46.367 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:19:46.369 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:19:46.369172437+08:00 2024-09-11 17:19:46.381 INFO 2981023 --- [host-check-agent-status1,670a3904810087b4,54085f2a0f7146ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:19:46.381 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:19:46.369+08:00 2024-09-11 17:19:46.383 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:19:46.384 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:19:46.383703047+08:00 2024-09-11 17:19:46.384 INFO 2981023 --- [host-check-agent-status1,670a3904810087b4,54085f2a0f7146ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:19:46.384 INFO 2981023 --- [host-check-agent-status1,670a3904810087b4,54085f2a0f7146ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:19:46.384 INFO 2981023 --- [host-check-agent-status1,670a3904810087b4,54085f2a0f7146ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:19:46.385 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:19:46.384+08:00 2024-09-11 17:19:46.386 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:19:46.369+08:00 and status AVAILABLE 2024-09-11 17:19:46.387 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:19:46.386723+08:00, lastProcessTime=2024-09-11T17:19:36.337251+08:00 2024-09-11 17:19:46.388 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:19:46.384+08:00 and status AVAILABLE 2024-09-11 17:19:46.389 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:19:46.390 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:19:46.390 INFO 2981023 --- [host-check-agent-status3,670a3904810087b4,ef7bb0bb367383d3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:19:46.392 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:19:46.392269025+08:00 2024-09-11 17:19:46.392 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:19:46.392628227+08:00 2024-09-11 17:19:46.394 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:19:46.392+08:00 2024-09-11 17:19:46.394 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:19:46.392+08:00 2024-09-11 17:19:46.395 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:19:46.392+08:00 and status AVAILABLE 2024-09-11 17:19:46.396 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.397 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:19:46.397 INFO 2981023 --- [host-check-agent-status3,670a3904810087b4,ef7bb0bb367383d3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:19:46.397 INFO 2981023 --- [host-check-agent-status3,670a3904810087b4,ef7bb0bb367383d3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:19:46.397 INFO 2981023 --- [host-check-agent-status3,670a3904810087b4,ef7bb0bb367383d3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:19:46.397 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:19:46.398 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:19:46.392+08:00 and status AVAILABLE 2024-09-11 17:19:46.398 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:19:46.405 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:19:46.407 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:19:46.409 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.409 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:19:46.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:19:46.411 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:19:46.415 INFO 2981023 --- [host-check-agent-status4,670a3904810087b4,09051c06ffd52eb3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:19:46.417 INFO 2981023 --- [ocp-schedules-15,7ef2f8742d0e018a,76d3e03e40736337] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:19:46.418 INFO 2981023 --- [host-check-agent-status4,670a3904810087b4,09051c06ffd52eb3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:19:46.418 INFO 2981023 --- [host-check-agent-status4,670a3904810087b4,09051c06ffd52eb3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:19:46.418 INFO 2981023 --- [host-check-agent-status4,670a3904810087b4,09051c06ffd52eb3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:19:46.467 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.467 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:46.472 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:19:46.477 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:19:46.478 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.479 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:19:46.479 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:19:46.480 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.484 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:19:46.486 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:19:46.487 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:19:46.488 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:19:46.488 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:19:46.489 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:19:46.489 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.490 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:46.496 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:19:46.497 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.498 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:46.502 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:19:46.524 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:19:46.554 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:19:46.566 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:19:46.576 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:19:46.585 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:19:46.590 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:19:46.590 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046282106000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:19:46.591 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.591 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:19:46.602 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:19:46.609 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:19:46.611 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:19:46.612 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:19:46.613 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.614 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.614 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:19:46.615 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:19:46.616 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.619 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:19:46.621 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:19:46.624 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:19:46.639 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:19:46.642 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:19:46.651 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:19:46.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.659 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.659 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:46.660 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:19:46.660 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.661 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:19:46.663 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:19:46.664 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:19:46.665 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:19:46.682 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.683 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:46.684 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:46.684 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:19:46.684 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:19:46.690 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:19:46.693 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:19:46.694 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:19:46.703 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.703 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.704 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:19:46.704 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:19:46.704 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:19:46.705 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.705 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:19:46.706 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:19:46.706 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.707 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:19:46.707 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:19:46.709 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:19:46.712 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:19:46.715 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:19:46.717 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.718 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.718 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:19:46.719 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:19:46.722 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:19:46.723 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:19:46.725 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:19:46.726 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:19:46.730 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:19:46.732 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:19:46.736 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:19:46.741 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.741 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.742 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:19:46.748 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:19:46.749 INFO 2981023 --- [host-check-host-status3,e6a77e194dde516e,74cbd16ce2cfa5a6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:19:46.755 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:19:46.758 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:19:46.761 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:19:46.763 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:19:46.770 INFO 2981023 --- [host-check-host-status4,e6a77e194dde516e,43015e7d61a6d745] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:19:46.776 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.776 INFO 2981023 --- [host-check-host-status2,e6a77e194dde516e,46c5408c19807c34] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:19:46.777 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.777 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:19:46.778 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:19:46.778 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.781 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:19:46.782 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:19:46.790 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:19:46.792 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.793 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.793 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:19:46.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:19:46.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:19:46.796 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:19:46.798 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.808 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:19:46.814 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.814 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:19:46.815 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.815 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:19:46.815 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:19:46.816 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:19:46.819 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:19:46.823 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:19:46.824 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:19:46.825 INFO 2981023 --- [host-check-host-status1,e6a77e194dde516e,387adade99a8fa96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:19:46.834 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:19:46.834 INFO 2981023 --- [ocp-schedules-3,82ee7188081065af,73dea922defccd93] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:19:46.834 INFO 2981023 --- [ocp-schedules-3,82ee7188081065af,73dea922defccd93] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:19:46.863 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:19:46.386723+08:00 2024-09-11 17:19:46.866 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:19:46.867 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:19:46.873 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.874 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.874 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:19:46.877 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.878 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.878 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:19:46.879 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:19:46.879 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.882 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:19:46.882 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:19:46.896 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:19:46.902 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:46.902 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:46.903 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:19:46.908 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:19:46.908 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726045812661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:19:46.908 INFO 2981023 --- [ocp-schedules-29,a0cffc02b97697f9,47cb481c29559ee4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:19:46.920 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:19:46.935 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:19:46.941 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:19:46.943 INFO 2981023 --- [ocp-schedules-8,7fccfdedc288bf0d,a47bd74ad20b0b9c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:19:46.973 INFO 2981023 --- [host-check-clockdiff1,98ae85b1608aafe1,523d04817a50531e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046386 0 0 , error= 2024-09-11 17:19:47.126 INFO 2981023 --- [ocp-async-10,a604abcba0290f05,aa75f0d7c18f2b0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,d1d478e5433f20fa,48c1b41ee5f67b72] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,d1d478e5433f20fa,48c1b41ee5f67b72] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:47.734 INFO 2981023 --- [ocp-async-8,b8a75855c6f43b87,1e001da55f7619d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:19:49.127 INFO 2981023 --- [ocp-async-4,ca8ce3c1feaba4f1,4a28b0cbe0359e93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:49.294 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,fdf02214abf62474,e38e9d94edc5bb91] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&version=2&ObClusterId=1719994437, client=10.31.35.10, traceId=fdf02214abf62474, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkprdocp,1719994437,2,{"ObRegion":"linkprdocp","ObCluster":"linkprdocp","ObRegionId":1719994437,"ObClusterId":1719994437,"Type":"PRIMARY","timestamp":1726046389293926,"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:19:49.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,fdf02214abf62474,e38e9d94edc5bb91] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkprdocp, obClusterId=1719994437, rootServerJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, groupName=linkprdocp, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:19:49.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,fdf02214abf62474,e38e9d94edc5bb91] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:19:49.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,fdf02214abf62474,e38e9d94edc5bb91] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkprdocp, clusterId=1719994437, affectRows=1 2024-09-11 17:19:49.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,fdf02214abf62474,e38e9d94edc5bb91] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.10, traceId=fdf02214abf62474, duration=8 ms] 2024-09-11 17:19:49.684 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:49.686 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:19:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:19:50.079 INFO 2981023 --- [ocp-schedules-22,efd6498020c08239,000742c68c8aa4d9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:19:50.079+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:19:50.083 INFO 2981023 --- [ocp-schedules-22,efd6498020c08239,000742c68c8aa4d9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:19:50.186 INFO 2981023 --- [ocp-schedules-18,4575be8a492451c6,efa83462761da8ad] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:19:50.230 INFO 2981023 --- [ocp-schedules-22,efd6498020c08239,000742c68c8aa4d9] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 28 to SqlAggregateData 2024-09-11 17:19:50.232 INFO 2981023 --- [ocp-schedules-22,efd6498020c08239,000742c68c8aa4d9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:19:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,59510cfe27e13baf,8da6b1b48619e0e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,59510cfe27e13baf,8da6b1b48619e0e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:50.546 INFO 2981023 --- [ocp-schedules-1,3ca2bcf34edc1c8c,bdab34371e86486c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:19:50.548 INFO 2981023 --- [ocp-schedules-1,3ca2bcf34edc1c8c,bdab34371e86486c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:19:50.595 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:19:50.595 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.596 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:19:50.597 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.599 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:19:50.599 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:19:50.599 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.600 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:19:50.602 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:19:50.602 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.602 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.605 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.606 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.610 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.611 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.612 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.613 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.613 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.614 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.616 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.616 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:19:50.623 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.623 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:19:50.625 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.625 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.627 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.627 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.630 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.630 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:19:50.636 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.636 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.636 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:19:50.636 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.637 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:19:50.637 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE17B1D729-0-0] 2024-09-11 17:19:50.637 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:19:50.638 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.638 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:19:50.641 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.641 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:19:50.642 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.642 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:19:50.643 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.643 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:19:50.644 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:19:50.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.648 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:19:50.649 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:19:50.649 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.650 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.650 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:19:50.650 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:19:50.650 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.651 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:19:50.651 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.652 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:19:50.652 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.653 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:19:50.653 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.654 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:19:50.654 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.655 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:19:50.655 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.656 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:19:50.656 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.657 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:19:50.657 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:19:50.661 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:19:50.662 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.663 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:19:50.663 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:50.663 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:19:50.664 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE17B1D733-0-0] 2024-09-11 17:19:50.664 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:19:50.667 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.667 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.669 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.670 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:19:50.670 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.671 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:19:50.671 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.672 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:19:50.672 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:50.672 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.673 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:19:50.674 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.674 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:19:50.675 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.675 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:19:50.676 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.676 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:19:50.676 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:19:50.678 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:19:50.681 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:19:50.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:19:50.685 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:19:50.685 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:19:50.685 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:19:50.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:19:50.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:19:50.691 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:19:50.692 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:19:50.694 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:19:50.694 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.694 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.695 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:19:50.699 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:19:50.701 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:19:50.701 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:19:50.701 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:19:50.705 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:19:50.706 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:19:50.707 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:50.707 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:19:50.715 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:19:50.716 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:19:50.716 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:19:50.716 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:19:50.716 INFO 2981023 --- [ocp-schedules-1,3ca2bcf34edc1c8c,bdab34371e86486c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:19:51.127 INFO 2981023 --- [ocp-async-7,637de36083059403,d41b21841599ccea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:51.353 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7b8e727258e0c797,8ae2538b0da659ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=7b8e727258e0c797, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:19:51.376 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7b8e727258e0c797,8ae2538b0da659ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=7b8e727258e0c797, duration=28 ms] 2024-09-11 17:19:51.489 INFO 2981023 --- [ocp-schedules-25,38fac5cf49812a02,503ec55c5f08f781] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:19:51.490 INFO 2981023 --- [ocp-schedules-25,38fac5cf49812a02,503ec55c5f08f781] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:19:51.490 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:19:51.490 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:19:51.490 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:19:51.490 INFO 2981023 --- [ocp-schedules-25,38fac5cf49812a02,503ec55c5f08f781] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:19:51.639 INFO 2981023 --- [ocp-schedules-22,3cba2a203eb96353,9a1fd8df0b0f4074] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:19:51.899 INFO 2981023 --- [ocp-schedules-7,c1ad3638ef3c60df,8e37f56e0d21dbe6] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:19:51.921 WARN 2981023 --- [ocp-schedules-7,c1ad3638ef3c60df,8e37f56e0d21dbe6] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:19:51.987 INFO 2981023 --- [host-check-clockdiff3,98ae85b1608aafe1,47baae3f464c958c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046391 1 1 , error= 2024-09-11 17:19:52.686 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:52.688 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:52.985 INFO 2981023 --- [ocp-schedules-23,4a0130ae3c89745f,95bc3bd0bfb6c05c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:19:52.985 INFO 2981023 --- [ocp-schedules-23,4a0130ae3c89745f,95bc3bd0bfb6c05c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:19:53.127 INFO 2981023 --- [ocp-async-10,56d0097e4d9a7073,09d8553859341a9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:53.293 INFO 2981023 --- [ocp-schedules-14,32e32b35d1631c55,e400de62c8899009] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:19:53.301 INFO 2981023 --- [ocp-schedules-14,32e32b35d1631c55,e400de62c8899009] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:19:53.301 INFO 2981023 --- [ocp-schedules-14,32e32b35d1631c55,e400de62c8899009] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:19:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,1972e7e4ec5043ce,5271d496bc1663be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,1972e7e4ec5043ce,5271d496bc1663be] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:54.133 INFO 2981023 --- [ocp-schedules-8,3d215b3c72bec953,5443970ef7b07780] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:19:55.063 INFO 2981023 --- [ocp-schedules-5,f6de740e74abad05,8d41ba99671a836c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:19:55.114 WARN 2981023 --- [scanner-13462030-1,f6de740e74abad05,3e24fde0896680c0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:19:55.127 INFO 2981023 --- [ocp-async-1,b041029fffdc8c5f,747bb2a1cb67ae48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:55.149 INFO 2981023 --- [alarm-detect-7,f6de740e74abad05,bbc8611ba1b19642] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643548, data is empty. 2024-09-11 17:19:55.154 INFO 2981023 --- [alarm-detect-7,f6de740e74abad05,bbc8611ba1b19642] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643576, data is empty. 2024-09-11 17:19:55.164 INFO 2981023 --- [alarm-detect-7,f6de740e74abad05,bbc8611ba1b19642] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643576, data is empty. 2024-09-11 17:19:55.173 INFO 2981023 --- [alarm-detect-9,f6de740e74abad05,f7a1d678a39611a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643548, data is empty. 2024-09-11 17:19:55.175 INFO 2981023 --- [alarm-detect-12,f6de740e74abad05,cda056618da8830f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643548, data is empty. 2024-09-11 17:19:55.181 INFO 2981023 --- [alarm-detect-9,f6de740e74abad05,f7a1d678a39611a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643576, data is empty. 2024-09-11 17:19:55.187 INFO 2981023 --- [alarm-detect-12,f6de740e74abad05,cda056618da8830f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643576, data is empty. 2024-09-11 17:19:55.187 INFO 2981023 --- [alarm-detect-9,f6de740e74abad05,f7a1d678a39611a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643576, data is empty. 2024-09-11 17:19:55.192 INFO 2981023 --- [alarm-detect-12,f6de740e74abad05,cda056618da8830f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643576, data is empty. 2024-09-11 17:19:55.210 INFO 2981023 --- [alarm-detect-14,f6de740e74abad05,1b3a7b4882380389] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=2, detectAlarmsCount=2, matchedTargets=[alarm_template_id=0:host=10.31.35.10-2, alarm_template_id=0:host=10.31.35.10-3, alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.6-2, alarm_template_id=0:host=10.31.35.5-2, alarm_template_id=0:host=10.31.35.7-3, alarm_template_id=0:host=10.31.35.7-2, alarm_template_id=0:host=10.31.35.6-3] 2024-09-11 17:19:55.295 INFO 2981023 --- [alarm-detect-6,f6de740e74abad05,a6ef3a158fd590f9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643548, data is empty. 2024-09-11 17:19:55.301 INFO 2981023 --- [alarm-detect-6,f6de740e74abad05,a6ef3a158fd590f9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643576, data is empty. 2024-09-11 17:19:55.307 INFO 2981023 --- [alarm-detect-6,f6de740e74abad05,a6ef3a158fd590f9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643576, data is empty. 2024-09-11 17:19:55.427 INFO 2981023 --- [ocp-schedules-21,793b5ebf2e8ae676,d5a055f79eb913b0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:19:55.427 INFO 2981023 --- [ocp-schedules-21,793b5ebf2e8ae676,d5a055f79eb913b0] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:19:55.447 INFO 2981023 --- [Collect all tenants major compactions2,793b5ebf2e8ae676,2b73ba258ab682f5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:19:55.449 INFO 2981023 --- [Collect all tenants major compactions1,793b5ebf2e8ae676,58626c2dd2e085bd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:19:55.449 INFO 2981023 --- [Collect all tenants major compactions3,793b5ebf2e8ae676,9e244dc2f50cd6d4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:19:55.449 INFO 2981023 --- [Collect all tenants major compactions4,793b5ebf2e8ae676,e557a4c79c733e90] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:19:55.450 INFO 2981023 --- [Collect all tenants major compactions6,793b5ebf2e8ae676,8b21859230661647] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:19:55.457 INFO 2981023 --- [Collect all tenants major compactions5,793b5ebf2e8ae676,d68426dd2593fa4d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:19:55.471 INFO 2981023 --- [Collect all tenants major compactions3,793b5ebf2e8ae676,9e244dc2f50cd6d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:55.471 INFO 2981023 --- [Collect all tenants major compactions4,793b5ebf2e8ae676,e557a4c79c733e90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:55.471 INFO 2981023 --- [Collect all tenants major compactions1,793b5ebf2e8ae676,58626c2dd2e085bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:55.471 INFO 2981023 --- [Collect all tenants major compactions2,793b5ebf2e8ae676,2b73ba258ab682f5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:55.472 INFO 2981023 --- [Collect all tenants major compactions5,793b5ebf2e8ae676,d68426dd2593fa4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:55.472 INFO 2981023 --- [Collect all tenants major compactions4,793b5ebf2e8ae676,e557a4c79c733e90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:19:55.472 INFO 2981023 --- [Collect all tenants major compactions1,793b5ebf2e8ae676,58626c2dd2e085bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:19:55.473 INFO 2981023 --- [Collect all tenants major compactions3,793b5ebf2e8ae676,9e244dc2f50cd6d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:19:55.473 INFO 2981023 --- [Collect all tenants major compactions5,793b5ebf2e8ae676,d68426dd2593fa4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:19:55.473 INFO 2981023 --- [Collect all tenants major compactions6,793b5ebf2e8ae676,8b21859230661647] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:19:55.474 INFO 2981023 --- [Collect all tenants major compactions2,793b5ebf2e8ae676,2b73ba258ab682f5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:19:55.475 INFO 2981023 --- [Collect all tenants major compactions6,793b5ebf2e8ae676,8b21859230661647] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:19:55.476 INFO 2981023 --- [Collect all tenants major compactions2,793b5ebf2e8ae676,2b73ba258ab682f5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:19:55.476 INFO 2981023 --- [Collect all tenants major compactions2,793b5ebf2e8ae676,2b73ba258ab682f5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:19:55.477 INFO 2981023 --- [Collect all tenants major compactions3,793b5ebf2e8ae676,9e244dc2f50cd6d4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:19:55.477 INFO 2981023 --- [Collect all tenants major compactions3,793b5ebf2e8ae676,9e244dc2f50cd6d4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:19:55.477 INFO 2981023 --- [Collect all tenants major compactions4,793b5ebf2e8ae676,e557a4c79c733e90] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:19:55.477 INFO 2981023 --- [Collect all tenants major compactions4,793b5ebf2e8ae676,e557a4c79c733e90] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:19:55.477 INFO 2981023 --- [Collect all tenants major compactions1,793b5ebf2e8ae676,58626c2dd2e085bd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:19:55.477 INFO 2981023 --- [Collect all tenants major compactions1,793b5ebf2e8ae676,58626c2dd2e085bd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:19:55.477 INFO 2981023 --- [Collect all tenants major compactions5,793b5ebf2e8ae676,d68426dd2593fa4d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:19:55.477 INFO 2981023 --- [Collect all tenants major compactions5,793b5ebf2e8ae676,d68426dd2593fa4d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:19:55.478 INFO 2981023 --- [Collect all tenants major compactions6,793b5ebf2e8ae676,8b21859230661647] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:19:55.478 INFO 2981023 --- [Collect all tenants major compactions6,793b5ebf2e8ae676,8b21859230661647] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:19:55.478 INFO 2981023 --- [ocp-schedules-21,793b5ebf2e8ae676,d5a055f79eb913b0] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:19:55.606 INFO 2981023 --- [ocp-schedules-8,797e3f670264b0da,c2bf1e8791874fae] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:19:55.606 INFO 2981023 --- [ocp-schedules-8,797e3f670264b0da,c2bf1e8791874fae] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:19:55.607 INFO 2981023 --- [ocp-schedules-8,797e3f670264b0da,c2bf1e8791874fae] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:19:55.607 INFO 2981023 --- [ocp-schedules-8,797e3f670264b0da,c2bf1e8791874fae] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:19:55.607 INFO 2981023 --- [ocp-schedules-8,797e3f670264b0da,c2bf1e8791874fae] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:19:55.607 INFO 2981023 --- [ocp-schedules-8,797e3f670264b0da,c2bf1e8791874fae] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:19:55.641 INFO 2981023 --- [ocp-schedules-19,421539afd4e9a537,59c1ecb359ed884a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:19:55.649 INFO 2981023 --- [ocp-schedules-19,421539afd4e9a537,59c1ecb359ed884a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:19:55.649 INFO 2981023 --- [ocp-schedules-19,421539afd4e9a537,59c1ecb359ed884a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:19:55.688 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:55.688 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,a111d39d60abd71e,1cac5deea6838669] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,a111d39d60abd71e,1cac5deea6838669] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:19:56.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:19:56.354541+08:00, lastProcessTime=2024-09-11T17:19:46.386723+08:00 2024-09-11 17:19:56.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=2, resultsCount=2 2024-09-11 17:19:56.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:19:56.354541+08:00 2024-09-11 17:19:57.127 INFO 2981023 --- [ocp-async-9,de6f6572d66a17ce,1bdebaea398bdab7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:57.734 INFO 2981023 --- [ocp-async-2,7b4828f92152e699,3f65df5f4e4be817] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:19:58.688 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:19:58.690 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:19:59.127 INFO 2981023 --- [ocp-async-10,597daf8874352541,9609edb2adf98f87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:19:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,1dc99bef775deda4,c79a4b1678f996bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046400, interval=60 2024-09-11 17:19:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,1dc99bef775deda4,c79a4b1678f996bf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:00.419 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:15:00.0, nextRun=2024-09-11 17:20:00.0, currentDate=Wed Sep 11 17:20:00 HKT 2024 2024-09-11 17:20:00.419 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:15:00.0, nextRunTime=2024-09-11 17:20:00.0} 2024-09-11 17:20:00.426 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:20:00.426 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:20:00.426 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:20:00.426+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:20:00.441 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:15:00.0, nextRun=2024-09-11 17:20:00.0, currentDate=Wed Sep 11 17:20:00 HKT 2024 2024-09-11 17:20:00.441 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:15:00.0, nextRunTime=2024-09-11 17:20:00.0} 2024-09-11 17:20:00.469 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 17:20:00.475 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:20:00.476 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:20:00.476 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:20:00.476 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a97cb33f4510337c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T17:20:00.476+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:20:01.127 INFO 2981023 --- [ocp-async-4,760ce49a0525b6d0,32a5781c9fa00172] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:01.357 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,367236e947e34692] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048157, operation:EXECUTE, state: RUNNING 2024-09-11 17:20:01.423 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,367236e947e34692] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048158, operation:EXECUTE, state: RUNNING 2024-09-11 17:20:01.434 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048157, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048177, task_operation=execute, latest_execution_start_time=2024-09-11T17:20:01.345+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048157}, listMap={}}, executor=10.31.35.10 2024-09-11 17:20:01.446 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048158, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048178, task_operation=execute, latest_execution_start_time=2024-09-11T17:20:01.420+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048158}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 17:20:01.447 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:20:01.452 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:20:01.472 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:20:01.473 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:20:01.484 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:20:01.488 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:20:01.488 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:20:01.505 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:20:01.511 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.perf.sql.internal.SqlParseService : parse 9 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:20:01.518 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:20:01.593 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:20:01.611 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:20:01.634 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:20:01.640 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:20:01.640 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:20:01.642 INFO 2981023 --- [ocp-schedules-11,7cc874ef83f18b75,ce0ca6ec1585e721] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:20:01.645 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:20:01.690 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:01.690 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:01.717 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2f09f30aac36bd49] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048158, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:20:01.758 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:20:01.800 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:20:01.820 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:20:01.835 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:20:01.872 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:20:01.883 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:20:01.897 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:20:01.903 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:20:01.904 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:20:01.908 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:20:01.979 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:20:02.024 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:20:02.040 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:20:02.056 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:20:02.071 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:20:02.076 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:20:02.085 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:20:02.087 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:20:02.088 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:20:02.101 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:20:02.109 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:20:02.120 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f43dc99fee6d4de5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048157, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:20:02.156 INFO 2981023 --- [ocp-schedules-21,95dd1c9d488fce0d,a906ed18d457d9c5] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:20:02.157 INFO 2981023 --- [ocp-schedules-21,95dd1c9d488fce0d,a906ed18d457d9c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:20:02.179 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:02.179 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:20:02.184 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:02.185 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:20:02.186 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046402186000} 2024-09-11 17:20:02.191 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:20:02.194 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000223, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:20:02.195 INFO 2981023 --- [ocp-schedules-21,95dd1c9d488fce0d,a906ed18d457d9c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:20:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,30778e1712673949,63a0c52426869912] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,30778e1712673949,63a0c52426869912] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:02.446 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048177, status:SUCCESS 2024-09-11 17:20:02.449 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048177, status:SUCCESS 2024-09-11 17:20:02.454 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048178, status:SUCCESS 2024-09-11 17:20:02.455 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048178, status:SUCCESS 2024-09-11 17:20:03.109 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 17:20:03.109 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 17:20:03.109 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 17:20:03.113 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000223, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 17:20:03.115 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 17:20:03.127 INFO 2981023 --- [ocp-async-7,f8f232b670d5440d,358bf8ca7e1662d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:03.705 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000223 2024-09-11 17:20:04.566 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,7289d8c80bf882e9] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:10:04.0, nextRun=2024-09-11 17:20:04.0, currentDate=Wed Sep 11 17:20:04 HKT 2024 2024-09-11 17:20:04.566 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,7289d8c80bf882e9] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-09-11 17:10:04.0, nextRunTime=2024-09-11 17:20:04.0} 2024-09-11 17:20:04.572 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,7289d8c80bf882e9] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-09-11 17:20:04.572 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,7289d8c80bf882e9] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:20:04.572 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,7289d8c80bf882e9] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:20:04.572+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:20:04.690 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:04.692 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:05.127 INFO 2981023 --- [ocp-async-8,b873d83829f7e965,930c8024427571da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,e7f9067b05d22e6b,b56887427eed91ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,e7f9067b05d22e6b,b56887427eed91ea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:05.346 INFO 2981023 --- [ocp-schedules-23,b188ef17f4a5db52,841be5dc0a6ae7dc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:20:05.389 WARN 2981023 --- [scanner-13462494-1,b188ef17f4a5db52,d4b18b931e811ef7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:20:05.472 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,1f3ee754cc7b9ecc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048159, operation:EXECUTE, state: RUNNING 2024-09-11 17:20:05.544 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048159, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048179, task_operation=execute, latest_execution_start_time=2024-09-11T17:20:05.465+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=4048159}, listMap={}}, executor=10.31.35.10 2024-09-11 17:20:05.557 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 17:20:05.558 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 17:20:05.559 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.560 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:20:05.563 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.563 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.565 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.565 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 17:20:05.578 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.579 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.579 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.580 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 17:20:05.588 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.588 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.589 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.589 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 17:20:05.600 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.600 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.600 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.601 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 17:20:05.610 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.610 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.610 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.611 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 17:20:05.614 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.615 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.615 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.615 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 17:20:05.618 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.618 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.619 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.619 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 17:20:05.622 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.622 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.622 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.623 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 17:20:05.626 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.627 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.627 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.627 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 17:20:05.631 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.631 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.632 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.632 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 17:20:05.635 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.635 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.636 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.636 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 17:20:05.642 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.642 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.643 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.643 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 17:20:05.652 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.652 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.653 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.653 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 17:20:05.656 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.656 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.657 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.657 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 17:20:05.661 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.661 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.661 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.662 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 17:20:05.665 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.665 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.666 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.666 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 17:20:05.668 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.669 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.669 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.670 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 17:20:05.672 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.672 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.673 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.673 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 17:20:05.676 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.676 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.677 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.677 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 17:20:05.680 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.680 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.681 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.681 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 17:20:05.684 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.684 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.684 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.685 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 17:20:05.687 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.688 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.688 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.689 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 17:20:05.691 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.691 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.692 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.692 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 17:20:05.695 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.695 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.696 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.696 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 17:20:05.699 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.699 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.699 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.700 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 17:20:05.702 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.702 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.703 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.703 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 17:20:05.706 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.707 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.707 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.708 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 17:20:05.710 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.711 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.711 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.711 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 17:20:05.714 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.714 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.715 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.715 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 17:20:05.718 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.718 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.718 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.719 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 17:20:05.721 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.722 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.722 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.722 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 17:20:05.725 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.725 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.726 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.726 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 17:20:05.729 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.729 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.730 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.730 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 17:20:05.733 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.733 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.733 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.734 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 17:20:05.738 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.739 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.739 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.739 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 17:20:05.743 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.744 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.744 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.744 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 17:20:05.747 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.748 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.748 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.748 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 17:20:05.754 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.754 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.755 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.755 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 17:20:05.757 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.757 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.758 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.758 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 17:20:05.760 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.761 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.761 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.761 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 17:20:05.764 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.764 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.764 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.765 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 17:20:05.767 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.767 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.767 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.768 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 17:20:05.774 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.774 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.775 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.775 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 17:20:05.791 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.791 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.792 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.792 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 17:20:05.797 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.797 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.798 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.798 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 17:20:05.802 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.802 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.803 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.803 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 17:20:05.807 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.807 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.808 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.808 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 17:20:05.810 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.811 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.811 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.812 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 17:20:05.814 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.815 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.815 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.816 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 17:20:05.818 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 17:20:05.818 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 17:20:05.822 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.822 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:20:05.824 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.824 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.825 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.825 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 17:20:05.829 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.829 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:05.829 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.830 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 17:20:05.832 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 17:20:05.843 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,239d35026fe45d8e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048159, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:20:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:20:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:20:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:20:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:20:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:20:06.002 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 17:20:06.002 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:20:06.002 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:20:06.004 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:06.004 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:20:06.005 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:06.005 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:20:06.062 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:17:41, start=2024-09-11T17:15:11, end=2024-09-11T17:17:41, updates=327, granule=120000ms, delay=150000ms 2024-09-11 17:20:06.069 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:17:41, target=2024-09-11T17:17:41, granule=120000ms, delay=0ms) 2024-09-11 17:20:06.069 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:17:41, target=2024-09-11T17:10:51, granule=600000ms, delay=410000ms) 2024-09-11 17:20:06.069 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=99ms, period=120000ms 2024-09-11 17:20:06.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:20:06.354494+08:00, lastProcessTime=2024-09-11T17:19:56.354541+08:00 2024-09-11 17:20:06.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:20:06.354494+08:00 2024-09-11 17:20:06.491 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048179, status:SUCCESS 2024-09-11 17:20:06.491 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048179, status:SUCCESS 2024-09-11 17:20:07.084 INFO 2981023 --- [ocp-schedules-21,5499c76f49bd8c9d,27f1edfbbd1496c5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:20:07.089 INFO 2981023 --- [ocp-schedules-23,679170e6380f8ec2,fbc838b99f518fdd] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:20:07.089+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:20:07.089 INFO 2981023 --- [ocp-schedules-23,679170e6380f8ec2,fbc838b99f518fdd] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:20:07.091 INFO 2981023 --- [ocp-schedules-23,679170e6380f8ec2,fbc838b99f518fdd] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:20:07.092 INFO 2981023 --- [ocp-schedules-23,679170e6380f8ec2,fbc838b99f518fdd] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:20:07.128 INFO 2981023 --- [ocp-async-3,f58ac1ca03f95882,e2a088a753cb8889] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:07.692 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:07.694 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:07.734 INFO 2981023 --- [ocp-async-4,7bdcd8c7ea13b464,0cc242f68f1a8262] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:20:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,8dcad89ce6a6b278,fb7853b270f0564e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:08.318 INFO 2981023 --- [prometheus-minute-schedule-1,8dcad89ce6a6b278,fb7853b270f0564e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:09.128 INFO 2981023 --- [ocp-async-9,29a303d1eb2bef71,ebe65a74a4d1c9fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:10.003 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:20:10.004 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:20:10.694 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:10.695 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:20:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:20:10.931 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,80e970b953417e1d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] update failed, sql:[alter system add server ? zone ?], error message:[PreparedStatementCallback; SQL [alter system add server ? zone ?]; (conn=3221488696) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=3221488696) Timeout] 2024-09-11 17:20:10.936 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,80e970b953417e1d] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2306-000621D2D953866A-0-0] 2024-09-11 17:20:10.940 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,80e970b953417e1d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=600116, sql=alter system add server ? zone ? 2024-09-11 17:20:10.943 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,80e970b953417e1d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:10.948 ERROR 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,80e970b953417e1d] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Timeout java.sql.SQLException: Timeout at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2347) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2210) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2098) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:247) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:170) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeUpdate(JDBC4PreparedStatement.java:204) at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeUpdate(DruidPooledPreparedStatement.java:255) at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:967) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:650) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:962) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1017) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1027) at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.updateInner(ConnectTemplate.java:293) at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.update(ConnectTemplate.java:282) at com.oceanbase.ocp.obsdk.operator.cluster.MysqlClusterOperator.addServer(MysqlClusterOperator.java:613) at com.oceanbase.ocp.service.task.business.server.ObServerTaskHandler.addObServer(ObServerTaskHandler.java:196) at com.oceanbase.ocp.service.task.business.server.AddObServerTask.run(AddObServerTask.java:53) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:20:10.972 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,80e970b953417e1d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048146, operation:EXECUTE, state: FAILED 2024-09-11 17:20:11.128 INFO 2981023 --- [ocp-async-6,701358813510a8a8,17fb8f8b578b0a1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,8074de9106cc6b54,6ee5b006624de1d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,8074de9106cc6b54,6ee5b006624de1d1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:11.479 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046340, nodeCount=1000, affectRows=1000 2024-09-11 17:20:11.481 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046340, nodeCount=999, validCount=999 2024-09-11 17:20:11.571 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046340, nodeCount=1000, affectRows=1000 2024-09-11 17:20:11.573 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046340, nodeCount=1000, validCount=1000 2024-09-11 17:20:11.584 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046340, nodeCount=49, affectRows=49 2024-09-11 17:20:11.584 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046340, nodeCount=47, validCount=47 2024-09-11 17:20:11.644 INFO 2981023 --- [ocp-schedules-29,6dae3090e12a8104,3d9cf8f06b6beb2c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:20:11.790 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=4048171, name='Add OB zone', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1000006, tenantId=null, hostId=null, creator='admin', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={cluster_name=linkrtd, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, prohibit_rollback=false, service_name=linkrtd:1719994443, ob_log_disk_path=/oceanbase/redo, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_cluster_id=1719994443, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, service_type=OB_CLUSTER, ob_disk_path_style=OBD, ob_svr_port=2882, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, host.2000008.zone=zone2, ob_run_user=admin, ob_data_disk_path=/data/oceanbase, zone2.idc=default_idc, zone2.region=sys_region, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, createTime=2024-09-11T17:07:30+08:00, startTime=2024-09-11T17:07:30.859+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=4048124, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=6, startTime=2024-09-11T17:09:05.240+08:00, endTime=2024-09-11T17:09:05.348+08:00}, SubtaskInstanceEntity{id=4048136, name=Set zone context, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.zone.SetZoneContextTask, seriesId=18, startTime=2024-09-11T17:07:37.520+08:00, endTime=2024-09-11T17:07:37.625+08:00}, SubtaskInstanceEntity{id=4048143, name=Restart observer process, state=PENDING, 2024-09-11 17:20:11.808 WARN 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:146) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:144) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:104) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:77) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:20:11.808 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048171, status:FAILED 2024-09-11 17:20:12.671 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0e4f1da7591659df,ea4e357b33f85c23] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&version=2&ObClusterId=1718689631, client=10.31.35.5, traceId=0e4f1da7591659df, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=rtdob,1718689631,2,{"ObRegion":"rtdob","ObCluster":"rtdob","ObRegionId":1718689631,"ObClusterId":1718689631,"Type":"PRIMARY","timestamp":1726046412658398,"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:20:12.679 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0e4f1da7591659df,ea4e357b33f85c23] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=rtdob, obClusterId=1718689631, rootServerJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, groupName=rtdob, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:20:12.679 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0e4f1da7591659df,ea4e357b33f85c23] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:20:12.679 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0e4f1da7591659df,ea4e357b33f85c23] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=rtdob, clusterId=1718689631, affectRows=1 2024-09-11 17:20:12.680 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0e4f1da7591659df,ea4e357b33f85c23] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.5, traceId=0e4f1da7591659df, duration=10 ms] 2024-09-11 17:20:12.697 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046340, nodeCount=480, affectRows=480 2024-09-11 17:20:13.128 INFO 2981023 --- [ocp-async-10,4ffb44498355f980,dd015e38fcb7ec17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:13.305 INFO 2981023 --- [ocp-schedules-23,24792f6c66d3e306,a762abc14acfe3cb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:20:13.317 INFO 2981023 --- [ocp-schedules-23,24792f6c66d3e306,a762abc14acfe3cb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:20:13.317 INFO 2981023 --- [ocp-schedules-23,24792f6c66d3e306,a762abc14acfe3cb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:20:13.477 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,742e9b21824340e0,a2e63c34eb7aa25a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=742e9b21824340e0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:20:13.521 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,742e9b21824340e0,a2e63c34eb7aa25a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=742e9b21824340e0, duration=53 ms] 2024-09-11 17:20:13.695 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:13.695 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,8469f58df50e5f22,4113f00d3d1e3abf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,25d9e2efe0bf155f,a0a877bc32c23c5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:20:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,8469f58df50e5f22,4113f00d3d1e3abf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:14.663 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,647a7c5eef3622b1,d4d2af29629ffcd3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=647a7c5eef3622b1, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:20:14.670 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,647a7c5eef3622b1,d4d2af29629ffcd3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=647a7c5eef3622b1, duration=9 ms] 2024-09-11 17:20:14.713 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046340, nodeCount=474, validCount=474 2024-09-11 17:20:15.128 INFO 2981023 --- [ocp-async-1,899f7a6a81d5181e,e5ddefe9d8e9439a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:15.579 INFO 2981023 --- [ocp-schedules-28,c94d4ea73adb6a64,dace93168eade98a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:20:15.651 INFO 2981023 --- [alarm-detect-6,c94d4ea73adb6a64,187ac66ce125c48f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643576, data is empty. 2024-09-11 17:20:15.656 INFO 2981023 --- [alarm-detect-6,c94d4ea73adb6a64,187ac66ce125c48f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643576, data is empty. 2024-09-11 17:20:15.661 INFO 2981023 --- [alarm-detect-6,c94d4ea73adb6a64,187ac66ce125c48f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643576, data is empty. 2024-09-11 17:20:15.667 WARN 2981023 --- [scanner-13464233-1,c94d4ea73adb6a64,45b796a081a3f211] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:20:15.675 INFO 2981023 --- [alarm-detect-4,c94d4ea73adb6a64,1a76a63a5e5b55c2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643576, data is empty. 2024-09-11 17:20:15.681 INFO 2981023 --- [alarm-detect-4,c94d4ea73adb6a64,1a76a63a5e5b55c2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643576, data is empty. 2024-09-11 17:20:15.686 INFO 2981023 --- [alarm-detect-2,c94d4ea73adb6a64,358d1f00b82ffb26] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643576, data is empty. 2024-09-11 17:20:15.687 INFO 2981023 --- [alarm-detect-4,c94d4ea73adb6a64,1a76a63a5e5b55c2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643576, data is empty. 2024-09-11 17:20:15.690 INFO 2981023 --- [alarm-detect-2,c94d4ea73adb6a64,358d1f00b82ffb26] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643576, data is empty. 2024-09-11 17:20:15.693 INFO 2981023 --- [alarm-detect-2,c94d4ea73adb6a64,358d1f00b82ffb26] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643576, data is empty. 2024-09-11 17:20:15.795 INFO 2981023 --- [alarm-detect-13,c94d4ea73adb6a64,30cae6e0f14b1b6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643576, data is empty. 2024-09-11 17:20:15.801 INFO 2981023 --- [alarm-detect-13,c94d4ea73adb6a64,30cae6e0f14b1b6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643576, data is empty. 2024-09-11 17:20:15.804 INFO 2981023 --- [alarm-detect-13,c94d4ea73adb6a64,30cae6e0f14b1b6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643576, data is empty. 2024-09-11 17:20:16.021 INFO 2981023 --- [ocp-schedules-18,f952c93cdf72ef4d,ed458120c9afbef8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:20:16.022 INFO 2981023 --- [host-check-clockdiff1,f952c93cdf72ef4d,e95fcc7dbce15ed4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:20:16.022 INFO 2981023 --- [host-check-clockdiff2,f952c93cdf72ef4d,4d169fc8f81c96aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:20:16.022 INFO 2981023 --- [ocp-schedules-3,e7224778640c3642,c0020e310ec0dab3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:20:16.022 INFO 2981023 --- [host-check-clockdiff3,f952c93cdf72ef4d,0f73f72a331ce9a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:20:16.023 INFO 2981023 --- [host-check-clockdiff4,f952c93cdf72ef4d,cbc119ee8f9e6715] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:20:16.024 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:20:16.025 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:20:16.025 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:20:16.025 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:20:16.026 INFO 2981023 --- [ocp-schedules-29,d4a8d153cfa2f506,a2542d42ca79c20b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:20:16.030 INFO 2981023 --- [host-check-agent-status1,d4a8d153cfa2f506,132de68146684297] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:20:16.030 INFO 2981023 --- [host-check-agent-status2,d4a8d153cfa2f506,b85a5c7fc5490f42] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:20:16.031 INFO 2981023 --- [host-check-agent-status3,d4a8d153cfa2f506,912341d9b002f5be] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:20:16.031 INFO 2981023 --- [host-check-agent-status4,d4a8d153cfa2f506,b3d8992ac244fcdd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:20:16.046 INFO 2981023 --- [host-check-clockdiff4,f952c93cdf72ef4d,cbc119ee8f9e6715] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046416 0 0 , error= 2024-09-11 17:20:16.061 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:20:16.066 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:20:16.068 INFO 2981023 --- [host-check-agent-status2,d4a8d153cfa2f506,b85a5c7fc5490f42] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:20:16.069 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:20:16.069 INFO 2981023 --- [host-check-agent-status1,d4a8d153cfa2f506,132de68146684297] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:20:16.071 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:20:16.072 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:20:16.073 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:20:16.077 INFO 2981023 --- [host-check-agent-status1,d4a8d153cfa2f506,132de68146684297] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:20:16.077 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:20:16.078 INFO 2981023 --- [host-check-agent-status2,d4a8d153cfa2f506,b85a5c7fc5490f42] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:20:16.079 INFO 2981023 --- [host-check-agent-status4,d4a8d153cfa2f506,b3d8992ac244fcdd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:20:16.080 INFO 2981023 --- [host-check-agent-status4,d4a8d153cfa2f506,b3d8992ac244fcdd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:20:16.080 INFO 2981023 --- [host-check-agent-status3,d4a8d153cfa2f506,912341d9b002f5be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:20:16.081 INFO 2981023 --- [host-check-agent-status3,d4a8d153cfa2f506,912341d9b002f5be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:20:16.089 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:20:16.189 INFO 2981023 --- [ocp-schedules-21,5b6a8cae8cb8d57e,ae61153ddc5d2a20] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:20:16.191 INFO 2981023 --- [ocp-schedules-15,ec94f8ae96740e3c,0ee0634e3ccaeab7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:20:16.198 INFO 2981023 --- [ocp-schedules-15,ec94f8ae96740e3c,0ee0634e3ccaeab7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:20:16.198 INFO 2981023 --- [ocp-schedules-1,600fb84809ac5996,89ee4a8bdfa1cd55] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:20:16.198 INFO 2981023 --- [ocp-schedules-8,bbaf0f4adf7a826d,244be55ef796744e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:20:16.206 INFO 2981023 --- [ocp-schedules-21,5b6a8cae8cb8d57e,ae61153ddc5d2a20] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:20:16.261 INFO 2981023 --- [ocp-schedules-1,600fb84809ac5996,89ee4a8bdfa1cd55] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:20:16.282 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046402186000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:20:16.283 INFO 2981023 --- [ocp-schedules-15,ec94f8ae96740e3c,0ee0634e3ccaeab7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:20:16.292 INFO 2981023 --- [ocp-schedules-12,5aca9ec9cd81afe0,346fc485f0e8641f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:20:16.297 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:20:16.299 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:20:16.299689978+08:00 2024-09-11 17:20:16.302 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:20:16.300+08:00 2024-09-11 17:20:16.307 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:20:16.300+08:00 and status AVAILABLE 2024-09-11 17:20:16.314 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:20:16.321 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:20:16.315943655+08:00 2024-09-11 17:20:16.323 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:20:16.321+08:00 2024-09-11 17:20:16.326 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:20:16.321+08:00 and status AVAILABLE 2024-09-11 17:20:16.327 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:20:16.329 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:20:16.329465481+08:00 2024-09-11 17:20:16.330 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:20:16.338 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:20:16.329+08:00 2024-09-11 17:20:16.341 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:20:16.342 INFO 2981023 --- [host-check-agent-status1,d4a8d153cfa2f506,132de68146684297] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:20:16.345 INFO 2981023 --- [host-check-agent-status1,d4a8d153cfa2f506,132de68146684297] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:20:16.345 INFO 2981023 --- [host-check-agent-status1,d4a8d153cfa2f506,132de68146684297] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:20:16.345 INFO 2981023 --- [host-check-agent-status1,d4a8d153cfa2f506,132de68146684297] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:20:16.357 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:20:16.329+08:00 and status AVAILABLE 2024-09-11 17:20:16.360 INFO 2981023 --- [host-check-agent-status3,d4a8d153cfa2f506,912341d9b002f5be] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:20:16.361 INFO 2981023 --- [host-check-agent-status2,d4a8d153cfa2f506,b85a5c7fc5490f42] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:20:16.364 INFO 2981023 --- [host-check-agent-status2,d4a8d153cfa2f506,b85a5c7fc5490f42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:20:16.364 INFO 2981023 --- [host-check-agent-status2,d4a8d153cfa2f506,b85a5c7fc5490f42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:20:16.364 INFO 2981023 --- [host-check-agent-status2,d4a8d153cfa2f506,b85a5c7fc5490f42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:20:16.365 INFO 2981023 --- [host-check-agent-status3,d4a8d153cfa2f506,912341d9b002f5be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:20:16.365 INFO 2981023 --- [host-check-agent-status3,d4a8d153cfa2f506,912341d9b002f5be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:20:16.365 INFO 2981023 --- [host-check-agent-status3,d4a8d153cfa2f506,912341d9b002f5be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:20:16.365 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:20:16.366 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:20:16.366619997+08:00 2024-09-11 17:20:16.368 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:20:16.366+08:00 2024-09-11 17:20:16.369 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:20:16.366+08:00 and status AVAILABLE 2024-09-11 17:20:16.373 INFO 2981023 --- [host-check-agent-status4,d4a8d153cfa2f506,b3d8992ac244fcdd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:20:16.375 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.375 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:16.380 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:20:16.380 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:20:16.381 INFO 2981023 --- [host-check-agent-status4,d4a8d153cfa2f506,b3d8992ac244fcdd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:20:16.382 INFO 2981023 --- [host-check-agent-status4,d4a8d153cfa2f506,b3d8992ac244fcdd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:20:16.382 INFO 2981023 --- [host-check-agent-status4,d4a8d153cfa2f506,b3d8992ac244fcdd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:20:16.384 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:20:16.389 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:20:16.389 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:20:16.391 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:20:16.391 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.392 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:20:16.392 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:20:16.403 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:20:16.405 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:20:16.404007+08:00, lastProcessTime=2024-09-11T17:20:06.354494+08:00 2024-09-11 17:20:16.406 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:20:16.408 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.409 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:20:16.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:20:16.410 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:20:16.410 INFO 2981023 --- [ocp-schedules-21,5b6a8cae8cb8d57e,ae61153ddc5d2a20] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:20:16.411 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:20:16.413 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.413 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:16.419 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:20:16.422 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.423 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:16.425 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:20:16.443 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:20:16.444 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:20:16.469 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:20:16.473 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:20:16.475 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:20:16.478 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:20:16.489 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:16.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:20:16.497 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:20:16.503 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:20:16.507 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:20:16.509 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:20:16.517 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:20:16.521 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:20:16.528 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:20:16.528 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046402186000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:20:16.531 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:20:16.536 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:16.537 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.537 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:20:16.540 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:20:16.555 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:20:16.558 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:16.559 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.560 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:16.560 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:20:16.561 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:20:16.561 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.563 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:20:16.563 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:20:16.564 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:20:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:20:16.573 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:16.574 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.574 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:16.575 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:20:16.575 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:20:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:20:16.577 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:20:16.578 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:20:16.583 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:20:16.583 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:20:16.585 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:20:16.586 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:20:16.587 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:20:16.589 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:20:16.598 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:20:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:20:16.600 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:20:16.602 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:20:16.609 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:20:16.610 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:16.610 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.611 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:20:16.612 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:20:16.613 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:16.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.614 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:20:16.614 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:20:16.614 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:20:16.617 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:20:16.619 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:20:16.619 INFO 2981023 --- [host-check-host-status1,e7224778640c3642,039512f7aa6f3704] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:20:16.624 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:20:16.624 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:20:16.624 INFO 2981023 --- [host-check-host-status4,e7224778640c3642,1dbbf99ffbea13f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:20:16.625 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:20:16.625 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:20:16.626 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:20:16.638 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:20:16.645 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:20:16.646 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:20:16.655 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:20:16.658 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:20:16.662 INFO 2981023 --- [host-check-host-status2,e7224778640c3642,168d81cfcf4145ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:20:16.666 INFO 2981023 --- [host-check-host-status3,e7224778640c3642,e74051e225791a8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:20:16.675 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:20:16.681 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:20:16.689 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:20:16.689 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:20:16.696 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:16.696 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:16.711 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:20:16.714 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:20:16.404007+08:00 2024-09-11 17:20:16.716 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:16.717 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:16.717 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:20:16.722 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:20:16.722 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:20:16.722 INFO 2981023 --- [ocp-schedules-1,600fb84809ac5996,89ee4a8bdfa1cd55] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:20:16.968 INFO 2981023 --- [host-check-clockdiff2,f952c93cdf72ef4d,4d169fc8f81c96aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046416 0 0 , error= 2024-09-11 17:20:17.128 INFO 2981023 --- [ocp-async-5,6d6f3290ea25cebc,74231bc7f5728222] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,4aabb7f1b9af516f,4cdc2a5096da22bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,4aabb7f1b9af516f,4cdc2a5096da22bc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:17.734 INFO 2981023 --- [ocp-async-9,a936cb83d694611f,da46c38e80abb9b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:20:18.559 INFO 2981023 --- [host-check-clockdiff1,f952c93cdf72ef4d,e95fcc7dbce15ed4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046418 0 0 , error= 2024-09-11 17:20:19.128 INFO 2981023 --- [ocp-async-10,3f49629c0b027441,cb076a3bb49fb0b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:19.696 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:19.698 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,286283bbf4a59986,1c8a6ebf57936fe7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,286283bbf4a59986,1c8a6ebf57936fe7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:21.128 INFO 2981023 --- [ocp-async-2,5b7116b99267d079,44c73f4c8441cb9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:21.647 INFO 2981023 --- [ocp-schedules-23,a250c0c4a267ed79,899a9fa5a4a702bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:20:21.997 INFO 2981023 --- [host-check-clockdiff3,f952c93cdf72ef4d,0f73f72a331ce9a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046421 1 1 , error= 2024-09-11 17:20:22.112 INFO 2981023 --- [ocp-schedules-16,287808061e771dfe,3e0d439522089487] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:20:22.112 INFO 2981023 --- [ocp-schedules-16,287808061e771dfe,3e0d439522089487] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:20:22.112 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:20:22.112 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:20:22.112 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:20:22.117 INFO 2981023 --- [ocp-schedules-16,287808061e771dfe,3e0d439522089487] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:20:22.117 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:20:22.117 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:20:22.117 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:20:22.121 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:20:22.121 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:20:22.121 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:20:22.121 INFO 2981023 --- [ocp-schedules-16,287808061e771dfe,3e0d439522089487] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:20:22.147 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:20:22.147 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:20:22.148 INFO 2981023 --- [ocp-schedules-16,287808061e771dfe,3e0d439522089487] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:20:22.148 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:20:22.150 INFO 2981023 --- [ocp-schedules-16,287808061e771dfe,3e0d439522089487] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:20:22.698 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:22.700 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:23.128 INFO 2981023 --- [ocp-async-5,833f8fc7fa759182,981c4a2a4e8b648a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,2675c4516e60256e,d473351b3f9f5283] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,2675c4516e60256e,d473351b3f9f5283] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:24.135 INFO 2981023 --- [ocp-schedules-21,7b3fb56d37252908,05985b7dd1967262] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:20:25.128 INFO 2981023 --- [ocp-async-4,411c189753477521,2779f7b1b7bf1f4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:25.700 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:25.701 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:25.852 INFO 2981023 --- [ocp-schedules-22,f4461cbae9eb6278,9568e98ed78bf386] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:20:25.900 WARN 2981023 --- [scanner-13464538-1,f4461cbae9eb6278,44e0e8fda8ca9a5b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:20:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,33e4f5520c8d8f42,1ea8c71667652ce4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,33e4f5520c8d8f42,1ea8c71667652ce4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:20:26.351106+08:00, lastProcessTime=2024-09-11T17:20:16.404007+08:00 2024-09-11 17:20:26.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:20:26.351106+08:00 2024-09-11 17:20:27.129 INFO 2981023 --- [ocp-async-3,198509ff62c4e078,69d481e9a1b5ba71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:27.735 INFO 2981023 --- [ocp-async-10,fa66a0422fba51d5,1d84b3ab8cad55ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:20:28.701 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:28.701 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:29.129 INFO 2981023 --- [ocp-async-7,7752fbbb37c31ca7,d1a19b14ff2fbda6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,2e6ade556527ba25,4e5f95ffd8fa028d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,2e6ade556527ba25,4e5f95ffd8fa028d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:31.129 INFO 2981023 --- [ocp-async-8,9a12df9ad3f4fa01,5b4ee49beaec31a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:31.650 INFO 2981023 --- [ocp-schedules-9,6a3a5b6bc631afe1,c75bb6a08e8a1bcc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:20:31.702 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:31.702 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,0a51b23a7d787bfc,b7a00af3353bd2f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,0a51b23a7d787bfc,b7a00af3353bd2f8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:33.131 INFO 2981023 --- [ocp-async-3,c71ddb09c42eb76f,bb65472b608545db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:33.328 INFO 2981023 --- [ocp-schedules-19,02b12c3ed5c876fd,e751d20dc54524fe] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:20:33.348 INFO 2981023 --- [ocp-schedules-19,02b12c3ed5c876fd,e751d20dc54524fe] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:20:33.348 INFO 2981023 --- [ocp-schedules-19,02b12c3ed5c876fd,e751d20dc54524fe] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:20:34.702 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:34.703 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:35.129 INFO 2981023 --- [ocp-async-5,635617e70a3ca18f,bee4b571ee93c4f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,c7c71b8ece8802b0,7a344843341fc11d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,c7c71b8ece8802b0,7a344843341fc11d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:36.049 INFO 2981023 --- [ocp-schedules-3,2ef6dcc0276adad6,c474192b0448f9b2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:20:36.062 WARN 2981023 --- [scanner-13464736-1,2ef6dcc0276adad6,29abc40c7a894c1f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:20:36.113 INFO 2981023 --- [alarm-detect-8,2ef6dcc0276adad6,8efc280762c2cc80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643576, data is empty. 2024-09-11 17:20:36.120 INFO 2981023 --- [alarm-detect-8,2ef6dcc0276adad6,8efc280762c2cc80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643576, data is empty. 2024-09-11 17:20:36.124 INFO 2981023 --- [alarm-detect-8,2ef6dcc0276adad6,8efc280762c2cc80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643576, data is empty. 2024-09-11 17:20:36.142 INFO 2981023 --- [alarm-detect-6,2ef6dcc0276adad6,51137079b4dfdefd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643576, data is empty. 2024-09-11 17:20:36.144 INFO 2981023 --- [alarm-detect-2,2ef6dcc0276adad6,f5aa1992d071bb5d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643576, data is empty. 2024-09-11 17:20:36.146 INFO 2981023 --- [alarm-detect-6,2ef6dcc0276adad6,51137079b4dfdefd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643576, data is empty. 2024-09-11 17:20:36.149 INFO 2981023 --- [alarm-detect-2,2ef6dcc0276adad6,f5aa1992d071bb5d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643576, data is empty. 2024-09-11 17:20:36.153 INFO 2981023 --- [alarm-detect-6,2ef6dcc0276adad6,51137079b4dfdefd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643576, data is empty. 2024-09-11 17:20:36.155 INFO 2981023 --- [alarm-detect-2,2ef6dcc0276adad6,f5aa1992d071bb5d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643576, data is empty. 2024-09-11 17:20:36.296 INFO 2981023 --- [alarm-detect-17,2ef6dcc0276adad6,cc2891b8d6314253] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643576, data is empty. 2024-09-11 17:20:36.298 INFO 2981023 --- [alarm-detect-17,2ef6dcc0276adad6,cc2891b8d6314253] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643576, data is empty. 2024-09-11 17:20:36.300 INFO 2981023 --- [alarm-detect-17,2ef6dcc0276adad6,cc2891b8d6314253] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643576, data is empty. 2024-09-11 17:20:36.348 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:20:36.347615+08:00, lastProcessTime=2024-09-11T17:20:26.351106+08:00 2024-09-11 17:20:36.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:20:36.347615+08:00 2024-09-11 17:20:37.034 INFO 2981023 --- [ocp-schedules-6,312eda85813bee41,3ac182183d87fafe] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:20:37.054 WARN 2981023 --- [ocp-schedules-6,312eda85813bee41,3ac182183d87fafe] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:20:37.129 INFO 2981023 --- [ocp-async-7,cb91152bffcfc31d,13dec7d78d06b16b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:37.314 INFO 2981023 --- [prometheus-cache-monitor1,6e7019a5beca9752,5512ae22a2073061] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:20:37.315 INFO 2981023 --- [prometheus-cache-monitor1,6e7019a5beca9752,5512ae22a2073061] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:20:37.703 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:37.703 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:37.735 INFO 2981023 --- [ocp-async-5,f70b67120e551d12,fadb0ae20834776f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:20:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,7368d21a02eac170,eb1413ab578832c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,7368d21a02eac170,eb1413ab578832c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:39.129 INFO 2981023 --- [ocp-async-10,323e1e914b2b8779,8b4de3d6ecce5fc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:39.243 INFO 2981023 --- [ocp-schedules-11,ce946361ad2a439f,cebee381b2c7af25] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:20:40.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:20:40.003 INFO 2981023 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:20:40.703 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:40.704 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:41.129 INFO 2981023 --- [ocp-async-2,8399413f1e1f3564,49bb33a1bfcdfcb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,3875111b60e6bde0,e0b3f514d458b475] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,3875111b60e6bde0,e0b3f514d458b475] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:41.652 INFO 2981023 --- [ocp-schedules-10,b1aed849af9bfa60,90c20bc75afccc34] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:20:43.129 INFO 2981023 --- [ocp-async-6,47b5210d76f2194b,4d9b56fbc4979acb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:43.704 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:43.705 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,273dc93cef6761e9,5d94834d2669e815] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,273dc93cef6761e9,5d94834d2669e815] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:45.101 INFO 2981023 --- [prometheus-second-schedule-1,c86aa175dd2047a8,88135d6ac061416f] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:20:45.130 INFO 2981023 --- [ocp-async-8,79b1a324c9311b87,cf339eb34fa06e95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:46.016 INFO 2981023 --- [ocp-schedules-23,ac7caf890164ddae,63579c7a5817e717] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:20:46.017 INFO 2981023 --- [host-check-agent-status1,ac7caf890164ddae,59fdc6f9c0aaec58] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:20:46.017 INFO 2981023 --- [host-check-agent-status3,ac7caf890164ddae,9dc3a91aa8742ce6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:20:46.018 INFO 2981023 --- [host-check-agent-status2,ac7caf890164ddae,c74d9820ea6b5a44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:20:46.018 INFO 2981023 --- [host-check-agent-status4,ac7caf890164ddae,617b30b8c821537c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:20:46.019 INFO 2981023 --- [ocp-schedules-27,e598c3b72afb97a1,e6a70ebe488d24f8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:20:46.019 INFO 2981023 --- [host-check-clockdiff1,e598c3b72afb97a1,a477bb693d5339a3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:20:46.019 INFO 2981023 --- [host-check-clockdiff2,e598c3b72afb97a1,cdc9a32b728755b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:20:46.019 INFO 2981023 --- [host-check-clockdiff3,e598c3b72afb97a1,80540924cc2fcd42] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:20:46.020 INFO 2981023 --- [host-check-clockdiff4,e598c3b72afb97a1,5f6e77d503dfcb5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:20:46.027 INFO 2981023 --- [ocp-schedules-12,9436fa12c1d2b6a8,6730f7309480adc9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:20:46.045 INFO 2981023 --- [host-check-clockdiff4,e598c3b72afb97a1,5f6e77d503dfcb5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046446 0 0 , error= 2024-09-11 17:20:46.047 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:20:46.047 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:20:46.047 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:20:46.047 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:20:46.063 INFO 2981023 --- [host-check-agent-status1,ac7caf890164ddae,59fdc6f9c0aaec58] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:20:46.063 INFO 2981023 --- [host-check-agent-status2,ac7caf890164ddae,c74d9820ea6b5a44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:20:46.065 INFO 2981023 --- [host-check-agent-status3,ac7caf890164ddae,9dc3a91aa8742ce6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:20:46.067 INFO 2981023 --- [host-check-agent-status4,ac7caf890164ddae,617b30b8c821537c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:20:46.068 INFO 2981023 --- [host-check-agent-status3,ac7caf890164ddae,9dc3a91aa8742ce6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:20:46.068 INFO 2981023 --- [host-check-agent-status1,ac7caf890164ddae,59fdc6f9c0aaec58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:20:46.070 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:20:46.070 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:20:46.071 INFO 2981023 --- [host-check-agent-status2,ac7caf890164ddae,c74d9820ea6b5a44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:20:46.073 INFO 2981023 --- [host-check-agent-status4,ac7caf890164ddae,617b30b8c821537c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:20:46.077 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:20:46.088 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:20:46.177 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:20:46.192 INFO 2981023 --- [ocp-schedules-13,459c3dd6063f6488,af2c5e37e0a6050b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:20:46.192 INFO 2981023 --- [ocp-schedules-1,2009a9406b039163,04e40977e85cfdeb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:20:46.194 INFO 2981023 --- [ocp-schedules-7,ca495c24d0a10d64,411c888a51ed5fca] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:20:46.194 INFO 2981023 --- [ocp-schedules-9,c94835d9734ff2cf,977a2c221b971ca9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:20:46.198 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:20:46.200 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:20:46.258 INFO 2981023 --- [ocp-schedules-19,59a4129a6ce13828,30320ed95dda8f82] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:20:46.258 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:20:46.259 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:20:46.266 INFO 2981023 --- [ocp-schedules-13,459c3dd6063f6488,af2c5e37e0a6050b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:20:46.271 INFO 2981023 --- [ocp-schedules-7,ca495c24d0a10d64,411c888a51ed5fca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:20:46.276 INFO 2981023 --- [ocp-schedules-1,2009a9406b039163,04e40977e85cfdeb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:20:46.281 INFO 2981023 --- [ocp-schedules-19,59a4129a6ce13828,30320ed95dda8f82] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:20:46.297 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046402186000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:20:46.302 INFO 2981023 --- [ocp-schedules-1,2009a9406b039163,04e40977e85cfdeb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:20:46.362 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:20:46.364 INFO 2981023 --- [ocp-schedules-29,b80acefac3161374,bb1df3b0076e718e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:20:46.366 INFO 2981023 --- [ocp-schedules-8,b2715bb6ae2a0cc8,0fe5732c8b7de9cd] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:20:46.377 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:20:46.376987+08:00, lastProcessTime=2024-09-11T17:20:36.347615+08:00 2024-09-11 17:20:46.388 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:20:46.390 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:20:46.390071721+08:00 2024-09-11 17:20:46.482 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:20:46.390+08:00 2024-09-11 17:20:46.489 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:20:46.390+08:00 and status AVAILABLE 2024-09-11 17:20:46.489 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:20:46.504 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:20:46.504681006+08:00 2024-09-11 17:20:46.506 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:20:46.507 WARN 2981023 --- [scanner-13465079-1,b80acefac3161374,763ec76db91bab87] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:20:46.555 INFO 2981023 --- [host-check-agent-status2,ac7caf890164ddae,c74d9820ea6b5a44] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:20:46.555 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:20:46.557 INFO 2981023 --- [host-check-agent-status4,ac7caf890164ddae,617b30b8c821537c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:20:46.559 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:20:46.558970056+08:00 2024-09-11 17:20:46.560 INFO 2981023 --- [host-check-agent-status1,ac7caf890164ddae,59fdc6f9c0aaec58] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:20:46.564 INFO 2981023 --- [host-check-agent-status2,ac7caf890164ddae,c74d9820ea6b5a44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:20:46.564 INFO 2981023 --- [host-check-agent-status2,ac7caf890164ddae,c74d9820ea6b5a44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:20:46.564 INFO 2981023 --- [host-check-agent-status2,ac7caf890164ddae,c74d9820ea6b5a44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:20:46.564 INFO 2981023 --- [host-check-agent-status4,ac7caf890164ddae,617b30b8c821537c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:20:46.564 INFO 2981023 --- [host-check-agent-status4,ac7caf890164ddae,617b30b8c821537c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:20:46.564 INFO 2981023 --- [host-check-agent-status4,ac7caf890164ddae,617b30b8c821537c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:20:46.570 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:20:46.559+08:00 2024-09-11 17:20:46.573 INFO 2981023 --- [host-check-agent-status1,ac7caf890164ddae,59fdc6f9c0aaec58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:20:46.573 INFO 2981023 --- [host-check-agent-status1,ac7caf890164ddae,59fdc6f9c0aaec58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:20:46.573 INFO 2981023 --- [host-check-agent-status1,ac7caf890164ddae,59fdc6f9c0aaec58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:20:46.582 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:46.582 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:20:46.559+08:00 and status AVAILABLE 2024-09-11 17:20:46.584 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:46.586 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:20:46.505+08:00 2024-09-11 17:20:46.589 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:20:46.505+08:00 and status AVAILABLE 2024-09-11 17:20:46.592 INFO 2981023 --- [host-check-agent-status3,ac7caf890164ddae,9dc3a91aa8742ce6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:20:46.596 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:20:46.597 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:20:46.597109557+08:00 2024-09-11 17:20:46.599 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:20:46.597+08:00 2024-09-11 17:20:46.599 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:46.599 INFO 2981023 --- [host-check-agent-status3,ac7caf890164ddae,9dc3a91aa8742ce6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:20:46.599 INFO 2981023 --- [host-check-agent-status3,ac7caf890164ddae,9dc3a91aa8742ce6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:20:46.599 INFO 2981023 --- [host-check-agent-status3,ac7caf890164ddae,9dc3a91aa8742ce6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:20:46.601 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:20:46.601 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:20:46.601 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:20:46.604 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:20:46.597+08:00 and status AVAILABLE 2024-09-11 17:20:46.636 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:20:46.643 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:46.644 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:20:46.644 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:20:46.705 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:46.871 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:47.130 INFO 2981023 --- [ocp-async-9,9180cac8645a87bf,8c49e374b3e0c0a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:47.182 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.182 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:20:47.183 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:20:47.183 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:20:47.183 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.186 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:20:47.189 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:20:47.194 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:20:47.194 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.196 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:47.197 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:20:47.199 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:20:47.215 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:20:47.218 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:20:47.228 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.229 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:47.244 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:20:47.251 INFO 2981023 --- [host-check-clockdiff1,e598c3b72afb97a1,a477bb693d5339a3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046446 0 0 , error= 2024-09-11 17:20:47.262 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:20:47.263 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:20:47.270 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:20:47.275 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:20:47.278 INFO 2981023 --- [ocp-schedules-13,459c3dd6063f6488,af2c5e37e0a6050b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:20:47.292 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:20:47.296 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:20:47.296 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.297 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.298 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:20:47.298 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:20:47.298 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.305 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:20:47.314 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:20:47.324 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:20:47.347 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.348 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.349 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:20:47.351 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:20:47.352 INFO 2981023 --- [prometheus-minute-schedule-1,45754f0efc513d4c,2b97646fe577534b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:47.353 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:20:47.360 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:20:47.362 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:20:47.367 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:20:47.367 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046402186000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:20:47.376 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:20:47.390 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:20:47.392 INFO 2981023 --- [prometheus-minute-schedule-1,45754f0efc513d4c,2b97646fe577534b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:47.408 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.413 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:20:47.416 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:20:47.420 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.421 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.421 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:20:47.423 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.423 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.426 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.426 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:20:47.426 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.427 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:20:47.427 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:47.427 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.427 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:20:47.428 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:20:47.428 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:20:47.428 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:20:46.376987+08:00 2024-09-11 17:20:47.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.431 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:20:47.431 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.433 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:20:47.438 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:20:47.445 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:20:47.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:20:47.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:20:47.461 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:20:47.476 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:20:47.476 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:20:47.480 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:20:47.481 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.483 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.483 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:20:47.484 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:20:47.486 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.489 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:20:47.492 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:20:47.499 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:20:47.499 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:20:47.501 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:20:47.504 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:20:47.504 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:20:47.507 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:20:47.507 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:20:47.510 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.515 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.516 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:20:47.518 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:20:47.526 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:20:47.527 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:20:47.556 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:20:47.558 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:20:47.559 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:20:47.562 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:20:47.568 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:20:47.568 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:20:47.574 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.575 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.575 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:20:47.576 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:20:47.576 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.579 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.579 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.580 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:20:47.580 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:20:47.590 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:20:47.590 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:20:47.591 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:20:47.593 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:20:47.595 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:20:47.595 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:20:47.597 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:20:47.599 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:20:47.602 INFO 2981023 --- [host-check-host-status3,9436fa12c1d2b6a8,c56c00f1ccecf690] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:20:47.616 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:20:47.623 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:20:47.624 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:20:47.627 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:20:47.636 INFO 2981023 --- [host-check-host-status1,9436fa12c1d2b6a8,f2b88acaa868f1f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:20:47.637 INFO 2981023 --- [host-check-host-status2,9436fa12c1d2b6a8,489ed00352c4ac17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:20:47.642 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:20:47.650 INFO 2981023 --- [host-check-host-status4,9436fa12c1d2b6a8,0240edd2ca23d7c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:20:47.658 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:20:47.659 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.660 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.660 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:20:47.661 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:20:47.662 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.670 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.670 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.671 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:20:47.679 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:20:47.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:20:47.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:20:47.685 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:47.685 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:47.686 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:20:47.692 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:20:47.692 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:20:47.693 INFO 2981023 --- [ocp-schedules-7,ca495c24d0a10d64,411c888a51ed5fca] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:20:47.711 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:20:47.727 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:20:47.732 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:20:47.735 INFO 2981023 --- [ocp-async-5,ece65e3379260710,d5c9b4059c3beb1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:20:47.736 INFO 2981023 --- [ocp-schedules-11,014b58624268f878,47dfd9428824e865] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:20:47.810 INFO 2981023 --- [host-check-clockdiff2,e598c3b72afb97a1,cdc9a32b728755b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046447 0 0 , error= 2024-09-11 17:20:49.130 INFO 2981023 --- [ocp-async-1,c6dd28c396e601af,0b81ec052b67625a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:49.871 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:49.873 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:20:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:20:50.178 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.183 INFO 2981023 --- [ocp-schedules-3,3f71a0784e64df3c,302c422f13631ee2] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:20:50.188 INFO 2981023 --- [ocp-schedules-5,631bdf66d6d4a69a,160adbcd8ed2e377] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:20:50.246 INFO 2981023 --- [ocp-schedules-21,0ed81ce8bdc95e3b,bfcdd1c77a211533] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:20:50.246+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:20:50.250 INFO 2981023 --- [ocp-schedules-21,0ed81ce8bdc95e3b,bfcdd1c77a211533] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:20:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,d639796320da08a6,ad21f5f88a61987e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,d639796320da08a6,ad21f5f88a61987e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:50.400 INFO 2981023 --- [ocp-schedules-21,0ed81ce8bdc95e3b,bfcdd1c77a211533] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 46 to SqlAggregateData 2024-09-11 17:20:50.403 INFO 2981023 --- [ocp-schedules-21,0ed81ce8bdc95e3b,bfcdd1c77a211533] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster linkrtd, exclude result is empty. 2024-09-11 17:20:50.716 INFO 2981023 --- [ocp-schedules-2,d6509996a33c7e8a,c0081b98f8428ce7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:20:50.717 INFO 2981023 --- [ocp-schedules-2,d6509996a33c7e8a,c0081b98f8428ce7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:20:50.742 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:20:50.743 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.746 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:20:50.746 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:20:50.746 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.748 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:20:50.749 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.749 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:20:50.750 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.751 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.752 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:20:50.754 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.755 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.756 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.756 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.756 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.757 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.763 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.763 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:20:50.764 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.764 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.770 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.770 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.771 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:20:50.771 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.775 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.776 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.776 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:20:50.776 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE18D1D6DC-0-0] 2024-09-11 17:20:50.777 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:20:50.777 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.777 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.777 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:20:50.777 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:20:50.782 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.782 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.783 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.783 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:20:50.788 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.788 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:20:50.794 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.794 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.794 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:20:50.795 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.795 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:20:50.795 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.795 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:20:50.796 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.796 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:20:50.796 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:20:50.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.797 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:20:50.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:20:50.798 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.798 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:20:50.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:20:50.800 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.800 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:20:50.801 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.801 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:20:50.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.801 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:20:50.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:20:50.802 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.802 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:20:50.802 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE18D1D6E6-0-0] 2024-09-11 17:20:50.802 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:20:50.803 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:20:50.803 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:20:50.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:20:50.806 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:20:50.808 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.808 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:20:50.809 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.809 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:20:50.810 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.811 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:20:50.811 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.812 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:20:50.812 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.813 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:20:50.814 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.814 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:20:50.814 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:20:50.816 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:20:50.817 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:50.818 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:50.818 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.819 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.820 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.820 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:20:50.826 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:20:50.827 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:20:50.828 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.829 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:20:50.829 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:20:50.829 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.830 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:20:50.831 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:20:50.833 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:20:50.833 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:20:50.833 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:20:50.835 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:20:50.836 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:20:50.836 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.836 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.838 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:20:50.840 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:20:50.840 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:50.840 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:20:50.841 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:20:50.843 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:20:50.843 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:20:50.843 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:20:50.849 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:20:50.850 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:20:50.850 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:20:50.850 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:20:50.850 INFO 2981023 --- [ocp-schedules-2,d6509996a33c7e8a,c0081b98f8428ce7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:20:51.130 INFO 2981023 --- [ocp-async-2,b183a3a6ebaa97b7,a1a108d8873ac5e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:51.491 INFO 2981023 --- [ocp-schedules-10,1ad242b1ff29e9dc,2986e7ebf553d76b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:20:51.492 INFO 2981023 --- [ocp-schedules-10,1ad242b1ff29e9dc,2986e7ebf553d76b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:20:51.492 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:20:51.492 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:20:51.492 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:20:51.493 INFO 2981023 --- [ocp-schedules-10,1ad242b1ff29e9dc,2986e7ebf553d76b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:20:51.655 INFO 2981023 --- [ocp-schedules-3,72736c82e576cf1b,dc718356dbef91ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:20:51.954 INFO 2981023 --- [ocp-schedules-4,2df056da8915f2cc,80ea56416d78e07e] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:20:51.971 WARN 2981023 --- [ocp-schedules-4,2df056da8915f2cc,80ea56416d78e07e] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:20:52.045 INFO 2981023 --- [host-check-clockdiff3,e598c3b72afb97a1,80540924cc2fcd42] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046452 1 1 , error= 2024-09-11 17:20:52.873 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:52.874 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:52.991 INFO 2981023 --- [ocp-schedules-2,6a08db1b74f178ed,8217521a47aa1248] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:20:52.991 INFO 2981023 --- [ocp-schedules-2,6a08db1b74f178ed,8217521a47aa1248] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:20:53.130 INFO 2981023 --- [ocp-async-7,fc9a8797ca4cb85c,b9293a2f8e9d7af4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,0e049da167ccb689,8848d0d381b3af2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,0e049da167ccb689,8848d0d381b3af2f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:53.351 INFO 2981023 --- [ocp-schedules-20,49a19ecf7761db5e,b7b8f73ea9ef40ec] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:20:53.360 INFO 2981023 --- [ocp-schedules-20,49a19ecf7761db5e,b7b8f73ea9ef40ec] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:20:53.360 INFO 2981023 --- [ocp-schedules-20,49a19ecf7761db5e,b7b8f73ea9ef40ec] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:20:54.138 INFO 2981023 --- [ocp-schedules-22,1c043cd6e6bb1e0c,89b20d19dc6cfdac] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:20:55.130 INFO 2981023 --- [ocp-async-4,0cde667e1112ee33,7aaa13fee89b1f9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:55.478 INFO 2981023 --- [ocp-schedules-12,a0585c03601df0e4,4bf20377a61d4338] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:20:55.478 INFO 2981023 --- [ocp-schedules-12,a0585c03601df0e4,4bf20377a61d4338] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:20:55.502 INFO 2981023 --- [Collect all tenants major compactions1,a0585c03601df0e4,aa06582d06955bda] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:20:55.503 INFO 2981023 --- [Collect all tenants major compactions5,a0585c03601df0e4,e7711b96e548afe7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:20:55.503 INFO 2981023 --- [Collect all tenants major compactions6,a0585c03601df0e4,afa783e2c3d49165] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:20:55.504 INFO 2981023 --- [Collect all tenants major compactions2,a0585c03601df0e4,c0e5d036fd05eacb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:20:55.504 INFO 2981023 --- [Collect all tenants major compactions3,a0585c03601df0e4,9068898102448bfe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:20:55.505 INFO 2981023 --- [Collect all tenants major compactions4,a0585c03601df0e4,f149969f35a16ce5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:20:55.520 INFO 2981023 --- [Collect all tenants major compactions4,a0585c03601df0e4,f149969f35a16ce5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:55.521 INFO 2981023 --- [Collect all tenants major compactions5,a0585c03601df0e4,e7711b96e548afe7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:55.521 INFO 2981023 --- [Collect all tenants major compactions3,a0585c03601df0e4,9068898102448bfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:55.521 INFO 2981023 --- [Collect all tenants major compactions1,a0585c03601df0e4,aa06582d06955bda] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:55.521 INFO 2981023 --- [Collect all tenants major compactions4,a0585c03601df0e4,f149969f35a16ce5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:20:55.522 INFO 2981023 --- [Collect all tenants major compactions5,a0585c03601df0e4,e7711b96e548afe7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:20:55.522 INFO 2981023 --- [Collect all tenants major compactions3,a0585c03601df0e4,9068898102448bfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:20:55.522 INFO 2981023 --- [Collect all tenants major compactions1,a0585c03601df0e4,aa06582d06955bda] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:20:55.523 INFO 2981023 --- [Collect all tenants major compactions2,a0585c03601df0e4,c0e5d036fd05eacb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:55.523 INFO 2981023 --- [Collect all tenants major compactions6,a0585c03601df0e4,afa783e2c3d49165] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:20:55.524 INFO 2981023 --- [Collect all tenants major compactions6,a0585c03601df0e4,afa783e2c3d49165] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:20:55.525 INFO 2981023 --- [Collect all tenants major compactions3,a0585c03601df0e4,9068898102448bfe] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:20:55.525 INFO 2981023 --- [Collect all tenants major compactions3,a0585c03601df0e4,9068898102448bfe] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:20:55.525 INFO 2981023 --- [Collect all tenants major compactions2,a0585c03601df0e4,c0e5d036fd05eacb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:20:55.525 INFO 2981023 --- [Collect all tenants major compactions4,a0585c03601df0e4,f149969f35a16ce5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:20:55.525 INFO 2981023 --- [Collect all tenants major compactions4,a0585c03601df0e4,f149969f35a16ce5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:20:55.526 INFO 2981023 --- [Collect all tenants major compactions1,a0585c03601df0e4,aa06582d06955bda] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:20:55.526 INFO 2981023 --- [Collect all tenants major compactions1,a0585c03601df0e4,aa06582d06955bda] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:20:55.526 INFO 2981023 --- [Collect all tenants major compactions5,a0585c03601df0e4,e7711b96e548afe7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:20:55.526 INFO 2981023 --- [Collect all tenants major compactions5,a0585c03601df0e4,e7711b96e548afe7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:20:55.526 INFO 2981023 --- [Collect all tenants major compactions6,a0585c03601df0e4,afa783e2c3d49165] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:20:55.526 INFO 2981023 --- [Collect all tenants major compactions6,a0585c03601df0e4,afa783e2c3d49165] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:20:55.527 INFO 2981023 --- [Collect all tenants major compactions2,a0585c03601df0e4,c0e5d036fd05eacb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:20:55.527 INFO 2981023 --- [Collect all tenants major compactions2,a0585c03601df0e4,c0e5d036fd05eacb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:20:55.528 INFO 2981023 --- [ocp-schedules-12,a0585c03601df0e4,4bf20377a61d4338] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:20:55.608 INFO 2981023 --- [ocp-schedules-23,b4cff8014702e267,a5161a438bb4f1f8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:20:55.608 INFO 2981023 --- [ocp-schedules-23,b4cff8014702e267,a5161a438bb4f1f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:20:55.609 INFO 2981023 --- [ocp-schedules-23,b4cff8014702e267,a5161a438bb4f1f8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:20:55.609 INFO 2981023 --- [ocp-schedules-23,b4cff8014702e267,a5161a438bb4f1f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:20:55.609 INFO 2981023 --- [ocp-schedules-23,b4cff8014702e267,a5161a438bb4f1f8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:20:55.609 INFO 2981023 --- [ocp-schedules-23,b4cff8014702e267,a5161a438bb4f1f8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:20:55.649 INFO 2981023 --- [ocp-schedules-11,c273c9296e9d4424,0480e091e65f6eb3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:20:55.655 INFO 2981023 --- [ocp-schedules-11,c273c9296e9d4424,0480e091e65f6eb3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:20:55.655 INFO 2981023 --- [ocp-schedules-11,c273c9296e9d4424,0480e091e65f6eb3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:20:55.874 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:55.875 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,ca8aaf4457cf80cb,dae2197aeb6d23b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,ca8aaf4457cf80cb,dae2197aeb6d23b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:20:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:20:56.352472+08:00, lastProcessTime=2024-09-11T17:20:46.376987+08:00 2024-09-11 17:20:56.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:20:56.352472+08:00 2024-09-11 17:20:57.130 INFO 2981023 --- [ocp-async-2,94797235c7dbd97b,4bf4a619f115b629] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:57.564 INFO 2981023 --- [ocp-schedules-20,5797304c19496417,ece8a4cdeba599f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:20:57.609 WARN 2981023 --- [scanner-13465303-1,5797304c19496417,6407cb3f2e818a09] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:20:57.637 INFO 2981023 --- [alarm-detect-5,5797304c19496417,06de37f79fadbfd2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643576, data is empty. 2024-09-11 17:20:57.643 INFO 2981023 --- [alarm-detect-5,5797304c19496417,06de37f79fadbfd2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643622, data is empty. 2024-09-11 17:20:57.648 INFO 2981023 --- [alarm-detect-5,5797304c19496417,06de37f79fadbfd2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643622, data is empty. 2024-09-11 17:20:57.664 INFO 2981023 --- [alarm-detect-5,5797304c19496417,4fac7e7bb7f98617] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643576, data is empty. 2024-09-11 17:20:57.667 INFO 2981023 --- [alarm-detect-5,5797304c19496417,4fac7e7bb7f98617] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643622, data is empty. 2024-09-11 17:20:57.672 INFO 2981023 --- [alarm-detect-5,5797304c19496417,4fac7e7bb7f98617] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643622, data is empty. 2024-09-11 17:20:57.674 INFO 2981023 --- [alarm-detect-10,5797304c19496417,47da667a4e0f7217] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643576, data is empty. 2024-09-11 17:20:57.679 INFO 2981023 --- [alarm-detect-10,5797304c19496417,47da667a4e0f7217] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643622, data is empty. 2024-09-11 17:20:57.684 INFO 2981023 --- [alarm-detect-10,5797304c19496417,47da667a4e0f7217] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643622, data is empty. 2024-09-11 17:20:57.735 INFO 2981023 --- [ocp-async-8,ac6e15a863e0c544,485f60d3a97b450f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:20:57.799 INFO 2981023 --- [alarm-detect-13,5797304c19496417,f3ab9f45e6f312cd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643576, data is empty. 2024-09-11 17:20:57.804 INFO 2981023 --- [alarm-detect-13,5797304c19496417,f3ab9f45e6f312cd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643622, data is empty. 2024-09-11 17:20:57.808 INFO 2981023 --- [alarm-detect-13,5797304c19496417,f3ab9f45e6f312cd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643622, data is empty. 2024-09-11 17:20:58.775 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:16:58.339, end=2024-09-11T17:18:58.771} 2024-09-11 17:20:58.777 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId 2024-09-11 17:20:58.778 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:20:58.779 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.339, end=2024-09-11T17:18:58.771},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:20:58.779 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:20:58.779 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:20:58.808 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:20:58.827 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058827000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.827Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:58.829 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:20:58.829 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:58.834 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.834 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.836 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:58.836 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:58.840 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.840 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.842 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:20:58.842 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.827, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:58.842 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:20:58.844 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 4 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.339, end=2024-09-11T17:18:58.771},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:20:58.847 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:20:58.847 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:20:58.875 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:20:58.879 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:20:58.890 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058890000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.890Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:58.894 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058894000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.894Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:58.896 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058896000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.896Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:58.898 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058898000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.898Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:58.898 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058898000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.898Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:58.901 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058901000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.901Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:58.904 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058904000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.904Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:58.905 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058905000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.905Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:58.909 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:20:58.909 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:58.911 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:58.911 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:58.918 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.918 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.919 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:58.919 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:58.919 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:58.919 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:58.922 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.922 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.929 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:58.929 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:58.931 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.931 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.931 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:58.931 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:58.932 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:58.932 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:58.933 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 58 2024-09-11 17:20:58.933 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:58.936 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:58.936 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:58.937 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.937 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.938 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:58.938 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:58.941 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:20:58.941 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:58.942 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:58.942 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:58.943 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.943 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.945 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.945 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.946 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:58.947 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.947 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.947 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:58.947 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:58.950 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:58.950 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:58.950 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.950 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.952 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.952 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.954 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.954 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.954 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:58.956 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:58.956 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:58.957 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:58.957 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.957 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.958 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.958 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.958 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.89, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:58.959 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.959 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.962 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:58.963 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.963 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.963 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:58.963 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:58.963 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:58.964 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.898, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:58.964 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.894, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:58.966 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:58.968 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.968 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.970 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.896, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:58.970 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.898, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:58.971 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:58.976 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.905, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:58.976 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.901, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:58.979 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:58.979 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:58.982 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:58.989 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.904, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:58.992 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960058992000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:58.992Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:59.001 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:20:59.001 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.001 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059001000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:59.001Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:59.005 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059005000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:59.005Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:59.007 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059007000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:59.007Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:59.010 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059010000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:59.010Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:59.011 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:59.011 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.014 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.015 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.015 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:20:59.015 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.019 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:59.019 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.021 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.021 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.021 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059021000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:59.021Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:59.023 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:20:59.023 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.024 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.024 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.028 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.028 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.031 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.031 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.032 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059032000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:59.032Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:59.032 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:59.032 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.032 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.032 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.032 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.032 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.034 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.034 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.036 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.036 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.036 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.041 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.041 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.043 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059043000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:59.043Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:59.044 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:59.044 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.044 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.044 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.045 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:58.992, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.045 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.045 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.046 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.046 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.048 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.048 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.051 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.051 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.052 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.052 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.054 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.054 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.059 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.060 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:59.060 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.062 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.001, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.062 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.005, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.062 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.062 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.063 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.063 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.065 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.065 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.065 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.007, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.066 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.066 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.066 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.067 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.071 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.021, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.072 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.072 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.072 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.01, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.073 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.073 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.074 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.074 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.075 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.075 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059075000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=C0995E24A01F869BEB311E8C790E6C86, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:20:59.075Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:20:59.080 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.080 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.081 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.032, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.081 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.083 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:20:59.083 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.085 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.043, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.088 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.088 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.090 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.090 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.094 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.094 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.095 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.098 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.075, end=2024-09-11T17:18:58.771}, clusterId=2, tenantId=6, sqlId=C0995E24A01F869BEB311E8C790E6C86, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.098 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:20:59.098 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.339, end=2024-09-11T17:18:58.771},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:20:59.099 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:20:59.099 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:20:59.099 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:20:59.100 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.339, end=2024-09-11T17:18:58.771},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:20:59.100 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:20:59.100 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:20:59.101 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:20:59.102 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.339, end=2024-09-11T17:18:58.771},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:20:59.103 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:20:59.103 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:20:59.114 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:20:59.130 INFO 2981023 --- [ocp-async-9,f7e629ed8c3283e9,264e2635ce59f162] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:20:59.138 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059138000, endTimeUs=1726046338771000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=007A9340F57820A7B16750F99E8718B3, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:20:59.138Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000028, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000008, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 17:20:59.139 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059139000, endTimeUs=1726046338771000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=E9E2014C8CE705871C555597A6A32456, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:20:59.139Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000028, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000008, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 17:20:59.142 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:20:59.142 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.143 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 3 2024-09-11 17:20:59.143 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.143 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960059143000, endTimeUs=1726046338771000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=23FE60A3371767FA5216686AD2237588, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:20:59.143Z, endTime=2024-09-11T09:18:58.771Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000028, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000008, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 17:20:59.146 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 17:20:59.146 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:20:59.151 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.151 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.151 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.151 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=007A9340F57820A7B16750F99E8718B3, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.153 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.153 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.154 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.154 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.154 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.154 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=23FE60A3371767FA5216686AD2237588, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.156 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:20:59.156 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:20:59.162 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.162 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.162 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.162 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=007A9340F57820A7B16750F99E8718B3, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.163 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:20:59.163 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.139, end=2024-09-11T17:18:58.771}, clusterId=1000006, tenantId=1000006, sqlId=E9E2014C8CE705871C555597A6A32456, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.163 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.165 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:20:59.165 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046338771000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=23FE60A3371767FA5216686AD2237588, sqlIdList=null, dynamicSql=false) 2024-09-11 17:20:59.166 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:20:59.167 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.138, end=2024-09-11T17:18:58.771}, clusterId=1000006, tenantId=1000006, sqlId=007A9340F57820A7B16750F99E8718B3, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.169 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:20:59.143, end=2024-09-11T17:18:58.771}, clusterId=1000006, tenantId=1000006, sqlId=23FE60A3371767FA5216686AD2237588, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:20:59.169 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:20:59.170 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.339, end=2024-09-11T17:18:58.771},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:20:59.170 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:20:59.170 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:20:59.170 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:20:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,a30cb25413b3eb30,f81a492289f770ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046460, interval=60 2024-09-11 17:20:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,a30cb25413b3eb30,f81a492289f770ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:01.130 INFO 2981023 --- [ocp-async-4,5f16b704ac0cfe84,55928edfebc847de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:01.657 INFO 2981023 --- [ocp-schedules-16,2f956c39ec093607,02e11e935b2f6da3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:21:01.879 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:01.880 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,4a6a45ffdf27712c,ecc06d8ab4f5d486] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,4a6a45ffdf27712c,ecc06d8ab4f5d486] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:03.131 INFO 2981023 --- [ocp-async-10,755269dcb797d15e,1a3a63be201b4789] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:04.880 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:04.881 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:05.131 INFO 2981023 --- [ocp-async-3,e262298ca70bbafd,9ab988ee30705958] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,6c4849340d09a09e,22cfc9553d8f6365] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,6c4849340d09a09e,22cfc9553d8f6365] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:21:06.336990+08:00, lastProcessTime=2024-09-11T17:20:56.352472+08:00 2024-09-11 17:21:06.465 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:21:06.336990+08:00 2024-09-11 17:21:07.100 INFO 2981023 --- [ocp-schedules-5,356773acc835d2ba,edacd85195a72e7d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:21:07.106 INFO 2981023 --- [ocp-schedules-21,f40d4fd9df512a19,dfe877af64ba0718] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:21:07.106+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:21:07.106 INFO 2981023 --- [ocp-schedules-21,f40d4fd9df512a19,dfe877af64ba0718] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:21:07.107 INFO 2981023 --- [ocp-schedules-21,f40d4fd9df512a19,dfe877af64ba0718] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:21:07.108 INFO 2981023 --- [ocp-schedules-21,f40d4fd9df512a19,dfe877af64ba0718] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:21:07.131 INFO 2981023 --- [ocp-async-9,8da282fc342cf611,c596976a4e005427] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:07.735 INFO 2981023 --- [ocp-async-1,00e49d3596c5edbe,b7ac5ffacbf4e431] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:21:07.844 INFO 2981023 --- [ocp-schedules-11,b4155b65d72c0690,e1f890c3e55677e0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:21:07.881 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:07.882 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:07.890 WARN 2981023 --- [scanner-13465583-1,b4155b65d72c0690,55d0171e800aab92] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:21:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,8b94f85a0d79e3b1,16f05d0ae0e46f18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,8b94f85a0d79e3b1,16f05d0ae0e46f18] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:09.131 INFO 2981023 --- [ocp-async-2,86ffc6bc16cd469a,4af8783c73af32e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:10.003 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:21:10.003 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:21:10.684 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9a269bd036d997b6,ed0027c50e57925b] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/4048171/rollback, client 10.31.33.41, traceId 9a269bd036d997b6] 2024-09-11 17:21:10.685 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9a269bd036d997b6,ed0027c50e57925b] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/4048171/rollback?taskInstanceId=4048171, client=10.31.33.41, traceId=9a269bd036d997b6, method=NoDataResponse com.oceanbase.ocp.server.common.controller.TaskController.rollbackTask(Long), args=4048171,] 2024-09-11 17:21:10.701 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9a269bd036d997b6,ed0027c50e57925b] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 4048146, current state: FAILED, new state: READY 2024-09-11 17:21:10.722 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9a269bd036d997b6,ed0027c50e57925b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/4048171/rollback client=10.31.33.41, traceId=9a269bd036d997b6, duration=41 ms] 2024-09-11 17:21:10.749 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,8a8573784a1984c4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048146, original=READY 2024-09-11 17:21:10.752 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,8a8573784a1984c4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048146, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:10.857 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048146, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Add observer, zone2.idc=default_idc, sub_task_instance_id=4048146, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:10.749+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:10.861 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.s.t.b.server.AddObServerTask : begin to roll back 2024-09-11 17:21:10.882 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:10.882 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:10.899 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.s.t.b.server.ObServerTaskHandler : begin to delete server, svrIp=10.31.35.7, svrPort=2882 2024-09-11 17:21:10.904 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:21:10.911 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.7, 2882] 2024-09-11 17:21:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:21:10.916 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.s.t.b.server.ObServerTaskHandler : target observer is not in cluster, no need to execute delete server command 2024-09-11 17:21:10.920 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.o.i.helper.OcpCacheHelperImpl : Ob cluster resource has been changed, cluster 1000006, tenant null, notify ocp cache 2024-09-11 17:21:10.923 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.s.t.b.server.AddObServerTask : Begin to wait for observer delete success 2024-09-11 17:21:10.938 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.s.t.b.server.ObServerTaskHandler : start to check server active 2024-09-11 17:21:10.942 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:10.949 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.7, 2882] 2024-09-11 17:21:10.953 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.s.t.b.server.ObServerTaskHandler : server 10.31.35.7, obServer=null 2024-09-11 17:21:10.959 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.s.t.b.server.AddObServerTask : Wait for observer delete success 2024-09-11 17:21:10.969 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,40eb80a627881c33] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048146, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:11.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,f42c953660d28b6f,95e20029fbd8b06e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.31.33.41, traceId=f42c953660d28b6f, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-09-11 17:21:11.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,f42c953660d28b6f,95e20029fbd8b06e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.31.33.41, traceId=f42c953660d28b6f, duration=6 ms] 2024-09-11 17:21:11.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,6bfdd30cafe308c5,ab56fb865ec10ecf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=6bfdd30cafe308c5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:21:11.127 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048131] 2024-09-11 17:21:11.131 INFO 2981023 --- [ocp-async-7,29ad795ccbc0e418,b05e19e2f77f7d7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:11.138 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,6bfdd30cafe308c5,ab56fb865ec10ecf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=6bfdd30cafe308c5, duration=18 ms] 2024-09-11 17:21:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,d0ccae845f7e5325,5e716473ce9d90bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:11.319 INFO 2981023 --- [prometheus-minute-schedule-1,d0ccae845f7e5325,5e716473ce9d90bd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:11.680 INFO 2981023 --- [ocp-schedules-14,97196ef2de77cfd6,bccab921ab3f792e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:21:11.692 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046400, nodeCount=480, affectRows=480 2024-09-11 17:21:11.762 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,3c856525a29515d8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048131, original=READY 2024-09-11 17:21:11.764 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,3c856525a29515d8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048131, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:11.771 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046400, nodeCount=1000, affectRows=1000 2024-09-11 17:21:11.774 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046400, nodeCount=999, validCount=999 2024-09-11 17:21:11.878 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,a4d0dd03eb28a9e9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048131, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048131, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:11.762+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:11.892 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,a4d0dd03eb28a9e9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048131, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:11.923 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046400, nodeCount=1000, affectRows=1000 2024-09-11 17:21:11.927 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046400, nodeCount=1000, validCount=1000 2024-09-11 17:21:11.938 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046400, nodeCount=48, affectRows=48 2024-09-11 17:21:11.939 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046400, nodeCount=46, validCount=46 2024-09-11 17:21:12.176 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048118] 2024-09-11 17:21:12.774 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,1fbd68014af2e21b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048118, original=READY 2024-09-11 17:21:12.776 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,1fbd68014af2e21b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048118, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:12.814 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,e8a6e09872e1a4eb,61b3aeaf230adf79] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments?taskInstanceId=4048171&subtaskInstanceId=4048146, client=10.31.33.41, traceId=e8a6e09872e1a4eb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048171,4048146,] 2024-09-11 17:21:12.821 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,e8a6e09872e1a4eb,61b3aeaf230adf79] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171/subtasks/4048146/logSegments client=10.31.33.41, traceId=e8a6e09872e1a4eb, duration=12 ms] 2024-09-11 17:21:12.880 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048118, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Start observer process with param, zone2.idc=default_idc, sub_task_instance_id=4048118, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:12.774+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:12.884 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] .t.b.s.StartObServerProcessWithParamTask : begin to rollback 2024-09-11 17:21:12.905 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:12.920 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:12.993 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/exists, request body:CheckProcessExistsRequest(name=observer), params:null 2024-09-11 17:21:13.007 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.s.t.b.server.ObServerTaskHandler : try to kill observer process,address is 10.31.35.7 2024-09-11 17:21:13.013 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:13.017 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:13.032 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/stop, request body:StopProcessRequest(process=FindProcessParam(findType=BY_NAME, name=observer, keyword=null, pid=null), force=true), params:null 2024-09-11 17:21:13.042 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.s.t.b.server.ObServerTaskHandler : wait for process observer to stop, waitTimes=1 2024-09-11 17:21:13.049 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] com.oceanbase.ocp.core.util.TaskUtils : [TaskUtils] wait 10 seconds 2024-09-11 17:21:13.131 INFO 2981023 --- [ocp-async-1,25eddfff01b2800b,f2973ccc1ad88454] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:13.362 INFO 2981023 --- [ocp-schedules-29,eb4a1232abba5a05,bb87ee33b93e4433] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:21:13.370 INFO 2981023 --- [ocp-schedules-29,eb4a1232abba5a05,bb87ee33b93e4433] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:21:13.370 INFO 2981023 --- [ocp-schedules-29,eb4a1232abba5a05,bb87ee33b93e4433] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:21:13.777 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046400, nodeCount=480, validCount=480 2024-09-11 17:21:13.801 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,f09af185a26ab342,592be572fc0cf51a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=f09af185a26ab342, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:21:13.822 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,f09af185a26ab342,592be572fc0cf51a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=f09af185a26ab342, duration=23 ms] 2024-09-11 17:21:13.883 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:13.884 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,c3df755ff89f77e6,fa9f9a3b714e3e15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,6ca8d93d50098295,542f343db784edf2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:21:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,c3df755ff89f77e6,fa9f9a3b714e3e15] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:15.131 INFO 2981023 --- [ocp-async-4,81549787c161de6b,ff0c3f79dc73b43f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:15.346 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6c43ddcddf1c5d8d,f5308e8efe13f5c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=6c43ddcddf1c5d8d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:21:15.356 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6c43ddcddf1c5d8d,f5308e8efe13f5c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=6c43ddcddf1c5d8d, duration=13 ms] 2024-09-11 17:21:16.019 INFO 2981023 --- [ocp-schedules-11,f6bf7df366a99a8c,6330971634e023ac] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:21:16.019 INFO 2981023 --- [host-check-clockdiff1,f6bf7df366a99a8c,91f00ce7ca45552f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:21:16.020 INFO 2981023 --- [host-check-clockdiff3,f6bf7df366a99a8c,603784011c81e595] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:21:16.020 INFO 2981023 --- [host-check-clockdiff2,f6bf7df366a99a8c,c8e8e4b34ea2e334] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:21:16.022 INFO 2981023 --- [host-check-clockdiff4,f6bf7df366a99a8c,e4c4b754b09b83e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:21:16.023 INFO 2981023 --- [ocp-schedules-20,1f4ae4a3e5c8bc80,06e0f25f9cdab54d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:21:16.028 INFO 2981023 --- [ocp-schedules-30,532684fac6e19ac0,17b10d0b3b105608] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:21:16.029 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:21:16.032 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:21:16.033 INFO 2981023 --- [host-check-agent-status1,532684fac6e19ac0,4af7dfa4b70731df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:21:16.033 INFO 2981023 --- [host-check-agent-status2,532684fac6e19ac0,9757544483640afa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:21:16.041 INFO 2981023 --- [host-check-clockdiff4,f6bf7df366a99a8c,e4c4b754b09b83e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046476 0 0 , error= 2024-09-11 17:21:16.041 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:21:16.042 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:21:16.042 INFO 2981023 --- [host-check-agent-status4,532684fac6e19ac0,092bac69c1ce25b2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:21:16.042 INFO 2981023 --- [host-check-agent-status3,532684fac6e19ac0,636bfa05dda0b8bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:21:16.050 INFO 2981023 --- [host-check-agent-status2,532684fac6e19ac0,9757544483640afa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:21:16.050 INFO 2981023 --- [host-check-agent-status1,532684fac6e19ac0,4af7dfa4b70731df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:21:16.051 INFO 2981023 --- [host-check-agent-status2,532684fac6e19ac0,9757544483640afa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:21:16.051 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:21:16.052 INFO 2981023 --- [host-check-agent-status1,532684fac6e19ac0,4af7dfa4b70731df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:21:16.053 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:21:16.053 INFO 2981023 --- [host-check-agent-status4,532684fac6e19ac0,092bac69c1ce25b2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:21:16.054 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:21:16.055 INFO 2981023 --- [host-check-agent-status3,532684fac6e19ac0,636bfa05dda0b8bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:16.055 INFO 2981023 --- [host-check-agent-status4,532684fac6e19ac0,092bac69c1ce25b2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:21:16.056 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:21:16.056 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:21:16.057 INFO 2981023 --- [host-check-agent-status3,532684fac6e19ac0,636bfa05dda0b8bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:16.058 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:16.059 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:16.059 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:21:16.078 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:21:16.081 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:21:16.080747908+08:00 2024-09-11 17:21:16.086 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:21:16.081+08:00 2024-09-11 17:21:16.087 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:21:16.090 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:21:16.081+08:00 and status AVAILABLE 2024-09-11 17:21:16.095 INFO 2981023 --- [host-check-agent-status2,532684fac6e19ac0,9757544483640afa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:21:16.097 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:21:16.100 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:21:16.089746286+08:00 2024-09-11 17:21:16.100 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:21:16.100100306+08:00 2024-09-11 17:21:16.102 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:21:16.100+08:00 2024-09-11 17:21:16.104 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:21:16.100+08:00 and status AVAILABLE 2024-09-11 17:21:16.105 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:21:16.100+08:00 2024-09-11 17:21:16.107 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:21:16.100+08:00 and status AVAILABLE 2024-09-11 17:21:16.107 INFO 2981023 --- [host-check-agent-status2,532684fac6e19ac0,9757544483640afa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:21:16.107 INFO 2981023 --- [host-check-agent-status2,532684fac6e19ac0,9757544483640afa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:21:16.107 INFO 2981023 --- [host-check-agent-status2,532684fac6e19ac0,9757544483640afa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:21:16.109 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:21:16.109 INFO 2981023 --- [host-check-agent-status4,532684fac6e19ac0,092bac69c1ce25b2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:21:16.112 INFO 2981023 --- [host-check-agent-status4,532684fac6e19ac0,092bac69c1ce25b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:21:16.112 INFO 2981023 --- [host-check-agent-status4,532684fac6e19ac0,092bac69c1ce25b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:21:16.112 INFO 2981023 --- [host-check-agent-status4,532684fac6e19ac0,092bac69c1ce25b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:21:16.115 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:21:16.133 INFO 2981023 --- [host-check-agent-status3,532684fac6e19ac0,636bfa05dda0b8bc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:21:16.133 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:21:16.137 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:21:16.138 INFO 2981023 --- [host-check-agent-status3,532684fac6e19ac0,636bfa05dda0b8bc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:21:16.138 INFO 2981023 --- [host-check-agent-status3,532684fac6e19ac0,636bfa05dda0b8bc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:21:16.138 INFO 2981023 --- [host-check-agent-status3,532684fac6e19ac0,636bfa05dda0b8bc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:21:16.139 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:21:16.139 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:21:16.140 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:21:16.147 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:21:16.150 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:21:16.151 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:21:16.148583962+08:00 2024-09-11 17:21:16.162 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:21:16.151+08:00 2024-09-11 17:21:16.163 INFO 2981023 --- [host-check-agent-status1,532684fac6e19ac0,4af7dfa4b70731df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:21:16.166 INFO 2981023 --- [host-check-agent-status1,532684fac6e19ac0,4af7dfa4b70731df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:21:16.166 INFO 2981023 --- [host-check-agent-status1,532684fac6e19ac0,4af7dfa4b70731df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:21:16.166 INFO 2981023 --- [host-check-agent-status1,532684fac6e19ac0,4af7dfa4b70731df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:21:16.173 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:21:16.173 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:21:16.151+08:00 and status AVAILABLE 2024-09-11 17:21:16.184 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:21:16.186 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:21:16.189 INFO 2981023 --- [ocp-schedules-22,bdd2e5ca32918cbd,781b36b93305a593] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:21:16.189 INFO 2981023 --- [ocp-schedules-24,e409c8f1f8689751,2f7ff1b720a4b755] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:21:16.189 INFO 2981023 --- [ocp-schedules-18,27ad483e78862b04,b65d45c135fd2885] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:21:16.189 INFO 2981023 --- [ocp-schedules-5,c0a913d0e0dd4fed,56c59ecef9e0d543] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:21:16.193 INFO 2981023 --- [ocp-schedules-24,e409c8f1f8689751,2f7ff1b720a4b755] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:21:16.202 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:21:16.203 INFO 2981023 --- [ocp-schedules-22,bdd2e5ca32918cbd,781b36b93305a593] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:21:16.203 INFO 2981023 --- [ocp-schedules-18,27ad483e78862b04,b65d45c135fd2885] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:21:16.207 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:21:16.214 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:21:16.223 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:16.223 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:21:16.225 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:16.230 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:21:16.236 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:21:16.241 INFO 2981023 --- [ocp-schedules-23,bac552c6a0306416,0f0f656bb7186192] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:21:16.245 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046402186000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:21:16.249 INFO 2981023 --- [ocp-schedules-24,e409c8f1f8689751,2f7ff1b720a4b755] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:21:16.270 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:21:16.289 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:21:16.291 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:21:16.293 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:21:16.300 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:21:16.302 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.302 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:21:16.303 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:21:16.303 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:21:16.306 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.306 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:21:16.307 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:21:16.311 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.313 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:16.313 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:21:16.314 INFO 2981023 --- [ocp-schedules-22,bdd2e5ca32918cbd,781b36b93305a593] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:21:16.324 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.324 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:16.324 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.325 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:21:16.325 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:21:16.325 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:16.333 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:21:16.338 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:21:16.348 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:21:16.350 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:21:16.367 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:21:16.369 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:21:16.368417+08:00, lastProcessTime=2024-09-11T17:21:06.336990+08:00 2024-09-11 17:21:16.369 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:21:16.371 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:16.372 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.372 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:21:16.374 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:21:16.376 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:21:16.384 INFO 2981023 --- [host-check-host-status1,1f4ae4a3e5c8bc80,bd3ae21c3b7caba2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:21:16.386 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:21:16.389 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:21:16.395 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:21:16.410 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:21:16.410 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046402186000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:21:16.416 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:16.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.417 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:21:16.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:16.418 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:21:16.418 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.419 INFO 2981023 --- [host-check-host-status2,1f4ae4a3e5c8bc80,044d6c9f94d73964] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:21:16.421 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:21:16.421 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:21:16.422 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:21:16.430 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:21:16.431 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:21:16.440 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:21:16.457 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:21:16.461 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:21:16.463 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:21:16.465 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:16.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:21:16.467 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:21:16.471 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:21:16.472 INFO 2981023 --- [host-check-host-status3,1f4ae4a3e5c8bc80,26b50b33a19b80ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:21:16.474 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:21:16.475 INFO 2981023 --- [host-check-host-status4,1f4ae4a3e5c8bc80,e2d0fa0c22aee8b5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:21:16.477 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:21:16.496 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:21:16.513 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:16.514 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.514 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:21:16.534 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:16.535 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.536 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:16.536 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:21:16.537 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.539 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:21:16.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:21:16.546 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:21:16.546 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:21:16.556 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:21:16.558 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:21:16.563 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:21:16.567 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:21:16.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:21:16.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:21:16.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:21:16.579 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:21:16.584 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:21:16.585 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:21:16.600 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:21:16.616 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:21:16.368417+08:00 2024-09-11 17:21:16.629 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:21:16.637 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:16.638 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:16.638 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:21:16.644 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:21:16.644 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:21:16.644 INFO 2981023 --- [ocp-schedules-18,27ad483e78862b04,b65d45c135fd2885] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:21:16.781 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,5d0ea805658f21cd,39bc218e9262f4ee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=5d0ea805658f21cd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:21:16.791 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,5d0ea805658f21cd,39bc218e9262f4ee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=5d0ea805658f21cd, duration=13 ms] 2024-09-11 17:21:16.884 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:16.885 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:17.131 INFO 2981023 --- [ocp-async-2,8642474637bb6c05,012ab154ecc15942] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,b0129af0cf11695f,04b3c7c61ef43aed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,b0129af0cf11695f,04b3c7c61ef43aed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:17.735 INFO 2981023 --- [ocp-async-4,6bf9c1e73b000a1e,32132fe28e25a6be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:21:18.031 INFO 2981023 --- [ocp-schedules-18,d2ad5ed982dae088,a810baa92ec737ef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:21:18.079 WARN 2981023 --- [scanner-13467315-1,d2ad5ed982dae088,6efa2791f7504b93] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:21:18.102 INFO 2981023 --- [alarm-detect-6,d2ad5ed982dae088,6de21706de850f34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643622, data is empty. 2024-09-11 17:21:18.105 INFO 2981023 --- [alarm-detect-6,d2ad5ed982dae088,6de21706de850f34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643622, data is empty. 2024-09-11 17:21:18.108 INFO 2981023 --- [alarm-detect-6,d2ad5ed982dae088,6de21706de850f34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643622, data is empty. 2024-09-11 17:21:18.137 INFO 2981023 --- [alarm-detect-13,d2ad5ed982dae088,15c238e6ad1f85d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643622, data is empty. 2024-09-11 17:21:18.137 INFO 2981023 --- [alarm-detect-20,d2ad5ed982dae088,99d6b5722fd5ad1e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643622, data is empty. 2024-09-11 17:21:18.141 INFO 2981023 --- [alarm-detect-20,d2ad5ed982dae088,99d6b5722fd5ad1e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643622, data is empty. 2024-09-11 17:21:18.141 INFO 2981023 --- [alarm-detect-13,d2ad5ed982dae088,15c238e6ad1f85d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643622, data is empty. 2024-09-11 17:21:18.144 INFO 2981023 --- [alarm-detect-13,d2ad5ed982dae088,15c238e6ad1f85d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643622, data is empty. 2024-09-11 17:21:18.147 INFO 2981023 --- [alarm-detect-20,d2ad5ed982dae088,99d6b5722fd5ad1e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643622, data is empty. 2024-09-11 17:21:18.209 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,c019b731a0561a09,649cd23fc395df52] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=c019b731a0561a09, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:21:18.247 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,c019b731a0561a09,649cd23fc395df52] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=c019b731a0561a09, duration=45 ms] 2024-09-11 17:21:18.275 INFO 2981023 --- [alarm-detect-1,d2ad5ed982dae088,802d10be2b3b1b74] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643622, data is empty. 2024-09-11 17:21:18.278 INFO 2981023 --- [alarm-detect-1,d2ad5ed982dae088,802d10be2b3b1b74] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643622, data is empty. 2024-09-11 17:21:18.282 INFO 2981023 --- [alarm-detect-1,d2ad5ed982dae088,802d10be2b3b1b74] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643622, data is empty. 2024-09-11 17:21:18.548 INFO 2981023 --- [host-check-clockdiff2,f6bf7df366a99a8c,c8e8e4b34ea2e334] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046478 0 0 , error= 2024-09-11 17:21:19.131 INFO 2981023 --- [ocp-async-9,b0e1d9cd6425b593,fd83af1e81cb6def] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:19.723 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,d9a7ee9720abdd76,a856e76531764836] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=d9a7ee9720abdd76, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:21:19.732 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,d9a7ee9720abdd76,a856e76531764836] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=d9a7ee9720abdd76, duration=12 ms] 2024-09-11 17:21:19.885 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:19.887 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:20.162 INFO 2981023 --- [host-check-clockdiff1,f6bf7df366a99a8c,91f00ce7ca45552f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046480 0 0 , error= 2024-09-11 17:21:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,e09537d7e5aea7fe,b0ef53c3c02c700e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,e09537d7e5aea7fe,b0ef53c3c02c700e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:21.132 INFO 2981023 --- [ocp-async-10,81d825e6ac9effb4,e2c6ac83be0e307f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:21.185 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,61edce4439d49fec,a99e75352718e921] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048171?taskInstanceId=4048171, client=10.31.33.41, traceId=61edce4439d49fec, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048171,] 2024-09-11 17:21:21.195 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,61edce4439d49fec,a99e75352718e921] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048171 client=10.31.33.41, traceId=61edce4439d49fec, duration=12 ms] 2024-09-11 17:21:21.683 INFO 2981023 --- [ocp-schedules-24,66c0d2a4721f8e97,8c361d2b01bd6918] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:21:21.999 INFO 2981023 --- [host-check-clockdiff3,f6bf7df366a99a8c,603784011c81e595] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046481 1 1 , error= 2024-09-11 17:21:22.154 INFO 2981023 --- [ocp-schedules-28,b9c8b43620a73fb3,3d9b851b3e20dfd7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:21:22.154 INFO 2981023 --- [ocp-schedules-28,b9c8b43620a73fb3,3d9b851b3e20dfd7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:21:22.154 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:21:22.154 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:21:22.154 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:21:22.159 INFO 2981023 --- [ocp-schedules-28,b9c8b43620a73fb3,3d9b851b3e20dfd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:22.159 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:21:22.159 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:21:22.159 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:21:22.163 INFO 2981023 --- [ocp-schedules-28,b9c8b43620a73fb3,3d9b851b3e20dfd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:22.163 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:21:22.164 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:21:22.168 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5abab3b88ba8b4c2,d73a4d23a75df50d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP&maxPoints=360, client=10.31.33.41, traceId=5abab3b88ba8b4c2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,] 2024-09-11 17:21:22.175 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:21:22.179 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,2d73c2e5ebde221c,95746a4bf1c9b973] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.31.33.41, traceId=2d73c2e5ebde221c, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2024-09-11 17:21:22.181 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5abab3b88ba8b4c2,d73a4d23a75df50d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=5abab3b88ba8b4c2, duration=18 ms] 2024-09-11 17:21:22.189 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,2d73c2e5ebde221c,95746a4bf1c9b973] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.31.33.41, traceId=2d73c2e5ebde221c, duration=13 ms] 2024-09-11 17:21:22.196 INFO 2981023 --- [ocp-schedules-28,b9c8b43620a73fb3,3d9b851b3e20dfd7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:21:22.196 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:21:22.198 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:21:22.199 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:21:22.201 INFO 2981023 --- [ocp-schedules-28,b9c8b43620a73fb3,3d9b851b3e20dfd7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:21:22.272 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,985e3a65ffcb1907,18facc9e87fcfe56] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.31.33.41, traceId=985e3a65ffcb1907, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2024-09-11 17:21:22.278 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,13802db5a27f2f86,7dd226eb40f0c8c8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=13802db5a27f2f86, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 17:21:22.278 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,301aaefe0f77c502,4a26abed49f1a020] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2024-09-11T16%3A21%3A21%2B08%3A00&endTime=2024-09-11T17%3A21%3A21%2B08%3A00&maxPoints=360, client=10.31.33.41, traceId=301aaefe0f77c502, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:21:21+08:00,2024-09-11T17:21:21+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2024-09-11 17:21:22.287 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,985e3a65ffcb1907,18facc9e87fcfe56] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=985e3a65ffcb1907, duration=17 ms] 2024-09-11 17:21:22.309 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,301aaefe0f77c502,4a26abed49f1a020] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=301aaefe0f77c502, duration=34 ms] 2024-09-11 17:21:22.315 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ddb75ad1fe181577,5ff985cdf1fa26d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.31.33.41, traceId=ddb75ad1fe181577, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2024-09-11 17:21:22.337 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: ob_cluster_status_check, cached timestamp: 1726039276 2024-09-11 17:21:22.337 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = ob_cluster_status_check 2024-09-11 17:21:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: ob_cluster_sync, cached timestamp: 1726039276 2024-09-11 17:21:22.339 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = ob_cluster_sync 2024-09-11 17:21:22.350 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,13802db5a27f2f86,7dd226eb40f0c8c8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=13802db5a27f2f86, duration=74 ms] 2024-09-11 17:21:22.362 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:22.362 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:22.364 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:21:22.364 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:21:22.377 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:22.379 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:21:22.383 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ddb75ad1fe181577,5ff985cdf1fa26d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.31.33.41, traceId=ddb75ad1fe181577, duration=75 ms] 2024-09-11 17:21:22.887 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:22.888 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:23.056 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:23.062 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:23.078 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=observer, skipPorts=false), params:null 2024-09-11 17:21:23.089 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.h.s.HostOperationServiceImpl : Process name: observer is not found in host 2000008 2024-09-11 17:21:23.093 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.s.t.b.server.ObServerTaskHandler : process observer has been killed 2024-09-11 17:21:23.099 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:23.104 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:23.118 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/exists, request body:CheckProcessExistsRequest(name=observer), params:null 2024-09-11 17:21:23.132 INFO 2981023 --- [ocp-async-2,82a804191e9de121,9d67daae6a114940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:23.130 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:23.135 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:23.162 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/ob/cleanDataFiles, request body:CleanFilesRequest(obClusterName=linkrtd, obPath=ObPath(installPath=/data/rtdob/oceanbase, dataPath=/data/oceanbase, logPath=/oceanbase/redo, diskPathStyle=null, runPath=/data/rtdob/oceanbase)), params:null 2024-09-11 17:21:23.170 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] .t.b.s.StartObServerProcessWithParamTask : rollback finished 2024-09-11 17:21:23.181 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,56afa42eecfe9999] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048118, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,0dd2d56e48490f62,98f73ff31e5de2a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:23.318 INFO 2981023 --- [prometheus-minute-schedule-1,0dd2d56e48490f62,98f73ff31e5de2a9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:23.515 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:21:23.515 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:21:23.518 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:21:23.518 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:21:23.518 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:21:23.518 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:21:23.520 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:21:23.520 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:21:23.520 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:21:23.520 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:21:23.541 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:21:23.545 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:21:23.545 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:21:23.545 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:21:23.545 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:21:23.545 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:21:23.549 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:21:23.691 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048124] 2024-09-11 17:21:23.804 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,031e733836e52c40] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048124, original=READY 2024-09-11 17:21:23.805 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,031e733836e52c40] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048124, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:23.812 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:21:23.906 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,e2e372372786acf6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048124, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048124, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:23.804+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:23.915 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,e2e372372786acf6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048124, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:24.091 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.571511291S 2024-09-11 17:21:24.091 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:21:24.141 INFO 2981023 --- [ocp-schedules-18,572ac6c145c7e758,4f5851cc0750e1e2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:21:24.143 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:21:24.143 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:21:24.158 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:21:24.192 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:21:24.192 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:21:24.192 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:21:24.192 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:21:24.192 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:21:24.192 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:21:24.198 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:21:24.285 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] o.s.boot.SpringApplication : Started application in 0.767 seconds (JVM running for 82284.157) 2024-09-11 17:21:24.285 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.767110549S 2024-09-11 17:21:24.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6dd6480c61d5c9a0,6f7ddba1d83be3d5] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&version=2&ObClusterId=1719994443, client=10.31.35.6, traceId=6dd6480c61d5c9a0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkrtd,1719994443,2,{"ObRegion":"linkrtd","ObCluster":"linkrtd","ObRegionId":1719994443,"ObClusterId":1719994443,"Type":"PRIMARY","timestamp":1726046484469270,"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:21:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6dd6480c61d5c9a0,6f7ddba1d83be3d5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkrtd, obClusterId=1719994443, rootServerJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, groupName=linkrtd, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:21:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6dd6480c61d5c9a0,6f7ddba1d83be3d5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:21:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6dd6480c61d5c9a0,6f7ddba1d83be3d5] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkrtd, clusterId=1719994443, affectRows=1 2024-09-11 17:21:24.480 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6dd6480c61d5c9a0,6f7ddba1d83be3d5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.6, traceId=6dd6480c61d5c9a0, duration=10 ms] 2024-09-11 17:21:24.490 INFO 2981023 --- [ocp-schedules-19,8f36879f93843b6a,82a30e0d48bff2e7] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:21:24.753 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048129] 2024-09-11 17:21:24.816 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,aefc00a790726770] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048129, original=READY 2024-09-11 17:21:24.820 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,aefc00a790726770] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048129, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:24.899 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2cc5fc60996dceab] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048129, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Prepare cgroup environment, zone2.idc=default_idc, sub_task_instance_id=4048129, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:24.816+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:24.907 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2cc5fc60996dceab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:24.911 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2cc5fc60996dceab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:24.927 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2cc5fc60996dceab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/ob/install/cleanCGroupEnv, request body:HandleCGroupEnvRequest(obVersion=4.2.4.0, obPath=ObPath(installPath=/data/rtdob/oceanbase, dataPath=/data/oceanbase, logPath=/oceanbase/redo, diskPathStyle=OBD, runPath=/data/rtdob/oceanbase), runUser=null), params:null 2024-09-11 17:21:24.944 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2cc5fc60996dceab] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048129, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:25.132 INFO 2981023 --- [ocp-async-7,68bc7e1ea7209a08,12f55f980119842b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:25.797 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048151] 2024-09-11 17:21:25.828 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,4499b295ea6b783f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048151, original=READY 2024-09-11 17:21:25.829 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,4499b295ea6b783f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048151, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:25.909 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:25.909 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:25.948 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,c782d77411f24756] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048151, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048151, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:25.828+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:25.958 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,c782d77411f24756] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048151, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,8526dc195888cc40,bd6aab629dd61839] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,8526dc195888cc40,bd6aab629dd61839] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:21:26.351255+08:00, lastProcessTime=2024-09-11T17:21:16.368417+08:00 2024-09-11 17:21:26.473 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:21:26.351255+08:00 2024-09-11 17:21:26.842 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048127] 2024-09-11 17:21:26.849 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,0a5d065165de02dd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048127, original=READY 2024-09-11 17:21:26.850 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,0a5d065165de02dd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048127, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:26.934 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,662ea64d592bf0a9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048127, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Make dir for ob, zone2.idc=default_idc, sub_task_instance_id=4048127, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:26.848+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:26.946 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,662ea64d592bf0a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:26.957 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,662ea64d592bf0a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:26.978 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,662ea64d592bf0a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/ob/install/cleanDirectories, request body:ObClusterInstallRequest(obClusterName=linkrtd, obVersion=null, obPath=ObPath(installPath=/data/rtdob/oceanbase, dataPath=/data/oceanbase, logPath=/oceanbase/redo, diskPathStyle=OBD, runPath=/data/rtdob/oceanbase), runUser=null), params:null 2024-09-11 17:21:26.990 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,662ea64d592bf0a9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048127, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:27.132 INFO 2981023 --- [ocp-async-1,c5d9479008639e18,7d13a9ea36ac172e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:27.735 INFO 2981023 --- [ocp-async-9,67c6bfa26fdf81d0,87f42605e59c8123] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:21:27.894 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048134] 2024-09-11 17:21:28.319 INFO 2981023 --- [ocp-schedules-25,76dcc58684c29256,c24b7750a8581ea0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:21:28.378 WARN 2981023 --- [scanner-13467687-1,76dcc58684c29256,f73816b0a8cc8b26] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:21:28.864 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,c5603dd923107fc9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048134, original=READY 2024-09-11 17:21:28.865 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,c5603dd923107fc9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048134, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:28.909 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:28.910 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:28.954 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,87a8518f0e06f89f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048134, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048134, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:28.864+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:28.965 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,87a8518f0e06f89f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048134, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:29.132 INFO 2981023 --- [ocp-async-5,88b841034403319d,e5945300cc805bdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,895cb676166ea70a,8d979a5d707603a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,895cb676166ea70a,8d979a5d707603a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:29.980 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048126] 2024-09-11 17:21:30.537 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,31952355b632a894,d5ed5a7008b5e37e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/users?page=1&size=10, client=10.31.33.41, traceId=31952355b632a894, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.IamController.listUsers(Pageable,String), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:21:30.557 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,31952355b632a894,d5ed5a7008b5e37e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/users client=10.31.33.41, traceId=31952355b632a894, duration=24 ms] 2024-09-11 17:21:30.567 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] s.b.a.a.m.OnlyOnceLoggingDenyMeterFilter : Reached the maximum number of URI tags for 'http.server.requests'. 2024-09-11 17:21:30.877 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9a068ed199c50f58] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048126, original=READY 2024-09-11 17:21:30.878 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9a068ed199c50f58] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048126, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:30.979 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048126, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Install OB rpm, zone2.idc=default_idc, sub_task_instance_id=4048126, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:30.877+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:30.991 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall, hostId=2000008, packageName=oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64 2024-09-11 17:21:31.000 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:31.007 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:31.025 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/package/uninstall, request body:UninstallPackageRequest(name=oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64, installPath=null), params:null 2024-09-11 17:21:31.132 INFO 2981023 --- [ocp-async-7,08b5ab19e4b0b8ab,234cc85c96840a19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:31.685 INFO 2981023 --- [ocp-schedules-19,37fd9a1ff79c5601,52413b4dfe459274] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:21:31.714 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,2ac9b0a169573fe2,8e6ee8c74a0e3197] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.31.33.41, traceId=2ac9b0a169573fe2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:21:31.714 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5823e7e71c96dcc6,682100899ea2e4ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.31.33.41, traceId=5823e7e71c96dcc6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2024-09-11 17:21:31.717 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-7,2ac9b0a169573fe2,8e6ee8c74a0e3197] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:21:31.724 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,cd795512752cc522,9442b4ce153686f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=OB&keyword=, client=10.31.33.41, traceId=cd795512752cc522, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=OB,,Page request [number: 0, size 5000, sort: createTime: DESC],] 2024-09-11 17:21:31.735 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,cd795512752cc522,9442b4ce153686f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.31.33.41, traceId=cd795512752cc522, duration=14 ms] 2024-09-11 17:21:31.756 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,2ac9b0a169573fe2,8e6ee8c74a0e3197] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=2ac9b0a169573fe2, duration=45 ms] 2024-09-11 17:21:31.768 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5823e7e71c96dcc6,682100899ea2e4ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.31.33.41, traceId=5823e7e71c96dcc6, duration=56 ms] 2024-09-11 17:21:31.910 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:31.910 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:32.043 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall do nothing, package not exists 2024-09-11 17:21:32.047 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall done 2024-09-11 17:21:32.051 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] .o.o.i.c.ClusterHostOperationServiceImpl : [cleanAllFiles]: generate agent async task token:b74ced54-d1ae-4526-a25e-83cdd1f4bf02 2024-09-11 17:21:32.058 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:32.064 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:32.079 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/ob/cleanAllFiles, request body:CleanFilesRequest(obClusterName=linkrtd, obPath=ObPath(installPath=/data/rtdob/oceanbase, dataPath=/data/oceanbase, logPath=/oceanbase/redo, diskPathStyle=OBD, runPath=/data/rtdob/oceanbase)), params:null 2024-09-11 17:21:32.087 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=b74ced54-d1ae-4526-a25e-83cdd1f4bf02), params:null 2024-09-11 17:21:32.092 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 17:21:32.096 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.ocp.common.lang.pattern.Retry : wait for 3 seconds 2024-09-11 17:21:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,31ba4045498bc9c0,225a2111147b8626] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,31ba4045498bc9c0,225a2111147b8626] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:33.132 INFO 2981023 --- [ocp-async-1,929e23b5fa164bde,9ccbb9b6dbe5c0cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:33.373 INFO 2981023 --- [ocp-schedules-28,e152d07f859e1025,e24b616ed45ed0e5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:21:33.381 INFO 2981023 --- [ocp-schedules-28,e152d07f859e1025,e24b616ed45ed0e5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:21:33.381 INFO 2981023 --- [ocp-schedules-28,e152d07f859e1025,e24b616ed45ed0e5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:21:34.910 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:34.911 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:35.100 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=b74ced54-d1ae-4526-a25e-83cdd1f4bf02), params:null 2024-09-11 17:21:35.107 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,null,, 2024-09-11 17:21:35.116 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,05fe35c24959e2b5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048126, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:35.132 INFO 2981023 --- [ocp-async-4,ed862707b705748a,cf4dc6d7eeebeaf8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:35.177 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048119] 2024-09-11 17:21:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,188632bdb32cc671,0e3e642ad889707c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,188632bdb32cc671,0e3e642ad889707c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:35.896 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,774828ed8bff3e02] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048119, original=READY 2024-09-11 17:21:35.897 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,774828ed8bff3e02] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048119, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:35.984 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,a047a902cdd0fbd5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048119, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048119, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:35.895+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:35.993 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,a047a902cdd0fbd5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048119, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:36.231 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048125] 2024-09-11 17:21:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:21:36.337612+08:00, lastProcessTime=2024-09-11T17:21:26.351255+08:00 2024-09-11 17:21:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:21:36.337612+08:00 2024-09-11 17:21:36.906 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,ef7bbdbaec8c9832] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048125, original=READY 2024-09-11 17:21:36.907 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,ef7bbdbaec8c9832] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048125, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:36.989 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048125, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Install dependencies, zone2.idc=default_idc, sub_task_instance_id=4048125, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:36.905+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:36.999 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall, hostId=2000008, packageName=oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64 2024-09-11 17:21:37.006 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:37.010 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:37.027 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/package/uninstall, request body:UninstallPackageRequest(name=oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64, installPath=null), params:null 2024-09-11 17:21:37.132 INFO 2981023 --- [ocp-async-2,ebd960ca15a39d67,bc0a909acbcc8bd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:37.189 INFO 2981023 --- [ocp-schedules-24,3d532f4c28a5843f,fd853027888318c6] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:21:37.208 WARN 2981023 --- [ocp-schedules-24,3d532f4c28a5843f,fd853027888318c6] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:21:37.314 INFO 2981023 --- [prometheus-cache-monitor1,07b2052c7c838cce,1b7232392b9b41ca] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:21:37.315 INFO 2981023 --- [prometheus-cache-monitor1,07b2052c7c838cce,1b7232392b9b41ca] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:21:37.735 INFO 2981023 --- [ocp-async-5,15053966ce5bca1e,a2cd7309b414002a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:21:37.911 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:37.912 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:38.044 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall do nothing, package not exists 2024-09-11 17:21:38.048 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall done 2024-09-11 17:21:38.052 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall, hostId=2000008, packageName=oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64 2024-09-11 17:21:38.058 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:38.067 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:38.100 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/package/uninstall, request body:UninstallPackageRequest(name=oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64, installPath=null), params:null 2024-09-11 17:21:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,35f4847c536b6af5,fc68083b4819bc79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:38.317 INFO 2981023 --- [prometheus-minute-schedule-1,35f4847c536b6af5,fc68083b4819bc79] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:38.507 INFO 2981023 --- [ocp-schedules-21,43d87a6eefeac1fd,d702eb95e50ec9a9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:21:38.553 WARN 2981023 --- [scanner-13467875-1,43d87a6eefeac1fd,e17b15cebf876f11] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:21:38.579 INFO 2981023 --- [alarm-detect-6,43d87a6eefeac1fd,8e3b48626bd07503] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643622, data is empty. 2024-09-11 17:21:38.582 INFO 2981023 --- [alarm-detect-6,43d87a6eefeac1fd,8e3b48626bd07503] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643622, data is empty. 2024-09-11 17:21:38.587 INFO 2981023 --- [alarm-detect-6,43d87a6eefeac1fd,8e3b48626bd07503] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643622, data is empty. 2024-09-11 17:21:38.604 INFO 2981023 --- [alarm-detect-11,43d87a6eefeac1fd,b34a9f48044e27c7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643622, data is empty. 2024-09-11 17:21:38.605 INFO 2981023 --- [alarm-detect-8,43d87a6eefeac1fd,ae0749229ee23b1e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643622, data is empty. 2024-09-11 17:21:38.612 INFO 2981023 --- [alarm-detect-11,43d87a6eefeac1fd,b34a9f48044e27c7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643622, data is empty. 2024-09-11 17:21:38.614 INFO 2981023 --- [alarm-detect-8,43d87a6eefeac1fd,ae0749229ee23b1e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643622, data is empty. 2024-09-11 17:21:38.618 INFO 2981023 --- [alarm-detect-11,43d87a6eefeac1fd,b34a9f48044e27c7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643622, data is empty. 2024-09-11 17:21:38.619 INFO 2981023 --- [alarm-detect-8,43d87a6eefeac1fd,ae0749229ee23b1e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643622, data is empty. 2024-09-11 17:21:38.706 INFO 2981023 --- [alarm-detect-20,43d87a6eefeac1fd,aaff04ba643806b2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643622, data is empty. 2024-09-11 17:21:38.709 INFO 2981023 --- [alarm-detect-20,43d87a6eefeac1fd,aaff04ba643806b2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643622, data is empty. 2024-09-11 17:21:38.712 INFO 2981023 --- [alarm-detect-20,43d87a6eefeac1fd,aaff04ba643806b2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643622, data is empty. 2024-09-11 17:21:39.117 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall do nothing, package not exists 2024-09-11 17:21:39.120 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall done 2024-09-11 17:21:39.129 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,7d1953eca0b91977] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048125, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:39.132 INFO 2981023 --- [ocp-async-8,2798794c93556d68,db59d0329c275fd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:39.370 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048142] 2024-09-11 17:21:39.685 INFO 2981023 --- [ocp-schedules-20,b4dc982146020fd4,a7585ccc8fd2a2a4] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:21:39.919 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,bfbae5c369fb55a8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048142, original=READY 2024-09-11 17:21:39.921 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,bfbae5c369fb55a8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048142, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:40.004 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:21:40.005 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:21:40.026 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,e968451b03d2ddbd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048142, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Set zone idc info, zone2.idc=default_idc, sub_task_instance_id=4048142, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:39.919+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:40.030 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,e968451b03d2ddbd] c.o.o.s.t.b.zone.SetZoneIdcInfoTask : need to do nothing when rollback 2024-09-11 17:21:40.040 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,e968451b03d2ddbd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048142, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:40.415 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048145] 2024-09-11 17:21:40.912 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:40.913 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:40.934 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,fc02de75828c5a96] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048145, original=READY 2024-09-11 17:21:40.935 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,fc02de75828c5a96] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048145, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:41.023 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048145, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Start zone, zone2.idc=default_idc, sub_task_instance_id=4048145, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:40.934+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:41.027 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.o.s.t.b.zone.ObZoneTaskHandler : begin to stop zone, clusterId=1000006, zone=zone2 2024-09-11 17:21:41.054 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:41.058 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:41.062 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:41.067 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [1800000000] 2024-09-11 17:21:41.072 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter system stop zone ?, args: [zone2] 2024-09-11 17:21:41.082 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:41.087 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.o.s.t.b.zone.ObZoneTaskHandler : stop zone success 2024-09-11 17:21:41.096 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,79636b3eacc80f31] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048145, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:41.133 INFO 2981023 --- [ocp-async-4,d3ab71ca80d1eec8,478c51f7a299f962] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,66f80b9c87cf157b,f4bdb148daba70ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,66f80b9c87cf157b,f4bdb148daba70ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:41.459 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048117] 2024-09-11 17:21:41.688 INFO 2981023 --- [ocp-schedules-15,cd8587bc1ef9efdf,ea7b7e98a0f324d7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:21:41.944 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9807de382baa713c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048117, original=READY 2024-09-11 17:21:41.945 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9807de382baa713c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048117, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:42.039 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048117, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Add zone, zone2.idc=default_idc, sub_task_instance_id=4048117, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:41.943+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:42.043 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.o.s.t.business.zone.AddObZoneTask : begin to rollback 2024-09-11 17:21:42.047 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.o.s.t.b.zone.ObZoneTaskHandler : begin to delete zone, clusterId=1000006, zone=zone2 2024-09-11 17:21:42.062 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:42.065 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:42.070 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:42.074 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter system delete zone ?, args: [zone2] 2024-09-11 17:21:42.083 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.o.s.t.b.zone.ObZoneTaskHandler : delete zone success 2024-09-11 17:21:42.087 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.o.o.i.helper.OcpCacheHelperImpl : Ob cluster resource has been changed, cluster 1000006, tenant null, notify ocp cache 2024-09-11 17:21:42.090 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.o.s.t.business.zone.AddObZoneTask : rollback finished 2024-09-11 17:21:42.102 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,2bfd4f9b25877a07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048117, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:42.504 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048136] 2024-09-11 17:21:42.956 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,78e8724057547ac6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048136, original=READY 2024-09-11 17:21:42.957 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,78e8724057547ac6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048136, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:43.042 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,90a9ceaa523b52c1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048136, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Set zone context, zone2.idc=default_idc, sub_task_instance_id=4048136, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:42.955+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:43.046 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,90a9ceaa523b52c1] c.o.o.s.t.b.zone.SetZoneContextTask : need to do nothing when rollback 2024-09-11 17:21:43.055 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,90a9ceaa523b52c1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048136, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:43.133 INFO 2981023 --- [ocp-async-2,c28acd74f7831381,72d8bc0645272166] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:43.553 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048120] 2024-09-11 17:21:43.913 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:43.913 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:43.966 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,b6069e2e8593e63a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048120, original=READY 2024-09-11 17:21:43.967 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,b6069e2e8593e63a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048120, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:44.058 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,14e568ae4011adb6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048120, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048120, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:43.966+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:44.072 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,14e568ae4011adb6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048120, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:44.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,9374e4a52fb672e4,fb260e060a1106ee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/properties?null, client=10.31.33.41, traceId=9374e4a52fb672e4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.PropertyController.findAllVisibleProperties(String,Pageable), args=Page request [number: 0, size 5000, sort: key: ASC],] 2024-09-11 17:21:44.128 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,9374e4a52fb672e4,fb260e060a1106ee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/properties client=10.31.33.41, traceId=9374e4a52fb672e4, duration=23 ms] 2024-09-11 17:21:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,155003c739daf76b,bdf3e9e662782632] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:44.317 INFO 2981023 --- [prometheus-minute-schedule-1,155003c739daf76b,bdf3e9e662782632] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:44.597 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048139] 2024-09-11 17:21:44.982 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,6596c997993a6b27] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048139, original=READY 2024-09-11 17:21:44.984 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,6596c997993a6b27] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048139, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:45.075 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,b2d79b9229608ba2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048139, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Pre check for install ob, zone2.idc=default_idc, sub_task_instance_id=4048139, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:44.982+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:45.079 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,b2d79b9229608ba2] c.o.o.s.t.b.cluster.InstallPreCheckTask : need to do nothing when rollback 2024-09-11 17:21:45.087 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,b2d79b9229608ba2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048139, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:45.101 INFO 2981023 --- [prometheus-second-schedule-1,fa58740795180d20,3ee8669f15b0d41b] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:21:45.133 INFO 2981023 --- [ocp-async-1,34f9e46b92e4fef6,e05a9ec661de75d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:45.641 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048150] 2024-09-11 17:21:45.995 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,95b96e07c8d0d9d8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048150, original=READY 2024-09-11 17:21:45.997 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,95b96e07c8d0d9d8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048150, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:46.022 INFO 2981023 --- [ocp-schedules-20,6c6255a73d2f26fd,4c71077e8a12a77d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:21:46.023 INFO 2981023 --- [host-check-clockdiff1,6c6255a73d2f26fd,b990118ae5507676] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:21:46.023 INFO 2981023 --- [host-check-clockdiff2,6c6255a73d2f26fd,4716cb5c08488930] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:21:46.026 INFO 2981023 --- [ocp-schedules-28,22a4cbcec8a119b1,9319900533fac14e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:21:46.030 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:21:46.036 INFO 2981023 --- [host-check-clockdiff3,6c6255a73d2f26fd,421d33d2e5c9571e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:21:46.037 INFO 2981023 --- [host-check-clockdiff4,6c6255a73d2f26fd,940d7eb59a2a33f2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:21:46.040 INFO 2981023 --- [ocp-schedules-7,63240e00e48da710,a8e58518c0781fee] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:21:46.040 INFO 2981023 --- [host-check-agent-status1,63240e00e48da710,ebcc990a90f88a95] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:21:46.041 INFO 2981023 --- [host-check-agent-status2,63240e00e48da710,b30e91a7696602b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:21:46.041 INFO 2981023 --- [host-check-agent-status3,63240e00e48da710,9026321aeb58c5c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:21:46.041 INFO 2981023 --- [host-check-agent-status4,63240e00e48da710,828ff4fddce89ecb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:21:46.044 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:21:46.046 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 17:21:46.049 INFO 2981023 --- [host-check-agent-status2,63240e00e48da710,b30e91a7696602b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:21:46.051 INFO 2981023 --- [host-check-agent-status1,63240e00e48da710,ebcc990a90f88a95] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:21:46.052 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:21:46.054 INFO 2981023 --- [host-check-agent-status1,63240e00e48da710,ebcc990a90f88a95] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:21:46.055 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:21:46.056 INFO 2981023 --- [host-check-agent-status3,63240e00e48da710,9026321aeb58c5c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:46.058 INFO 2981023 --- [host-check-agent-status2,63240e00e48da710,b30e91a7696602b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:21:46.063 INFO 2981023 --- [host-check-agent-status3,63240e00e48da710,9026321aeb58c5c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:46.064 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:21:46.066 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:21:46.067 INFO 2981023 --- [host-check-agent-status4,63240e00e48da710,828ff4fddce89ecb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:21:46.067 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:21:46.067 INFO 2981023 --- [host-check-clockdiff2,6c6255a73d2f26fd,4716cb5c08488930] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046506 0 0 , error= 2024-09-11 17:21:46.068 INFO 2981023 --- [host-check-clockdiff4,6c6255a73d2f26fd,940d7eb59a2a33f2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046506 0 0 , error= 2024-09-11 17:21:46.071 INFO 2981023 --- [host-check-agent-status4,63240e00e48da710,828ff4fddce89ecb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:21:46.081 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:46.085 INFO 2981023 --- [host-check-agent-status2,63240e00e48da710,b30e91a7696602b9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:21:46.093 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:46.100 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:21:46.103 INFO 2981023 --- [host-check-agent-status2,63240e00e48da710,b30e91a7696602b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:21:46.103 INFO 2981023 --- [host-check-agent-status2,63240e00e48da710,b30e91a7696602b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:21:46.103 INFO 2981023 --- [host-check-agent-status2,63240e00e48da710,b30e91a7696602b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:21:46.109 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:21:46.109 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:21:46.112 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:21:46.110141221+08:00 2024-09-11 17:21:46.122 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:21:46.112+08:00 2024-09-11 17:21:46.128 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:21:46.112+08:00 and status AVAILABLE 2024-09-11 17:21:46.137 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:21:46.160 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:21:46.165 INFO 2981023 --- [host-check-agent-status3,63240e00e48da710,9026321aeb58c5c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:21:46.170 INFO 2981023 --- [ocp-schedules-17,eb761178eda27fe9,a4fcd4329b4d02a4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:21:46.174 INFO 2981023 --- [host-check-agent-status3,63240e00e48da710,9026321aeb58c5c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:21:46.174 INFO 2981023 --- [host-check-agent-status3,63240e00e48da710,9026321aeb58c5c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:21:46.174 INFO 2981023 --- [host-check-agent-status3,63240e00e48da710,9026321aeb58c5c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:21:46.185 INFO 2981023 --- [ocp-schedules-17,eb761178eda27fe9,a4fcd4329b4d02a4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:21:46.187 INFO 2981023 --- [ocp-schedules-5,75cb4943c75604c3,a3fc215e006c5de1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:21:46.188 INFO 2981023 --- [ocp-schedules-16,d0d6284f9886bef9,771bf233380aae92] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:21:46.195 INFO 2981023 --- [ocp-schedules-5,75cb4943c75604c3,a3fc215e006c5de1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:21:46.195 INFO 2981023 --- [ocp-schedules-16,d0d6284f9886bef9,771bf233380aae92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:21:46.196 INFO 2981023 --- [ocp-schedules-24,f87314a9a7143261,d917c892c559a2ac] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:21:46.196 INFO 2981023 --- [ocp-schedules-2,e9eb2f43cb14c04b,5fcec9ce10d2d795] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:21:46.197 INFO 2981023 --- [host-check-agent-status4,63240e00e48da710,828ff4fddce89ecb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:21:46.168 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,b862a8e8fd12b002] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048150, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Reserve host, zone2.idc=default_idc, sub_task_instance_id=4048150, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:45.995+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:46.199 INFO 2981023 --- [ocp-schedules-24,f87314a9a7143261,d917c892c559a2ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:21:46.199 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:21:46.203 INFO 2981023 --- [host-check-agent-status4,63240e00e48da710,828ff4fddce89ecb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:21:46.203 INFO 2981023 --- [host-check-agent-status4,63240e00e48da710,828ff4fddce89ecb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:21:46.203 INFO 2981023 --- [host-check-agent-status4,63240e00e48da710,828ff4fddce89ecb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:21:46.203 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:21:46.223 INFO 2981023 --- [host-check-agent-status1,63240e00e48da710,ebcc990a90f88a95] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:21:46.225 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:21:46.226 INFO 2981023 --- [ocp-schedules-14,9e9cb678e6596f93,70c95b4e3b570287] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:21:46.226 INFO 2981023 --- [ocp-schedules-14,9e9cb678e6596f93,70c95b4e3b570287] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:21:46.226 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:21:46.225884540+08:00 2024-09-11 17:21:46.229 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:21:46.226+08:00 2024-09-11 17:21:46.231 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:21:46.231 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:21:46.226+08:00 and status AVAILABLE 2024-09-11 17:21:46.197 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,b862a8e8fd12b002] c.o.o.s.t.business.host.ReserveHostTask : unreserve host 2000008 2024-09-11 17:21:46.234 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:21:46.232718242+08:00 2024-09-11 17:21:46.240 INFO 2981023 --- [host-check-agent-status1,63240e00e48da710,ebcc990a90f88a95] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:21:46.240 INFO 2981023 --- [host-check-agent-status1,63240e00e48da710,ebcc990a90f88a95] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:21:46.240 INFO 2981023 --- [host-check-agent-status1,63240e00e48da710,ebcc990a90f88a95] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:21:46.241 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:21:46.234+08:00 2024-09-11 17:21:46.252 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:21:46.257 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:21:46.234+08:00 and status AVAILABLE 2024-09-11 17:21:46.259 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:21:46.259 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:21:46.259659998+08:00 2024-09-11 17:21:46.260 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:21:46.261 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:21:46.259+08:00 2024-09-11 17:21:46.264 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:21:46.259+08:00 and status AVAILABLE 2024-09-11 17:21:46.271 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.271 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:21:46.271 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:21:46.272 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:21:46.238 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,b862a8e8fd12b002] c.o.o.c.h.m.HostServiceManagerImpl : Removing service begin: service [OB_CLUSTER:linkrtd:1719994443], host [2000008:10.31.35.7] 2024-09-11 17:21:46.273 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:21:46.279 INFO 2981023 --- [ocp-schedules-18,eb155121c3e8d528,9b5bfb60fc94e656] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:21:46.281 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:21:46.281 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.283 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:46.289 INFO 2981023 --- [ocp-schedules-12,74ab763af738c025,7549b35e5ad83882] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:21:46.290 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:21:46.290 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:21:46.292 INFO 2981023 --- [ocp-schedules-12,74ab763af738c025,7549b35e5ad83882] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:21:46.298 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:21:46.272 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,b862a8e8fd12b002] c.o.o.c.h.m.HostServiceManagerImpl : Removing service: no services left on host [2000008:10.31.35.7], status changed from [ONLINE] to [AVAILABLE] 2024-09-11 17:21:46.309 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:21:46.318 INFO 2981023 --- [ocp-schedules-16,d0d6284f9886bef9,771bf233380aae92] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:21:46.322 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.327 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:21:46.328 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:21:46.329 INFO 2981023 --- [ocp-schedules-5,75cb4943c75604c3,a3fc215e006c5de1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:21:46.331 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:21:46.317 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,b862a8e8fd12b002] c.o.o.c.h.m.HostServiceManagerImpl : Removing service done: service [OB_CLUSTER:linkrtd:1719994443], host [2000008:10.31.35.7], current status [AVAILABLE] 2024-09-11 17:21:46.332 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:21:46.332 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:21:46.335 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:21:46.335 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:21:46.340 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:21:46.341 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.343 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:46.367 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.368 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.369 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:46.374 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.374 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:21:46.378 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,b862a8e8fd12b002] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048150, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:46.379 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:21:46.385 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:21:46.383834+08:00, lastProcessTime=2024-09-11T17:21:36.337612+08:00 2024-09-11 17:21:46.387 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.387 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.388 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:21:46.388 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:21:46.393 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8d05fc55085379bb,6f10acb25bd71d4d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=8d05fc55085379bb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:21:46.393 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8d05fc55085379bb,6f10acb25bd71d4d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=8d05fc55085379bb, duration=1 ms] 2024-09-11 17:21:46.394 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:21:46.394 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,b3ff81681ce9ac91,56ead19ff5be88ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=b3ff81681ce9ac91, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:21:46.394 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,b3ff81681ce9ac91,56ead19ff5be88ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=b3ff81681ce9ac91, duration=0 ms] 2024-09-11 17:21:46.396 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.396 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.396 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:21:46.399 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:21:46.399 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:21:46.400 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.400 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:21:46.401 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:21:46.401 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.404 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.404 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.407 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:21:46.407 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:21:46.411 INFO 2981023 --- [ocp-schedules-12,74ab763af738c025,7549b35e5ad83882] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:21:46.411 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:21:46.413 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:21:46.422 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.422 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.423 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:46.423 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:21:46.423 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.425 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:21:46.426 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:21:46.426 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:21:46.427 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:21:46.427 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:21:46.436 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:21:46.443 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:21:46.443 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:21:46.445 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:21:46.450 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 17:21:46.450 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:21:46.456 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:21:46.464 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:21:46.471 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:21:46.471 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:21:46.489 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:21:46.523 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:21:46.528 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:21:46.541 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.541 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.542 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:21:46.542 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:21:46.544 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:21:46.547 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:21:46.557 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:21:46.559 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:21:46.561 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:21:46.561 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.562 INFO 2981023 --- [pool-10-thread-9,565ab1f405efd8ac,54a3eb071247a263] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:21:46.563 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.564 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:21:46.564 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:21:46.565 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:21:46.565 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.566 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,25995f2f01054024,a5f3d3617d58ced2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tags?page=1&size=10, client=10.31.33.41, traceId=25995f2f01054024, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TagController.listTags(String,Boolean,Pageable), args=true,Page request [number: 0, size 10, sort: key: DESC,value: DESC],] 2024-09-11 17:21:46.587 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:21:46.592 INFO 2981023 --- [pool-10-thread-11,565ab1f405efd8ac,d0c45aea8dd6fcba] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:21:46.595 INFO 2981023 --- [pool-10-thread-9,565ab1f405efd8ac,54a3eb071247a263] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:21:46.595 INFO 2981023 --- [pool-10-thread-9,565ab1f405efd8ac,54a3eb071247a263] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:21:46.605 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:21:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:21:46.621 INFO 2981023 --- [pool-10-thread-12,565ab1f405efd8ac,647fcfb1b1ac3d62] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:21:46.627 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:21:46.638 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:21:46.641 INFO 2981023 --- [pool-10-thread-10,565ab1f405efd8ac,927805d76b40d39a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:21:46.642 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:21:46.644 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:21:46.649 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:21:46.649 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:21:46.652 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:21:46.652 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:21:46.673 INFO 2981023 --- [ForkJoinPool-6-worker-6,565ab1f405efd8ac,23dd246c5d2aa1a6] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:21:46.673 INFO 2981023 --- [ForkJoinPool-6-worker-6,565ab1f405efd8ac,23dd246c5d2aa1a6] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:21:46.677 INFO 2981023 --- [host-check-host-status2,22a4cbcec8a119b1,a33cd4120a72dee5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:21:46.679 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:21:46.679 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.681 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.682 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:21:46.687 INFO 2981023 --- [pool-10-thread-10,565ab1f405efd8ac,927805d76b40d39a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:21:46.692 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:21:46.703 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:21:46.713 WARN 2981023 --- [ForkJoinPool-6-worker-6,565ab1f405efd8ac,23dd246c5d2aa1a6] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:21:46.727 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:21:46.731 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.732 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:21:46.733 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.733 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:21:46.748 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.749 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:21:46.750 INFO 2981023 --- [host-check-host-status1,22a4cbcec8a119b1,6c3ee9faaec8c10d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:21:46.761 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.762 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.769 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:21:46.771 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.771 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.771 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:21:46.772 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:21:46.772 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:21:46.773 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:21:46.774 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.776 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.776 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.782 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,25995f2f01054024,a5f3d3617d58ced2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tags client=10.31.33.41, traceId=25995f2f01054024, duration=227 ms] 2024-09-11 17:21:46.782 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:21:46.783 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:21:46.784 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:21:46.784 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:21:46.789 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.790 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:21:46.793 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:21:46.793 INFO 2981023 --- [host-check-host-status3,22a4cbcec8a119b1,d4e09992abcf6297] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:21:46.794 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.801 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:21:46.805 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:21:46.805 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:21:46.808 INFO 2981023 --- [ForkJoinPool-6-worker-10,565ab1f405efd8ac,faf8a55f9f8d7782] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.810 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.810 INFO 2981023 --- [ForkJoinPool-6-worker-10,565ab1f405efd8ac,faf8a55f9f8d7782] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:21:46.811 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:21:46.812 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:21:46.827 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:21:46.829 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:21:46.834 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.835 INFO 2981023 --- [ocp-schedules-15,48051322d5de3e92,fb60ca80c0fc3031] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2050,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:21:46.835 INFO 2981023 --- [ocp-schedules-15,48051322d5de3e92,fb60ca80c0fc3031] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:21:46.835 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:21:46, 2024-09-11 17:21:46] 2024-09-11 17:21:46.836 INFO 2981023 --- [host-check-host-status4,22a4cbcec8a119b1,ca4baf52fd0739ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:21:46.837 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:21:46.837 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:21:46.837 INFO 2981023 --- [ForkJoinPool-6-worker-15,565ab1f405efd8ac,5bfdd8c99dffaaf7] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:21:46.843 INFO 2981023 --- [ForkJoinPool-6-worker-10,565ab1f405efd8ac,faf8a55f9f8d7782] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:21:46.844 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.844 INFO 2981023 --- [ForkJoinPool-6-worker-10,565ab1f405efd8ac,faf8a55f9f8d7782] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.844 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.849 INFO 2981023 --- [ForkJoinPool-6-worker-10,565ab1f405efd8ac,faf8a55f9f8d7782] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:21:46.849 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.849 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:21:46.849 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:21:46.849 INFO 2981023 --- [ForkJoinPool-6-worker-10,565ab1f405efd8ac,faf8a55f9f8d7782] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.850 INFO 2981023 --- [ForkJoinPool-6-worker-10,565ab1f405efd8ac,faf8a55f9f8d7782] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:21:46.851 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:21:46.851 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.852 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.852 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.853 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:21:46.853 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:21:46.853 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:21:46.854 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.854 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:21:46.855 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.855 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:21:46.855 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:21:46.865 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:21:46.865 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:21:46.865 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.865 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:21:46.866 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:21:46.867 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:21:46.871 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:21:46.872 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:21:46.893 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:21:46.893 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.893 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.894 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:21:46, 2024-09-11 17:21:46] 2024-09-11 17:21:46.895 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:21:46.896 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:21:46.897 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.900 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.900 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:21:46.901 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:21:46.901 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.904 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.909 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.914 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:46.914 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:21:46.916 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:46.917 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:21:46.919 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:21:46.919 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:21:46.920 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.920 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.921 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:21:46, 2024-09-11 17:21:46] 2024-09-11 17:21:46.922 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:21:46.922 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:21:46.922 INFO 2981023 --- [ForkJoinPool-6-worker-13,565ab1f405efd8ac,d3350c3fae3e503c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:21:46.923 INFO 2981023 --- [pool-10-thread-11,565ab1f405efd8ac,d0c45aea8dd6fcba] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:21:46.923 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048137] 2024-09-11 17:21:46.923 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.924 INFO 2981023 --- [ForkJoinPool-6-worker-8,565ab1f405efd8ac,a0f97dca09344a34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:21:46.925 INFO 2981023 --- [pool-10-thread-12,565ab1f405efd8ac,647fcfb1b1ac3d62] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:21:46.930 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:21:46.938 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.939 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.939 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:21:46.939 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:21:46.949 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:21:46.949 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:21:46.950 INFO 2981023 --- [ocp-schedules-24,f87314a9a7143261,d917c892c559a2ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:21:46.954 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:46.955 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.955 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:21:46.956 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:21:46.956 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:46.998 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:21:46.383834+08:00 2024-09-11 17:21:47.005 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:21:47.006 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,0614e4369ee7c2f2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048137, original=READY 2024-09-11 17:21:47.008 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,0614e4369ee7c2f2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048137, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:47.023 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:21:47.031 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:21:47.035 INFO 2981023 --- [ocp-schedules-3,b39c73b4ec0224a4,df08e62b8af60829] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:21:47.052 INFO 2981023 --- [pool-10-thread-10,565ab1f405efd8ac,927805d76b40d39a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:21:47.057 INFO 2981023 --- [pool-10-thread-10,565ab1f405efd8ac,927805d76b40d39a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:21:47.103 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,55f6426df7af7e07] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048137, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048137, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:47.006+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:47.112 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,55f6426df7af7e07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048137, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:47.133 INFO 2981023 --- [ocp-async-3,a9eabccbd6c5bbe5,0770338bae999c17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,7f928b1972044bce,325b58e9eed8f06c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,7f928b1972044bce,325b58e9eed8f06c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:47.735 INFO 2981023 --- [ocp-async-4,7fddc964c9158082,a5ecfce2f4c9d5bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:21:47.975 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048141] 2024-09-11 17:21:48.023 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9eb7ef6a70e8b4a5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048141, original=READY 2024-09-11 17:21:48.024 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9eb7ef6a70e8b4a5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048141, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:48.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,9fe0e93dd694c16d,f3fdd2f5c1a95e2b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/dashboards?null, client=10.31.33.41, traceId=9fe0e93dd694c16d, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorDashboardsController.queryMonitorDashboards(Pageable), args=Page request [number: 0, size 5000, sort: order: ASC],] 2024-09-11 17:21:48.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,9fe0e93dd694c16d,f3fdd2f5c1a95e2b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/dashboards client=10.31.33.41, traceId=9fe0e93dd694c16d, duration=15 ms] 2024-09-11 17:21:48.113 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,a5b03def8e4acd6b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048141, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Check ob server network, zone2.idc=default_idc, sub_task_instance_id=4048141, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:48.023+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:48.127 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,a5b03def8e4acd6b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048141, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:48.576 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,442f88dad9bc7cd5,08dfa2ec0d3765e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/obproxy/clusters?null, client=10.31.33.41, traceId=442f88dad9bc7cd5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObproxyController.listObproxyClusters(Pageable,List,String,List,Boolean), args=Page request [number: 0, size 5000, sort: id: ASC],false,] 2024-09-11 17:21:48.577 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05807f180e78bcdd,9a7e583d9b840fed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.31.33.41, traceId=05807f180e78bcdd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:21:48.578 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,bd1fa36c7767dd3d,8d08a7ccfbf980d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.31.33.41, traceId=bd1fa36c7767dd3d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 17:21:48.578 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05807f180e78bcdd,9a7e583d9b840fed] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:21:48.581 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,70537844b94a7ce6,0bd4b2a1e949d65f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?null, client=10.31.33.41, traceId=70537844b94a7ce6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List,WithRemoteTenant,List), args=Page request [number: 0, size 5000, sort: id: DESC],NONE,] 2024-09-11 17:21:48.586 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,442f88dad9bc7cd5,08dfa2ec0d3765e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/obproxy/clusters client=10.31.33.41, traceId=442f88dad9bc7cd5, duration=13 ms] 2024-09-11 17:21:48.593 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,bd1fa36c7767dd3d,8d08a7ccfbf980d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=bd1fa36c7767dd3d, duration=17 ms] 2024-09-11 17:21:48.604 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,70537844b94a7ce6,0bd4b2a1e949d65f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=10.31.33.41, traceId=70537844b94a7ce6, duration=26 ms] 2024-09-11 17:21:48.618 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05807f180e78bcdd,9a7e583d9b840fed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=05807f180e78bcdd, duration=43 ms] 2024-09-11 17:21:48.758 INFO 2981023 --- [ocp-schedules-16,5cffd8194b37a3c5,b5254df894177d5e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:21:48.811 WARN 2981023 --- [scanner-13468187-1,5cffd8194b37a3c5,73eed4de2eeb5057] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:21:48.822 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,55ee2d0ed43b73be,bdf9a3e7be68ee38] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?null, client=10.31.33.41, traceId=55ee2d0ed43b73be, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List,WithRemoteTenant,List), args=Page request [number: 0, size 5000, sort: id: DESC],NONE,] 2024-09-11 17:21:48.822 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,987b9e66b4ac5e0e,6e1e0230f3ef77ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/templates?null, client=10.31.33.41, traceId=987b9e66b4ac5e0e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.getAlarmTemplates(String,String,Boolean,Boolean,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:21:48.841 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,55ee2d0ed43b73be,bdf9a3e7be68ee38] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=10.31.33.41, traceId=55ee2d0ed43b73be, duration=24 ms] 2024-09-11 17:21:48.848 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,bb68a4c4e11357c9,a6317a54fb1dcf6c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/groups?null, client=10.31.33.41, traceId=bb68a4c4e11357c9, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarmGroups(Integer,Integer), args=] 2024-09-11 17:21:48.855 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,987b9e66b4ac5e0e,6e1e0230f3ef77ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/templates client=10.31.33.41, traceId=987b9e66b4ac5e0e, duration=41 ms] 2024-09-11 17:21:48.887 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,bb68a4c4e11357c9,a6317a54fb1dcf6c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/groups client=10.31.33.41, traceId=bb68a4c4e11357c9, duration=63 ms] 2024-09-11 17:21:49.024 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,9d3e4cf1afbe19b7,58b3edf9fc476667] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?page=1&size=10&collapsed=false, client=10.31.33.41, traceId=9d3e4cf1afbe19b7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:21:49.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 17:21:49.100 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/arbitration/services, client=10.31.33.41, traceId=] 2024-09-11 17:21:49.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 17:21:49.120 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[4048135] 2024-09-11 17:21:49.136 INFO 2981023 --- [ocp-async-2,0c9b93c56f3c20f4,57e7324124534fb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:49.246 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,9d3e4cf1afbe19b7,58b3edf9fc476667] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.31.33.41, traceId=9d3e4cf1afbe19b7, duration=244 ms] 2024-09-11 17:21:49.469 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ddd9f109e0d06ea6,5cde377631629554] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.31.33.41, traceId=ddd9f109e0d06ea6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:21:49.472 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ddd9f109e0d06ea6,5cde377631629554] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.31.33.41, traceId=ddd9f109e0d06ea6, duration=5 ms] 2024-09-11 17:21:49.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6451b79aeaefd248,9244a9fd9146a563] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/regions?null, client=10.31.33.41, traceId=6451b79aeaefd248, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listRegions(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:21:49.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0f5fca93068c720b,e2aba3a3e3ec4bd7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=0f5fca93068c720b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:21:49.482 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6451b79aeaefd248,9244a9fd9146a563] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/regions client=10.31.33.41, traceId=6451b79aeaefd248, duration=4 ms] 2024-09-11 17:21:49.483 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0f5fca93068c720b,e2aba3a3e3ec4bd7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=0f5fca93068c720b, duration=6 ms] 2024-09-11 17:21:49.515 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f0555a3cbc2b6daa,eeb419c3c740299a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=f0555a3cbc2b6daa, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:21:49.517 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f0555a3cbc2b6daa,eeb419c3c740299a] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:21:49.527 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f0555a3cbc2b6daa,eeb419c3c740299a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=f0555a3cbc2b6daa, duration=14 ms] 2024-09-11 17:21:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:21:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:21:49.916 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:49.918 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:50.065 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,b3b012fc65e676d2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=4048135, original=READY 2024-09-11 17:21:50.066 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,b3b012fc65e676d2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048135, operation:ROLLBACK, state: ROLLING_BACK 2024-09-11 17:21:50.148 WARN 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,952d4e6ff206b32a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=4048135, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048171, task_operation=rollback, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, host.2000008.zone=zone2, sub_task_instance_name=Prepare add zone, zone2.idc=default_idc, sub_task_instance_id=4048135, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:21:50.065+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000028], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000008], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, operation=ROLLBACK, executor=10.31.35.10 2024-09-11 17:21:50.152 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,952d4e6ff206b32a] c.o.o.s.t.b.zone.PrepareAddObZoneTask : begin to rollback 2024-09-11 17:21:50.162 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,952d4e6ff206b32a] c.o.o.s.t.b.zone.PrepareAddObZoneTask : clusterId=1000006, set status to RUNNING 2024-09-11 17:21:50.173 INFO 2981023 --- [host-check-clockdiff1,6c6255a73d2f26fd,b990118ae5507676] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046510 0 0 , error= 2024-09-11 17:21:50.190 INFO 2981023 --- [ocp-schedules-11,94b7e98204d1f812,115c8f7e7ca00fa6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:21:50.191 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,952d4e6ff206b32a] c.o.o.s.t.b.zone.PrepareAddObZoneTask : rollback finished. 2024-09-11 17:21:50.200 INFO 2981023 --- [manual-subtask-executor16,271ff184e32b2c91,952d4e6ff206b32a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048135, operation:ROLLBACK, state: PENDING 2024-09-11 17:21:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,f98c737cffc53a76,849e4b5d1f486331] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,f98c737cffc53a76,849e4b5d1f486331] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:50.418 INFO 2981023 --- [ocp-schedules-14,b0b199fd9554f9ef,cd88e31f9a45de0c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:21:50.418+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:21:50.421 INFO 2981023 --- [ocp-schedules-14,b0b199fd9554f9ef,cd88e31f9a45de0c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:21:50.535 INFO 2981023 --- [ocp-schedules-14,b0b199fd9554f9ef,cd88e31f9a45de0c] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 41 to SqlAggregateData 2024-09-11 17:21:50.537 INFO 2981023 --- [ocp-schedules-14,b0b199fd9554f9ef,cd88e31f9a45de0c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:21:50.850 INFO 2981023 --- [ocp-schedules-23,e4deb197aa3626c8,ee16397511227055] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:21:50.851 INFO 2981023 --- [ocp-schedules-23,e4deb197aa3626c8,ee16397511227055] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:21:50.886 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:21:50.886 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.888 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:21:50.888 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:21:50.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.889 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:21:50.892 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:21:50.892 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:21:50.895 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.895 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.896 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.897 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.897 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.897 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.901 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.902 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.907 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.907 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.908 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.908 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:21:50.911 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.911 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:21:50.920 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:21:50.922 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.922 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.922 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:21:50.923 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE19D1D726-0-0] 2024-09-11 17:21:50.923 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:21:50.923 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.923 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:21:50.925 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.925 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.926 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.926 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:21:50.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:21:50.938 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.938 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.939 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:21:50.939 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:21:50.939 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.940 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.940 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:21:50.940 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:21:50.941 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.941 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.941 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:21:50.942 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.942 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:21:50.942 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:21:50.942 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:21:50.943 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.943 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:21:50.944 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.944 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:21:50.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:21:50.946 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.946 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.946 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:21:50.946 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:21:50.947 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:21:50.947 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.947 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE19D1D730-0-0] 2024-09-11 17:21:50.947 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:21:50.947 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:21:50.947 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:21:50.950 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:21:50.950 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:21:50.954 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.955 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:21:50.955 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.956 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:21:50.956 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.957 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:21:50.957 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.958 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:21:50.958 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.959 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:21:50.959 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.960 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:21:50.960 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:21:50.962 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:50.962 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:21:50.962 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:50.963 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.964 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.964 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.964 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:21:50.971 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:21:50.972 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:21:50.973 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.974 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:21:50.974 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:21:50.974 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.974 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:21:50.976 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:21:50.978 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:21:50.978 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:21:50.978 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:21:50.980 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:21:50.982 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:21:50.982 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.982 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.984 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:21:50.986 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:21:50.986 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:50.987 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:21:50.987 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:21:50.989 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:21:50.989 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:21:50.989 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:21:50.994 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:21:50.995 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:21:50.995 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:21:50.995 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:21:50.996 INFO 2981023 --- [ocp-schedules-23,e4deb197aa3626c8,ee16397511227055] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:21:51.136 INFO 2981023 --- [ocp-async-1,8c90278115e280c1,cfa39b43fde7feab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:51.232 WARN 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:146) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:144) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:104) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:77) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:21:51.493 INFO 2981023 --- [ocp-schedules-19,47ccf284d0bf1996,b6af157cc05c7993] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:21:51.494 INFO 2981023 --- [ocp-schedules-19,47ccf284d0bf1996,b6af157cc05c7993] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:21:51.495 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:21:51.495 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:21:51.495 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:21:51.495 INFO 2981023 --- [ocp-schedules-19,47ccf284d0bf1996,b6af157cc05c7993] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:21:51.691 INFO 2981023 --- [ocp-schedules-11,f2ecfb53003b985b,9aa0eab6409b4f21] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:21:51.991 INFO 2981023 --- [host-check-clockdiff3,6c6255a73d2f26fd,421d33d2e5c9571e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046511 1 1 , error= 2024-09-11 17:21:52.000 INFO 2981023 --- [ocp-schedules-9,9bffcd0f30857912,801468a072c836af] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:21:52.018 WARN 2981023 --- [ocp-schedules-9,9bffcd0f30857912,801468a072c836af] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:21:52.918 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:52.920 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:52.997 INFO 2981023 --- [ocp-schedules-16,d887480b941afa01,9c110dba720d127e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:21:52.997 INFO 2981023 --- [ocp-schedules-16,d887480b941afa01,9c110dba720d127e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:21:53.136 INFO 2981023 --- [ocp-async-5,29c32f4487fad3ec,70d529d58ba1916c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,988125d551cde4af,ca4afddf852b2bc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,988125d551cde4af,ca4afddf852b2bc5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:53.384 INFO 2981023 --- [ocp-schedules-1,9b728425e12b662f,f9fa7a15ca69ecd2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:21:53.395 INFO 2981023 --- [ocp-schedules-1,9b728425e12b662f,f9fa7a15ca69ecd2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:21:53.395 INFO 2981023 --- [ocp-schedules-1,9b728425e12b662f,f9fa7a15ca69ecd2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:21:54.144 INFO 2981023 --- [ocp-schedules-19,b986ed830b9d4d08,27b1bfe897015a40] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:21:54.647 INFO 2981023 --- [ocp-schedules-5,3e6584a357e5165e,0d09c0a5f1ecc032] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:21:54.647 INFO 2981023 --- [ocp-schedules-5,3e6584a357e5165e,0d09c0a5f1ecc032] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:21:54.652 INFO 2981023 --- [ocp-schedules-5,3e6584a357e5165e,0d09c0a5f1ecc032] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:21:54.652 INFO 2981023 --- [ocp-schedules-5,3e6584a357e5165e,0d09c0a5f1ecc032] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:21:54.652 INFO 2981023 --- [ocp-schedules-5,3e6584a357e5165e,0d09c0a5f1ecc032] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:21:54.662 INFO 2981023 --- [ocp-schedules-5,3e6584a357e5165e,0d09c0a5f1ecc032] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:21:54.662 INFO 2981023 --- [ocp-schedules-5,3e6584a357e5165e,0d09c0a5f1ecc032] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:21:54.662 INFO 2981023 --- [ocp-schedules-5,3e6584a357e5165e,0d09c0a5f1ecc032] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:21:54.673 INFO 2981023 --- [ocp-schedules-5,3e6584a357e5165e,0d09c0a5f1ecc032] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:21:55.136 INFO 2981023 --- [ocp-async-6,60d21ffc0ab76fa9,f0813d7da2481abe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:55.528 INFO 2981023 --- [ocp-schedules-7,7a08c40a09b14987,7be9efd13d1258a8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:21:55.528 INFO 2981023 --- [ocp-schedules-7,7a08c40a09b14987,7be9efd13d1258a8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:21:55.548 INFO 2981023 --- [Collect all tenants major compactions2,7a08c40a09b14987,d663fb37a8281401] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:21:55.548 INFO 2981023 --- [Collect all tenants major compactions3,7a08c40a09b14987,879553c82fc8543e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:21:55.549 INFO 2981023 --- [Collect all tenants major compactions1,7a08c40a09b14987,c48b49a0baf58b59] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:21:55.549 INFO 2981023 --- [Collect all tenants major compactions4,7a08c40a09b14987,65e3fd0b91595a2a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:21:55.550 INFO 2981023 --- [Collect all tenants major compactions6,7a08c40a09b14987,820278fc063f75bf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:21:55.551 INFO 2981023 --- [Collect all tenants major compactions5,7a08c40a09b14987,22600725c718108f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:21:55.567 INFO 2981023 --- [Collect all tenants major compactions4,7a08c40a09b14987,65e3fd0b91595a2a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:55.567 INFO 2981023 --- [Collect all tenants major compactions6,7a08c40a09b14987,820278fc063f75bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:55.567 INFO 2981023 --- [Collect all tenants major compactions2,7a08c40a09b14987,d663fb37a8281401] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:55.568 INFO 2981023 --- [Collect all tenants major compactions1,7a08c40a09b14987,c48b49a0baf58b59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:55.568 INFO 2981023 --- [Collect all tenants major compactions4,7a08c40a09b14987,65e3fd0b91595a2a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:21:55.568 INFO 2981023 --- [Collect all tenants major compactions2,7a08c40a09b14987,d663fb37a8281401] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:21:55.568 INFO 2981023 --- [Collect all tenants major compactions5,7a08c40a09b14987,22600725c718108f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:55.568 INFO 2981023 --- [Collect all tenants major compactions6,7a08c40a09b14987,820278fc063f75bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:21:55.569 INFO 2981023 --- [Collect all tenants major compactions3,7a08c40a09b14987,879553c82fc8543e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:21:55.570 INFO 2981023 --- [Collect all tenants major compactions5,7a08c40a09b14987,22600725c718108f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:21:55.570 INFO 2981023 --- [Collect all tenants major compactions3,7a08c40a09b14987,879553c82fc8543e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:21:55.570 INFO 2981023 --- [Collect all tenants major compactions1,7a08c40a09b14987,c48b49a0baf58b59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:21:55.572 INFO 2981023 --- [Collect all tenants major compactions4,7a08c40a09b14987,65e3fd0b91595a2a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:21:55.572 INFO 2981023 --- [Collect all tenants major compactions5,7a08c40a09b14987,22600725c718108f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:21:55.573 INFO 2981023 --- [Collect all tenants major compactions3,7a08c40a09b14987,879553c82fc8543e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:21:55.573 INFO 2981023 --- [Collect all tenants major compactions1,7a08c40a09b14987,c48b49a0baf58b59] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:21:55.574 INFO 2981023 --- [Collect all tenants major compactions6,7a08c40a09b14987,820278fc063f75bf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:21:55.574 INFO 2981023 --- [Collect all tenants major compactions2,7a08c40a09b14987,d663fb37a8281401] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:21:55.577 INFO 2981023 --- [Collect all tenants major compactions4,7a08c40a09b14987,65e3fd0b91595a2a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:21:55.579 INFO 2981023 --- [Collect all tenants major compactions2,7a08c40a09b14987,d663fb37a8281401] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:21:55.580 INFO 2981023 --- [Collect all tenants major compactions3,7a08c40a09b14987,879553c82fc8543e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:21:55.581 INFO 2981023 --- [Collect all tenants major compactions1,7a08c40a09b14987,c48b49a0baf58b59] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:21:55.582 INFO 2981023 --- [Collect all tenants major compactions5,7a08c40a09b14987,22600725c718108f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:21:55.583 INFO 2981023 --- [Collect all tenants major compactions6,7a08c40a09b14987,820278fc063f75bf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:21:55.583 INFO 2981023 --- [ocp-schedules-7,7a08c40a09b14987,7be9efd13d1258a8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:21:55.610 INFO 2981023 --- [ocp-schedules-19,6f6cee67f5c26621,d54156abf938ceda] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:21:55.610 INFO 2981023 --- [ocp-schedules-19,6f6cee67f5c26621,d54156abf938ceda] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:21:55.613 INFO 2981023 --- [ocp-schedules-19,6f6cee67f5c26621,d54156abf938ceda] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:21:55.613 INFO 2981023 --- [ocp-schedules-19,6f6cee67f5c26621,d54156abf938ceda] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:21:55.615 INFO 2981023 --- [ocp-schedules-19,6f6cee67f5c26621,d54156abf938ceda] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:21:55.615 INFO 2981023 --- [ocp-schedules-19,6f6cee67f5c26621,d54156abf938ceda] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:21:55.655 INFO 2981023 --- [ocp-schedules-27,8c8d22f1d7500207,38809d996181b4a2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:21:55.662 INFO 2981023 --- [ocp-schedules-27,8c8d22f1d7500207,38809d996181b4a2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:21:55.662 INFO 2981023 --- [ocp-schedules-27,8c8d22f1d7500207,38809d996181b4a2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:21:55.920 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:55.920 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,454c810b8dffff25,00009462118078cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,454c810b8dffff25,00009462118078cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:56.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:21:56.353764+08:00, lastProcessTime=2024-09-11T17:21:46.383834+08:00 2024-09-11 17:21:56.484 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:21:56.353764+08:00 2024-09-11 17:21:57.136 INFO 2981023 --- [ocp-async-8,3e8d03c1e838e68d,69a66e84d19250ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:57.736 INFO 2981023 --- [ocp-async-2,535b7a367d69a599,6ae790d441def74b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:21:58.920 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:21:58.921 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:21:59.136 INFO 2981023 --- [ocp-async-5,e191885fc81ccd4d,87292a4673b12b9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:21:59.140 INFO 2981023 --- [ocp-schedules-5,f17cf9f8ba6c59d3,8bbd0727e2511139] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:21:59.176 WARN 2981023 --- [scanner-13468376-1,f17cf9f8ba6c59d3,32fe4d336de04b74] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:21:59.217 INFO 2981023 --- [alarm-detect-10,f17cf9f8ba6c59d3,ef86dcc7e6bff545] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643622, data is empty. 2024-09-11 17:21:59.220 INFO 2981023 --- [alarm-detect-10,f17cf9f8ba6c59d3,ef86dcc7e6bff545] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643663, data is empty. 2024-09-11 17:21:59.223 INFO 2981023 --- [alarm-detect-10,f17cf9f8ba6c59d3,ef86dcc7e6bff545] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643663, data is empty. 2024-09-11 17:21:59.235 INFO 2981023 --- [alarm-detect-10,f17cf9f8ba6c59d3,cd24252bffda256d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643622, data is empty. 2024-09-11 17:21:59.236 INFO 2981023 --- [alarm-detect-15,f17cf9f8ba6c59d3,226797932858aff6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643622, data is empty. 2024-09-11 17:21:59.239 INFO 2981023 --- [alarm-detect-10,f17cf9f8ba6c59d3,cd24252bffda256d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643663, data is empty. 2024-09-11 17:21:59.240 INFO 2981023 --- [alarm-detect-15,f17cf9f8ba6c59d3,226797932858aff6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643663, data is empty. 2024-09-11 17:21:59.246 INFO 2981023 --- [alarm-detect-10,f17cf9f8ba6c59d3,cd24252bffda256d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643663, data is empty. 2024-09-11 17:21:59.247 INFO 2981023 --- [alarm-detect-15,f17cf9f8ba6c59d3,226797932858aff6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643663, data is empty. 2024-09-11 17:21:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,32cf02d250a373a5,904e984a636b1ec0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046520, interval=60 2024-09-11 17:21:59.321 INFO 2981023 --- [prometheus-minute-schedule-1,32cf02d250a373a5,904e984a636b1ec0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:21:59.372 INFO 2981023 --- [alarm-detect-19,f17cf9f8ba6c59d3,028bc3011ee3b046] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643622, data is empty. 2024-09-11 17:21:59.376 INFO 2981023 --- [alarm-detect-19,f17cf9f8ba6c59d3,028bc3011ee3b046] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643663, data is empty. 2024-09-11 17:21:59.378 INFO 2981023 --- [alarm-detect-19,f17cf9f8ba6c59d3,028bc3011ee3b046] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643663, data is empty. 2024-09-11 17:22:01.136 INFO 2981023 --- [ocp-async-7,e5c5355408e921b2,0d596a281f761c6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:01.693 INFO 2981023 --- [ocp-schedules-28,8f39a5cadbee396c,735a92487f72adaa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:22:01.921 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:01.922 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:01.951 INFO 2981023 --- [ocp-schedules-12,67e33db736d801f0,d475b94732bd772d] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:22:01.951 INFO 2981023 --- [ocp-schedules-12,67e33db736d801f0,d475b94732bd772d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:22:02.195 INFO 2981023 --- [ocp-schedules-25,7fa4bbffac4e188b,2ea5156cc44089ba] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:22:02.196 INFO 2981023 --- [ocp-schedules-25,7fa4bbffac4e188b,2ea5156cc44089ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:02.248 INFO 2981023 --- [ocp-schedules-25,7fa4bbffac4e188b,2ea5156cc44089ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:22:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,c4b447c37aa9e38a,47a41e0971f16206] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:02.331 INFO 2981023 --- [prometheus-minute-schedule-1,c4b447c37aa9e38a,47a41e0971f16206] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:03.136 INFO 2981023 --- [ocp-async-1,56ea058f11f4d8df,5620285787e52820] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:03.538 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f03c9b5fc7b9794d,035de855b98a93c5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?status=OFFLINE&includedServiceType=OB_PROXY, client=10.31.33.41, traceId=f03c9b5fc7b9794d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[OFFLINE],[OB_PROXY],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:22:03.542 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f03c9b5fc7b9794d,035de855b98a93c5] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:22:03.547 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,44791d727f056129,9d04c0368999bcc1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=ODP&type=TOP&maxPoints=360, client=10.31.33.41, traceId=44791d727f056129, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,ODP,] 2024-09-11 17:22:03.555 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,8ddba43d19298924,8761d1bd70cf453a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=8ddba43d19298924, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:22:03.556 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,8ddba43d19298924,8761d1bd70cf453a] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:22:03.565 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f03c9b5fc7b9794d,035de855b98a93c5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=f03c9b5fc7b9794d, duration=31 ms] 2024-09-11 17:22:03.567 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,44791d727f056129,9d04c0368999bcc1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=44791d727f056129, duration=23 ms] 2024-09-11 17:22:03.577 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05d906a8fced8043,c135885a9b6300d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/obproxy/clusters?page=1&size=10, client=10.31.33.41, traceId=05d906a8fced8043, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObproxyController.listObproxyClusters(Pageable,List,String,List,Boolean), args=Page request [number: 0, size 10, sort: id: ASC],false,] 2024-09-11 17:22:03.590 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05d906a8fced8043,c135885a9b6300d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/obproxy/clusters client=10.31.33.41, traceId=05d906a8fced8043, duration=16 ms] 2024-09-11 17:22:03.597 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,8ddba43d19298924,8761d1bd70cf453a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=8ddba43d19298924, duration=47 ms] 2024-09-11 17:22:03.972 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,65c3390bf9ae9f31,06fbe8474fccc38a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-09-11T16%3A22%3A03%2B08%3A00&endTime=2024-09-11T17%3A22%3A03%2B08%3A00&metrics=current_session_server%2Cserver_connection_used_percent&queryAsync=true&labels=app%3AODP&groupBy=app%2Cobproxy_cluster&limit=5&maxPoints=360, client=10.31.33.41, traceId=65c3390bf9ae9f31, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:22:03+08:00,2024-09-11T17:22:03+08:00,current_session_server,server_connection_used_percent,app,obproxy_cluster,0,0,5,360,app:ODP,true,] 2024-09-11 17:22:03.972 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-6,65c3390bf9ae9f31,06fbe8474fccc38a] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = current_session_server, labels = app="ODP", groupBy = app,obproxy_cluster, interval = 30 2024-09-11 17:22:03.976 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,65c3390bf9ae9f31,06fbe8474fccc38a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=65c3390bf9ae9f31, duration=6 ms] 2024-09-11 17:22:03.987 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b9ecc44a77dc60f6,1096a83afd5a9f3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-09-11T16%3A22%3A03%2B08%3A00&endTime=2024-09-11T17%3A22%3A03%2B08%3A00&metrics=current_session_client%2Cclient_connection_used_percent&queryAsync=true&labels=app%3AODP&groupBy=app%2Cobproxy_cluster&limit=5&maxPoints=360, client=10.31.33.41, traceId=b9ecc44a77dc60f6, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:22:03+08:00,2024-09-11T17:22:03+08:00,current_session_client,client_connection_used_percent,app,obproxy_cluster,0,0,5,360,app:ODP,true,] 2024-09-11 17:22:03.987 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b9ecc44a77dc60f6,1096a83afd5a9f3c] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = current_session_client, labels = app="ODP", groupBy = app,obproxy_cluster, interval = 30 2024-09-11 17:22:03.989 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b9ecc44a77dc60f6,1096a83afd5a9f3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=b9ecc44a77dc60f6, duration=4 ms] 2024-09-11 17:22:04.013 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,468e7f2aa4a6401e,60689411ee8aa5ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-09-11T16%3A22%3A03%2B08%3A00&endTime=2024-09-11T17%3A22%3A03%2B08%3A00&metrics=request_total&queryAsync=true&labels=app%3AODP&groupBy=app%2Cobproxy_cluster&limit=5&maxPoints=360, client=10.31.33.41, traceId=468e7f2aa4a6401e, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:22:03+08:00,2024-09-11T17:22:03+08:00,request_total,app,obproxy_cluster,0,0,5,360,app:ODP,true,] 2024-09-11 17:22:04.013 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-4,468e7f2aa4a6401e,60689411ee8aa5ae] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = request_total, labels = app="ODP", groupBy = app,obproxy_cluster, interval = 30 2024-09-11 17:22:04.014 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,468e7f2aa4a6401e,60689411ee8aa5ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=468e7f2aa4a6401e, duration=3 ms] 2024-09-11 17:22:04.051 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,400774c52297cf2e,cd25e7aa8c7da76e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-09-11T16%3A22%3A03%2B08%3A00&endTime=2024-09-11T17%3A22%3A03%2B08%3A00&metrics=transaction_total&queryAsync=true&labels=app%3AODP&groupBy=app%2Cobproxy_cluster&limit=5&maxPoints=360, client=10.31.33.41, traceId=400774c52297cf2e, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:22:03+08:00,2024-09-11T17:22:03+08:00,transaction_total,app,obproxy_cluster,0,0,5,360,app:ODP,true,] 2024-09-11 17:22:04.051 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-9,400774c52297cf2e,cd25e7aa8c7da76e] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = transaction_total, labels = app="ODP", groupBy = app,obproxy_cluster, interval = 30 2024-09-11 17:22:04.052 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,400774c52297cf2e,cd25e7aa8c7da76e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=400774c52297cf2e, duration=3 ms] 2024-09-11 17:22:04.922 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:04.923 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:05.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,1a6eb52bc48eef0f,e3b0ca5f889a294e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&type=TOP&maxPoints=360, client=10.31.33.41, traceId=1a6eb52bc48eef0f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,TENANT,] 2024-09-11 17:22:05.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,1a6eb52bc48eef0f,e3b0ca5f889a294e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=1a6eb52bc48eef0f, duration=7 ms] 2024-09-11 17:22:05.136 INFO 2981023 --- [ocp-async-4,5719267cf0e68853,3b6e019f51a0fa9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:05.143 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,d43a9c8be115a3f4,48a3b6c98d924bd5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenantsView?page=1&size=10, client=10.31.33.41, traceId=d43a9c8be115a3f4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenantsView(Pageable,String,List,Boolean,String,Boolean,List,List), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:22:05.162 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,d43a9c8be115a3f4,48a3b6c98d924bd5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenantsView client=10.31.33.41, traceId=d43a9c8be115a3f4, duration=22 ms] 2024-09-11 17:22:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,86c6232488ec2d68,5b44a70be9f33e59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,86c6232488ec2d68,5b44a70be9f33e59] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:05.493 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b00dcd6988124b51,dff9460e5359585d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-09-11T16%3A22%3A05%2B08%3A00&endTime=2024-09-11T17%3A22%3A05%2B08%3A00&metrics=tps&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.31.33.41, traceId=b00dcd6988124b51, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:22:05+08:00,2024-09-11T17:22:05+08:00,tps,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2024-09-11 17:22:05.519 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b00dcd6988124b51,dff9460e5359585d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=b00dcd6988124b51, duration=28 ms] 2024-09-11 17:22:05.529 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,76dae4544366126a,1adf6deb48b35682] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-09-11T16%3A22%3A05%2B08%3A00&endTime=2024-09-11T17%3A22%3A05%2B08%3A00&metrics=sql_all_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.31.33.41, traceId=76dae4544366126a, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:22:05+08:00,2024-09-11T17:22:05+08:00,sql_all_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2024-09-11 17:22:05.547 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,76dae4544366126a,1adf6deb48b35682] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=76dae4544366126a, duration=20 ms] 2024-09-11 17:22:05.586 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a398a4e5e83baf21,ee531fa63ceb8cc9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-09-11T16%3A22%3A05%2B08%3A00&endTime=2024-09-11T17%3A22%3A05%2B08%3A00&metrics=sql_all_count&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.31.33.41, traceId=a398a4e5e83baf21, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:22:05+08:00,2024-09-11T17:22:05+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2024-09-11 17:22:05.602 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a398a4e5e83baf21,ee531fa63ceb8cc9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=a398a4e5e83baf21, duration=18 ms] 2024-09-11 17:22:05.608 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,10f16d17c92b9010,f90a309a53f29c6a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2024-09-11T16%3A22%3A05%2B08%3A00&endTime=2024-09-11T17%3A22%3A05%2B08%3A00&metrics=tps_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.31.33.41, traceId=10f16d17c92b9010, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:22:05+08:00,2024-09-11T17:22:05+08:00,tps_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2024-09-11 17:22:05.626 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,10f16d17c92b9010,f90a309a53f29c6a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=10f16d17c92b9010, duration=20 ms] 2024-09-11 17:22:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:22:05.971 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:05.971 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:22:05.972 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:22:05.988 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:22:05.993 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:22:05.998 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:22:06.003 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:22:06.003 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:22:06.003 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:22:06.006 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T17:19:41, target=2024-09-11T17:17:41, granule=120000ms, delay=120000ms) 2024-09-11 17:22:06.006 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:17:41, target=2024-09-11T17:10:51, granule=600000ms, delay=410000ms) 2024-09-11 17:22:06.006 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:06.007 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:22:06.007 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:06.008 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:22:06.049 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:19:40, start=2024-09-11T17:01:11, end=2024-09-11T17:03:41, updates=143, granule=120000ms, delay=1109000ms 2024-09-11 17:22:06.075 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - source=2024-09-11T17:19:40, start=2024-09-11T17:03:40, end=2024-09-11T17:06:10, updates=128, granule=120000ms, delay=960000ms 2024-09-11 17:22:06.097 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=2) - source=2024-09-11T17:19:40, start=2024-09-11T17:06:10, end=2024-09-11T17:08:40, updates=127, granule=120000ms, delay=810000ms 2024-09-11 17:22:06.120 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=3) - source=2024-09-11T17:19:40, start=2024-09-11T17:08:11, end=2024-09-11T17:10:41, updates=113, granule=120000ms, delay=689000ms 2024-09-11 17:22:06.148 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=4) - source=2024-09-11T17:19:40, start=2024-09-11T17:10:40, end=2024-09-11T17:13:10, updates=136, granule=120000ms, delay=540000ms 2024-09-11 17:22:06.170 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=5) - source=2024-09-11T17:19:40, start=2024-09-11T17:13:10, end=2024-09-11T17:15:40, updates=109, granule=120000ms, delay=390000ms 2024-09-11 17:22:06.193 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=6) - source=2024-09-11T17:19:40, start=2024-09-11T17:15:40, end=2024-09-11T17:18:10, updates=148, granule=120000ms, delay=240000ms 2024-09-11 17:22:06.202 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=7) - not enough cached interval (source=2024-09-11T17:19:40, target=2024-09-11T17:18:10, granule=120000ms, delay=90000ms) 2024-09-11 17:22:06.238 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - source=2024-09-11T17:18:10, start=2024-09-11T16:50:55, end=2024-09-11T17:00:55, updates=120, granule=600000ms, delay=1635000ms 2024-09-11 17:22:06.267 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=1->2, c=1) - source=2024-09-11T17:18:10, start=2024-09-11T16:59:54, end=2024-09-11T17:09:54, updates=163, granule=600000ms, delay=1096000ms 2024-09-11 17:22:06.268 ERROR 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup after (cluster=linkrtd:1719994443, server=1, level=1->2, c=1) - no data for end-time 2024-09-11 17:22:06.273 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=301ms, period=120000ms 2024-09-11 17:22:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:22:06.353329+08:00, lastProcessTime=2024-09-11T17:21:56.353764+08:00 2024-09-11 17:22:06.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:22:06.353329+08:00 2024-09-11 17:22:07.116 INFO 2981023 --- [ocp-schedules-19,65a82fd02829eac8,ac4c7536f333978e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:22:07.120 INFO 2981023 --- [ocp-schedules-20,a87c716c6608bef1,da580a45135e7e49] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:22:07.120+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:22:07.121 INFO 2981023 --- [ocp-schedules-20,a87c716c6608bef1,da580a45135e7e49] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:22:07.122 INFO 2981023 --- [ocp-schedules-20,a87c716c6608bef1,da580a45135e7e49] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:22:07.123 INFO 2981023 --- [ocp-schedules-20,a87c716c6608bef1,da580a45135e7e49] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:22:07.137 INFO 2981023 --- [ocp-async-5,ee24c4f39178d4f8,58b028551796daa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:07.736 INFO 2981023 --- [ocp-async-9,aa386b65a85fcc93,a2b81c6bdb27f86d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:22:07.923 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:07.923 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,85d691135fd753fa,d791b5ae17a2b4b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,85d691135fd753fa,d791b5ae17a2b4b7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:09.093 INFO 2981023 --- [ocp-schedules-29,83c8b0128808ff39,25c6ad75043748bc] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-09-11 17:22:09.093 INFO 2981023 --- [ocp-schedules-29,83c8b0128808ff39,25c6ad75043748bc] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-09-11 17:22:09.093 INFO 2981023 --- [ocp-schedules-29,83c8b0128808ff39,25c6ad75043748bc] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=11634 2024-09-11 17:22:09.139 INFO 2981023 --- [ocp-async-1,9ea1b31926f0de32,6e0def6c57b6593c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:09.313 INFO 2981023 --- [ocp-schedules-29,83c8b0128808ff39,25c6ad75043748bc] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=11634 2024-09-11 17:22:09.425 INFO 2981023 --- [ocp-schedules-19,ffd2e0e48ca6f953,17ad96784fcc2051] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:22:09.467 WARN 2981023 --- [scanner-13468720-1,ffd2e0e48ca6f953,68421fe7c81ab9d4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:22:10.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:22:10.003 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:22:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:22:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:22:10.923 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:10.924 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:11.137 INFO 2981023 --- [ocp-async-4,a121ff422b4d056f,2f5715443f4eb34e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,08ae122998a15361,dfc88240a3d848cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,08ae122998a15361,dfc88240a3d848cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:11.696 INFO 2981023 --- [ocp-schedules-11,4c7e8a0e9cf22647,3725f3c5e4edd18c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:22:12.072 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046460, nodeCount=1000, affectRows=1000 2024-09-11 17:22:12.075 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046460, nodeCount=999, validCount=999 2024-09-11 17:22:12.186 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046460, nodeCount=1000, affectRows=1000 2024-09-11 17:22:12.190 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046460, nodeCount=1000, validCount=1000 2024-09-11 17:22:12.197 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046460, nodeCount=50, affectRows=50 2024-09-11 17:22:12.197 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046460, nodeCount=48, validCount=48 2024-09-11 17:22:12.346 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:22:12.360 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:12.363 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:12.419 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:12.420 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:12.431 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:22:12.432 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:22:12.443 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:12.444 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:12.455 INFO 2981023 --- [ocp-schedules-3,f6cb58df021f0d9b,82e8292731e8823d] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=5 2024-09-11 17:22:13.137 INFO 2981023 --- [ocp-async-5,cf4b9ce3baf7883c,ceb70c83c9ffb118] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:13.398 INFO 2981023 --- [ocp-schedules-25,b27802a5136901c6,7770a29813e2754e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:22:13.406 INFO 2981023 --- [ocp-schedules-25,b27802a5136901c6,7770a29813e2754e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:22:13.406 INFO 2981023 --- [ocp-schedules-25,b27802a5136901c6,7770a29813e2754e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:22:13.855 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046460, nodeCount=480, affectRows=480 2024-09-11 17:22:13.924 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:13.925 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,19a8c3bf436b344c,2b446fcbfb62c16e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,a9a6712bb308902b,a696b27dfb4aa65f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:22:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,19a8c3bf436b344c,2b446fcbfb62c16e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:15.137 INFO 2981023 --- [ocp-async-10,053caff044e0deec,c3552e9e1a2fa69d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:15.422 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,110738b04745d064,58f2524de1d8e64c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/unitSpecs?page=1&size=10, client=10.31.33.41, traceId=110738b04745d064, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObUnitSpecController.listUnitSpecs(Pageable,String,List,Long), args=Page request [number: 0, size 10, sort: id: ASC],] 2024-09-11 17:22:15.432 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,110738b04745d064,58f2524de1d8e64c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/unitSpecs client=10.31.33.41, traceId=110738b04745d064, duration=12 ms] 2024-09-11 17:22:15.547 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046460, nodeCount=480, validCount=480 2024-09-11 17:22:15.997 INFO 2981023 --- [ocp-schedules-16,cdcf1df1f36711dd,340a9cb343d74a5b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:22:15.997 INFO 2981023 --- [host-check-clockdiff1,cdcf1df1f36711dd,938879f12c3405d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:22:15.998 INFO 2981023 --- [host-check-clockdiff2,cdcf1df1f36711dd,0a22dccdf3fcf08f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:22:15.998 INFO 2981023 --- [host-check-clockdiff3,cdcf1df1f36711dd,12bd0f5d7e55275e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:22:15.998 INFO 2981023 --- [host-check-clockdiff4,cdcf1df1f36711dd,8c6d0e377e472041] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:22:16.008 INFO 2981023 --- [ocp-schedules-24,e2d2144ec97eab39,5472304f509e3b72] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:22:16.009 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:22:16.009 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:22:16.009 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:22:16.018 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:22:16.031 INFO 2981023 --- [host-check-clockdiff4,cdcf1df1f36711dd,8c6d0e377e472041] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046536 0 0 , error= 2024-09-11 17:22:16.043 INFO 2981023 --- [ocp-schedules-20,fbfac7241812c9fb,0c87ca476bebe24e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:22:16.044 INFO 2981023 --- [host-check-agent-status1,fbfac7241812c9fb,8a8f103327d8e79b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:22:16.044 INFO 2981023 --- [host-check-agent-status2,fbfac7241812c9fb,c49b3dfde24acb01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:22:16.044 INFO 2981023 --- [host-check-agent-status3,fbfac7241812c9fb,2780ae86fa083d5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:22:16.044 INFO 2981023 --- [host-check-agent-status4,fbfac7241812c9fb,1a23b3ee51f68968] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:22:16.047 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:16.048 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:16.049 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:16.050 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:16.050 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:16.051 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:22:16.051 INFO 2981023 --- [host-check-agent-status1,fbfac7241812c9fb,8a8f103327d8e79b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:16.052 INFO 2981023 --- [host-check-agent-status1,fbfac7241812c9fb,8a8f103327d8e79b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:16.052 INFO 2981023 --- [host-check-agent-status2,fbfac7241812c9fb,c49b3dfde24acb01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:16.052 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:22:16.053 INFO 2981023 --- [host-check-agent-status2,fbfac7241812c9fb,c49b3dfde24acb01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:16.055 INFO 2981023 --- [host-check-agent-status3,fbfac7241812c9fb,2780ae86fa083d5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:22:16.056 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:16.056 INFO 2981023 --- [host-check-agent-status3,fbfac7241812c9fb,2780ae86fa083d5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:22:16.057 INFO 2981023 --- [host-check-agent-status4,fbfac7241812c9fb,1a23b3ee51f68968] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:16.058 INFO 2981023 --- [host-check-agent-status4,fbfac7241812c9fb,1a23b3ee51f68968] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:16.079 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:22:16.081 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:22:16.082 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:22:16.081625116+08:00 2024-09-11 17:22:16.082 INFO 2981023 --- [host-check-agent-status2,fbfac7241812c9fb,c49b3dfde24acb01] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:22:16.083 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:22:16.082+08:00 2024-09-11 17:22:16.083 INFO 2981023 --- [host-check-agent-status3,fbfac7241812c9fb,2780ae86fa083d5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:22:16.084 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:22:16.083751378+08:00 2024-09-11 17:22:16.084 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:22:16.084 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:22:16.084+08:00 2024-09-11 17:22:16.085 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:22:16.082+08:00 and status AVAILABLE 2024-09-11 17:22:16.085 INFO 2981023 --- [host-check-agent-status2,fbfac7241812c9fb,c49b3dfde24acb01] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:22:16.085 INFO 2981023 --- [host-check-agent-status2,fbfac7241812c9fb,c49b3dfde24acb01] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:22:16.085 INFO 2981023 --- [host-check-agent-status2,fbfac7241812c9fb,c49b3dfde24acb01] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:22:16.086 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:22:16.086799351+08:00 2024-09-11 17:22:16.086 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:22:16.084+08:00 and status AVAILABLE 2024-09-11 17:22:16.089 INFO 2981023 --- [host-check-agent-status3,fbfac7241812c9fb,2780ae86fa083d5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:22:16.089 INFO 2981023 --- [host-check-agent-status3,fbfac7241812c9fb,2780ae86fa083d5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:22:16.089 INFO 2981023 --- [host-check-agent-status3,fbfac7241812c9fb,2780ae86fa083d5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:22:16.089 INFO 2981023 --- [host-check-agent-status4,fbfac7241812c9fb,1a23b3ee51f68968] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:22:16.089 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:22:16.086+08:00 2024-09-11 17:22:16.090 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:22:16.086+08:00 and status AVAILABLE 2024-09-11 17:22:16.098 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:22:16.101 INFO 2981023 --- [host-check-agent-status4,fbfac7241812c9fb,1a23b3ee51f68968] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:22:16.101 INFO 2981023 --- [host-check-agent-status4,fbfac7241812c9fb,1a23b3ee51f68968] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:22:16.101 INFO 2981023 --- [host-check-agent-status4,fbfac7241812c9fb,1a23b3ee51f68968] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:22:16.101 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:22:16.101314921+08:00 2024-09-11 17:22:16.101 INFO 2981023 --- [host-check-agent-status1,fbfac7241812c9fb,8a8f103327d8e79b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:22:16.101 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:22:16.101+08:00 2024-09-11 17:22:16.103 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:22:16.101+08:00 and status AVAILABLE 2024-09-11 17:22:16.103 INFO 2981023 --- [host-check-agent-status1,fbfac7241812c9fb,8a8f103327d8e79b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:22:16.104 INFO 2981023 --- [host-check-agent-status1,fbfac7241812c9fb,8a8f103327d8e79b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:22:16.104 INFO 2981023 --- [host-check-agent-status1,fbfac7241812c9fb,8a8f103327d8e79b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:22:16.105 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:22:16.126 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:16.127 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:16.135 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:22:16.137 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:22:16.145 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:22:16.145 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:16.147 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:22:16.147 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:16.153 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:22:16.166 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:22:16.170 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:22:16.179 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:16.180 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:16.187 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:22:16.187 INFO 2981023 --- [ocp-schedules-26,245d6e15856f68b0,7fa151677664ee10] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:22:16.187 INFO 2981023 --- [ocp-schedules-27,3344989cc313d1b3,a6486bb0ad093e28] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:22:16.194 INFO 2981023 --- [ocp-schedules-9,a203e78e185cd94d,a039bf44b90f832b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:22:16.194 INFO 2981023 --- [ocp-schedules-12,080a5373054d83a6,2b6f42bf24932028] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:22:16.195 INFO 2981023 --- [ocp-schedules-27,3344989cc313d1b3,a6486bb0ad093e28] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:16.196 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:22:16.201 INFO 2981023 --- [ocp-schedules-26,245d6e15856f68b0,7fa151677664ee10] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:16.203 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:22:16.204 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:22:16.210 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:22:16.211 INFO 2981023 --- [ocp-schedules-9,a203e78e185cd94d,a039bf44b90f832b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:16.226 INFO 2981023 --- [ocp-schedules-5,c3c10b24e7052f7d,e7d38fd67747d8a2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:22:16.230 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:22:16.236 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:22:16.247 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:22:16.250 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:22:16.272 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:22:16.281 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.281 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:16.283 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:22:16.284 INFO 2981023 --- [ocp-schedules-27,3344989cc313d1b3,a6486bb0ad093e28] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:22:16.286 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:22:16.301 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:22:16.307 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:22:16.310 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.310 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:16.310 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:22:16.320 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:22:16.323 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.324 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:16.325 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:22:16.326 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.327 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:16.328 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.328 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.329 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:22:16.332 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:22:16.335 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.336 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:16.336 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:22:16.336 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:22:16.338 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.339 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:16.339 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:22:16.340 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:22:16.340 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:22:16.342 INFO 2981023 --- [ocp-schedules-26,245d6e15856f68b0,7fa151677664ee10] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:22:16.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:22:16.352660+08:00, lastProcessTime=2024-09-11T17:22:06.353329+08:00 2024-09-11 17:22:16.356 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:22:16.368 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:22:16.369 INFO 2981023 --- [host-check-host-status3,e2d2144ec97eab39,d9c8d1e2d5905f2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:22:16.373 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:22:16.376 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.378 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.378 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:16.379 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:22:16.382 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:22:16.382 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.385 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:22:16.386 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:22:16.396 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:16.396 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:22:16.398 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:22:16.402 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:22:16.406 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:22:16.422 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.423 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.423 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:22:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:16.429 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:22:16.430 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:22:16.439 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:22:16.440 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.440 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.441 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:16.442 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:22:16.444 INFO 2981023 --- [host-check-host-status1,e2d2144ec97eab39,b2c1f824c33e4837] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:22:16.444 INFO 2981023 --- [host-check-host-status2,e2d2144ec97eab39,f9c6e01e851bcbaf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:22:16.447 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:22:16.450 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:16.452 INFO 2981023 --- [host-check-host-status4,e2d2144ec97eab39,db38a34a6868a501] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:22:16.452 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:16.461 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.462 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.462 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:16.463 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.466 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:22:16.467 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:22:16.474 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:22:16.476 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.476 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.477 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:22:16.481 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:16.481 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:22:16.497 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:16.504 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.504 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.505 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:16.505 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:22:16.506 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.508 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:22:16.508 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.509 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:22:16.509 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.510 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:16.510 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:22:16.514 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:22:16.517 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:16.517 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:22:16.519 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:16.520 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:16.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:16.538 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.539 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.539 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:16.540 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:22:16.540 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:22:16.543 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:22:16.548 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:22:16.549 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:16.550 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:16.556 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.556 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:22:16.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:22:16.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:22:16.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:22:16.594 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:22:16.596 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:22:16.352660+08:00 2024-09-11 17:22:16.599 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.600 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.601 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:22:16.602 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:22:16.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:22:16.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:22:16.608 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:16.608 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:16.609 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:22:16.614 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:22:16.614 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:22:16.614 INFO 2981023 --- [ocp-schedules-9,a203e78e185cd94d,a039bf44b90f832b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:22:16.925 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:16.925 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:17.086 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e2088d56caa77ada,e34a3771b8d8caf5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants/parameterTemplates?page=1&size=10, client=10.31.33.41, traceId=e2088d56caa77ada, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantParameterController.listTenantParameterTemplates(String,List,Long,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:22:17.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e2088d56caa77ada,e34a3771b8d8caf5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants/parameterTemplates client=10.31.33.41, traceId=e2088d56caa77ada, duration=15 ms] 2024-09-11 17:22:17.137 INFO 2981023 --- [ocp-async-3,a70e639e4d642261,1bda0d54b4b18abe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,c4f8973a10ee3ce3,172bc3b750fe2b99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,c4f8973a10ee3ce3,172bc3b750fe2b99] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:17.736 INFO 2981023 --- [ocp-async-1,e270b6393dcf9a57,bb1f02cc0e263991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:22:17.787 INFO 2981023 --- [host-check-clockdiff2,cdcf1df1f36711dd,0a22dccdf3fcf08f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046537 0 0 , error= 2024-09-11 17:22:17.792 INFO 2981023 --- [host-check-clockdiff1,cdcf1df1f36711dd,938879f12c3405d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046537 0 0 , error= 2024-09-11 17:22:18.046 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3c38c7c1d4b1ef26,ddb78658219511ad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP&maxPoints=360, client=10.31.33.41, traceId=3c38c7c1d4b1ef26, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,] 2024-09-11 17:22:18.048 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2af5c94498f94a94,6c3b0a0bcd0b9fe3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.31.33.41, traceId=2af5c94498f94a94, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2024-09-11 17:22:18.050 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3c38c7c1d4b1ef26,ddb78658219511ad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=3c38c7c1d4b1ef26, duration=7 ms] 2024-09-11 17:22:18.056 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7ade7e908182df37,47845f8347793fc4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.31.33.41, traceId=7ade7e908182df37, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2024-09-11 17:22:18.058 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2af5c94498f94a94,6c3b0a0bcd0b9fe3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=2af5c94498f94a94, duration=14 ms] 2024-09-11 17:22:18.061 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f87b64636f0808a9,ab11cc2a2f033b99] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=f87b64636f0808a9, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 17:22:18.062 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,641e30f10baa889c,35d7b8a2c549f0b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2024-09-11T16%3A22%3A17%2B08%3A00&endTime=2024-09-11T17%3A22%3A17%2B08%3A00&maxPoints=360, client=10.31.33.41, traceId=641e30f10baa889c, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:22:17+08:00,2024-09-11T17:22:17+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2024-09-11 17:22:18.067 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7ade7e908182df37,47845f8347793fc4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.31.33.41, traceId=7ade7e908182df37, duration=14 ms] 2024-09-11 17:22:18.082 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,641e30f10baa889c,35d7b8a2c549f0b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=641e30f10baa889c, duration=29 ms] 2024-09-11 17:22:18.088 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,79b6ecac9c267766,d6c7908b14ab2607] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.31.33.41, traceId=79b6ecac9c267766, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2024-09-11 17:22:18.095 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,79b6ecac9c267766,d6c7908b14ab2607] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.31.33.41, traceId=79b6ecac9c267766, duration=10 ms] 2024-09-11 17:22:18.098 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f87b64636f0808a9,ab11cc2a2f033b99] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=f87b64636f0808a9, duration=40 ms] 2024-09-11 17:22:19.137 INFO 2981023 --- [ocp-async-6,872a2d5eda586ce4,8339a55ddb7eda5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:19.641 INFO 2981023 --- [ocp-schedules-19,cd3f314cb0a9a4a9,3d1d2a163def6983] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:22:19.680 WARN 2981023 --- [scanner-13470413-1,cd3f314cb0a9a4a9,501f0475e69e0b28] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:22:19.712 INFO 2981023 --- [alarm-detect-10,cd3f314cb0a9a4a9,4b804dd2994ae417] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643663, data is empty. 2024-09-11 17:22:19.716 INFO 2981023 --- [alarm-detect-10,cd3f314cb0a9a4a9,4b804dd2994ae417] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643663, data is empty. 2024-09-11 17:22:19.719 INFO 2981023 --- [alarm-detect-10,cd3f314cb0a9a4a9,4b804dd2994ae417] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643663, data is empty. 2024-09-11 17:22:19.732 INFO 2981023 --- [alarm-detect-4,cd3f314cb0a9a4a9,cc02cac2f9621e34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643663, data is empty. 2024-09-11 17:22:19.736 INFO 2981023 --- [alarm-detect-4,cd3f314cb0a9a4a9,cc02cac2f9621e34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643663, data is empty. 2024-09-11 17:22:19.741 INFO 2981023 --- [alarm-detect-15,cd3f314cb0a9a4a9,82262851bf42b050] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643663, data is empty. 2024-09-11 17:22:19.743 INFO 2981023 --- [alarm-detect-4,cd3f314cb0a9a4a9,cc02cac2f9621e34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643663, data is empty. 2024-09-11 17:22:19.745 INFO 2981023 --- [alarm-detect-15,cd3f314cb0a9a4a9,82262851bf42b050] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643663, data is empty. 2024-09-11 17:22:19.750 INFO 2981023 --- [alarm-detect-15,cd3f314cb0a9a4a9,82262851bf42b050] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643663, data is empty. 2024-09-11 17:22:19.860 INFO 2981023 --- [alarm-detect-7,cd3f314cb0a9a4a9,2daf9af4f5ebcd2d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643663, data is empty. 2024-09-11 17:22:19.864 INFO 2981023 --- [alarm-detect-7,cd3f314cb0a9a4a9,2daf9af4f5ebcd2d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643663, data is empty. 2024-09-11 17:22:19.870 INFO 2981023 --- [alarm-detect-7,cd3f314cb0a9a4a9,2daf9af4f5ebcd2d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643663, data is empty. 2024-09-11 17:22:19.926 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:19.926 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,6d112d19cc3f54d3,8acad9d0badb6a60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:20.315 INFO 2981023 --- [prometheus-minute-schedule-1,6d112d19cc3f54d3,8acad9d0badb6a60] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:21.137 INFO 2981023 --- [ocp-async-2,15b23a61cb3a0200,93f617024857f42a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:21.699 INFO 2981023 --- [ocp-schedules-1,c1252a9cc929d4c0,d3a9da2765caf3ac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:22:21.993 INFO 2981023 --- [host-check-clockdiff3,cdcf1df1f36711dd,12bd0f5d7e55275e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046541 1 1 , error= 2024-09-11 17:22:22.205 INFO 2981023 --- [ocp-schedules-13,8cb8248f42de4ac5,5658796d5cecd74c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:22:22.205 INFO 2981023 --- [ocp-schedules-13,8cb8248f42de4ac5,5658796d5cecd74c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:22:22.205 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:22:22.205 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:22:22.205 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:22:22.208 INFO 2981023 --- [ocp-schedules-13,8cb8248f42de4ac5,5658796d5cecd74c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:22:22.209 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:22.209 INFO 2981023 --- [ocp-schedules-13,8cb8248f42de4ac5,5658796d5cecd74c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:22:22.210 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:22.210 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:22.211 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:22.211 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:22.212 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:22.226 INFO 2981023 --- [ocp-schedules-13,8cb8248f42de4ac5,5658796d5cecd74c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:22:22.227 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:22:22.230 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:22:22.232 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:22:22.236 INFO 2981023 --- [ocp-schedules-13,8cb8248f42de4ac5,5658796d5cecd74c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:22:22.926 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:22.928 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:23.138 INFO 2981023 --- [ocp-async-3,af6c10c9fbd7d474,eb62f2362aff0745] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,db4a9aca8b6286a7,cdee369738ef008a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,db4a9aca8b6286a7,cdee369738ef008a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:24.147 INFO 2981023 --- [ocp-schedules-4,d114cc78836d7b7a,2f0ab9b1800b87f0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:22:25.138 INFO 2981023 --- [ocp-async-9,753bbf6b5610f00d,97fe79e3b846a0d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:25.928 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:25.928 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,cefcd7aed09fa34e,adc9714a49322ea4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,cefcd7aed09fa34e,adc9714a49322ea4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:22:26.351926+08:00, lastProcessTime=2024-09-11T17:22:16.352660+08:00 2024-09-11 17:22:26.372 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,cace994a692bc5f2,33716da79a7b07ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=cace994a692bc5f2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:22:26.373 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,23650396c2d471cc,1b8a238d473b53e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=23650396c2d471cc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:22:26.378 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,23650396c2d471cc,1b8a238d473b53e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=23650396c2d471cc, duration=11 ms] 2024-09-11 17:22:26.382 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,cace994a692bc5f2,33716da79a7b07ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=cace994a692bc5f2, duration=14 ms] 2024-09-11 17:22:26.384 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3ef86bcd49947cb4,c54b39d7a8e13950] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=3ef86bcd49947cb4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:22:26.384 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,f0f24d5971abcb75,3a40019189833f33] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=f0f24d5971abcb75, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 17:22:26.385 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,9268ef605bb55e2c,ba348b2b22a14a0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=9268ef605bb55e2c, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 17:22:26.386 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3ef86bcd49947cb4,c54b39d7a8e13950] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=3ef86bcd49947cb4, duration=4 ms] 2024-09-11 17:22:26.387 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,9268ef605bb55e2c,ba348b2b22a14a0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=9268ef605bb55e2c, duration=5 ms] 2024-09-11 17:22:26.424 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,f0f24d5971abcb75,3a40019189833f33] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=f0f24d5971abcb75, duration=42 ms] 2024-09-11 17:22:26.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:22:26.351926+08:00 2024-09-11 17:22:26.500 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bf6102e3e0617bc6,1fc67b4989ceb197] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_TENANT&type=TOP&maxPoints=360, client=10.31.33.41, traceId=bf6102e3e0617bc6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_TENANT,] 2024-09-11 17:22:26.500 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,16cef78f1cf01d67,87e101ba40bc4678] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_HOST&type=TOP&maxPoints=360, client=10.31.33.41, traceId=16cef78f1cf01d67, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_HOST,] 2024-09-11 17:22:26.501 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,beb87c3d7fb92636,29986ccb4cfae4e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_cpu_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=beb87c3d7fb92636, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_cpu_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 17:22:26.503 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,cf8d397a8240f609,b9b90f184c17e147] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_disk_used_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=cf8d397a8240f609, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_disk_used_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 17:22:26.505 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bf6102e3e0617bc6,1fc67b4989ceb197] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=bf6102e3e0617bc6, duration=8 ms] 2024-09-11 17:22:26.507 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,cf8d397a8240f609,b9b90f184c17e147] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=cf8d397a8240f609, duration=7 ms] 2024-09-11 17:22:26.510 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7de4deb7c60330e7,dba254bd53f8e125] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_ODP&type=TOP&maxPoints=360, client=10.31.33.41, traceId=7de4deb7c60330e7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_ODP,] 2024-09-11 17:22:26.510 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,16cef78f1cf01d67,87e101ba40bc4678] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=16cef78f1cf01d67, duration=12 ms] 2024-09-11 17:22:26.512 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,beb87c3d7fb92636,29986ccb4cfae4e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=beb87c3d7fb92636, duration=14 ms] 2024-09-11 17:22:26.513 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7de4deb7c60330e7,dba254bd53f8e125] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=7de4deb7c60330e7, duration=6 ms] 2024-09-11 17:22:26.553 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2acd5585c4de61c9,48cd35109058a83c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_memory_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=2acd5585c4de61c9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_memory_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 17:22:26.556 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2acd5585c4de61c9,48cd35109058a83c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=2acd5585c4de61c9, duration=5 ms] 2024-09-11 17:22:26.626 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c1bee100b044b48d,2a3f8751f796a863] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_memory_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=c1bee100b044b48d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_memory_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 17:22:26.627 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,8148c6e050ee3eed,14dd021985109238] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_cpu_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=8148c6e050ee3eed, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_cpu_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 17:22:26.631 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8dc921660a26d871,ced1809275fd8884] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_tenant_disk_usage_v4&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=8dc921660a26d871, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_tenant_disk_usage_v4,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 17:22:26.636 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,4c2d25ddb3571054,7b7323a5c5f1371a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?foldStandby=false, client=10.31.33.41, traceId=4c2d25ddb3571054, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 17:22:26.636 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8dc921660a26d871,ced1809275fd8884] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=8dc921660a26d871, duration=9 ms] 2024-09-11 17:22:26.643 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c1bee100b044b48d,2a3f8751f796a863] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=c1bee100b044b48d, duration=20 ms] 2024-09-11 17:22:26.643 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,8148c6e050ee3eed,14dd021985109238] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=8148c6e050ee3eed, duration=18 ms] 2024-09-11 17:22:26.646 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,4c2d25ddb3571054,7b7323a5c5f1371a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=4c2d25ddb3571054, duration=13 ms] 2024-09-11 17:22:27.138 INFO 2981023 --- [ocp-async-7,41d1b9b66a28cee1,3bc5159e53750c40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:27.736 INFO 2981023 --- [ocp-async-6,12c5259c85aaa9d3,16b01833182adbae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:22:28.928 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:28.929 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:29.138 INFO 2981023 --- [ocp-async-8,7009c0317355c2dc,a43e3ef7fbf84db5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,a3d7ccf880f6bda8,99769679f96f2b45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,a3d7ccf880f6bda8,99769679f96f2b45] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:29.915 INFO 2981023 --- [ocp-schedules-15,ad2c8614bb8b9076,c202b10556436bb8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:22:29.959 WARN 2981023 --- [scanner-13470782-1,ad2c8614bb8b9076,654e2423aa992e6b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:22:31.138 INFO 2981023 --- [ocp-async-9,4192baaa5094a551,580c74ffd777352f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:31.702 INFO 2981023 --- [ocp-schedules-15,540c8a0ed98b8f6d,6c9ebeeead8a696d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:22:31.908 INFO 2981023 --- [ocp-schedules-9,2003cb7a01d81289,1e6d16aabc2e14b5] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY,C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkprdocp-1719994437/H_10.31.35.10,C_linkrtd-1719994443/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631/H_10.31.35.5,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7, current targets ,,,,,C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY,C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkprdocp-1719994437/H_10.31.35.10,C_linkrtd-1719994443/H_10.31.35.6,C_rtdob-1718689631/H_10.31.35.5,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:31.929 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:31.930 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,01a30e45b2003bf7,666432b70cd9c970] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,01a30e45b2003bf7,666432b70cd9c970] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:33.138 INFO 2981023 --- [ocp-async-7,87255f252810edcb,9826f8b0825a95aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:33.409 INFO 2981023 --- [ocp-schedules-26,8b0b38ec43323441,1b4f0edbdacdf152] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:22:33.418 INFO 2981023 --- [ocp-schedules-26,8b0b38ec43323441,1b4f0edbdacdf152] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:22:33.418 INFO 2981023 --- [ocp-schedules-26,8b0b38ec43323441,1b4f0edbdacdf152] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:22:34.930 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:34.930 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:35.138 INFO 2981023 --- [ocp-async-1,682c1d3ce554fbf0,bafbb9e68a4a83fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,35f756c4ffbffdbe,e59c05b58766ce10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,35f756c4ffbffdbe,e59c05b58766ce10] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:22:36.337349+08:00, lastProcessTime=2024-09-11T17:22:26.351926+08:00 2024-09-11 17:22:36.341 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1beedc7238fdfcf4,c4bc3a81fed05a4f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=1beedc7238fdfcf4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:22:36.341 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a3978949e7ede3a5,d1cc5d7757b8d5fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=a3978949e7ede3a5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:22:36.342 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1beedc7238fdfcf4,c4bc3a81fed05a4f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=1beedc7238fdfcf4, duration=4 ms] 2024-09-11 17:22:36.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a3978949e7ede3a5,d1cc5d7757b8d5fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=a3978949e7ede3a5, duration=6 ms] 2024-09-11 17:22:36.354 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f4c66cd9abf65e1a,528b22a670e9f90a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=f4c66cd9abf65e1a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:22:36.354 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f4c66cd9abf65e1a,528b22a670e9f90a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=f4c66cd9abf65e1a, duration=6 ms] 2024-09-11 17:22:36.470 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:22:36.337349+08:00 2024-09-11 17:22:37.138 INFO 2981023 --- [ocp-async-5,7041b32661cf8c3c,d72919129e581422] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:37.314 INFO 2981023 --- [prometheus-cache-monitor1,7e2094f16557c6a3,bd990dde79e91828] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:22:37.315 INFO 2981023 --- [prometheus-cache-monitor1,7e2094f16557c6a3,bd990dde79e91828] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:22:37.341 INFO 2981023 --- [ocp-schedules-12,efd86023e1bbfc5b,00fd1f7b83ef0b00] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:22:37.360 WARN 2981023 --- [ocp-schedules-12,efd86023e1bbfc5b,00fd1f7b83ef0b00] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:22:37.736 INFO 2981023 --- [ocp-async-8,fb43548ee84e45ca,6fc64c06fbea9b73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:22:37.930 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:37.931 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,0c5613831922ee66,4444680806c7b41b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,0c5613831922ee66,4444680806c7b41b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:39.138 INFO 2981023 --- [ocp-async-7,b4ef07d3f39d7e00,141046c7a5d70c34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:40.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:22:40.003 INFO 2981023 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:22:40.107 INFO 2981023 --- [ocp-schedules-11,ae1897eb082f662e,dd8c2efe29c7b594] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:22:40.154 INFO 2981023 --- [ocp-schedules-1,84a206485018db35,f2afd7022da32992] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:22:40.204 WARN 2981023 --- [scanner-13471004-1,ae1897eb082f662e,a50ad274bdc3d722] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:22:40.226 INFO 2981023 --- [alarm-detect-13,ae1897eb082f662e,0495824dcda04011] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643663, data is empty. 2024-09-11 17:22:40.229 INFO 2981023 --- [alarm-detect-13,ae1897eb082f662e,0495824dcda04011] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643663, data is empty. 2024-09-11 17:22:40.233 INFO 2981023 --- [alarm-detect-13,ae1897eb082f662e,0495824dcda04011] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643663, data is empty. 2024-09-11 17:22:40.267 INFO 2981023 --- [alarm-detect-17,ae1897eb082f662e,89c26584458a8df9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643663, data is empty. 2024-09-11 17:22:40.271 INFO 2981023 --- [alarm-detect-15,ae1897eb082f662e,d15416ba2d093928] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643663, data is empty. 2024-09-11 17:22:40.272 INFO 2981023 --- [alarm-detect-17,ae1897eb082f662e,89c26584458a8df9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643663, data is empty. 2024-09-11 17:22:40.276 INFO 2981023 --- [alarm-detect-15,ae1897eb082f662e,d15416ba2d093928] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643663, data is empty. 2024-09-11 17:22:40.281 INFO 2981023 --- [alarm-detect-17,ae1897eb082f662e,89c26584458a8df9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643663, data is empty. 2024-09-11 17:22:40.284 INFO 2981023 --- [alarm-detect-15,ae1897eb082f662e,d15416ba2d093928] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643663, data is empty. 2024-09-11 17:22:40.391 INFO 2981023 --- [alarm-detect-17,ae1897eb082f662e,651bd36027747426] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643663, data is empty. 2024-09-11 17:22:40.394 INFO 2981023 --- [alarm-detect-17,ae1897eb082f662e,651bd36027747426] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643663, data is empty. 2024-09-11 17:22:40.399 INFO 2981023 --- [alarm-detect-17,ae1897eb082f662e,651bd36027747426] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643663, data is empty. 2024-09-11 17:22:40.931 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:40.932 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:41.138 INFO 2981023 --- [ocp-async-3,0ced8a6f982e6622,aa8e9acae79703ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,953c90e7c2b37326,98f06ddc1599eb2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,953c90e7c2b37326,98f06ddc1599eb2d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:41.704 INFO 2981023 --- [ocp-schedules-18,03b57b88bf40678b,551ed322553c25ed] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:22:42.382 INFO 2981023 --- [ocp-schedules-30,8b6c4e95b2384568,c4174840aba069fa] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-09-11 17:22:42.384 INFO 2981023 --- [ocp-schedules-30,8b6c4e95b2384568,c4174840aba069fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:42.441 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:42.443 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:42.443 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:42.445 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:07:40.80042, 2024-09-11 17:22:42.42, 1000] 2024-09-11 17:22:42.445 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 02:03:06.281634, 2024-09-11 17:22:42.418, 1000] 2024-09-11 17:22:42.447 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 10:40:18.658505, 2024-09-11 17:22:42.419, 1000] 2024-09-11 17:22:42.448 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-09-11 02:03:06.281634, endTime 2024-09-11 17:22:42.418 2024-09-11 17:22:42.449 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2024-09-11 10:40:18.658505, endTime 2024-09-11 17:22:42.419 2024-09-11 17:22:42.457 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 4 system events to database for cluster 1000006 2024-09-11 17:22:42.457 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 4 system events for cluster 1000006, startTime 2024-09-11 17:07:40.80042, endTime 2024-09-11 17:22:42.42, limit 1000, offset 0 2024-09-11 17:22:42.464 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:42.465 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:42.469 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 17:07:40.80042, 2024-09-11 17:22:42.42, 1000, 1000] 2024-09-11 17:22:42.469 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000006, startTime 2024-09-11 17:07:40.80042, endTime 2024-09-11 17:22:42.42 2024-09-11 17:22:42.489 INFO 2981023 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:42.489 INFO 2981023 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:42.490 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:42.494 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:42.495 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:42.495 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:42.523 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 08:00:10.321716, 2024-09-11 17:22:42.482, 1000] 2024-09-11 17:22:42.524 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-09-11 08:00:10.321716, endTime 2024-09-11 17:22:42.482 2024-09-11 17:22:42.527 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 16:48:30.801707, 2024-09-11 17:22:42.482, 1000] 2024-09-11 17:22:42.528 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2024-09-11 16:48:30.801707, endTime 2024-09-11 17:22:42.482 2024-09-11 17:22:42.534 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:12:38.669736, 2024-09-11 17:22:42.483, 1000] 2024-09-11 17:22:42.622 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1000 server events for cluster 1000006, startTime 2024-09-11 17:12:38.669736, endTime 2024-09-11 17:22:42.483, limit 1000, offset 0 2024-09-11 17:22:43.035 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1000 server events to database for cluster 1000006 2024-09-11 17:22:43.042 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:43.042 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:43.063 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 17:12:38.669736, 2024-09-11 17:22:42.483, 1000, 1000] 2024-09-11 17:22:43.141 INFO 2981023 --- [ocp-async-4,fb5c9155d7e9361c,bb310f3ff01935cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:43.153 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1000 server events for cluster 1000006, startTime 2024-09-11 17:12:38.669736, endTime 2024-09-11 17:22:42.483, limit 1000, offset 1000 2024-09-11 17:22:43.597 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1000 server events to database for cluster 1000006 2024-09-11 17:22:43.602 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:43.603 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:43.621 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 17:12:38.669736, 2024-09-11 17:22:42.483, 1000, 2000] 2024-09-11 17:22:43.708 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1000 server events for cluster 1000006, startTime 2024-09-11 17:12:38.669736, endTime 2024-09-11 17:22:42.483, limit 1000, offset 2000 2024-09-11 17:22:43.932 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:43.933 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:44.112 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1000 server events to database for cluster 1000006 2024-09-11 17:22:44.117 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:44.118 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:44.137 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 17:12:38.669736, 2024-09-11 17:22:42.483, 1000, 3000] 2024-09-11 17:22:44.228 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1000 server events for cluster 1000006, startTime 2024-09-11 17:12:38.669736, endTime 2024-09-11 17:22:42.483, limit 1000, offset 3000 2024-09-11 17:22:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,93604820e0b8715b,21ffa1c06915f6ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,93604820e0b8715b,21ffa1c06915f6ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:44.663 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1000 server events to database for cluster 1000006 2024-09-11 17:22:44.669 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:44.670 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:44.689 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 17:12:38.669736, 2024-09-11 17:22:42.483, 1000, 4000] 2024-09-11 17:22:44.731 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 442 server events for cluster 1000006, startTime 2024-09-11 17:12:38.669736, endTime 2024-09-11 17:22:42.483, limit 1000, offset 4000 2024-09-11 17:22:44.914 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 442 server events to database for cluster 1000006 2024-09-11 17:22:44.919 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:44.921 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:44.938 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 17:12:38.669736, 2024-09-11 17:22:42.483, 1000, 5000] 2024-09-11 17:22:44.944 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000006, startTime 2024-09-11 17:12:38.669736, endTime 2024-09-11 17:22:42.483 2024-09-11 17:22:44.944 INFO 2981023 --- [ocp-schedules-30,8b6c4e95b2384568,c4174840aba069fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, tryCount=3, successCount=3 2024-09-11 17:22:45.101 INFO 2981023 --- [prometheus-second-schedule-1,80e93373e70e2411,b2f1ba410cdf1301] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:22:45.139 INFO 2981023 --- [ocp-async-9,20743e5d9731fbbe,c2907305215da796] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:46.019 INFO 2981023 --- [ocp-schedules-14,e11965396e3f37c0,f152ed9571a6ffb1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:22:46.020 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:22:46.021 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:22:46.021 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:22:46.021 INFO 2981023 --- [ocp-schedules-10,bc3e041a2ecf7497,4c2e9235a6c54e02] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:22:46.022 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:22:46.022 INFO 2981023 --- [host-check-clockdiff1,bc3e041a2ecf7497,8f0046690c5a9da0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:22:46.023 INFO 2981023 --- [host-check-clockdiff2,bc3e041a2ecf7497,a87df1d0cf474862] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:22:46.023 INFO 2981023 --- [host-check-clockdiff3,bc3e041a2ecf7497,0731dbe95396aa46] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:22:46.024 INFO 2981023 --- [host-check-clockdiff4,bc3e041a2ecf7497,374745010bf33450] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:22:46.028 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:46.029 INFO 2981023 --- [ocp-schedules-27,93871b35e9d5c9d3,281de015e61985e1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:22:46.035 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:46.037 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:46.038 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:46.040 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:46.041 INFO 2981023 --- [host-check-agent-status1,93871b35e9d5c9d3,59ce4817ebce9a2c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:22:46.042 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:46.042 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:22:46.042 INFO 2981023 --- [host-check-agent-status2,93871b35e9d5c9d3,01d3b45c764e1c61] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:22:46.043 INFO 2981023 --- [host-check-agent-status3,93871b35e9d5c9d3,1258bd15f0e9df1b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:22:46.043 INFO 2981023 --- [host-check-agent-status4,93871b35e9d5c9d3,5368fa8e2775651c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:22:46.044 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:22:46.041 INFO 2981023 --- [host-check-clockdiff4,bc3e041a2ecf7497,374745010bf33450] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046566 0 0 , error= 2024-09-11 17:22:46.051 INFO 2981023 --- [host-check-agent-status1,93871b35e9d5c9d3,59ce4817ebce9a2c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:46.053 INFO 2981023 --- [host-check-agent-status1,93871b35e9d5c9d3,59ce4817ebce9a2c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:46.053 INFO 2981023 --- [host-check-agent-status3,93871b35e9d5c9d3,1258bd15f0e9df1b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:22:46.054 INFO 2981023 --- [host-check-agent-status3,93871b35e9d5c9d3,1258bd15f0e9df1b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:22:46.057 INFO 2981023 --- [host-check-agent-status2,93871b35e9d5c9d3,01d3b45c764e1c61] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:46.059 INFO 2981023 --- [host-check-agent-status2,93871b35e9d5c9d3,01d3b45c764e1c61] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:46.060 INFO 2981023 --- [host-check-agent-status4,93871b35e9d5c9d3,5368fa8e2775651c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:46.064 INFO 2981023 --- [host-check-agent-status4,93871b35e9d5c9d3,5368fa8e2775651c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:46.081 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:22:46.082 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:22:46.083 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:22:46.083200276+08:00 2024-09-11 17:22:46.084 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:22:46.083+08:00 2024-09-11 17:22:46.084 INFO 2981023 --- [host-check-agent-status3,93871b35e9d5c9d3,1258bd15f0e9df1b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:22:46.085 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:22:46.084635946+08:00 2024-09-11 17:22:46.086 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:22:46.085+08:00 2024-09-11 17:22:46.086 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:22:46.083+08:00 and status AVAILABLE 2024-09-11 17:22:46.087 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:22:46.085+08:00 and status AVAILABLE 2024-09-11 17:22:46.089 INFO 2981023 --- [host-check-agent-status1,93871b35e9d5c9d3,59ce4817ebce9a2c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:22:46.089 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:22:46.090 INFO 2981023 --- [host-check-agent-status3,93871b35e9d5c9d3,1258bd15f0e9df1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:22:46.090 INFO 2981023 --- [host-check-agent-status3,93871b35e9d5c9d3,1258bd15f0e9df1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:22:46.090 INFO 2981023 --- [host-check-agent-status3,93871b35e9d5c9d3,1258bd15f0e9df1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:22:46.090 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:22:46.090563386+08:00 2024-09-11 17:22:46.091 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:22:46.090+08:00 2024-09-11 17:22:46.091 INFO 2981023 --- [host-check-agent-status2,93871b35e9d5c9d3,01d3b45c764e1c61] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:22:46.092 INFO 2981023 --- [host-check-agent-status4,93871b35e9d5c9d3,5368fa8e2775651c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:22:46.093 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:22:46.090+08:00 and status AVAILABLE 2024-09-11 17:22:46.093 INFO 2981023 --- [host-check-agent-status1,93871b35e9d5c9d3,59ce4817ebce9a2c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:22:46.093 INFO 2981023 --- [host-check-agent-status1,93871b35e9d5c9d3,59ce4817ebce9a2c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:22:46.093 INFO 2981023 --- [host-check-agent-status1,93871b35e9d5c9d3,59ce4817ebce9a2c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:22:46.094 INFO 2981023 --- [host-check-agent-status2,93871b35e9d5c9d3,01d3b45c764e1c61] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:22:46.094 INFO 2981023 --- [host-check-agent-status2,93871b35e9d5c9d3,01d3b45c764e1c61] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:22:46.094 INFO 2981023 --- [host-check-agent-status2,93871b35e9d5c9d3,01d3b45c764e1c61] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:22:46.095 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:22:46.096 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:22:46.096512576+08:00 2024-09-11 17:22:46.098 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:22:46.096+08:00 2024-09-11 17:22:46.099 INFO 2981023 --- [host-check-agent-status4,93871b35e9d5c9d3,5368fa8e2775651c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:22:46.099 INFO 2981023 --- [host-check-agent-status4,93871b35e9d5c9d3,5368fa8e2775651c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:22:46.099 INFO 2981023 --- [host-check-agent-status4,93871b35e9d5c9d3,5368fa8e2775651c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:22:46.099 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:22:46.100 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:22:46.100 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:22:46.096+08:00 and status AVAILABLE 2024-09-11 17:22:46.105 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:22:46.131 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:22:46.132 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:46.133 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:46.137 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:46.138 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:46.138 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:46.139 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:46.160 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:22:46.163 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:22:46.163 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:22:46.174 INFO 2981023 --- [ocp-schedules-18,8e251790a1f39313,a67091b5a6be6041] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:22:46.187 INFO 2981023 --- [ocp-schedules-23,df752f5f53494d08,61eba0b971b79a07] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:22:46.187 INFO 2981023 --- [ocp-schedules-11,92b0f1a613aea3ba,1e076c022491e85d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:22:46.187 INFO 2981023 --- [ocp-schedules-25,5abf73c6424c7ca4,ba53e6f7e0f6ae66] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:22:46.188 INFO 2981023 --- [ocp-schedules-22,4c376c5bd7f70137,4aba7bc2e66d6c34] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:22:46.189 INFO 2981023 --- [ocp-schedules-23,df752f5f53494d08,61eba0b971b79a07] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:46.190 INFO 2981023 --- [ocp-schedules-25,5abf73c6424c7ca4,ba53e6f7e0f6ae66] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:46.194 INFO 2981023 --- [ocp-schedules-11,92b0f1a613aea3ba,1e076c022491e85d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:46.197 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:22:46.201 INFO 2981023 --- [ocp-schedules-18,8e251790a1f39313,a67091b5a6be6041] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:22:46.207 INFO 2981023 --- [ocp-schedules-11,92b0f1a613aea3ba,1e076c022491e85d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:22:46.212 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:22:46.220 INFO 2981023 --- [ocp-schedules-17,4fd077cc4c124f28,de99e3228a7b1dcb] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:22:46.221 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:22:46.225 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:22:46.243 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:22:46.247 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:22:46.256 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:22:46.267 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:22:46.280 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:22:46.284 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:22:46.293 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.294 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:46.295 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:22:46.296 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:22:46.302 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.303 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:46.309 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:22:46.311 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.312 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:46.312 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.312 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:46.313 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.313 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:22:46.313 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:46.314 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:22:46.315 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:22:46.315 INFO 2981023 --- [ocp-schedules-23,df752f5f53494d08,61eba0b971b79a07] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:22:46.321 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:22:46.325 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.326 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:22:46.326 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.327 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:22:46.327 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:22:46.329 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:46.329 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.345 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:22:46.346 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.347 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.347 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:22:46.347 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:22:46.348 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:22:46.346852+08:00, lastProcessTime=2024-09-11T17:22:36.337349+08:00 2024-09-11 17:22:46.349 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:22:46.354 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:22:46.355 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:22:46.373 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:22:46.383 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:22:46.400 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:22:46.402 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:22:46.405 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:22:46.409 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.410 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:22:46.410 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.410 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.411 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:22:46.411 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.411 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.412 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.412 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:22:46.413 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:22:46.413 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:46.413 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.414 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:22:46.414 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.414 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:22:46.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:22:46.418 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:22:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:22:46.432 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:22:46.436 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:22:46.439 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:22:46.445 INFO 2981023 --- [host-check-host-status3,e11965396e3f37c0,c05ba4a5163810e2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:22:46.450 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:22:46.453 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:46.459 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:22:46.464 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:22:46.464 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.465 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:46.466 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:22:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:22:46.469 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.470 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:46.472 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.475 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:22:46.475 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:22:46.483 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:22:46.483 INFO 2981023 --- [host-check-host-status2,e11965396e3f37c0,156de38fd3cde4fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:22:46.484 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:22:46.485 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.489 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:22:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:22:46.491 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:46.491 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:22:46.501 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.501 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:22:46.502 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.502 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:22:46.506 INFO 2981023 --- [host-check-host-status1,e11965396e3f37c0,745f1c11ec8ecc84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:22:46.509 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.510 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.510 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:22:46.510 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:22:46.510 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:22:46.511 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.511 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:22:46.512 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:22:46.513 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.513 INFO 2981023 --- [host-check-host-status4,e11965396e3f37c0,d31b61fabe86d0c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:22:46.518 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:22:46.528 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.532 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.532 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:22:46.533 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:22:46.533 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.536 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:22:46.536 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:22:46.544 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.544 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.545 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:46.546 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:22:46.551 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:22:46.551 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:46.551 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:22:46.554 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:22:46.558 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:22:46.559 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:22:46.567 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:22:46.580 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:22:46.582 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.583 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.584 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.584 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.584 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:22:46.584 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:22:46.585 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:22:46.585 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.585 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:22:46.589 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:22:46.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:22:46.593 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:22:46.595 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:22:46.599 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:22:46.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:22:46.613 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:22:46.613 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:22:46.617 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:22:46.620 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:22:46.634 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.635 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.635 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:22:46.636 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:22:46.636 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.640 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:22:46.644 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:22:46.346852+08:00 2024-09-11 17:22:46.649 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:22:46.650 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.651 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.651 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:22:46.658 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:46.659 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:22:46.659 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:22:46.664 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:46.664 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:22:46.670 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:22:46.670 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:22:46.670 INFO 2981023 --- [ocp-schedules-25,5abf73c6424c7ca4,ba53e6f7e0f6ae66] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:22:46.681 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:22:46.697 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:22:46.702 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:22:46.705 INFO 2981023 --- [ocp-schedules-4,be9110269e321bb8,ca213ca98918ecd6] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:22:46.933 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:46.933 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:46.979 INFO 2981023 --- [host-check-clockdiff1,bc3e041a2ecf7497,8f0046690c5a9da0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046566 0 0 , error= 2024-09-11 17:22:46.979 INFO 2981023 --- [host-check-clockdiff2,bc3e041a2ecf7497,a87df1d0cf474862] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046566 0 0 , error= 2024-09-11 17:22:47.139 INFO 2981023 --- [ocp-async-2,94bbe9fb5a493926,38d3386725439528] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,2530eb48bd940656,ede115efd1adb612] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,2530eb48bd940656,ede115efd1adb612] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:47.331 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,c9c33d4701c4eb2c,a1786390c8efe778] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=c9c33d4701c4eb2c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:22:47.331 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,38d7b5f0fdec27e2,eaaec76aa2da175f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=38d7b5f0fdec27e2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:22:47.336 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,c9c33d4701c4eb2c,a1786390c8efe778] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=c9c33d4701c4eb2c, duration=7 ms] 2024-09-11 17:22:47.336 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,38d7b5f0fdec27e2,eaaec76aa2da175f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=38d7b5f0fdec27e2, duration=6 ms] 2024-09-11 17:22:47.341 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7507993eb8c42a47,74c9a74f196f573a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=7507993eb8c42a47, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:22:47.341 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7507993eb8c42a47,74c9a74f196f573a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=7507993eb8c42a47, duration=2 ms] 2024-09-11 17:22:47.429 INFO 2981023 --- [ocp-schedules-22,cd0c934894329f69,e1d7e27a5b22cd25] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-09-11 17:22:47.429 INFO 2981023 --- [ocp-schedules-22,cd0c934894329f69,e1d7e27a5b22cd25] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-09-11 17:22:47.736 INFO 2981023 --- [ocp-async-1,b8e8dd7eddc2209c,05f7c7b866f53e00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:22:49.139 INFO 2981023 --- [ocp-async-10,4ff4581c49f24144,c811d0989dd52089] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:22:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:22:49.933 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:49.934 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:50.192 INFO 2981023 --- [ocp-schedules-27,69e1d8a5dd7444d6,7f0bfa9bb7705a0c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:22:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,ec44591318a58fe8,20c61141da992bb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,ec44591318a58fe8,20c61141da992bb2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:50.459 INFO 2981023 --- [ocp-schedules-5,866f47a3e68600be,14668216f1586fbd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:22:50.511 WARN 2981023 --- [scanner-13471312-1,866f47a3e68600be,fe2dafcc3faa9fde] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:22:50.572 INFO 2981023 --- [ocp-schedules-2,6a25533f3cafadff,fdc0f6bdfd7ca679] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:22:50.572+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:22:50.583 INFO 2981023 --- [ocp-schedules-2,6a25533f3cafadff,fdc0f6bdfd7ca679] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:22:50.726 INFO 2981023 --- [ocp-schedules-2,6a25533f3cafadff,fdc0f6bdfd7ca679] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 46 to SqlAggregateData 2024-09-11 17:22:50.728 INFO 2981023 --- [ocp-schedules-2,6a25533f3cafadff,fdc0f6bdfd7ca679] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:22:50.996 INFO 2981023 --- [ocp-schedules-1,fca944c92671215a,fe15284432f2b2c6] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:22:50.997 INFO 2981023 --- [ocp-schedules-1,fca944c92671215a,fe15284432f2b2c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:51.022 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:22:51.022 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.023 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:22:51.023 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:22:51.023 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.023 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.025 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:22:51.027 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:22:51.027 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:22:51.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.030 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.031 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.031 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.031 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.037 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.037 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.041 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.042 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.043 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.044 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.044 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.044 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:22:51.051 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.051 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.051 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:22:51.052 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1AE1D6DB-0-0] 2024-09-11 17:22:51.052 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:22:51.052 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.052 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.052 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:22:51.052 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:22:51.057 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.058 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.058 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.058 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:22:51.063 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.063 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:22:51.065 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.065 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:22:51.066 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.067 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:22:51.067 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.068 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:22:51.068 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.069 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:22:51.069 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:22:51.073 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.074 INFO 2981023 --- [host-check-clockdiff3,bc3e041a2ecf7497,0731dbe95396aa46] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046571 1 1 , error= 2024-09-11 17:22:51.074 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:22:51.076 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:22:51.076 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:22:51.076 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1AE1D6E5-0-0] 2024-09-11 17:22:51.076 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:22:51.080 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.081 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:22:51.082 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.082 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.083 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:22:51.083 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.083 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:22:51.084 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.084 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:22:51.084 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:22:51.084 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.085 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:22:51.085 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:22:51.086 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.086 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.086 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:22:51.086 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:22:51.087 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.087 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.087 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:22:51.088 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:22:51.088 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.088 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.089 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:22:51.089 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:22:51.089 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:22:51.089 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.090 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:51.090 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:22:51.090 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.091 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:22:51.091 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:22:51.091 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:22:51.091 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.092 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.093 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:22:51.102 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:51.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:22:51.105 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.105 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.105 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:22:51.106 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:22:51.107 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:22:51.109 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:22:51.109 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:22:51.109 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:22:51.110 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:22:51.111 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:22:51.111 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.112 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:22:51.116 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:22:51.116 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:22:51.117 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:22:51.117 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.118 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:22:51.118 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.118 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:51.118 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:22:51.141 INFO 2981023 --- [ocp-async-7,ae4ad2b1d8dabbb3,9c2e0e7a91edd61d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:51.144 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:22:51.145 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:22:51.145 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:22:51.145 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:22:51.150 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:22:51.152 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:22:51.152 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:22:51.152 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:22:51.152 INFO 2981023 --- [ocp-schedules-1,fca944c92671215a,fe15284432f2b2c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:22:51.495 INFO 2981023 --- [ocp-schedules-7,afbcafbdee82414d,3a02c3fbef85ef97] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:22:51.498 INFO 2981023 --- [ocp-schedules-7,afbcafbdee82414d,3a02c3fbef85ef97] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:22:51.498 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:22:51.498 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:22:51.498 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:22:51.499 INFO 2981023 --- [ocp-schedules-7,afbcafbdee82414d,3a02c3fbef85ef97] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:22:51.707 INFO 2981023 --- [ocp-schedules-28,7efc055c0dc42674,a1447f66c1c326a2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:22:52.043 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:22:52.061 WARN 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2024-09-11 17:22:52.075 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2024-09-11 17:22:52.076 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2024-09-11 17:22:52.077 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2024-09-11 17:22:52.078 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2024-09-11 17:22:52.079 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2024-09-11 17:22:52.080 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2024-09-11 17:22:52.081 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2024-09-11 17:22:52.082 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2024-09-11 17:22:52.083 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2024-09-11 17:22:52.084 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 3 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2024-09-11 17:22:52.085 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2024-09-11 17:22:52.086 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2024-09-11 17:22:52.087 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2024-09-11 17:22:52.088 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2024-09-11 17:22:52.089 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2024-09-11 17:22:52.090 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2024-09-11 17:22:52.091 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2024-09-11 17:22:52.092 INFO 2981023 --- [ocp-schedules-5,d7d4529d3fba28cd,9f1f658d2aa850d1] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=126, updatedCount=126, failedToUpdateCount=0 2024-09-11 17:22:52.934 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:52.936 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:53.004 INFO 2981023 --- [ocp-schedules-14,c0b6c5dcf896dcd4,043e77b508946bce] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:22:53.004 INFO 2981023 --- [ocp-schedules-14,c0b6c5dcf896dcd4,043e77b508946bce] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:22:53.141 INFO 2981023 --- [ocp-async-2,342dcdeddf0df079,f48e4fd50b7e861a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,8224ec96c8c6156f,99eca4fdded94594] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,8224ec96c8c6156f,99eca4fdded94594] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:53.420 INFO 2981023 --- [ocp-schedules-28,52348e2658486c68,e6efa91319fb928e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:22:53.428 INFO 2981023 --- [ocp-schedules-28,52348e2658486c68,e6efa91319fb928e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:22:53.428 INFO 2981023 --- [ocp-schedules-28,52348e2658486c68,e6efa91319fb928e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:22:54.150 INFO 2981023 --- [ocp-schedules-23,ba3e7927d8e5b0b8,0258708bc85f27c3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:22:55.141 INFO 2981023 --- [ocp-async-4,6a724906fa82aac6,35c994e5983cab65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:55.584 INFO 2981023 --- [ocp-schedules-12,ba452908d92ad341,381cb06ee3e95e51] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:22:55.584 INFO 2981023 --- [ocp-schedules-12,ba452908d92ad341,381cb06ee3e95e51] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:22:55.604 INFO 2981023 --- [Collect all tenants major compactions2,ba452908d92ad341,d63fe5fe3b86dec6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:22:55.604 INFO 2981023 --- [Collect all tenants major compactions3,ba452908d92ad341,109db141d9267a4e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:22:55.606 INFO 2981023 --- [Collect all tenants major compactions6,ba452908d92ad341,ff256ab1d7da40cc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:22:55.607 INFO 2981023 --- [Collect all tenants major compactions4,ba452908d92ad341,48a4e07fd7a3800d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:22:55.608 INFO 2981023 --- [Collect all tenants major compactions1,ba452908d92ad341,37dfcc05f08311fe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:22:55.610 INFO 2981023 --- [Collect all tenants major compactions5,ba452908d92ad341,237ca35af0872ddb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:22:55.617 INFO 2981023 --- [ocp-schedules-5,82c86f14eb1a8c44,9e6ce0ef02d674b1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:22:55.617 INFO 2981023 --- [ocp-schedules-5,82c86f14eb1a8c44,9e6ce0ef02d674b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:22:55.618 INFO 2981023 --- [ocp-schedules-5,82c86f14eb1a8c44,9e6ce0ef02d674b1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:22:55.618 INFO 2981023 --- [ocp-schedules-5,82c86f14eb1a8c44,9e6ce0ef02d674b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:22:55.618 INFO 2981023 --- [ocp-schedules-5,82c86f14eb1a8c44,9e6ce0ef02d674b1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:22:55.618 INFO 2981023 --- [ocp-schedules-5,82c86f14eb1a8c44,9e6ce0ef02d674b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:22:55.621 INFO 2981023 --- [Collect all tenants major compactions3,ba452908d92ad341,109db141d9267a4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:55.622 INFO 2981023 --- [Collect all tenants major compactions5,ba452908d92ad341,237ca35af0872ddb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:55.622 INFO 2981023 --- [Collect all tenants major compactions4,ba452908d92ad341,48a4e07fd7a3800d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:55.622 INFO 2981023 --- [Collect all tenants major compactions2,ba452908d92ad341,d63fe5fe3b86dec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:55.622 INFO 2981023 --- [Collect all tenants major compactions3,ba452908d92ad341,109db141d9267a4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:22:55.623 INFO 2981023 --- [Collect all tenants major compactions4,ba452908d92ad341,48a4e07fd7a3800d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:22:55.623 INFO 2981023 --- [Collect all tenants major compactions5,ba452908d92ad341,237ca35af0872ddb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:22:55.624 INFO 2981023 --- [Collect all tenants major compactions6,ba452908d92ad341,ff256ab1d7da40cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:55.624 INFO 2981023 --- [Collect all tenants major compactions2,ba452908d92ad341,d63fe5fe3b86dec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:22:55.625 INFO 2981023 --- [Collect all tenants major compactions6,ba452908d92ad341,ff256ab1d7da40cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:22:55.627 INFO 2981023 --- [Collect all tenants major compactions4,ba452908d92ad341,48a4e07fd7a3800d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:22:55.627 INFO 2981023 --- [Collect all tenants major compactions3,ba452908d92ad341,109db141d9267a4e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:22:55.627 INFO 2981023 --- [Collect all tenants major compactions3,ba452908d92ad341,109db141d9267a4e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:22:55.627 INFO 2981023 --- [Collect all tenants major compactions4,ba452908d92ad341,48a4e07fd7a3800d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:22:55.627 INFO 2981023 --- [Collect all tenants major compactions2,ba452908d92ad341,d63fe5fe3b86dec6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:22:55.627 INFO 2981023 --- [Collect all tenants major compactions2,ba452908d92ad341,d63fe5fe3b86dec6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:22:55.627 INFO 2981023 --- [Collect all tenants major compactions5,ba452908d92ad341,237ca35af0872ddb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:22:55.627 INFO 2981023 --- [Collect all tenants major compactions5,ba452908d92ad341,237ca35af0872ddb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:22:55.628 INFO 2981023 --- [Collect all tenants major compactions6,ba452908d92ad341,ff256ab1d7da40cc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:22:55.628 INFO 2981023 --- [Collect all tenants major compactions6,ba452908d92ad341,ff256ab1d7da40cc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:22:55.628 INFO 2981023 --- [Collect all tenants major compactions1,ba452908d92ad341,37dfcc05f08311fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:22:55.629 INFO 2981023 --- [Collect all tenants major compactions1,ba452908d92ad341,37dfcc05f08311fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:22:55.630 INFO 2981023 --- [Collect all tenants major compactions1,ba452908d92ad341,37dfcc05f08311fe] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:22:55.630 INFO 2981023 --- [Collect all tenants major compactions1,ba452908d92ad341,37dfcc05f08311fe] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:22:55.631 INFO 2981023 --- [ocp-schedules-12,ba452908d92ad341,381cb06ee3e95e51] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:22:55.662 INFO 2981023 --- [ocp-schedules-5,dabe23d5bcc2f934,bb11c3dddbb9e09d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:22:55.668 INFO 2981023 --- [ocp-schedules-5,dabe23d5bcc2f934,bb11c3dddbb9e09d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:22:55.668 INFO 2981023 --- [ocp-schedules-5,dabe23d5bcc2f934,bb11c3dddbb9e09d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:22:55.936 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:55.936 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,425e230ae470ef0b,8364c36fa9975cbe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,425e230ae470ef0b,8364c36fa9975cbe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:22:56.351274+08:00, lastProcessTime=2024-09-11T17:22:46.346852+08:00 2024-09-11 17:22:56.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:22:56.351274+08:00 2024-09-11 17:22:57.141 INFO 2981023 --- [ocp-async-9,9e30fb0ab18d61d0,adf6e0a7765acd66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:57.601 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e77722eec7d802fe,00291ca8051cec1c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=e77722eec7d802fe, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:22:57.601 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fee691496eb005cd,de2a152909dabd98] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=fee691496eb005cd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:22:57.605 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e77722eec7d802fe,00291ca8051cec1c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=e77722eec7d802fe, duration=9 ms] 2024-09-11 17:22:57.609 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fee691496eb005cd,de2a152909dabd98] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=fee691496eb005cd, duration=12 ms] 2024-09-11 17:22:57.613 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,038bd07eb5c73ab8,72f1eb0456b4d9c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=038bd07eb5c73ab8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:22:57.614 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,038bd07eb5c73ab8,72f1eb0456b4d9c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=038bd07eb5c73ab8, duration=4 ms] 2024-09-11 17:22:57.736 INFO 2981023 --- [ocp-async-6,d86f8882b948e498,aab3522791957552] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:22:58.105 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.107 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdoc 2024-09-11 17:22:58.114 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.119 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.120 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.125 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.130 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.133 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.137 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.140 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.143 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.146 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.149 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.151 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.152 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.153 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.153 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.154 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.155 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.155 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.156 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.157 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.161 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.164 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.167 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.169 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:16:58.004, end=2024-09-11T17:21:58.101} 2024-09-11 17:22:58.170 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:22:58.936 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:22:58.937 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:22:59.142 INFO 2981023 --- [ocp-async-2,e54ff05ddb50a8f2,a01880d6e32e4f05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:22:59.192 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:18:58.771, end=2024-09-11T17:20:59.189} 2024-09-11 17:22:59.195 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=l 2024-09-11 17:22:59.195 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:22:59.196 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:18:58.771, end=2024-09-11T17:20:59.189},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:22:59.196 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:22:59.196 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:22:59.197 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:22:59.199 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:18:58.771, end=2024-09-11T17:20:59.189},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:22:59.199 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:22:59.199 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:22:59.209 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:22:59.225 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179225000, endTimeUs=1726046459189000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=BD5DCCE7A516959882352D2E7F3E65DA, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:22:59.225Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.227 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:22:59.227 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.232 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.232 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=BD5DCCE7A516959882352D2E7F3E65DA, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.234 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.234 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.239 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.239 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=BD5DCCE7A516959882352D2E7F3E65DA, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.240 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:22:59.240 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.225, end=2024-09-11T17:20:59.189}, clusterId=1000006, tenantId=1000006, sqlId=BD5DCCE7A516959882352D2E7F3E65DA, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.240 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:22:59.242 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:18:58.771, end=2024-09-11T17:20:59.189},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:22:59.242 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:22:59.242 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:22:59.288 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179288000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.288Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.290 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179290000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.290Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.290 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179290000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.290Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.292 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179292000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.292Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.297 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179297000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.297Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.299 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179299000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.299Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.300 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179300000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.300Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.305 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179305000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.305Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,e2005b759c35534d,b9b8712ff2ad2796] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046580, interval=60 2024-09-11 17:22:59.318 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 58 2024-09-11 17:22:59.318 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.320 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:22:59.320 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.322 INFO 2981023 --- [prometheus-minute-schedule-1,e2005b759c35534d,b9b8712ff2ad2796] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:22:59.339 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:22:59.339 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.344 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:22:59.344 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.349 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.349 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.350 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.350 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.353 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.353 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.353 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.353 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.359 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:22:59.359 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.363 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.363 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.364 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:22:59.364 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.366 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.366 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.368 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:22:59.368 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.373 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.373 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.373 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.373 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.375 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.376 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:22:59.376 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.378 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.378 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.379 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.380 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.383 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.29, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.384 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.384 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.389 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.389 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.394 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.394 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.394 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.394 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.396 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.398 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.398 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.398 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.398 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.398 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.398 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.400 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.400 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.401 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.401 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.401 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.402 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.403 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.403 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.404 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.404 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.404 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.404 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.404 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.288, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.407 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.408 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.413 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.413 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.413 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.413 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.415 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.29, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.416 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.292, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.417 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.418 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.420 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.3, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.423 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.299, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.424 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.305, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.424 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.297, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.435 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179435000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.435Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.448 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:22:59.448 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.451 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179451000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.451Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.456 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179456000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.456Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.461 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179461000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=C0995E24A01F869BEB311E8C790E6C86, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.461Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.462 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:22:59.462 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.463 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179463000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.463Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.467 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179467000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.467Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.469 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:22:59.469 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.471 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.471 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.471 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:22:59.471 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.473 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179473000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.473Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.474 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179474000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84D9CB28179A8A6A515E8623A68C21C2, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.474Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.477 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:22:59.477 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.477 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.477 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.479 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.479 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.479 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.479 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.484 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:22:59.484 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.485 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:22:59.485 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.487 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.487 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.488 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 18 2024-09-11 17:22:59.488 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.490 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.490 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.492 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.492 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.492 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.492 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.493 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.493 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.493 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.493 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.495 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.495 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.496 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.496 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.498 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.498 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.500 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.500 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84D9CB28179A8A6A515E8623A68C21C2, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.502 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.451, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.503 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.503 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.504 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.435, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.505 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.505 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.505 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.505 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.505 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.505 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.505 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.505 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.506 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.507 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.507 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.507 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.508 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.510 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.512 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.461, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=C0995E24A01F869BEB311E8C790E6C86, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.512 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.512 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.514 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.514 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84D9CB28179A8A6A515E8623A68C21C2, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.515 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.516 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.456, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.516 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.517 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.517 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.518 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.518 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.519 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.519 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.463, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.519 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:22:59.521 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.474, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=84D9CB28179A8A6A515E8623A68C21C2, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.522 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.473, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.522 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.467, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.522 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:22:59.523 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:18:58.771, end=2024-09-11T17:20:59.189},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:22:59.523 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:22:59.523 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:22:59.524 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:22:59.525 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:18:58.771, end=2024-09-11T17:20:59.189},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:22:59.525 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:22:59.525 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:22:59.534 INFO 2981023 --- [plan_change_detector18,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:22:59.547 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960179547000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:22:59.547Z, endTime=2024-09-11T09:20:59.189Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:22:59.549 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:22:59.549 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:22:59.553 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.553 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.555 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:22:59.555 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:22:59.559 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:22:59.559 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046459189000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:22:59.560 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:22:59.560 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:22:59.547, end=2024-09-11T17:20:59.189}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:22:59.560 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:22:59.561 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:18:58.771, end=2024-09-11T17:20:59.189},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:22:59.561 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:22:59.561 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:22:59.561 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:23:00.678 INFO 2981023 --- [ocp-schedules-5,739f9f5037118b6e,da3a0019f6a4a832] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:23:00.711 WARN 2981023 --- [scanner-13471540-1,739f9f5037118b6e,b664782121317d2f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:23:00.733 INFO 2981023 --- [alarm-detect-2,739f9f5037118b6e,aef66536ab92710c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643663, data is empty. 2024-09-11 17:23:00.735 INFO 2981023 --- [alarm-detect-2,739f9f5037118b6e,aef66536ab92710c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643709, data is empty. 2024-09-11 17:23:00.738 INFO 2981023 --- [alarm-detect-2,739f9f5037118b6e,aef66536ab92710c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643709, data is empty. 2024-09-11 17:23:00.780 INFO 2981023 --- [alarm-detect-4,739f9f5037118b6e,6de20a6d46f0a011] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643663, data is empty. 2024-09-11 17:23:00.780 INFO 2981023 --- [alarm-detect-8,739f9f5037118b6e,3214a278ce549222] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643663, data is empty. 2024-09-11 17:23:00.782 INFO 2981023 --- [alarm-detect-4,739f9f5037118b6e,6de20a6d46f0a011] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643709, data is empty. 2024-09-11 17:23:00.783 INFO 2981023 --- [alarm-detect-8,739f9f5037118b6e,3214a278ce549222] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643709, data is empty. 2024-09-11 17:23:00.788 INFO 2981023 --- [alarm-detect-4,739f9f5037118b6e,6de20a6d46f0a011] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643709, data is empty. 2024-09-11 17:23:00.788 INFO 2981023 --- [alarm-detect-8,739f9f5037118b6e,3214a278ce549222] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643709, data is empty. 2024-09-11 17:23:00.876 INFO 2981023 --- [alarm-detect-6,739f9f5037118b6e,f11e73921d695a96] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643663, data is empty. 2024-09-11 17:23:00.880 INFO 2981023 --- [alarm-detect-6,739f9f5037118b6e,f11e73921d695a96] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643709, data is empty. 2024-09-11 17:23:00.883 INFO 2981023 --- [alarm-detect-6,739f9f5037118b6e,f11e73921d695a96] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643709, data is empty. 2024-09-11 17:23:01.142 INFO 2981023 --- [ocp-async-4,651d8602ba37d473,196142019d88b29b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:01.709 INFO 2981023 --- [ocp-schedules-23,f55cf4903f92ed3f,34e93c241e384f4d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:23:01.937 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:01.938 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,605183ff79058d59,5bea969566ac188e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,605183ff79058d59,5bea969566ac188e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:03.142 INFO 2981023 --- [ocp-async-9,220d1a18b38e9240,c3d2d0d781ad4036] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:04.938 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:04.939 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:05.142 INFO 2981023 --- [ocp-async-8,df9825e96cedf587,74b7ea8d3c7577bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,c539e3adfe1e7211,f8a569ff3827bd81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,c539e3adfe1e7211,f8a569ff3827bd81] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:23:06.336851+08:00, lastProcessTime=2024-09-11T17:22:56.351274+08:00 2024-09-11 17:23:06.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:23:06.336851+08:00 2024-09-11 17:23:07.133 INFO 2981023 --- [ocp-schedules-17,0ff1c3043909e3be,8e35fd32a2dfd82a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:23:07.136 INFO 2981023 --- [ocp-schedules-20,87e3741364edf040,29bfaacd156008a0] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:23:07.136+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:23:07.136 INFO 2981023 --- [ocp-schedules-20,87e3741364edf040,29bfaacd156008a0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:23:07.138 INFO 2981023 --- [ocp-schedules-20,87e3741364edf040,29bfaacd156008a0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:23:07.139 INFO 2981023 --- [ocp-schedules-20,87e3741364edf040,29bfaacd156008a0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:23:07.142 INFO 2981023 --- [ocp-async-10,7df60ccac154aade,dd24792a6ca041bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:07.736 INFO 2981023 --- [ocp-async-2,f6a0f88b5b6d080d,8152443efeb3f51b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:23:07.939 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:07.939 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,098450f8ce0e7657,9899e017a4e5d1c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,098450f8ce0e7657,9899e017a4e5d1c0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:09.142 INFO 2981023 --- [ocp-async-7,42dc7fefa612c7dc,890bd73f8f71535e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:10.004 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:23:10.004 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:23:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:23:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:23:10.939 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:10.940 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:10.942 INFO 2981023 --- [ocp-schedules-13,8e30fba8dea9b15e,2821c0dfe97be80c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:23:10.988 WARN 2981023 --- [scanner-13471893-1,8e30fba8dea9b15e,5ecc5e8bfc375e60] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:23:11.142 INFO 2981023 --- [ocp-async-8,4d35e7307cdd05c4,72ba12e51fac043b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,50b48431c0309301,bc4ecfb3def73ebe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,50b48431c0309301,bc4ecfb3def73ebe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:11.711 INFO 2981023 --- [ocp-schedules-5,b3b21ed2a067acf6,413635eb50f2f918] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:23:12.300 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046520, nodeCount=1000, affectRows=1000 2024-09-11 17:23:12.302 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046520, nodeCount=999, validCount=999 2024-09-11 17:23:12.361 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046520, nodeCount=1000, affectRows=1000 2024-09-11 17:23:12.363 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046520, nodeCount=1000, validCount=1000 2024-09-11 17:23:12.370 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046520, nodeCount=48, affectRows=48 2024-09-11 17:23:12.370 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046520, nodeCount=48, validCount=48 2024-09-11 17:23:13.142 INFO 2981023 --- [ocp-async-5,3325d8a7c5dda275,e67570f7098b49e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:13.429 INFO 2981023 --- [ocp-schedules-18,29a1693d8ccd57b4,5db69b2ab2ee9883] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:23:13.434 INFO 2981023 --- [ocp-schedules-18,29a1693d8ccd57b4,5db69b2ab2ee9883] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:23:13.434 INFO 2981023 --- [ocp-schedules-18,29a1693d8ccd57b4,5db69b2ab2ee9883] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:23:13.940 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:13.941 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,c8fe310ff5e51bcd,9f95665f232311c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,c8e16c2114b0022e,36bac8a4453c4bf2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:23:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,c8fe310ff5e51bcd,9f95665f232311c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:14.789 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046520, nodeCount=480, affectRows=480 2024-09-11 17:23:15.142 INFO 2981023 --- [ocp-async-6,938257f39ce0e5e7,79dae88ba2505f44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:16.027 INFO 2981023 --- [ocp-schedules-28,30a2e2a5c8ae7ace,6b8d1ac429fd0404] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:23:16.027 INFO 2981023 --- [host-check-clockdiff1,30a2e2a5c8ae7ace,3c9f1473091dc77f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:23:16.027 INFO 2981023 --- [host-check-clockdiff2,30a2e2a5c8ae7ace,1f821cc6da895492] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:23:16.027 INFO 2981023 --- [host-check-clockdiff3,30a2e2a5c8ae7ace,78cb302c4f8e944f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:23:16.032 INFO 2981023 --- [ocp-schedules-13,fff2086dc6485716,1651e1e1631947d3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:23:16.032 INFO 2981023 --- [host-check-clockdiff4,30a2e2a5c8ae7ace,0134a883d0b3d941] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:23:16.034 INFO 2981023 --- [ocp-schedules-5,b94daee6461604b3,d722964d6fd824b6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:23:16.034 INFO 2981023 --- [host-check-agent-status1,fff2086dc6485716,b3163a395d337e21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:23:16.035 INFO 2981023 --- [host-check-agent-status2,fff2086dc6485716,f73e581a2893dca9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:23:16.035 INFO 2981023 --- [host-check-agent-status3,fff2086dc6485716,72bb890d63c6658a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:23:16.073 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:23:16.073 INFO 2981023 --- [host-check-clockdiff4,30a2e2a5c8ae7ace,0134a883d0b3d941] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046596 0 0 , error= 2024-09-11 17:23:16.073 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:23:16.073 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:23:16.075 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:23:16.082 INFO 2981023 --- [host-check-agent-status4,fff2086dc6485716,f1a981172d0416de] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:23:16.083 INFO 2981023 --- [host-check-clockdiff2,30a2e2a5c8ae7ace,1f821cc6da895492] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046596 0 0 , error= 2024-09-11 17:23:16.111 INFO 2981023 --- [host-check-agent-status2,fff2086dc6485716,f73e581a2893dca9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:23:16.112 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:23:16.112 INFO 2981023 --- [host-check-agent-status1,fff2086dc6485716,b3163a395d337e21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:23:16.116 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:23:16.116 INFO 2981023 --- [host-check-agent-status1,fff2086dc6485716,b3163a395d337e21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:23:16.116 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:23:16.117 INFO 2981023 --- [host-check-agent-status2,fff2086dc6485716,f73e581a2893dca9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:23:16.121 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:23:16.122 INFO 2981023 --- [host-check-agent-status3,fff2086dc6485716,72bb890d63c6658a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:23:16.126 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:23:16.133 INFO 2981023 --- [host-check-agent-status3,fff2086dc6485716,72bb890d63c6658a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:23:16.137 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:23:16.147 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:23:16.185 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:23:16.189 INFO 2981023 --- [ocp-schedules-8,2fb2f9f0c34d5eff,cf349002856a9b3d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:23:16.189 INFO 2981023 --- [ocp-schedules-26,ff546af9c11f0e3d,73c28e38f9d4c646] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:23:16.189 INFO 2981023 --- [host-check-agent-status4,fff2086dc6485716,f1a981172d0416de] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:23:16.189 INFO 2981023 --- [ocp-schedules-10,8678267846667d77,4c3e5e50fcc320a3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:23:16.189 INFO 2981023 --- [ocp-schedules-19,0b076c1efd01d9ae,28e196fd65908420] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:23:16.195 INFO 2981023 --- [host-check-agent-status4,fff2086dc6485716,f1a981172d0416de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:23:16.195 INFO 2981023 --- [ocp-schedules-10,8678267846667d77,4c3e5e50fcc320a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:23:16.195 INFO 2981023 --- [ocp-schedules-8,2fb2f9f0c34d5eff,cf349002856a9b3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:23:16.204 INFO 2981023 --- [ocp-schedules-26,ff546af9c11f0e3d,73c28e38f9d4c646] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:23:16.221 INFO 2981023 --- [ocp-schedules-26,ff546af9c11f0e3d,73c28e38f9d4c646] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:23:16.357 INFO 2981023 --- [ocp-schedules-9,d94a9765510747cd,aff677fdae6ad723] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:23:16.382 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:23:16.381384+08:00, lastProcessTime=2024-09-11T17:23:06.336851+08:00 2024-09-11 17:23:16.384 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:23:16.386 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:23:16.386813754+08:00 2024-09-11 17:23:16.389 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:23:16.387+08:00 2024-09-11 17:23:16.391 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:23:16.387+08:00 and status AVAILABLE 2024-09-11 17:23:16.396 INFO 2981023 --- [host-check-agent-status3,fff2086dc6485716,72bb890d63c6658a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:23:16.396 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:23:16.398 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:23:16.398458796+08:00 2024-09-11 17:23:16.399 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:23:16.398+08:00 2024-09-11 17:23:16.399 INFO 2981023 --- [host-check-agent-status4,fff2086dc6485716,f1a981172d0416de] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:23:16.399 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:23:16.400 INFO 2981023 --- [host-check-agent-status3,fff2086dc6485716,72bb890d63c6658a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:23:16.400 INFO 2981023 --- [host-check-agent-status3,fff2086dc6485716,72bb890d63c6658a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:23:16.400 INFO 2981023 --- [host-check-agent-status3,fff2086dc6485716,72bb890d63c6658a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:23:16.400 INFO 2981023 --- [host-check-agent-status1,fff2086dc6485716,b3163a395d337e21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:23:16.400 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.401 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:16.402 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:23:16.398+08:00 and status AVAILABLE 2024-09-11 17:23:16.402 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:23:16.402 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.402 INFO 2981023 --- [host-check-agent-status4,fff2086dc6485716,f1a981172d0416de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:23:16.402 INFO 2981023 --- [host-check-agent-status4,fff2086dc6485716,f1a981172d0416de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:23:16.402 INFO 2981023 --- [host-check-agent-status4,fff2086dc6485716,f1a981172d0416de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:23:16.402 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:16.403 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:23:16.403 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:23:16.401395249+08:00 2024-09-11 17:23:16.403 INFO 2981023 --- [host-check-agent-status1,fff2086dc6485716,b3163a395d337e21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:23:16.403 INFO 2981023 --- [host-check-agent-status1,fff2086dc6485716,b3163a395d337e21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:23:16.403 INFO 2981023 --- [host-check-agent-status1,fff2086dc6485716,b3163a395d337e21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:23:16.403 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:23:16.403+08:00 2024-09-11 17:23:16.406 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:23:16.403+08:00 and status AVAILABLE 2024-09-11 17:23:16.407 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:23:16.407 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:23:16.409 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:23:16.408509177+08:00 2024-09-11 17:23:16.409 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:23:16.409+08:00 2024-09-11 17:23:16.411 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:23:16.409+08:00 and status AVAILABLE 2024-09-11 17:23:16.412 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:23:16.414 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:23:16.415 INFO 2981023 --- [host-check-agent-status2,fff2086dc6485716,f73e581a2893dca9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:23:16.416 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.416 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:16.417 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:16.420 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:23:16.424 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.425 INFO 2981023 --- [ocp-schedules-8,2fb2f9f0c34d5eff,cf349002856a9b3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:23:16.425 INFO 2981023 --- [host-check-agent-status2,fff2086dc6485716,f73e581a2893dca9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:23:16.425 INFO 2981023 --- [host-check-agent-status2,fff2086dc6485716,f73e581a2893dca9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:23:16.425 INFO 2981023 --- [host-check-agent-status2,fff2086dc6485716,f73e581a2893dca9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:23:16.426 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:16.426 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:23:16.465 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.466 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:16.467 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:23:16.468 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:23:16.468 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:23:16.468 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:23:16.468 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:23:16.469 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:23:16.469 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:23:16.470 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:23:16.493 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:23:16.497 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:23:16.501 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.501 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.501 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:23:16.506 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:23:16.509 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:23:16.518 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:23:16.526 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:23:16.526 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:23:16.531 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:23:16.536 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:23:16.566 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:23:16.572 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.573 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.574 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:23:16.583 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.584 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.584 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:16.585 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:23:16.585 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:23:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.586 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:23:16.588 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:23:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:23:16.590 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:23:16.597 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:23:16.605 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.607 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:23:16.607 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:16.608 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.609 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:23:16.611 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:23:16.612 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:23:16.621 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:23:16.622 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:16.622 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:23:16.635 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:23:16.636 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:23:16.643 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:23:16.649 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:16.649 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:16.653 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.654 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.654 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:23:16.660 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:23:16.662 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:23:16.662 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.663 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.663 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.664 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.664 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:16.664 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:16.665 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:23:16.665 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:23:16.668 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:23:16.668 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:23:16.670 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:23:16.674 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:23:16.676 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:23:16.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:23:16.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:23:16.680 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:23:16.683 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:23:16.688 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:23:16.692 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.694 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.694 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:16.695 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:23:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.697 INFO 2981023 --- [host-check-host-status1,b94daee6461604b3,5b8b72502f7bdbb0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:23:16.698 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:23:16.698 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:23:16.698 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:23:16.701 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:23:16.705 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:23:16.708 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:23:16.711 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:23:16.713 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:16.714 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:23:16.734 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:16.745 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:23:16.381384+08:00 2024-09-11 17:23:16.745 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:16.747 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:23:16.750 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:23:16.751 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:23:16.751 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:23:16.756 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:23:16.757 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:23:16.760 INFO 2981023 --- [host-check-host-status4,b94daee6461604b3,b45a1b790e6b92ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:23:16.765 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:23:16.773 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:23:16.775 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:23:16.780 INFO 2981023 --- [host-check-host-status2,b94daee6461604b3,8c6d7d6f6a216d41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:23:16.784 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:23:16.786 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:23:16.787 INFO 2981023 --- [host-check-host-status3,b94daee6461604b3,7ea5b69f726399d1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:23:16.792 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.793 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.793 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:23:16.793 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.794 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.794 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:23:16.801 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:23:16.801 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:23:16.802 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:23:16.802 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:23:16.803 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:23:16.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:16.809 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:16.810 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:23:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:23:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:23:16.815 INFO 2981023 --- [ocp-schedules-10,8678267846667d77,4c3e5e50fcc320a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:23:16.941 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:16.942 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:16.971 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046520, nodeCount=480, validCount=480 2024-09-11 17:23:17.143 INFO 2981023 --- [ocp-async-3,d27e8a90e20e35c4,645ed82ee865da6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,9f3bb1535dc23906,aa0b5e2c8e9c4213] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,9f3bb1535dc23906,aa0b5e2c8e9c4213] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:17.737 INFO 2981023 --- [ocp-async-7,ca9c9cdc1da633e5,b32d635e59a329fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:23:19.143 INFO 2981023 --- [ocp-async-5,87af56d63db5f4f5,523a7b4379481d28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:19.219 INFO 2981023 --- [host-check-clockdiff1,30a2e2a5c8ae7ace,3c9f1473091dc77f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046599 0 0 , error= 2024-09-11 17:23:19.942 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:19.942 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,cc69e9a020e1b264,ed27cc717751a9e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,cc69e9a020e1b264,ed27cc717751a9e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:21.143 INFO 2981023 --- [ocp-async-9,4cc0915adca34089,5a6e5ea4f19a371a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:21.153 INFO 2981023 --- [ocp-schedules-21,0e04149026e1cb1d,92646eb61502745f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:23:21.161 WARN 2981023 --- [scanner-13473539-1,0e04149026e1cb1d,a48ec9d294920cf2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:23:21.222 INFO 2981023 --- [alarm-detect-6,0e04149026e1cb1d,041f4d573ed2a0a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643709, data is empty. 2024-09-11 17:23:21.227 INFO 2981023 --- [alarm-detect-6,0e04149026e1cb1d,041f4d573ed2a0a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643709, data is empty. 2024-09-11 17:23:21.230 INFO 2981023 --- [alarm-detect-6,0e04149026e1cb1d,041f4d573ed2a0a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643709, data is empty. 2024-09-11 17:23:21.253 INFO 2981023 --- [alarm-detect-7,0e04149026e1cb1d,7f6c5c2f5ce79103] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643709, data is empty. 2024-09-11 17:23:21.255 INFO 2981023 --- [alarm-detect-20,0e04149026e1cb1d,8b9c984b8e7c6d64] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643709, data is empty. 2024-09-11 17:23:21.259 INFO 2981023 --- [alarm-detect-7,0e04149026e1cb1d,7f6c5c2f5ce79103] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643709, data is empty. 2024-09-11 17:23:21.262 INFO 2981023 --- [alarm-detect-20,0e04149026e1cb1d,8b9c984b8e7c6d64] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643709, data is empty. 2024-09-11 17:23:21.269 INFO 2981023 --- [alarm-detect-7,0e04149026e1cb1d,7f6c5c2f5ce79103] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643709, data is empty. 2024-09-11 17:23:21.269 INFO 2981023 --- [alarm-detect-20,0e04149026e1cb1d,8b9c984b8e7c6d64] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643709, data is empty. 2024-09-11 17:23:21.363 INFO 2981023 --- [alarm-detect-20,0e04149026e1cb1d,f1e3ce45bb5c412e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643709, data is empty. 2024-09-11 17:23:21.366 INFO 2981023 --- [alarm-detect-20,0e04149026e1cb1d,f1e3ce45bb5c412e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643709, data is empty. 2024-09-11 17:23:21.370 INFO 2981023 --- [alarm-detect-20,0e04149026e1cb1d,f1e3ce45bb5c412e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643709, data is empty. 2024-09-11 17:23:21.498 INFO 2981023 --- [ocp-schedules-20,8bcff7dfda5685a8,dc1024103e050d7a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 17:23:21.498 INFO 2981023 --- [ocp-schedules-20,8bcff7dfda5685a8,dc1024103e050d7a] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 17:23:21.511 INFO 2981023 --- [pool-459-thread-2,8bcff7dfda5685a8,0f3ed67295c1ae55] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:23:21.511 INFO 2981023 --- [pool-459-thread-1,8bcff7dfda5685a8,484c1a1d770467e2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:23:21.511 INFO 2981023 --- [pool-459-thread-3,8bcff7dfda5685a8,78d548a0a8c4c382] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:23:21.530 INFO 2981023 --- [pool-459-thread-2,8bcff7dfda5685a8,0f3ed67295c1ae55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:21.532 INFO 2981023 --- [pool-459-thread-3,8bcff7dfda5685a8,78d548a0a8c4c382] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:21.532 INFO 2981023 --- [pool-459-thread-1,8bcff7dfda5685a8,484c1a1d770467e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:21.533 INFO 2981023 --- [pool-459-thread-2,8bcff7dfda5685a8,0f3ed67295c1ae55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046001000000, 1726046601000000] 2024-09-11 17:23:21.534 INFO 2981023 --- [pool-459-thread-3,8bcff7dfda5685a8,78d548a0a8c4c382] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046001000000, 1726046601000000] 2024-09-11 17:23:21.535 INFO 2981023 --- [pool-459-thread-1,8bcff7dfda5685a8,484c1a1d770467e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046001000000, 1726046601000000] 2024-09-11 17:23:21.538 INFO 2981023 --- [pool-459-thread-3,8bcff7dfda5685a8,78d548a0a8c4c382] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:23:21.619 INFO 2981023 --- [pool-459-thread-1,8bcff7dfda5685a8,484c1a1d770467e2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:23:21.676 INFO 2981023 --- [pool-459-thread-2,8bcff7dfda5685a8,0f3ed67295c1ae55] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:23:21.676 INFO 2981023 --- [ocp-schedules-20,8bcff7dfda5685a8,dc1024103e050d7a] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 17:23:21.714 INFO 2981023 --- [ocp-schedules-7,738148223918cd85,1f29a51dc5fa7568] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:23:21.994 INFO 2981023 --- [host-check-clockdiff3,30a2e2a5c8ae7ace,78cb302c4f8e944f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046601 1 1 , error= 2024-09-11 17:23:22.240 INFO 2981023 --- [ocp-schedules-26,b60b16b5de6c2513,c315237603c6c4ae] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:23:22.240 INFO 2981023 --- [ocp-schedules-26,b60b16b5de6c2513,c315237603c6c4ae] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:23:22.240 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:23:22.240 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:23:22.240 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:23:22.244 INFO 2981023 --- [ocp-schedules-26,b60b16b5de6c2513,c315237603c6c4ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:23:22.245 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:23:22.245 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:23:22.245 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:23:22.246 INFO 2981023 --- [ocp-schedules-26,b60b16b5de6c2513,c315237603c6c4ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:23:22.246 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:23:22.246 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:23:22.246 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:23:22.263 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:23:22.266 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:23:22.267 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:23:22.269 INFO 2981023 --- [ocp-schedules-26,b60b16b5de6c2513,c315237603c6c4ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:23:22.271 INFO 2981023 --- [ocp-schedules-26,b60b16b5de6c2513,c315237603c6c4ae] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:23:22.942 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:22.943 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:23.143 INFO 2981023 --- [ocp-async-3,38875ad295cfc724,8bdfb4131ac497a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,1948e701e7d13f3c,dfff49f95510450c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,1948e701e7d13f3c,dfff49f95510450c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:24.152 INFO 2981023 --- [ocp-schedules-12,42b5210c607d56e1,728f9a9fac4770f7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:23:25.143 INFO 2981023 --- [ocp-async-10,1e5d1fd43a59ec75,1a3de39b878ab3d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:25.943 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:25.943 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:26.299 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.301 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkp 2024-09-11 17:23:26.311 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.311 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:23:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,28c4dba7f517da15,7a96b643cb31af87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,28c4dba7f517da15,7a96b643cb31af87] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:26.319 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.330 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.335 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:23:26.336524+08:00, lastProcessTime=2024-09-11T17:23:16.381384+08:00 2024-09-11 17:23:26.340 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.341 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:23:26.343 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.347 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 10 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.386 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.387 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.394 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.395 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.396 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.397 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.397 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.398 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.399 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.400 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.401 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.406 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.406 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:23:26.409 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.414 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 4 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.419 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.425 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.429 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 36 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.484 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 8 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.509 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 170 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.530 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:23:26.336524+08:00 2024-09-11 17:23:26.536 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:17:26.122, end=2024-09-11T17:22:26.295} 2024-09-11 17:23:26.537 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 4 dubious sqls 2024-09-11 17:23:27.143 INFO 2981023 --- [ocp-async-6,85bf27053ce49dea,8171ce3ba5c8d872] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:27.737 INFO 2981023 --- [ocp-async-5,ef764ce8aeec5514,17180ec0875ab18b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:23:28.944 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:28.944 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:29.143 INFO 2981023 --- [ocp-async-3,cfe8cb3b2f26e960,0ea09109c39af286] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,636084ec0a451bd9,157bdbd8c0c7524e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,636084ec0a451bd9,157bdbd8c0c7524e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:31.143 INFO 2981023 --- [ocp-async-2,087604066744e6ba,519276caab05d945] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:31.408 INFO 2981023 --- [ocp-schedules-13,412b5d44070fd6ff,1fe8d6a2bdb1bbf0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:23:31.458 WARN 2981023 --- [scanner-13473853-1,412b5d44070fd6ff,a720b30bd34ea568] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:23:31.517 INFO 2981023 --- [alarm-detect-9,412b5d44070fd6ff,b1a06aada84df7e7] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007100] will be resolved. 2024-09-11 17:23:31.517 INFO 2981023 --- [alarm-detect-9,412b5d44070fd6ff,b1a06aada84df7e7] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007100] will be resolved. 2024-09-11 17:23:31.577 INFO 2981023 --- [alarm-detect-9,412b5d44070fd6ff,b1a06aada84df7e7] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 17:23:31.716 INFO 2981023 --- [ocp-schedules-26,d824b35c3978478a,98da7566bd16d931] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:23:31.944 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:31.945 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,35f62018e1e0d259,c7538f206e4a6d31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,35f62018e1e0d259,c7538f206e4a6d31] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:33.143 INFO 2981023 --- [ocp-async-6,82d2f142d8d52db2,7e47dcfeb9eaf09f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:33.437 INFO 2981023 --- [ocp-schedules-19,a0cca887a4852272,907286a054758234] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:23:33.446 INFO 2981023 --- [ocp-schedules-19,a0cca887a4852272,907286a054758234] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:23:33.446 INFO 2981023 --- [ocp-schedules-19,a0cca887a4852272,907286a054758234] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:23:34.945 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:34.946 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:35.143 INFO 2981023 --- [ocp-async-1,638458dff4a1818d,68e949cda114c102] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,75321eafffcacd8c,6dfa8203579be973] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:35.317 INFO 2981023 --- [prometheus-minute-schedule-1,75321eafffcacd8c,6dfa8203579be973] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:36.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:23:36.354020+08:00, lastProcessTime=2024-09-11T17:23:26.336524+08:00 2024-09-11 17:23:36.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:23:36.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:23:36.354020+08:00 2024-09-11 17:23:37.143 INFO 2981023 --- [ocp-async-8,21a0970b542e92e0,4832b630163f8879] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:37.314 INFO 2981023 --- [prometheus-cache-monitor1,136214733d2e2355,4c7183b96e1289a0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:23:37.315 INFO 2981023 --- [prometheus-cache-monitor1,136214733d2e2355,4c7183b96e1289a0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:23:37.493 INFO 2981023 --- [ocp-schedules-1,e8bfca55abd8b903,28857c24ccc2e6a9] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:23:37.510 WARN 2981023 --- [ocp-schedules-1,e8bfca55abd8b903,28857c24ccc2e6a9] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:23:37.737 INFO 2981023 --- [ocp-async-1,053c57ea27ab693a,035bdfb536ef6cc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:23:37.946 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:37.948 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,4378762dd837ad15,b18ecfe0da982f83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:38.317 INFO 2981023 --- [prometheus-minute-schedule-1,4378762dd837ad15,b18ecfe0da982f83] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:39.144 INFO 2981023 --- [ocp-async-5,1d59110d9eadeab4,17b4aead8a9c73f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:40.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:23:40.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:23:40.767 INFO 2981023 --- [ocp-schedules-13,2e20ee34044ffa44,d377a1ae7928f545] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:23:40.948 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:40.949 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:41.144 INFO 2981023 --- [ocp-async-7,92ed7339e6bd6108,0ca2c99e3b5eb940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,99bc628c5e492d54,bd2d765d51aafed5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:41.319 INFO 2981023 --- [prometheus-minute-schedule-1,99bc628c5e492d54,bd2d765d51aafed5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:41.617 INFO 2981023 --- [ocp-schedules-17,99d70b567d94998f,f27a65993fa4ff4f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:23:41.666 WARN 2981023 --- [scanner-13474119-1,99d70b567d94998f,276b9baccc793b9a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:23:41.686 INFO 2981023 --- [alarm-detect-19,99d70b567d94998f,bc192f87e1cb3038] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643709, data is empty. 2024-09-11 17:23:41.693 INFO 2981023 --- [alarm-detect-19,99d70b567d94998f,bc192f87e1cb3038] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643709, data is empty. 2024-09-11 17:23:41.700 INFO 2981023 --- [alarm-detect-19,99d70b567d94998f,bc192f87e1cb3038] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643709, data is empty. 2024-09-11 17:23:41.714 INFO 2981023 --- [alarm-detect-13,99d70b567d94998f,b01ae08b2365ba0e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643709, data is empty. 2024-09-11 17:23:41.719 INFO 2981023 --- [ocp-schedules-23,511a8872de0e5770,35e2a1c75b03ba06] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:23:41.720 INFO 2981023 --- [alarm-detect-13,99d70b567d94998f,b01ae08b2365ba0e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643709, data is empty. 2024-09-11 17:23:41.720 INFO 2981023 --- [alarm-detect-10,99d70b567d94998f,bd7f58f35ca086d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643709, data is empty. 2024-09-11 17:23:41.726 INFO 2981023 --- [alarm-detect-13,99d70b567d94998f,b01ae08b2365ba0e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643709, data is empty. 2024-09-11 17:23:41.728 INFO 2981023 --- [alarm-detect-10,99d70b567d94998f,bd7f58f35ca086d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643709, data is empty. 2024-09-11 17:23:41.732 INFO 2981023 --- [alarm-detect-10,99d70b567d94998f,bd7f58f35ca086d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643709, data is empty. 2024-09-11 17:23:41.833 INFO 2981023 --- [alarm-detect-2,99d70b567d94998f,0cec77ce1394cc16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643709, data is empty. 2024-09-11 17:23:41.836 INFO 2981023 --- [alarm-detect-2,99d70b567d94998f,0cec77ce1394cc16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643709, data is empty. 2024-09-11 17:23:41.842 INFO 2981023 --- [alarm-detect-2,99d70b567d94998f,0cec77ce1394cc16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643709, data is empty. 2024-09-11 17:23:43.144 INFO 2981023 --- [ocp-async-1,4780b89148e129e3,aa1fc90511b97bc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:43.949 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:43.949 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,16064b02de349680,e97b7679b717d2ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,16064b02de349680,e97b7679b717d2ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:45.101 INFO 2981023 --- [prometheus-second-schedule-1,eb18983098d03fda,72b23457d923e35d] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:23:45.144 INFO 2981023 --- [ocp-async-6,7b96a8949d8dd034,f00a77ba26acd0f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:46.014 INFO 2981023 --- [ocp-schedules-3,7ef5ad0762007a31,e8fa29d44ccf22fe] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:23:46.014 INFO 2981023 --- [host-check-clockdiff1,7ef5ad0762007a31,d760459739197711] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:23:46.015 INFO 2981023 --- [host-check-clockdiff2,7ef5ad0762007a31,3f1cec55af0fee89] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:23:46.015 INFO 2981023 --- [host-check-clockdiff3,7ef5ad0762007a31,d6b7485e64244ece] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:23:46.018 INFO 2981023 --- [host-check-clockdiff4,7ef5ad0762007a31,572f2363f36fa90c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:23:46.025 INFO 2981023 --- [ocp-schedules-18,0ecb3d545ea213df,ad4979154f33894b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:23:46.029 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:23:46.029 INFO 2981023 --- [ocp-schedules-22,d44d4461040444be,359333b97bf5c593] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:23:46.032 INFO 2981023 --- [host-check-agent-status1,d44d4461040444be,c63897449ab4b0b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:23:46.032 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:23:46.032 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:23:46.031 INFO 2981023 --- [host-check-clockdiff1,7ef5ad0762007a31,d760459739197711] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046626 0 0 , error= 2024-09-11 17:23:46.046 INFO 2981023 --- [host-check-agent-status2,d44d4461040444be,d529e11ba4fe53df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:23:46.046 INFO 2981023 --- [host-check-agent-status3,d44d4461040444be,070e269019466ea7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:23:46.046 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:23:46.046 INFO 2981023 --- [host-check-agent-status4,d44d4461040444be,f023e2e793ee5d86] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:23:46.053 INFO 2981023 --- [host-check-clockdiff4,7ef5ad0762007a31,572f2363f36fa90c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046626 0 0 , error= 2024-09-11 17:23:46.057 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:23:46.057 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:23:46.060 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:23:46.060 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:23:46.076 INFO 2981023 --- [host-check-agent-status1,d44d4461040444be,c63897449ab4b0b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:23:46.077 INFO 2981023 --- [host-check-agent-status1,d44d4461040444be,c63897449ab4b0b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:23:46.160 INFO 2981023 --- [ocp-schedules-1,5c36508d5a1f242e,bdcaf1de458c3a04] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:23:46.160 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:23:46.165 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:23:46.188 INFO 2981023 --- [ocp-schedules-26,38cd54d2a1305168,9ff779fe20a597c5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:23:46.189 INFO 2981023 --- [ocp-schedules-8,52d60d61572feac6,1df03af8b40076cd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:23:46.190 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:23:46.190 INFO 2981023 --- [ocp-schedules-11,aa2a7cf2d371c0c1,c4b8b5f1131e6c29] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:23:46.190 INFO 2981023 --- [ocp-schedules-10,729b9d7a0c5fef9b,cf76526e13656fa4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:23:46.192 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:23:46.194 INFO 2981023 --- [ocp-schedules-26,38cd54d2a1305168,9ff779fe20a597c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:23:46.195 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:23:46.198 INFO 2981023 --- [host-check-agent-status2,d44d4461040444be,d529e11ba4fe53df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:23:46.256 INFO 2981023 --- [ocp-schedules-8,52d60d61572feac6,1df03af8b40076cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:23:46.257 INFO 2981023 --- [ocp-schedules-11,aa2a7cf2d371c0c1,c4b8b5f1131e6c29] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:23:46.258 INFO 2981023 --- [host-check-agent-status4,d44d4461040444be,f023e2e793ee5d86] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:23:46.263 INFO 2981023 --- [host-check-agent-status2,d44d4461040444be,d529e11ba4fe53df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:23:46.269 INFO 2981023 --- [host-check-agent-status4,d44d4461040444be,f023e2e793ee5d86] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:23:46.277 INFO 2981023 --- [ocp-schedules-1,5c36508d5a1f242e,bdcaf1de458c3a04] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:23:46.277 INFO 2981023 --- [host-check-agent-status3,d44d4461040444be,070e269019466ea7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:23:46.289 INFO 2981023 --- [host-check-agent-status3,d44d4461040444be,070e269019466ea7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:23:46.293 INFO 2981023 --- [ocp-schedules-8,52d60d61572feac6,1df03af8b40076cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:23:46.297 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:23:46.300 INFO 2981023 --- [ocp-schedules-27,46f0666146ac70a5,56aa10c9a525d7c5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:23:46.370 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:23:46.375 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:23:46.376 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:23:46.377 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:23:46.376678954+08:00 2024-09-11 17:23:46.378 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:23:46.378176706+08:00 2024-09-11 17:23:46.378 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:23:46.378+08:00 2024-09-11 17:23:46.380 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:23:46.377+08:00 2024-09-11 17:23:46.381 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:23:46.378+08:00 and status AVAILABLE 2024-09-11 17:23:46.383 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:23:46.381133140+08:00 2024-09-11 17:23:46.384 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:23:46.377+08:00 and status AVAILABLE 2024-09-11 17:23:46.385 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:23:46.383+08:00 2024-09-11 17:23:46.387 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:23:46.383+08:00 and status AVAILABLE 2024-09-11 17:23:46.389 INFO 2981023 --- [host-check-agent-status1,d44d4461040444be,c63897449ab4b0b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:23:46.390 INFO 2981023 --- [host-check-agent-status2,d44d4461040444be,d529e11ba4fe53df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:23:46.390 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:23:46.390 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:23:46.392 INFO 2981023 --- [host-check-agent-status1,d44d4461040444be,c63897449ab4b0b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:23:46.392 INFO 2981023 --- [host-check-agent-status1,d44d4461040444be,c63897449ab4b0b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:23:46.392 INFO 2981023 --- [host-check-agent-status1,d44d4461040444be,c63897449ab4b0b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:23:46.392 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:23:46.392164922+08:00 2024-09-11 17:23:46.393 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:23:46.392750+08:00, lastProcessTime=2024-09-11T17:23:36.354020+08:00 2024-09-11 17:23:46.393 INFO 2981023 --- [host-check-agent-status4,d44d4461040444be,f023e2e793ee5d86] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:23:46.393 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:23:46.392+08:00 2024-09-11 17:23:46.397 INFO 2981023 --- [host-check-agent-status2,d44d4461040444be,d529e11ba4fe53df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:23:46.397 INFO 2981023 --- [host-check-agent-status2,d44d4461040444be,d529e11ba4fe53df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:23:46.397 INFO 2981023 --- [host-check-agent-status2,d44d4461040444be,d529e11ba4fe53df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:23:46.397 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:23:46.392+08:00 and status AVAILABLE 2024-09-11 17:23:46.398 INFO 2981023 --- [host-check-agent-status4,d44d4461040444be,f023e2e793ee5d86] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:23:46.398 INFO 2981023 --- [host-check-agent-status4,d44d4461040444be,f023e2e793ee5d86] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:23:46.398 INFO 2981023 --- [host-check-agent-status4,d44d4461040444be,f023e2e793ee5d86] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:23:46.400 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:23:46.406 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:23:46.406 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.407 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:46.407 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:23:46.462 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:23:46.469 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:23:46.469 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.470 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:46.470 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.470 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:23:46.471 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:23:46.471 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:46.472 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:23:46.473 INFO 2981023 --- [ocp-schedules-26,38cd54d2a1305168,9ff779fe20a597c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:23:46.473 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:23:46.477 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.477 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:46.477 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:23:46.479 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:23:46.479 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:23:46.482 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:23:46.482 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:23:46.483 INFO 2981023 --- [host-check-agent-status3,d44d4461040444be,070e269019466ea7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:23:46.487 INFO 2981023 --- [host-check-agent-status3,d44d4461040444be,070e269019466ea7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:23:46.487 INFO 2981023 --- [host-check-agent-status3,d44d4461040444be,070e269019466ea7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:23:46.487 INFO 2981023 --- [host-check-agent-status3,d44d4461040444be,070e269019466ea7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:23:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.491 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:46.493 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.494 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:23:46.495 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:23:46.495 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.501 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:23:46.509 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:23:46.519 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:23:46.547 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:23:46.564 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:23:46.566 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:23:46.571 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:23:46.583 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:23:46.592 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:23:46.592 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.593 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.593 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.593 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:23:46.594 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.595 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:23:46.597 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:23:46.598 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:23:46.663 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.663 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:23:46.663 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.664 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:23:46.664 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:23:46.664 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.681 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.681 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:46.682 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:23:46.682 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.684 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.685 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.685 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:23:46.685 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:46.686 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:23:46.687 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:23:46.688 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.691 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:23:46.691 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:23:46.692 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:23:46.699 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:23:46.706 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:46.706 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:23:46.707 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:23:46.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:46.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:23:46.727 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:23:46.766 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:23:46.773 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:23:46.774 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:23:46.775 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:46.782 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.782 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.783 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:23:46.786 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.786 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.787 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:46.787 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:23:46.788 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:23:46.788 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:46.797 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:23:46.798 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.801 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:23:46.803 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:23:46.805 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:23:46.805 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.806 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.806 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:46.807 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:23:46.835 INFO 2981023 --- [ocp-schedules-6,13804b520162b5f9,b21290b15be55ff4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:23:46.835 INFO 2981023 --- [ocp-schedules-6,13804b520162b5f9,b21290b15be55ff4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:23:46.856 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:23:46.858 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:23:46.860 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:23:46.862 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.864 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.864 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:23:46.865 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:23:46.865 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.865 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:23:46.866 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:23:46.867 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.870 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:23:46.870 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:23:46.871 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:23:46.872 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:23:46.872 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:23:46.893 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:23:46.895 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:23:46.895 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:23:46.895 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.898 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:23:46.899 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:23:46.903 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:23:46.904 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:23:46.904 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:23:46.909 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:23:46.930 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:23:46.950 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:46.958 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:46.958 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:23:46.963 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:23:46.964 INFO 2981023 --- [host-check-host-status4,0ecb3d545ea213df,5c5162d46737f417] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:23:46.965 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:23:46.969 INFO 2981023 --- [host-check-clockdiff2,7ef5ad0762007a31,3f1cec55af0fee89] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046626 0 0 , error= 2024-09-11 17:23:46.975 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:46.980 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:23:46.981 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:23:46.981 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:23:46.983 INFO 2981023 --- [host-check-host-status3,0ecb3d545ea213df,049958ce09751bed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:23:46.988 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.989 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:46.989 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:23:46.990 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:23:46.999 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:46.999 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:47.000 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:23:47.003 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:47.004 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:47.006 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:23:47.006 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:23:47.008 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:23:47.010 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:23:47.013 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:23:47.057 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:23:47.057 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:23:47.059 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:23:47.063 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:47.067 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:23:47.069 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:23:47.072 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:47.073 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:23:47.073 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:23:47.073 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:47.073 INFO 2981023 --- [host-check-host-status1,0ecb3d545ea213df,a4d986abcb63d06f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:23:47.077 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:23:47.089 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:23:47.091 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:23:47.096 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:23:46.392750+08:00 2024-09-11 17:23:47.114 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:23:47.120 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:23:47.144 INFO 2981023 --- [ocp-async-10,ca5bf48826585550,20fb2a5e3be6d6ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:47.160 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:47.160 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:47.161 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:23:47.163 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:23:47.169 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:23:47.169 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:23:47.169 INFO 2981023 --- [ocp-schedules-11,aa2a7cf2d371c0c1,c4b8b5f1131e6c29] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:23:47.174 INFO 2981023 --- [host-check-host-status2,0ecb3d545ea213df,738ac318be167733] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:23:47.189 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:47.189 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:47.190 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:23:47.190 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:23:47.190 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:47.241 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:23:47.270 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:23:47.276 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:23:47.279 INFO 2981023 --- [ocp-schedules-2,dcab37543fd7de3c,862db11686eb6c77] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:23:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,5e2cc01be5d700c9,c52af47e5b9bb802] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:47.315 INFO 2981023 --- [prometheus-minute-schedule-1,5e2cc01be5d700c9,c52af47e5b9bb802] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:47.737 INFO 2981023 --- [ocp-async-7,95be1cb2b569626f,61edd74f53d2f1bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:23:49.144 INFO 2981023 --- [ocp-async-2,c636adaf9ef94206,9cc4d947f4a1bdac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:23:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:23:49.975 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:49.977 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:50.185 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:23:50.190 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:23:50.190 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.190 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.190 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.190 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.191 INFO 2981023 --- [ocp-schedules-15,34b09b61a4aa5696,3df2f77c262942c0] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:23:50.194 INFO 2981023 --- [ocp-schedules-14,21628c4613648d71,5604abb993261804] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:23:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,a173ccb4df7c7075,a8368927b21c7ff1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,a173ccb4df7c7075,a8368927b21c7ff1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:50.749 INFO 2981023 --- [ocp-schedules-9,a3b1688f96919748,acff5a59e1f55476] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:23:50.749+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:23:50.751 INFO 2981023 --- [ocp-schedules-9,a3b1688f96919748,acff5a59e1f55476] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:23:50.784 INFO 2981023 --- [ocp-schedules-9,a3b1688f96919748,acff5a59e1f55476] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 21 to SqlAggregateData 2024-09-11 17:23:50.785 INFO 2981023 --- [ocp-schedules-9,a3b1688f96919748,acff5a59e1f55476] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:23:51.144 INFO 2981023 --- [ocp-async-6,2240661fd7da2238,7773b3e3e1037c3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:51.152 INFO 2981023 --- [ocp-schedules-7,5fbdd55342ee3169,3cb9323a13be4b28] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:23:51.153 INFO 2981023 --- [ocp-schedules-7,5fbdd55342ee3169,3cb9323a13be4b28] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:23:51.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:23:51.189 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:23:51.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.189 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.191 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:23:51.191 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.193 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:23:51.193 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:23:51.194 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:23:51.197 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.197 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.198 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.198 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.198 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.199 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.207 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.207 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:23:51.210 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.211 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.212 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.213 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.214 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.214 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:23:51.219 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.220 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:23:51.225 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.225 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.225 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.225 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:23:51.226 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.226 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1BD1D6DA-0-0] 2024-09-11 17:23:51.226 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:23:51.226 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.226 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:23:51.227 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.228 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:23:51.233 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.234 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:23:51.235 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.236 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:23:51.236 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.237 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:23:51.237 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:23:51.239 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.239 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:23:51.241 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.241 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:23:51.241 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:23:51.242 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.242 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:23:51.243 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.244 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:23:51.244 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.245 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:23:51.245 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.246 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:23:51.246 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.247 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:23:51.247 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.248 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:23:51.248 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.249 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:23:51.250 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.250 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:23:51.251 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:23:51.252 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.252 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:23:51.253 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:23:51.253 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:23:51.253 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1BD1D6E4-0-0] 2024-09-11 17:23:51.253 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:23:51.254 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:51.256 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.256 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.260 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.261 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:23:51.263 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.263 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:23:51.264 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.264 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:23:51.265 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.266 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:23:51.266 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.267 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:51.267 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:23:51.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:23:51.269 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:23:51.269 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.269 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:23:51.270 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:23:51.271 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:23:51.273 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:23:51.273 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:23:51.273 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:23:51.277 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:23:51.280 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:23:51.280 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.280 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:23:51.285 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:23:51.286 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:23:51.287 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.287 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:23:51.288 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:23:51.288 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.289 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.293 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:23:51.295 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:23:51.295 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:23:51.295 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:23:51.296 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:23:51.297 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:23:51.297 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:51.298 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:23:51.308 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:23:51.310 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:23:51.310 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:23:51.310 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:23:51.310 INFO 2981023 --- [ocp-schedules-7,5fbdd55342ee3169,3cb9323a13be4b28] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:23:51.499 INFO 2981023 --- [ocp-schedules-10,a628ded1b31e6c67,e4c80af5f9355e33] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:23:51.500 INFO 2981023 --- [ocp-schedules-10,a628ded1b31e6c67,e4c80af5f9355e33] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:23:51.500 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:23:51.500 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:23:51.500 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:23:51.501 INFO 2981023 --- [ocp-schedules-10,a628ded1b31e6c67,e4c80af5f9355e33] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:23:51.729 INFO 2981023 --- [ocp-schedules-14,8f13baf4a77ae996,60632587c367f6b4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:23:51.881 INFO 2981023 --- [ocp-schedules-22,94d69cf85bcf8fef,0c1c99dfed0c2d32] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:23:51.937 WARN 2981023 --- [scanner-13474399-1,94d69cf85bcf8fef,91ad7442fa26d3a4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:23:52.022 INFO 2981023 --- [host-check-clockdiff3,7ef5ad0762007a31,d6b7485e64244ece] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046631 1 1 , error= 2024-09-11 17:23:52.104 INFO 2981023 --- [ocp-schedules-8,ebb4eef706914026,2654d2f6852f407a] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:23:52.121 WARN 2981023 --- [ocp-schedules-8,ebb4eef706914026,2654d2f6852f407a] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:23:52.977 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:52.979 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:53.010 INFO 2981023 --- [ocp-schedules-20,f288643ac88cffb5,334c5aa78227ea69] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:23:53.010 INFO 2981023 --- [ocp-schedules-20,f288643ac88cffb5,334c5aa78227ea69] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:23:53.144 INFO 2981023 --- [ocp-async-3,8d175963afbbdf72,5a50a9c082aae154] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,589551f895fce23e,7fea4e4baf051a4d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,589551f895fce23e,7fea4e4baf051a4d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:53.449 INFO 2981023 --- [ocp-schedules-29,d55de1f16ee9d382,6a9d04b2d2efef09] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:23:53.457 INFO 2981023 --- [ocp-schedules-29,d55de1f16ee9d382,6a9d04b2d2efef09] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:23:53.457 INFO 2981023 --- [ocp-schedules-29,d55de1f16ee9d382,6a9d04b2d2efef09] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:23:54.155 INFO 2981023 --- [ocp-schedules-9,db0862cf892a4993,0e4181722178492e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:23:54.510 INFO 2981023 --- [ocp-schedules-30,5434f2c63d2e548e,bb11ffc80ce7a96d] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T17:23:54.510+08:00, tenantSize=6 2024-09-11 17:23:54.513 INFO 2981023 --- [ocp-schedules-30,5434f2c63d2e548e,bb11ffc80ce7a96d] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726046634510000 2024-09-11 17:23:54.513 INFO 2981023 --- [ocp-schedules-30,5434f2c63d2e548e,bb11ffc80ce7a96d] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726046334461000 2024-09-11 17:23:54.529 INFO 2981023 --- [ocp-schedules-30,5434f2c63d2e548e,bb11ffc80ce7a96d] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 17:23:55.145 INFO 2981023 --- [ocp-async-1,d96e3b11aa92467f,e22740b6c1816af6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:55.619 INFO 2981023 --- [ocp-schedules-13,70729262e1c1a168,6be1284df05050ec] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:23:55.619 INFO 2981023 --- [ocp-schedules-13,70729262e1c1a168,6be1284df05050ec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:23:55.620 INFO 2981023 --- [ocp-schedules-13,70729262e1c1a168,6be1284df05050ec] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:23:55.620 INFO 2981023 --- [ocp-schedules-13,70729262e1c1a168,6be1284df05050ec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:23:55.620 INFO 2981023 --- [ocp-schedules-13,70729262e1c1a168,6be1284df05050ec] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:23:55.620 INFO 2981023 --- [ocp-schedules-13,70729262e1c1a168,6be1284df05050ec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:23:55.631 INFO 2981023 --- [ocp-schedules-12,4b0571af00748ab5,48c61695aa68457b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:23:55.631 INFO 2981023 --- [ocp-schedules-12,4b0571af00748ab5,48c61695aa68457b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:23:55.646 INFO 2981023 --- [Collect all tenants major compactions3,4b0571af00748ab5,e645607316396ff2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:23:55.647 INFO 2981023 --- [Collect all tenants major compactions4,4b0571af00748ab5,401c813a570a7aae] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:23:55.648 INFO 2981023 --- [Collect all tenants major compactions2,4b0571af00748ab5,d150696a3d9f5cfc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:23:55.648 INFO 2981023 --- [Collect all tenants major compactions5,4b0571af00748ab5,b76d8c7112cf9981] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:23:55.649 INFO 2981023 --- [Collect all tenants major compactions6,4b0571af00748ab5,78bed89fd2021020] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:23:55.649 INFO 2981023 --- [Collect all tenants major compactions1,4b0571af00748ab5,b93fffa49d2d1c43] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:23:55.665 INFO 2981023 --- [Collect all tenants major compactions5,4b0571af00748ab5,b76d8c7112cf9981] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:55.666 INFO 2981023 --- [Collect all tenants major compactions4,4b0571af00748ab5,401c813a570a7aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:55.666 INFO 2981023 --- [Collect all tenants major compactions5,4b0571af00748ab5,b76d8c7112cf9981] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:23:55.666 INFO 2981023 --- [Collect all tenants major compactions3,4b0571af00748ab5,e645607316396ff2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:55.667 INFO 2981023 --- [Collect all tenants major compactions2,4b0571af00748ab5,d150696a3d9f5cfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:55.667 INFO 2981023 --- [Collect all tenants major compactions4,4b0571af00748ab5,401c813a570a7aae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:23:55.667 INFO 2981023 --- [Collect all tenants major compactions1,4b0571af00748ab5,b93fffa49d2d1c43] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:55.668 INFO 2981023 --- [ocp-schedules-9,87fec8d107733a71,cbf991cd9b9cb2f3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:23:55.668 INFO 2981023 --- [Collect all tenants major compactions2,4b0571af00748ab5,d150696a3d9f5cfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:23:55.668 INFO 2981023 --- [Collect all tenants major compactions6,4b0571af00748ab5,78bed89fd2021020] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:23:55.669 INFO 2981023 --- [Collect all tenants major compactions3,4b0571af00748ab5,e645607316396ff2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:23:55.670 INFO 2981023 --- [Collect all tenants major compactions6,4b0571af00748ab5,78bed89fd2021020] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:23:55.671 INFO 2981023 --- [Collect all tenants major compactions1,4b0571af00748ab5,b93fffa49d2d1c43] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:23:55.672 INFO 2981023 --- [Collect all tenants major compactions3,4b0571af00748ab5,e645607316396ff2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:23:55.672 INFO 2981023 --- [Collect all tenants major compactions3,4b0571af00748ab5,e645607316396ff2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:23:55.672 INFO 2981023 --- [Collect all tenants major compactions2,4b0571af00748ab5,d150696a3d9f5cfc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:23:55.672 INFO 2981023 --- [Collect all tenants major compactions4,4b0571af00748ab5,401c813a570a7aae] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:23:55.672 INFO 2981023 --- [Collect all tenants major compactions2,4b0571af00748ab5,d150696a3d9f5cfc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:23:55.672 INFO 2981023 --- [Collect all tenants major compactions4,4b0571af00748ab5,401c813a570a7aae] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:23:55.673 INFO 2981023 --- [Collect all tenants major compactions5,4b0571af00748ab5,b76d8c7112cf9981] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:23:55.673 INFO 2981023 --- [Collect all tenants major compactions5,4b0571af00748ab5,b76d8c7112cf9981] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:23:55.673 INFO 2981023 --- [Collect all tenants major compactions1,4b0571af00748ab5,b93fffa49d2d1c43] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:23:55.673 INFO 2981023 --- [Collect all tenants major compactions1,4b0571af00748ab5,b93fffa49d2d1c43] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:23:55.674 INFO 2981023 --- [Collect all tenants major compactions6,4b0571af00748ab5,78bed89fd2021020] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:23:55.674 INFO 2981023 --- [Collect all tenants major compactions6,4b0571af00748ab5,78bed89fd2021020] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:23:55.675 INFO 2981023 --- [ocp-schedules-12,4b0571af00748ab5,48c61695aa68457b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:23:55.677 INFO 2981023 --- [ocp-schedules-9,87fec8d107733a71,cbf991cd9b9cb2f3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:23:55.677 INFO 2981023 --- [ocp-schedules-9,87fec8d107733a71,cbf991cd9b9cb2f3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:23:55.979 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:55.980 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,09f844a35382beb6,dd5bcd5433d8a870] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,09f844a35382beb6,dd5bcd5433d8a870] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:23:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:23:56.351817+08:00, lastProcessTime=2024-09-11T17:23:46.392750+08:00 2024-09-11 17:23:56.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:23:56.351817+08:00 2024-09-11 17:23:57.145 INFO 2981023 --- [ocp-async-6,c82b80e33b69ec69,fed80fdf6322f845] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:57.737 INFO 2981023 --- [ocp-async-2,bc8ef860913c0383,fed05be0be8c4257] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:23:58.337 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,e57c9d61f824b11e,95c73d28e7edb574] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=e57c9d61f824b11e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:23:58.337 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3b1467ea05147637,7b0f9bfce8abd997] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=3b1467ea05147637, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:23:58.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3b1467ea05147637,7b0f9bfce8abd997] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=3b1467ea05147637, duration=5 ms] 2024-09-11 17:23:58.340 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,e57c9d61f824b11e,95c73d28e7edb574] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=e57c9d61f824b11e, duration=7 ms] 2024-09-11 17:23:58.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,112dd1c6db0bbd92,cfdeeda049b464d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=112dd1c6db0bbd92, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:23:58.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,112dd1c6db0bbd92,cfdeeda049b464d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=112dd1c6db0bbd92, duration=2 ms] 2024-09-11 17:23:58.980 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:23:58.981 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:23:59.145 INFO 2981023 --- [ocp-async-3,158d5bf5a4462ce6,9d33705ea5ebb442] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:23:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,bd839b6591da239e,7ab429b771e8024f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046640, interval=60 2024-09-11 17:23:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,bd839b6591da239e,7ab429b771e8024f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:01.145 INFO 2981023 --- [ocp-async-4,7dabe1dcbacc488e,d5cb30480dcf800b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:01.731 INFO 2981023 --- [ocp-schedules-21,f80fafd068b26d92,ea48b83268de5be3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:24:01.981 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:01.982 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:02.078 INFO 2981023 --- [ocp-schedules-19,639460e2e7bcc107,9b88ea72e14533f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:24:02.120 WARN 2981023 --- [scanner-13474654-1,639460e2e7bcc107,5ef2669ec98618c4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:24:02.146 INFO 2981023 --- [alarm-detect-11,639460e2e7bcc107,510deb3f47619ff0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643709, data is empty. 2024-09-11 17:24:02.158 INFO 2981023 --- [alarm-detect-11,639460e2e7bcc107,510deb3f47619ff0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643730, data is empty. 2024-09-11 17:24:02.161 INFO 2981023 --- [alarm-detect-11,639460e2e7bcc107,510deb3f47619ff0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643730, data is empty. 2024-09-11 17:24:02.207 INFO 2981023 --- [alarm-detect-9,639460e2e7bcc107,c9f427de390cff07] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643709, data is empty. 2024-09-11 17:24:02.208 INFO 2981023 --- [alarm-detect-19,639460e2e7bcc107,4c447b49e18d98f8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643709, data is empty. 2024-09-11 17:24:02.211 INFO 2981023 --- [alarm-detect-9,639460e2e7bcc107,c9f427de390cff07] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643730, data is empty. 2024-09-11 17:24:02.217 INFO 2981023 --- [alarm-detect-19,639460e2e7bcc107,4c447b49e18d98f8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643730, data is empty. 2024-09-11 17:24:02.217 INFO 2981023 --- [alarm-detect-9,639460e2e7bcc107,c9f427de390cff07] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643730, data is empty. 2024-09-11 17:24:02.222 INFO 2981023 --- [alarm-detect-19,639460e2e7bcc107,4c447b49e18d98f8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643730, data is empty. 2024-09-11 17:24:02.248 INFO 2981023 --- [ocp-schedules-23,f8d6b9d07caae019,dc6de5027383629e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:24:02.249 INFO 2981023 --- [ocp-schedules-23,f8d6b9d07caae019,dc6de5027383629e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:24:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,3fd8eed19c0a21f3,274e2947c8bf08cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:02.317 INFO 2981023 --- [prometheus-minute-schedule-1,3fd8eed19c0a21f3,274e2947c8bf08cd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:02.345 INFO 2981023 --- [alarm-detect-10,639460e2e7bcc107,2b75c9ac10ba778b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046610 ~ 1726046632 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046625, value=2.467)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.467, values={1726046625=2.467}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:24:02.345 INFO 2981023 --- [alarm-detect-10,639460e2e7bcc107,2b75c9ac10ba778b] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.10-3, currentState=Pending 2024-09-11 17:24:02.358 INFO 2981023 --- [ocp-schedules-23,f8d6b9d07caae019,dc6de5027383629e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:24:02.407 INFO 2981023 --- [alarm-detect-18,639460e2e7bcc107,7f320c5a7e94e8cc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643709, data is empty. 2024-09-11 17:24:02.410 INFO 2981023 --- [alarm-detect-18,639460e2e7bcc107,7f320c5a7e94e8cc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643730, data is empty. 2024-09-11 17:24:02.414 INFO 2981023 --- [alarm-detect-18,639460e2e7bcc107,7f320c5a7e94e8cc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643730, data is empty. 2024-09-11 17:24:03.145 INFO 2981023 --- [ocp-async-6,e5315fe6789dd15f,d43f0b939aa5ac10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:04.982 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:04.982 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:05.145 INFO 2981023 --- [ocp-async-9,6a18753d9e607590,9ba1d162196cb98c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,b3875573aba886af,84adcfd09dcc72cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,b3875573aba886af,84adcfd09dcc72cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:24:05.981 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:24:05.986 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:24:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:24:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:24:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:24:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:24:05.999 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:05.999 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:06.000 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:24:06.000 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:24:06.000 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:06.001 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:06.001 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:06.002 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:06.051 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:21:40, start=2024-09-11T17:18:10, end=2024-09-11T17:20:40, updates=112, granule=120000ms, delay=210000ms 2024-09-11 17:24:06.057 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:21:40, target=2024-09-11T17:20:40, granule=120000ms, delay=60000ms) 2024-09-11 17:24:06.115 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - source=2024-09-11T17:20:40, start=2024-09-11T16:59:54, end=2024-09-11T17:09:54, updates=326, granule=600000ms, delay=1246000ms 2024-09-11 17:24:06.119 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:20:40, target=2024-09-11T17:09:41, granule=600000ms, delay=659000ms) 2024-09-11 17:24:06.138 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:21:41, start=2024-09-11T17:17:41, end=2024-09-11T17:20:11, updates=279, granule=120000ms, delay=240000ms 2024-09-11 17:24:06.144 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:21:41, target=2024-09-11T17:20:11, granule=120000ms, delay=90000ms) 2024-09-11 17:24:06.144 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:20:11, target=2024-09-11T17:10:51, granule=600000ms, delay=560000ms) 2024-09-11 17:24:06.144 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=177ms, period=120000ms 2024-09-11 17:24:06.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:24:06.353366+08:00, lastProcessTime=2024-09-11T17:23:56.351817+08:00 2024-09-11 17:24:06.509 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:24:06.353366+08:00 2024-09-11 17:24:07.145 INFO 2981023 --- [ocp-async-8,59958d5bf0c5199c,52baa1fe9808c38d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:07.147 INFO 2981023 --- [ocp-schedules-24,f070404c13f47110,6829088a494c9a14] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:24:07.153 INFO 2981023 --- [ocp-schedules-29,725c82b0f9276b74,9daa773fa66b2090] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:24:07.153+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:24:07.154 INFO 2981023 --- [ocp-schedules-29,725c82b0f9276b74,9daa773fa66b2090] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:24:07.155 INFO 2981023 --- [ocp-schedules-29,725c82b0f9276b74,9daa773fa66b2090] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:24:07.155 INFO 2981023 --- [ocp-schedules-29,725c82b0f9276b74,9daa773fa66b2090] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:24:07.737 INFO 2981023 --- [ocp-async-3,4275d80827a395fd,0430e463e05911ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:24:07.982 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:07.983 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,8653acb6afae3e50,891da165051deb53] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,8653acb6afae3e50,891da165051deb53] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:09.145 INFO 2981023 --- [ocp-async-6,0273164d4290150c,7c8385bd7502e4d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:24:10.003 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:24:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:24:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:24:10.983 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:10.984 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:11.146 INFO 2981023 --- [ocp-async-9,c7c2aec5cc48ca07,462454091c0895c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,6096163a6a490543,4814dd6b99db4146] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:11.320 INFO 2981023 --- [prometheus-minute-schedule-1,6096163a6a490543,4814dd6b99db4146] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:11.734 INFO 2981023 --- [ocp-schedules-29,b9ef6e93760a56d7,acd36430c134a07b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:24:11.800 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7e74b45884f22f23,3d549a68b0d9f8d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=7e74b45884f22f23, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:24:11.800 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,8ef53829798cfb26,7f7c40caa0c0aa23] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=8ef53829798cfb26, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:24:11.801 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7e74b45884f22f23,3d549a68b0d9f8d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=7e74b45884f22f23, duration=6 ms] 2024-09-11 17:24:11.803 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,8ef53829798cfb26,7f7c40caa0c0aa23] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=8ef53829798cfb26, duration=7 ms] 2024-09-11 17:24:11.817 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,78f9ae81d70fd902,551dc6ae341ac817] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=78f9ae81d70fd902, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:24:11.818 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,78f9ae81d70fd902,551dc6ae341ac817] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=78f9ae81d70fd902, duration=12 ms] 2024-09-11 17:24:12.027 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046580, nodeCount=480, affectRows=480 2024-09-11 17:24:12.462 INFO 2981023 --- [ocp-schedules-20,6401425c3dee33ed,e9699cbc68d1b738] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:24:12.515 WARN 2981023 --- [scanner-13476465-1,6401425c3dee33ed,d077598ccd849051] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:24:12.728 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046580, nodeCount=1000, affectRows=1000 2024-09-11 17:24:12.730 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046580, nodeCount=999, validCount=999 2024-09-11 17:24:12.839 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046580, nodeCount=1000, affectRows=1000 2024-09-11 17:24:12.842 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046580, nodeCount=1000, validCount=1000 2024-09-11 17:24:12.852 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046580, nodeCount=48, affectRows=48 2024-09-11 17:24:12.852 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046580, nodeCount=48, validCount=48 2024-09-11 17:24:13.146 INFO 2981023 --- [ocp-async-1,a5dba1535ee742e9,a02cbaed28b3bda7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:13.460 INFO 2981023 --- [ocp-schedules-9,3852af70c0fc409d,a885ddfbcd56288f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:24:13.469 INFO 2981023 --- [ocp-schedules-9,3852af70c0fc409d,a885ddfbcd56288f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:24:13.469 INFO 2981023 --- [ocp-schedules-9,3852af70c0fc409d,a885ddfbcd56288f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:24:13.971 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046580, nodeCount=480, validCount=480 2024-09-11 17:24:13.984 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:13.984 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,320c1826ab5e19b8,b8ad58eb2681d41e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,0dc5c12b4058abc0,e51ddc0587b9362d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:24:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,320c1826ab5e19b8,b8ad58eb2681d41e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:15.146 INFO 2981023 --- [ocp-async-5,37592121d6640a42,11fef872a5da8861] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:16.019 INFO 2981023 --- [ocp-schedules-26,482c04e0566d516b,f1c92261579a3cbe] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:24:16.020 INFO 2981023 --- [host-check-clockdiff1,482c04e0566d516b,9efbe2bfd4900eb3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:24:16.021 INFO 2981023 --- [host-check-clockdiff2,482c04e0566d516b,e3e4a5fd5f256801] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:24:16.021 INFO 2981023 --- [host-check-clockdiff3,482c04e0566d516b,312950d244592121] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:24:16.023 INFO 2981023 --- [host-check-clockdiff4,482c04e0566d516b,caa0e5275613805b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:24:16.026 INFO 2981023 --- [ocp-schedules-10,204889aa68896477,882e1c3b95c97161] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:24:16.027 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:24:16.027 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:24:16.028 INFO 2981023 --- [ocp-schedules-18,9f1a41930f32b470,7cbb2448934f510b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:24:16.028 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:24:16.032 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:24:16.032 INFO 2981023 --- [host-check-agent-status1,9f1a41930f32b470,9f745d01e282af08] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:24:16.032 INFO 2981023 --- [host-check-agent-status3,9f1a41930f32b470,d987dcecd7fbfa75] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:24:16.033 INFO 2981023 --- [host-check-agent-status2,9f1a41930f32b470,a8d64f8b46e66d0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:24:16.033 INFO 2981023 --- [host-check-agent-status4,9f1a41930f32b470,0d20d6f599754485] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:24:16.050 INFO 2981023 --- [host-check-clockdiff2,482c04e0566d516b,e3e4a5fd5f256801] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046656 0 0 , error= 2024-09-11 17:24:16.052 INFO 2981023 --- [host-check-clockdiff4,482c04e0566d516b,caa0e5275613805b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046656 0 0 , error= 2024-09-11 17:24:16.062 INFO 2981023 --- [host-check-agent-status1,9f1a41930f32b470,9f745d01e282af08] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:16.066 INFO 2981023 --- [host-check-agent-status1,9f1a41930f32b470,9f745d01e282af08] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:16.069 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:16.070 INFO 2981023 --- [host-check-agent-status3,9f1a41930f32b470,d987dcecd7fbfa75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:24:16.071 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:16.073 INFO 2981023 --- [host-check-agent-status3,9f1a41930f32b470,d987dcecd7fbfa75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:24:16.077 INFO 2981023 --- [host-check-agent-status4,9f1a41930f32b470,0d20d6f599754485] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:16.080 INFO 2981023 --- [host-check-agent-status4,9f1a41930f32b470,0d20d6f599754485] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:16.088 INFO 2981023 --- [host-check-agent-status2,9f1a41930f32b470,a8d64f8b46e66d0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:16.089 INFO 2981023 --- [host-check-agent-status2,9f1a41930f32b470,a8d64f8b46e66d0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:16.187 INFO 2981023 --- [ocp-schedules-27,c41b1b2f803f6596,7e8b75f569230163] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:24:16.189 INFO 2981023 --- [ocp-schedules-27,c41b1b2f803f6596,7e8b75f569230163] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:24:16.192 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:16.197 INFO 2981023 --- [ocp-schedules-16,d591eeb1f911a77f,3fdccc33cbc256a6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:24:16.197 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:16.198 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:24:16.199 INFO 2981023 --- [ocp-schedules-5,5f16e8f5ecfbba5e,557f02afdf2b1c86] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:24:16.199 INFO 2981023 --- [ocp-schedules-2,53ee0e68b96837e8,9de1a175d935891b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:24:16.272 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:24:16.273 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:16.273 INFO 2981023 --- [ocp-schedules-5,5f16e8f5ecfbba5e,557f02afdf2b1c86] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:24:16.274 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:16.280 INFO 2981023 --- [ocp-schedules-16,d591eeb1f911a77f,3fdccc33cbc256a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:24:16.288 INFO 2981023 --- [ocp-schedules-16,d591eeb1f911a77f,3fdccc33cbc256a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:24:16.359 INFO 2981023 --- [ocp-schedules-14,044295968b630e62,9cba591b6e241f18] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:24:16.370 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:24:16.367073+08:00, lastProcessTime=2024-09-11T17:24:06.353366+08:00 2024-09-11 17:24:16.371 INFO 2981023 --- [host-check-agent-status3,9f1a41930f32b470,d987dcecd7fbfa75] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:24:16.373 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:24:16.375 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:24:16.374444841+08:00 2024-09-11 17:24:16.377 INFO 2981023 --- [host-check-agent-status3,9f1a41930f32b470,d987dcecd7fbfa75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:24:16.377 INFO 2981023 --- [host-check-agent-status3,9f1a41930f32b470,d987dcecd7fbfa75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:24:16.377 INFO 2981023 --- [host-check-agent-status3,9f1a41930f32b470,d987dcecd7fbfa75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:24:16.377 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:24:16.375+08:00 2024-09-11 17:24:16.380 INFO 2981023 --- [host-check-agent-status2,9f1a41930f32b470,a8d64f8b46e66d0c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:24:16.380 INFO 2981023 --- [host-check-agent-status1,9f1a41930f32b470,9f745d01e282af08] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:24:16.381 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:24:16.375+08:00 and status AVAILABLE 2024-09-11 17:24:16.383 INFO 2981023 --- [host-check-agent-status2,9f1a41930f32b470,a8d64f8b46e66d0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:24:16.383 INFO 2981023 --- [host-check-agent-status2,9f1a41930f32b470,a8d64f8b46e66d0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:24:16.383 INFO 2981023 --- [host-check-agent-status2,9f1a41930f32b470,a8d64f8b46e66d0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:24:16.384 INFO 2981023 --- [host-check-agent-status1,9f1a41930f32b470,9f745d01e282af08] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:24:16.384 INFO 2981023 --- [host-check-agent-status1,9f1a41930f32b470,9f745d01e282af08] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:24:16.384 INFO 2981023 --- [host-check-agent-status1,9f1a41930f32b470,9f745d01e282af08] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:24:16.385 INFO 2981023 --- [host-check-agent-status4,9f1a41930f32b470,0d20d6f599754485] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:24:16.386 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:24:16.386 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:24:16.387 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.387 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:24:16.387769417+08:00 2024-09-11 17:24:16.388 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:16.388 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:24:16.388165779+08:00 2024-09-11 17:24:16.388 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:24:16.388 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:24:16.389 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:24:16.388+08:00 2024-09-11 17:24:16.389 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:24:16.387+08:00 2024-09-11 17:24:16.389 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:24:16.389100210+08:00 2024-09-11 17:24:16.390 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:24:16.389+08:00 2024-09-11 17:24:16.390 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.391 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:24:16.388+08:00 and status AVAILABLE 2024-09-11 17:24:16.391 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:24:16.387+08:00 and status AVAILABLE 2024-09-11 17:24:16.391 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:16.391 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.392 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:24:16.392 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:16.392 INFO 2981023 --- [host-check-agent-status4,9f1a41930f32b470,0d20d6f599754485] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:24:16.392 INFO 2981023 --- [host-check-agent-status4,9f1a41930f32b470,0d20d6f599754485] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:24:16.392 INFO 2981023 --- [host-check-agent-status4,9f1a41930f32b470,0d20d6f599754485] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:24:16.392 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:24:16.389+08:00 and status AVAILABLE 2024-09-11 17:24:16.393 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:24:16.394 INFO 2981023 --- [ocp-schedules-27,c41b1b2f803f6596,7e8b75f569230163] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:24:16.394 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:24:16.401 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:24:16.403 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.403 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.403 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:16.403 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:16.410 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.410 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:16.413 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:24:16.453 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:24:16.464 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:16.465 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:16.469 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:24:16.469 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:16.469 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:16.470 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:24:16.471 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:16.471 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:16.493 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:24:16.498 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:24:16.499 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.499 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.499 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:24:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:24:16.501 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:24:16.521 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:24:16.527 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.528 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.529 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:24:16.529 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:24:16.529 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:16.530 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:24:16.531 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.533 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:24:16.533 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:24:16.534 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:24:16.560 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:24:16.568 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:24:16.570 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:24:16.574 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:24:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:24:16.579 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.580 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.581 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:24:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:16.607 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.608 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.609 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:16.610 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.613 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:24:16.613 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:24:16.614 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:24:16.614 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:24:16.619 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:24:16.634 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:24:16.641 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:24:16.647 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:24:16.666 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:16.666 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:24:16.669 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:24:16.670 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:24:16.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.677 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:24:16.678 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:16.679 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:24:16.689 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:24:16.690 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:24:16.692 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:16.698 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.698 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.699 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:24:16.701 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:16.702 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:16.709 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:24:16.709 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.710 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:24:16.710 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.711 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:16.712 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:24:16.720 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:24:16.720 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:24:16.723 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:24:16.724 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:16.725 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:24:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.727 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:16.727 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.727 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:16.728 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:24:16.729 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.732 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:24:16.732 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:24:16.735 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:24:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:24:16.743 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:24:16.743 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:24:16.749 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:24:16.755 INFO 2981023 --- [host-check-host-status1,204889aa68896477,226a3d8cf6b7d2d4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:24:16.758 INFO 2981023 --- [host-check-host-status2,204889aa68896477,55a0ee64db1b735d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:24:16.760 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:24:16.766 INFO 2981023 --- [host-check-host-status3,204889aa68896477,cdd2c152ed8351ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:24:16.767 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.780 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:16.780 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:24:16.786 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:24:16.791 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:16.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:16.803 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:24:16.814 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:24:16.818 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:24:16.819 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:24:16.822 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.823 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.823 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:24:16.826 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.827 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.827 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:24:16.367073+08:00 2024-09-11 17:24:16.828 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:24:16.829 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:24:16.830 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:24:16.830 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:24:16.834 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:24:16.834 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:24:16.836 INFO 2981023 --- [host-check-host-status4,204889aa68896477,327e29e28242acca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:24:16.853 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:24:16.858 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:16.859 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:16.859 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:24:16.864 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:24:16.864 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:24:16.864 INFO 2981023 --- [ocp-schedules-5,5f16e8f5ecfbba5e,557f02afdf2b1c86] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:24:16.984 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:16.985 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:17.146 INFO 2981023 --- [ocp-async-10,704bf494733b316e,85933ec2cbc7c39f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,b76f0f18a6547253,1f38b9642bf661fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,b76f0f18a6547253,1f38b9642bf661fe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:17.737 INFO 2981023 --- [ocp-async-6,ec41dbc3767877cb,579ca5d78c096839] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:24:18.953 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:24:18.953 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000008 2024-09-11 17:24:18.953 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000009 2024-09-11 17:24:18.953 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000007 2024-09-11 17:24:18.954 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2024-09-11 17:24:18.992 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:24:18.994 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:24:19.004 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:24:19.016 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:24:19.019 WARN 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:24:19.021 WARN 2981023 --- [ForkJoinPool.commonPool-worker-3,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:24:19.025 WARN 2981023 --- [ForkJoinPool.commonPool-worker-1,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_all_error_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.040 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000008 agent sql collect property 2024-09-11 17:24:19.041 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:24:19.042 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:24:19.043 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:24:19.043 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:24:19.043 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000009 agent sql collect property 2024-09-11 17:24:19.049 WARN 2981023 --- [ForkJoinPool.commonPool-worker-2,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:24:19.050 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:24:19.053 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2000009 to set sql collect property inactive 2024-09-11 17:24:19.054 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:19.057 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:19.061 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:24:19.061 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:24:19.062 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:24:19.062 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2024-09-11 17:24:19.072 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:19.074 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:19.077 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.ob.trace.status=inactive), (monagent.pipeline.ob.log.status=inactive)]), params:null 2024-09-11 17:24:19.078 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:24:19.078 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:24:19.078 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:24:19.079 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000007 agent sql collect property 2024-09-11 17:24:19.082 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994437), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.10), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkprdocp), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"include 2024-09-11 17:24:19.088 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:19.091 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:19.100 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1718689631), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.5), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=rtdob), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"in 2024-09-11 17:24:19.107 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000009, do nothing 2024-09-11 17:24:19.111 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.6), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4 2024-09-11 17:24:19.114 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2024-09-11 17:24:19.126 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000007, do nothing 2024-09-11 17:24:19.146 INFO 2981023 --- [ocp-async-1,a20355b79708d773,1538fd9b9a23529d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:19.291 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000008, do nothing 2024-09-11 17:24:19.293 INFO 2981023 --- [ocp-schedules-5,5cd998d0444ae459,2744621a11248fb6] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-09-11 17:24:19.985 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:19.986 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:20.160 INFO 2981023 --- [host-check-clockdiff1,482c04e0566d516b,9efbe2bfd4900eb3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046660 0 0 , error= 2024-09-11 17:24:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,2c4ae5051b0708d7,2121e154d258d07d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,2c4ae5051b0708d7,2121e154d258d07d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:21.146 INFO 2981023 --- [ocp-async-5,f5a500b177e4017e,d82ba475ba264985] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:21.736 INFO 2981023 --- [ocp-schedules-20,69abc8450ef101b2,c3548169ef65e2b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:24:21.996 INFO 2981023 --- [host-check-clockdiff3,482c04e0566d516b,312950d244592121] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046661 1 1 , error= 2024-09-11 17:24:22.275 INFO 2981023 --- [ocp-schedules-22,4949c644336ea8c1,7cefa342d7eea5c0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:24:22.275 INFO 2981023 --- [ocp-schedules-22,4949c644336ea8c1,7cefa342d7eea5c0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:24:22.275 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:24:22.275 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:24:22.275 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:24:22.279 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:22.280 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:22.281 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:22.281 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:22.282 INFO 2981023 --- [ocp-schedules-22,4949c644336ea8c1,7cefa342d7eea5c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:24:22.282 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:22.283 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:22.283 INFO 2981023 --- [ocp-schedules-22,4949c644336ea8c1,7cefa342d7eea5c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:24:22.301 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:24:22.301 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:24:22.302 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:24:22.304 INFO 2981023 --- [ocp-schedules-22,4949c644336ea8c1,7cefa342d7eea5c0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:24:22.306 INFO 2981023 --- [ocp-schedules-22,4949c644336ea8c1,7cefa342d7eea5c0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:24:22.697 INFO 2981023 --- [ocp-schedules-17,3df48f13a13ca050,0294c5c5f2f26c9b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:24:22.738 WARN 2981023 --- [scanner-13476679-1,3df48f13a13ca050,d3c22a47776c4daf] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:24:22.764 INFO 2981023 --- [alarm-detect-13,3df48f13a13ca050,951d9d01329f3691] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643730, data is empty. 2024-09-11 17:24:22.766 INFO 2981023 --- [alarm-detect-13,3df48f13a13ca050,951d9d01329f3691] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643730, data is empty. 2024-09-11 17:24:22.769 INFO 2981023 --- [alarm-detect-13,3df48f13a13ca050,951d9d01329f3691] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643730, data is empty. 2024-09-11 17:24:22.802 INFO 2981023 --- [alarm-detect-14,3df48f13a13ca050,dce099d37c75f26f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643730, data is empty. 2024-09-11 17:24:22.807 INFO 2981023 --- [alarm-detect-8,3df48f13a13ca050,ff7ddc776e1adbf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643730, data is empty. 2024-09-11 17:24:22.811 INFO 2981023 --- [alarm-detect-14,3df48f13a13ca050,dce099d37c75f26f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643730, data is empty. 2024-09-11 17:24:22.812 INFO 2981023 --- [alarm-detect-8,3df48f13a13ca050,ff7ddc776e1adbf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643730, data is empty. 2024-09-11 17:24:22.814 INFO 2981023 --- [alarm-detect-14,3df48f13a13ca050,dce099d37c75f26f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643730, data is empty. 2024-09-11 17:24:22.814 INFO 2981023 --- [alarm-detect-8,3df48f13a13ca050,ff7ddc776e1adbf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643730, data is empty. 2024-09-11 17:24:22.863 INFO 2981023 --- [alarm-detect-4,3df48f13a13ca050,054aa76840e0ea05] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046625 ~ 1726046652 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046640, value=2.015)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.015, values={1726046640=2.015}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:24:22.872 INFO 2981023 --- [alarm-detect-4,3df48f13a13ca050,054aa76840e0ea05] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007099] will be resolved. 2024-09-11 17:24:22.872 INFO 2981023 --- [alarm-detect-4,3df48f13a13ca050,054aa76840e0ea05] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007099] will be resolved. 2024-09-11 17:24:22.907 INFO 2981023 --- [alarm-detect-4,3df48f13a13ca050,054aa76840e0ea05] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 17:24:22.951 INFO 2981023 --- [alarm-detect-4,3df48f13a13ca050,e164800ebee539ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643730, data is empty. 2024-09-11 17:24:22.956 INFO 2981023 --- [alarm-detect-4,3df48f13a13ca050,e164800ebee539ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643730, data is empty. 2024-09-11 17:24:22.961 INFO 2981023 --- [alarm-detect-4,3df48f13a13ca050,e164800ebee539ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643730, data is empty. 2024-09-11 17:24:22.987 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:22.989 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:23.146 INFO 2981023 --- [ocp-async-10,a343b061c2afec17,01a99f1cff26a5e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,3dfff1ee42405759,78411718809078e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,3dfff1ee42405759,78411718809078e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:24.159 INFO 2981023 --- [ocp-schedules-2,869a55594b339c33,2341822052403d3c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:24:24.490 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:24:24.490 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:24:24.493 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:24:24.493 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:24:24.493 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:24:24.493 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:24:24.495 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:24:24.495 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:24:24.495 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:24:24.495 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:24:24.519 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:24:24.522 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:24:24.522 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:24:24.522 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:24:24.522 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:24:24.522 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:24:24.526 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:24:24.769 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:24:25.052 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.557124501S 2024-09-11 17:24:25.052 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:24:25.098 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:24:25.098 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:24:25.114 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:24:25.159 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:24:25.159 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:24:25.159 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:24:25.159 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:24:25.159 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:24:25.159 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:24:25.165 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:24:25.252 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] o.s.boot.SpringApplication : Started application in 0.758 seconds (JVM running for 82465.124) 2024-09-11 17:24:25.252 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.759022689S 2024-09-11 17:24:25.459 INFO 2981023 --- [ocp-schedules-4,4ac2c5a18d1265ea,9eaf0ff72d7c5a97] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:24:25.989 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:25.989 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,040c3086aa90668d,40c8eb17c6b7919d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,040c3086aa90668d,40c8eb17c6b7919d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:26.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:24:26.350666+08:00, lastProcessTime=2024-09-11T17:24:16.367073+08:00 2024-09-11 17:24:26.469 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:24:26.477 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:24:26.350666+08:00 2024-09-11 17:24:26.571 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 17:24:26.571 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 17:24:26.572 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.572 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:24:26.573 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.573 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.573 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.574 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 17:24:26.589 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.590 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.590 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.590 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 17:24:26.599 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.599 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.600 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.600 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 17:24:26.609 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.609 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.609 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.610 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 17:24:26.619 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.619 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.620 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.620 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 17:24:26.623 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.624 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.624 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.625 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 17:24:26.627 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.628 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.628 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.629 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 17:24:26.632 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.633 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.633 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.633 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 17:24:26.638 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.638 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.639 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.639 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 17:24:26.642 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.642 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.643 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.643 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 17:24:26.646 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.647 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.647 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.647 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 17:24:26.653 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.653 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.654 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.654 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 17:24:26.663 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.663 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.664 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.664 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 17:24:26.667 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.667 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.667 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.668 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 17:24:26.671 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.672 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.672 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.672 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 17:24:26.675 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.676 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.676 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.677 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 17:24:26.680 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.680 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.681 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.681 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 17:24:26.684 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.684 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.685 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.685 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 17:24:26.688 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.689 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.689 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.689 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 17:24:26.693 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.693 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.694 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.694 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 17:24:26.697 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.697 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.697 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.698 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 17:24:26.700 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.701 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.701 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.701 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 17:24:26.704 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.704 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.705 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.705 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 17:24:26.708 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.708 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.708 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.709 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 17:24:26.711 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.712 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.712 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.712 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 17:24:26.715 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.715 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.716 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.716 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 17:24:26.719 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.719 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.719 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.719 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 17:24:26.722 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.723 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.723 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.723 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 17:24:26.726 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.726 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.727 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.727 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 17:24:26.730 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.730 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.730 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.731 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 17:24:26.733 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.733 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.734 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.734 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 17:24:26.737 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.737 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.737 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.738 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 17:24:26.740 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.740 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.741 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.741 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 17:24:26.744 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.744 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.744 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.745 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 17:24:26.749 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.750 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.750 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.750 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 17:24:26.754 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.754 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.755 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.755 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 17:24:26.758 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.758 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.759 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.759 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 17:24:26.762 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.762 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.763 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.763 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 17:24:26.765 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.765 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.766 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.766 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 17:24:26.768 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.769 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.769 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.769 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 17:24:26.771 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.772 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.772 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.772 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 17:24:26.774 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.775 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.775 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.775 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 17:24:26.782 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.782 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.782 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.783 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 17:24:26.797 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.797 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.798 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.798 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 17:24:26.802 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.802 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.803 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.803 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 17:24:26.807 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.808 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.808 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.808 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 17:24:26.812 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.812 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.813 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.813 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 17:24:26.816 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.816 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.817 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.819 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 17:24:26.822 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.822 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.822 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.823 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 17:24:26.825 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 17:24:26.825 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 17:24:26.826 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.826 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:24:26.828 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.829 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.829 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.830 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 17:24:26.834 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.834 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:24:26.835 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:26.835 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 17:24:26.838 INFO 2981023 --- [ocp-schedules-6,b4c3a2a6925635a7,d551e5165c61a2da] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 17:24:27.147 INFO 2981023 --- [ocp-async-4,5222a28186d5d82f,f8cd749a7a103b26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:27.737 INFO 2981023 --- [ocp-async-1,ad9e58abfc706a9b,018584a4b3067fda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:24:28.989 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:28.991 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:29.147 INFO 2981023 --- [ocp-async-10,2961cdff09c06dd9,800dfb06a7fd347b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,6f7a85de09b44613,cc1904ce78478e23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,6f7a85de09b44613,cc1904ce78478e23] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:31.147 INFO 2981023 --- [ocp-async-8,09edcf0dbcc1c5f1,379c511bcd5d47b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:31.739 INFO 2981023 --- [ocp-schedules-29,8c4fe8e6703c31e0,d245d22df613f42f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:24:31.991 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:31.992 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,351037a5ab3fa27e,797154dc625e8825] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,351037a5ab3fa27e,797154dc625e8825] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:33.006 INFO 2981023 --- [ocp-schedules-5,cb9d2859ec984eb3,bf551bb10bf02b71] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:24:33.090 INFO 2981023 --- [alarm-detect-10,cb9d2859ec984eb3,3f88251014013d09] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046640 ~ 1726046663 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726046655, value=1.625)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.625, values={1726046655=1.625}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:24:33.107 WARN 2981023 --- [scanner-13477022-1,cb9d2859ec984eb3,05ca18d06387c0c9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:24:33.147 INFO 2981023 --- [ocp-async-2,52b3abbcd93c16ca,44d8571b8fea184e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:33.472 INFO 2981023 --- [ocp-schedules-29,f606feac6587c6e0,067376ec87a102b0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:24:33.480 INFO 2981023 --- [ocp-schedules-29,f606feac6587c6e0,067376ec87a102b0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:24:33.480 INFO 2981023 --- [ocp-schedules-29,f606feac6587c6e0,067376ec87a102b0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:24:34.992 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:34.993 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:35.147 INFO 2981023 --- [ocp-async-6,c86653c432c2b01e,8e872268a74e423c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,4f4746e9e65dfce3,95ebb7bc91fed300] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:35.317 INFO 2981023 --- [prometheus-minute-schedule-1,4f4746e9e65dfce3,95ebb7bc91fed300] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:36.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:24:36.350973+08:00, lastProcessTime=2024-09-11T17:24:26.350666+08:00 2024-09-11 17:24:36.495 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:24:36.350973+08:00 2024-09-11 17:24:37.148 INFO 2981023 --- [ocp-async-3,ee6617845137b288,e65e53850c1f0192] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:37.314 INFO 2981023 --- [prometheus-cache-monitor1,cbd9e82dc857c8c9,b3126932dc4583e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:24:37.315 INFO 2981023 --- [prometheus-cache-monitor1,cbd9e82dc857c8c9,b3126932dc4583e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:24:37.664 INFO 2981023 --- [ocp-schedules-30,b5f7325f8e1b438d,2ae1d02dd6bb5e46] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:24:37.682 WARN 2981023 --- [ocp-schedules-30,b5f7325f8e1b438d,2ae1d02dd6bb5e46] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:24:37.738 INFO 2981023 --- [ocp-async-6,033dce4f6cd6a62f,b58f9534228a8040] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:24:37.994 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:37.995 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,f6686203779a8fe9,68b50303b76ee6b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,f6686203779a8fe9,68b50303b76ee6b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:39.148 INFO 2981023 --- [ocp-async-2,068a2272c55f861b,131ad506b1a5e8eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:40.003 INFO 2981023 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:24:40.003 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:24:40.995 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:40.996 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:41.148 INFO 2981023 --- [ocp-async-10,804d6d161491049e,4d2f179053579c88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:41.217 INFO 2981023 --- [ocp-schedules-2,8524de5228fa1973,9bfd469fc0bd6796] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:24:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,b2298c29b9d02022,bdba4ce2ec41d96d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,b2298c29b9d02022,bdba4ce2ec41d96d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:41.742 INFO 2981023 --- [ocp-schedules-18,ca831ea834be0f65,f5ded493aced7ab7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:24:43.148 INFO 2981023 --- [ocp-async-6,c0b4490c11ab6757,35ddca12b1bf5b88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:43.202 INFO 2981023 --- [ocp-schedules-18,dbdeba90176f0386,ccb7aae8e01e2a6d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:24:43.250 WARN 2981023 --- [scanner-13477224-1,dbdeba90176f0386,f7bea7ad6b7867bb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:24:43.265 INFO 2981023 --- [alarm-detect-9,dbdeba90176f0386,c4527365037450a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643730, data is empty. 2024-09-11 17:24:43.270 INFO 2981023 --- [alarm-detect-9,dbdeba90176f0386,c4527365037450a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643730, data is empty. 2024-09-11 17:24:43.275 INFO 2981023 --- [alarm-detect-9,dbdeba90176f0386,c4527365037450a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643730, data is empty. 2024-09-11 17:24:43.292 INFO 2981023 --- [alarm-detect-6,dbdeba90176f0386,1ee5fcb3c2699998] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643730, data is empty. 2024-09-11 17:24:43.293 INFO 2981023 --- [alarm-detect-18,dbdeba90176f0386,575839ccd2437a56] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643730, data is empty. 2024-09-11 17:24:43.296 INFO 2981023 --- [alarm-detect-6,dbdeba90176f0386,1ee5fcb3c2699998] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643730, data is empty. 2024-09-11 17:24:43.296 INFO 2981023 --- [alarm-detect-18,dbdeba90176f0386,575839ccd2437a56] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643730, data is empty. 2024-09-11 17:24:43.299 INFO 2981023 --- [alarm-detect-6,dbdeba90176f0386,1ee5fcb3c2699998] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643730, data is empty. 2024-09-11 17:24:43.300 INFO 2981023 --- [alarm-detect-18,dbdeba90176f0386,575839ccd2437a56] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643730, data is empty. 2024-09-11 17:24:43.409 INFO 2981023 --- [alarm-detect-11,dbdeba90176f0386,3354ba11f4a45b42] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643730, data is empty. 2024-09-11 17:24:43.417 INFO 2981023 --- [alarm-detect-11,dbdeba90176f0386,3354ba11f4a45b42] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643730, data is empty. 2024-09-11 17:24:43.423 INFO 2981023 --- [alarm-detect-11,dbdeba90176f0386,3354ba11f4a45b42] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643730, data is empty. 2024-09-11 17:24:43.996 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:43.996 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,4e088a1bfdc56b6c,91001399184d1077] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,4e088a1bfdc56b6c,91001399184d1077] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:45.101 INFO 2981023 --- [prometheus-second-schedule-1,cc74248a42f3ea3d,d1849b556fc08371] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:24:45.148 INFO 2981023 --- [ocp-async-1,0b4e0893276544b2,73d2cff011dc3b34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:46.020 INFO 2981023 --- [ocp-schedules-14,c4482f3a64a73c0e,cee251446d7a594f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:24:46.020 INFO 2981023 --- [ocp-schedules-13,5874318f11dcb065,77c33b39274bb83d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:24:46.020 INFO 2981023 --- [host-check-clockdiff1,c4482f3a64a73c0e,07c6e0c4dc64d5b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:24:46.021 INFO 2981023 --- [host-check-agent-status1,5874318f11dcb065,614d026907804d68] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:24:46.023 INFO 2981023 --- [host-check-clockdiff2,c4482f3a64a73c0e,c89430d4aed4edd2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:24:46.024 INFO 2981023 --- [host-check-agent-status2,5874318f11dcb065,4a6b5758cb21caa4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:24:46.024 INFO 2981023 --- [host-check-agent-status3,5874318f11dcb065,36a58f14987f09aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:24:46.032 INFO 2981023 --- [host-check-clockdiff3,c4482f3a64a73c0e,f6b357745355dbfb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:24:46.032 INFO 2981023 --- [host-check-agent-status4,5874318f11dcb065,597606d257504cd3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:24:46.047 INFO 2981023 --- [host-check-clockdiff4,c4482f3a64a73c0e,f1d81bde73251909] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:24:46.049 INFO 2981023 --- [host-check-clockdiff1,c4482f3a64a73c0e,07c6e0c4dc64d5b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046686 0 0 , error= 2024-09-11 17:24:46.051 INFO 2981023 --- [ocp-schedules-15,3b8f44dabbecd53a,b973f27fb6a584ed] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:24:46.059 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:24:46.060 INFO 2981023 --- [host-check-clockdiff2,c4482f3a64a73c0e,c89430d4aed4edd2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046686 0 0 , error= 2024-09-11 17:24:46.061 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:24:46.061 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:24:46.063 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:24:46.071 INFO 2981023 --- [host-check-agent-status1,5874318f11dcb065,614d026907804d68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:46.073 INFO 2981023 --- [host-check-clockdiff4,c4482f3a64a73c0e,f1d81bde73251909] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046686 0 0 , error= 2024-09-11 17:24:46.077 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:24:46.077 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:46.077 INFO 2981023 --- [host-check-agent-status1,5874318f11dcb065,614d026907804d68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:46.078 INFO 2981023 --- [host-check-agent-status2,5874318f11dcb065,4a6b5758cb21caa4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:46.083 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:46.085 INFO 2981023 --- [host-check-agent-status3,5874318f11dcb065,36a58f14987f09aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:24:46.087 INFO 2981023 --- [host-check-agent-status3,5874318f11dcb065,36a58f14987f09aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:24:46.087 INFO 2981023 --- [host-check-agent-status2,5874318f11dcb065,4a6b5758cb21caa4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:46.091 INFO 2981023 --- [host-check-agent-status4,5874318f11dcb065,597606d257504cd3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:46.091 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:46.092 INFO 2981023 --- [host-check-agent-status4,5874318f11dcb065,597606d257504cd3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:46.092 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:24:46.103 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:46.162 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:24:46.192 INFO 2981023 --- [ocp-schedules-27,39b75c3bc794060f,17769a11eaf7c592] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:24:46.198 INFO 2981023 --- [ocp-schedules-1,01bfcbdf7a804a73,ac059d533c736ab2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:24:46.199 INFO 2981023 --- [ocp-schedules-30,2445ab6331e2a87f,3caffd58a464e3ef] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:24:46.199 INFO 2981023 --- [ocp-schedules-9,e9b5f822a67da69f,46575121e7c3ee85] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:24:46.199 INFO 2981023 --- [ocp-schedules-5,70237737a93da7f5,8e4d2a45f0d93ef7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:24:46.202 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:46.202 INFO 2981023 --- [ocp-schedules-27,39b75c3bc794060f,17769a11eaf7c592] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:24:46.265 INFO 2981023 --- [ocp-schedules-9,e9b5f822a67da69f,46575121e7c3ee85] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:24:46.278 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:46.290 INFO 2981023 --- [ocp-schedules-19,58929adba61cf10a,acc4d0926c3f7a8b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:24:46.293 INFO 2981023 --- [ocp-schedules-30,2445ab6331e2a87f,3caffd58a464e3ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:24:46.295 INFO 2981023 --- [ocp-schedules-30,2445ab6331e2a87f,3caffd58a464e3ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:24:46.364 INFO 2981023 --- [ocp-schedules-1,01bfcbdf7a804a73,ac059d533c736ab2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:24:46.398 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:24:46.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:24:46.458870+08:00, lastProcessTime=2024-09-11T17:24:36.350973+08:00 2024-09-11 17:24:46.462 INFO 2981023 --- [host-check-agent-status3,5874318f11dcb065,36a58f14987f09aa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:24:46.465 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:24:46.465 INFO 2981023 --- [host-check-agent-status2,5874318f11dcb065,4a6b5758cb21caa4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:24:46.466 INFO 2981023 --- [host-check-agent-status4,5874318f11dcb065,597606d257504cd3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:24:46.467 INFO 2981023 --- [host-check-agent-status3,5874318f11dcb065,36a58f14987f09aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:24:46.467 INFO 2981023 --- [host-check-agent-status3,5874318f11dcb065,36a58f14987f09aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:24:46.467 INFO 2981023 --- [host-check-agent-status3,5874318f11dcb065,36a58f14987f09aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:24:46.468 INFO 2981023 --- [host-check-agent-status1,5874318f11dcb065,614d026907804d68] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:24:46.469 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:24:46.469060252+08:00 2024-09-11 17:24:46.469 INFO 2981023 --- [host-check-agent-status2,5874318f11dcb065,4a6b5758cb21caa4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:24:46.469 INFO 2981023 --- [host-check-agent-status2,5874318f11dcb065,4a6b5758cb21caa4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:24:46.469 INFO 2981023 --- [host-check-agent-status2,5874318f11dcb065,4a6b5758cb21caa4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:24:46.471 INFO 2981023 --- [host-check-agent-status4,5874318f11dcb065,597606d257504cd3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:24:46.471 INFO 2981023 --- [host-check-agent-status4,5874318f11dcb065,597606d257504cd3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:24:46.471 INFO 2981023 --- [host-check-agent-status4,5874318f11dcb065,597606d257504cd3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:24:46.471 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:24:46.469+08:00 2024-09-11 17:24:46.473 INFO 2981023 --- [host-check-agent-status1,5874318f11dcb065,614d026907804d68] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:24:46.473 INFO 2981023 --- [host-check-agent-status1,5874318f11dcb065,614d026907804d68] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:24:46.473 INFO 2981023 --- [host-check-agent-status1,5874318f11dcb065,614d026907804d68] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:24:46.474 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:24:46.475 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:24:46.475046082+08:00 2024-09-11 17:24:46.476 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:24:46.475+08:00 2024-09-11 17:24:46.478 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:24:46.469+08:00 and status AVAILABLE 2024-09-11 17:24:46.480 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:24:46.475+08:00 and status AVAILABLE 2024-09-11 17:24:46.485 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:24:46.485 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:24:46.485507322+08:00 2024-09-11 17:24:46.486 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:24:46.486 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:24:46.485+08:00 2024-09-11 17:24:46.487 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:24:46.486610926+08:00 2024-09-11 17:24:46.487 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.487 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.488 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:24:46.485+08:00 and status AVAILABLE 2024-09-11 17:24:46.488 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:46.489 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:46.489 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:24:46.490 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:24:46.487+08:00 2024-09-11 17:24:46.490 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.490 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:24:46.490 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:46.491 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:24:46.492 INFO 2981023 --- [ocp-schedules-27,39b75c3bc794060f,17769a11eaf7c592] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:24:46.492 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:24:46.487+08:00 and status AVAILABLE 2024-09-11 17:24:46.493 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:24:46.494 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:24:46.503 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:24:46.507 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.512 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:46.513 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.513 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:24:46.513 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:46.514 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.514 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:46.563 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:24:46.563 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:46.565 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:46.566 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:46.566 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:46.566 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:24:46.575 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.580 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.581 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:24:46.582 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:24:46.582 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.587 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:46.589 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:46.595 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:24:46.597 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:24:46.599 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:24:46.612 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.616 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.617 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:24:46.618 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:24:46.622 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:24:46.626 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:24:46.627 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:24:46.650 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:24:46.651 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:24:46.659 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:24:46.671 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:24:46.677 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.683 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.684 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:24:46.687 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:24:46.688 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:24:46.705 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.707 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.708 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:46.708 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:24:46.709 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:24:46.712 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.714 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.715 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.715 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:46.715 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:24:46.716 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:24:46.716 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.719 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:24:46.720 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:24:46.726 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.728 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.728 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:24:46.729 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:24:46.729 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:24:46.729 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.730 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:24:46.748 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:24:46.753 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:46.753 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:24:46.755 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:46.755 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:24:46.756 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:24:46.759 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:24:46.760 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:24:46.771 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:24:46.772 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:24:46.773 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:24:46.780 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:24:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:46.791 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.792 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:24:46.792 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.792 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:24:46.792 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.797 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:24:46.798 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.800 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.800 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:46.801 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:24:46.801 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.801 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.802 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:46.804 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:24:46.805 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:24:46.805 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:24:46.806 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:24:46.810 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.811 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:24:46.811 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:24:46.811 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:24:46.812 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.815 INFO 2981023 --- [host-check-host-status3,3b8f44dabbecd53a,4b98ab846c9cbbec] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:24:46.818 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:24:46.818 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:24:46.820 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:24:46.823 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.824 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:24:46.824 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.825 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:46.826 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:24:46.828 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:24:46.829 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:24:46.833 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:24:46.833 INFO 2981023 --- [host-check-host-status2,3b8f44dabbecd53a,4d3b5358f6857dc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:24:46.834 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.835 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.835 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:24:46.836 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:24:46.838 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.839 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:24:46.841 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:24:46.842 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:24:46.853 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:24:46.854 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:24:46.855 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:24:46.861 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:46.861 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:24:46.872 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:24:46.876 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:24:46.877 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:24:46.880 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:24:46.882 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:24:46.895 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.895 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.896 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:24:46.896 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.896 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:24:46.897 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.897 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:24:46.898 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:24:46.898 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.900 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:24:46.900 INFO 2981023 --- [host-check-host-status1,3b8f44dabbecd53a,f6dd74923127e7e4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:24:46.904 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:24:46.905 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:24:46.912 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:24:46.915 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:24:46.919 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:24:46.458870+08:00 2024-09-11 17:24:46.922 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:24:46.930 INFO 2981023 --- [host-check-host-status4,3b8f44dabbecd53a,6a5283c66030218b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:24:46.933 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:24:46.935 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:24:46.945 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.949 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.950 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:24:46.956 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:24:46.957 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.958 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.958 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:24:46.959 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:24:46.960 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.960 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:24:46.960 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:24:46.961 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:24:46.964 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.964 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.965 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:24:46.971 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:46.971 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:46.971 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:24:46.971 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:24:46.972 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:24:46.977 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:24:46.977 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:24:46.977 INFO 2981023 --- [ocp-schedules-9,e9b5f822a67da69f,46575121e7c3ee85] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:24:46.996 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:46.997 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:47.002 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:24:47.018 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:24:47.024 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:24:47.027 INFO 2981023 --- [ocp-schedules-11,21c6ba2e105ba1a1,befc93921b3c687c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:24:47.148 INFO 2981023 --- [ocp-async-7,baaa48c1c2c74e37,fca40a7afc2a3f3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,019957adb9756862,c7db8aa69032d215] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,019957adb9756862,c7db8aa69032d215] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:47.738 INFO 2981023 --- [ocp-async-5,18033024cacb031e,6cdf28b145984c17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:24:49.148 INFO 2981023 --- [ocp-async-8,5944b6c09ab74be2,297fd2013b34edca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:24:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:24:49.997 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:50.000 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:50.197 INFO 2981023 --- [ocp-schedules-28,f02fb9b7bdfcc8e3,3ccc753a2c7581f9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:24:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,69b8b02de983e35b,61646576e6592f04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,69b8b02de983e35b,61646576e6592f04] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:50.806 INFO 2981023 --- [ocp-schedules-24,bc566a4085ef739f,b26a0c2a36a8228e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:24:50.806+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:24:50.808 INFO 2981023 --- [ocp-schedules-24,bc566a4085ef739f,b26a0c2a36a8228e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:24:50.958 INFO 2981023 --- [ocp-schedules-24,bc566a4085ef739f,b26a0c2a36a8228e] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 21 to SqlAggregateData 2024-09-11 17:24:50.960 INFO 2981023 --- [ocp-schedules-24,bc566a4085ef739f,b26a0c2a36a8228e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:24:51.148 INFO 2981023 --- [ocp-async-1,cac56090732589bc,b315c797644df6fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:51.311 INFO 2981023 --- [ocp-schedules-1,911d58f6d19792fe,67ed2e1753b2615b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:24:51.312 INFO 2981023 --- [ocp-schedules-1,911d58f6d19792fe,67ed2e1753b2615b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:24:51.350 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:24:51.350 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.352 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:24:51.353 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.354 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:24:51.355 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:24:51.355 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.356 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:24:51.359 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:24:51.360 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.361 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.361 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.361 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.363 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.363 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.367 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.367 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.373 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.373 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:24:51.375 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.376 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.381 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.381 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:24:51.386 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.387 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:24:51.389 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.389 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.390 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:24:51.390 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.390 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1CD1D734-0-0] 2024-09-11 17:24:51.390 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:24:51.390 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.390 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.391 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:24:51.392 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.393 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:24:51.401 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.402 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:24:51.402 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.402 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.403 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:24:51.403 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:24:51.404 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.404 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:24:51.405 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:24:51.407 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.408 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:24:51.410 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.410 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:24:51.411 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.411 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:24:51.412 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:24:51.412 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.413 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:24:51.415 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.416 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:24:51.417 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.417 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.417 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:24:51.417 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:24:51.418 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:24:51.418 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1CD1D73E-0-0] 2024-09-11 17:24:51.418 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.418 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:24:51.419 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:24:51.420 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.420 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:24:51.421 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.422 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:24:51.422 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:24:51.424 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:24:51.425 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.425 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:51.426 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:24:51.426 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.427 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:24:51.427 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.428 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.428 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.428 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:24:51.429 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.430 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:24:51.430 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.431 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:24:51.432 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.432 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:24:51.433 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:24:51.435 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:24:51.437 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:51.439 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.439 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:24:51.443 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:24:51.446 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:24:51.449 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:24:51.449 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:24:51.449 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:24:51.450 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:24:51.450 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.450 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:24:51.451 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:24:51.453 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.453 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:24:51.457 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:24:51.459 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:24:51.459 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.460 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.465 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:24:51.466 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:24:51.467 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:24:51.467 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:51.467 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:24:51.468 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:24:51.468 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:24:51.468 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:24:51.477 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:24:51.479 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:24:51.479 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:24:51.479 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:24:51.479 INFO 2981023 --- [ocp-schedules-1,911d58f6d19792fe,67ed2e1753b2615b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:24:51.501 INFO 2981023 --- [ocp-schedules-4,66fcdcb92f0b975d,80f8d700bb2b9cab] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:24:51.502 INFO 2981023 --- [ocp-schedules-4,66fcdcb92f0b975d,80f8d700bb2b9cab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:24:51.502 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:24:51.503 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:24:51.503 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:24:51.504 INFO 2981023 --- [ocp-schedules-4,66fcdcb92f0b975d,80f8d700bb2b9cab] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:24:51.744 INFO 2981023 --- [ocp-schedules-8,7db32701df09d86c,c45774ac9a3c4c2a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:24:51.962 INFO 2981023 --- [host-check-clockdiff3,c4482f3a64a73c0e,f6b357745355dbfb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046691 1 1 , error= 2024-09-11 17:24:52.148 INFO 2981023 --- [ocp-schedules-27,67dda519c235e674,f4aed13d6912cd34] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:24:52.167 WARN 2981023 --- [ocp-schedules-27,67dda519c235e674,f4aed13d6912cd34] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:24:52.466 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9cfd48b2142e28e2,8a5a83e6531a64fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=9cfd48b2142e28e2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:24:52.467 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,d922082c65e72b07,b395fae8887d1e3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=d922082c65e72b07, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:24:52.468 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,d922082c65e72b07,b395fae8887d1e3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=d922082c65e72b07, duration=8 ms] 2024-09-11 17:24:52.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9cfd48b2142e28e2,8a5a83e6531a64fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=9cfd48b2142e28e2, duration=10 ms] 2024-09-11 17:24:52.473 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c8669daf1709d2e6,1c16b2f5b885d462] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=c8669daf1709d2e6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:24:52.474 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c8669daf1709d2e6,1c16b2f5b885d462] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=c8669daf1709d2e6, duration=3 ms] 2024-09-11 17:24:53.000 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:53.001 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:53.018 INFO 2981023 --- [ocp-schedules-21,2dfe42ab9aa45974,eac9cfd9ca668701] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:24:53.018 INFO 2981023 --- [ocp-schedules-21,2dfe42ab9aa45974,eac9cfd9ca668701] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:24:53.149 INFO 2981023 --- [ocp-async-5,00516f5da4309e1d,1630c916f378a19d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,32308173ae272eb3,4edd5a74ff6baf20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,32308173ae272eb3,4edd5a74ff6baf20] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:53.457 INFO 2981023 --- [ocp-schedules-8,404b5abf2b820dfd,fe819a143ca84962] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:24:53.491 INFO 2981023 --- [ocp-schedules-22,191ec9c2c299d71a,066cff4275090869] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:24:53.505 INFO 2981023 --- [ocp-schedules-22,191ec9c2c299d71a,066cff4275090869] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:24:53.505 INFO 2981023 --- [ocp-schedules-22,191ec9c2c299d71a,066cff4275090869] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:24:53.513 WARN 2981023 --- [scanner-13477503-1,404b5abf2b820dfd,885efa75c5a22f94] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:24:54.162 INFO 2981023 --- [ocp-schedules-27,41ce7f043330e221,90f622af1fdf8890] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:24:54.673 INFO 2981023 --- [ocp-schedules-3,fe7b6a284188241c,7aec4955b38982a4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:24:54.673 INFO 2981023 --- [ocp-schedules-3,fe7b6a284188241c,7aec4955b38982a4] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:24:54.678 INFO 2981023 --- [ocp-schedules-3,fe7b6a284188241c,7aec4955b38982a4] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:24:54.678 INFO 2981023 --- [ocp-schedules-3,fe7b6a284188241c,7aec4955b38982a4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:24:54.678 INFO 2981023 --- [ocp-schedules-3,fe7b6a284188241c,7aec4955b38982a4] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:24:54.688 INFO 2981023 --- [ocp-schedules-3,fe7b6a284188241c,7aec4955b38982a4] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:24:54.688 INFO 2981023 --- [ocp-schedules-3,fe7b6a284188241c,7aec4955b38982a4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:24:54.688 INFO 2981023 --- [ocp-schedules-3,fe7b6a284188241c,7aec4955b38982a4] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:24:54.699 INFO 2981023 --- [ocp-schedules-3,fe7b6a284188241c,7aec4955b38982a4] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:24:55.149 INFO 2981023 --- [ocp-async-6,7225715dd2345483,666286a04fa0e5b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:55.620 INFO 2981023 --- [ocp-schedules-6,a59ff27aa485f018,0e30db3d1cbdc7e5] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:24:55.620 INFO 2981023 --- [ocp-schedules-6,a59ff27aa485f018,0e30db3d1cbdc7e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:24:55.623 INFO 2981023 --- [ocp-schedules-6,a59ff27aa485f018,0e30db3d1cbdc7e5] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:24:55.623 INFO 2981023 --- [ocp-schedules-6,a59ff27aa485f018,0e30db3d1cbdc7e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:24:55.625 INFO 2981023 --- [ocp-schedules-6,a59ff27aa485f018,0e30db3d1cbdc7e5] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:24:55.625 INFO 2981023 --- [ocp-schedules-6,a59ff27aa485f018,0e30db3d1cbdc7e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:24:55.675 INFO 2981023 --- [ocp-schedules-27,c121e5f1044ecabf,0c6c9966f3009e68] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:24:55.675 INFO 2981023 --- [ocp-schedules-27,c121e5f1044ecabf,0c6c9966f3009e68] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:24:55.677 INFO 2981023 --- [ocp-schedules-1,2d93a3af2ea74973,7921110d706fe651] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:24:55.686 INFO 2981023 --- [ocp-schedules-1,2d93a3af2ea74973,7921110d706fe651] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:24:55.686 INFO 2981023 --- [ocp-schedules-1,2d93a3af2ea74973,7921110d706fe651] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:24:55.697 INFO 2981023 --- [Collect all tenants major compactions3,c121e5f1044ecabf,9131d244cd140705] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:24:55.697 INFO 2981023 --- [Collect all tenants major compactions5,c121e5f1044ecabf,1d16706363a63f55] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:24:55.698 INFO 2981023 --- [Collect all tenants major compactions4,c121e5f1044ecabf,4a65f8a52935dc84] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:24:55.699 INFO 2981023 --- [Collect all tenants major compactions1,c121e5f1044ecabf,3c38e6be9c3db06d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:24:55.700 INFO 2981023 --- [Collect all tenants major compactions6,c121e5f1044ecabf,fc5282a5bffd4508] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:24:55.701 INFO 2981023 --- [Collect all tenants major compactions2,c121e5f1044ecabf,44bab4a16463d458] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:24:55.717 INFO 2981023 --- [Collect all tenants major compactions3,c121e5f1044ecabf,9131d244cd140705] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:55.719 INFO 2981023 --- [Collect all tenants major compactions3,c121e5f1044ecabf,9131d244cd140705] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:24:55.721 INFO 2981023 --- [Collect all tenants major compactions1,c121e5f1044ecabf,3c38e6be9c3db06d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:55.721 INFO 2981023 --- [Collect all tenants major compactions3,c121e5f1044ecabf,9131d244cd140705] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:24:55.722 INFO 2981023 --- [Collect all tenants major compactions1,c121e5f1044ecabf,3c38e6be9c3db06d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:24:55.722 INFO 2981023 --- [Collect all tenants major compactions2,c121e5f1044ecabf,44bab4a16463d458] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:55.723 INFO 2981023 --- [Collect all tenants major compactions1,c121e5f1044ecabf,3c38e6be9c3db06d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:24:55.724 INFO 2981023 --- [Collect all tenants major compactions6,c121e5f1044ecabf,fc5282a5bffd4508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:55.724 INFO 2981023 --- [Collect all tenants major compactions3,c121e5f1044ecabf,9131d244cd140705] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:24:55.725 INFO 2981023 --- [Collect all tenants major compactions4,c121e5f1044ecabf,4a65f8a52935dc84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:55.725 INFO 2981023 --- [Collect all tenants major compactions2,c121e5f1044ecabf,44bab4a16463d458] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:24:55.726 INFO 2981023 --- [Collect all tenants major compactions5,c121e5f1044ecabf,1d16706363a63f55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:24:55.726 INFO 2981023 --- [Collect all tenants major compactions4,c121e5f1044ecabf,4a65f8a52935dc84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:24:55.726 INFO 2981023 --- [Collect all tenants major compactions6,c121e5f1044ecabf,fc5282a5bffd4508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:24:55.726 INFO 2981023 --- [Collect all tenants major compactions1,c121e5f1044ecabf,3c38e6be9c3db06d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:24:55.727 INFO 2981023 --- [Collect all tenants major compactions2,c121e5f1044ecabf,44bab4a16463d458] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:24:55.727 INFO 2981023 --- [Collect all tenants major compactions2,c121e5f1044ecabf,44bab4a16463d458] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:24:55.728 INFO 2981023 --- [Collect all tenants major compactions5,c121e5f1044ecabf,1d16706363a63f55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:24:55.728 INFO 2981023 --- [Collect all tenants major compactions4,c121e5f1044ecabf,4a65f8a52935dc84] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:24:55.728 INFO 2981023 --- [Collect all tenants major compactions4,c121e5f1044ecabf,4a65f8a52935dc84] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:24:55.728 INFO 2981023 --- [Collect all tenants major compactions6,c121e5f1044ecabf,fc5282a5bffd4508] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:24:55.728 INFO 2981023 --- [Collect all tenants major compactions6,c121e5f1044ecabf,fc5282a5bffd4508] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:24:55.729 INFO 2981023 --- [Collect all tenants major compactions5,c121e5f1044ecabf,1d16706363a63f55] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:24:55.729 INFO 2981023 --- [Collect all tenants major compactions5,c121e5f1044ecabf,1d16706363a63f55] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:24:55.730 INFO 2981023 --- [ocp-schedules-27,c121e5f1044ecabf,0c6c9966f3009e68] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:24:56.001 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:56.002 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,9483bedee1031916,d2e0c97f9c52bd64] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,9483bedee1031916,d2e0c97f9c52bd64] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:24:56.351536+08:00, lastProcessTime=2024-09-11T17:24:46.458870+08:00 2024-09-11 17:24:56.450 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,532dd30d53c81edd,95e61c2753ef09bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=532dd30d53c81edd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:24:56.452 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,93624ba58de19015,dd47befc0333d7b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=93624ba58de19015, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:24:56.453 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,93624ba58de19015,dd47befc0333d7b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=93624ba58de19015, duration=8 ms] 2024-09-11 17:24:56.459 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,d9def37093a9f43b,23aaf0ed350de7ee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=d9def37093a9f43b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:24:56.463 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,d9def37093a9f43b,23aaf0ed350de7ee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=d9def37093a9f43b, duration=6 ms] 2024-09-11 17:24:56.466 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,532dd30d53c81edd,95e61c2753ef09bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=532dd30d53c81edd, duration=20 ms] 2024-09-11 17:24:56.493 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:24:56.351536+08:00 2024-09-11 17:24:57.149 INFO 2981023 --- [ocp-async-1,be0b8b5a021fd288,ba38c5e6b10f7ca3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:57.738 INFO 2981023 --- [ocp-async-4,7495e5ce0e38ef92,4543f8fb5f212eb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:24:57.903 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,647e74e488f6fe38,3297ecf94192ebdb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.31.33.41, traceId=647e74e488f6fe38, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-09-11 17:24:57.906 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,647e74e488f6fe38,3297ecf94192ebdb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.31.33.41, traceId=647e74e488f6fe38, duration=5 ms] 2024-09-11 17:24:59.002 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:24:59.004 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:24:59.149 INFO 2981023 --- [ocp-async-5,0b523078739dc8f0,8dd512c479286ab5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:24:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,f81bce8d823e7e88,70615e4ca93c7778] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046700, interval=60 2024-09-11 17:24:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,f81bce8d823e7e88,70615e4ca93c7778] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:24:59.581 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:20:59.189, end=2024-09-11T17:22:59.577} 2024-09-11 17:24:59.585 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, 2024-09-11 17:24:59.586 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:24:59.589 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:20:59.189, end=2024-09-11T17:22:59.577},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:24:59.589 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:24:59.589 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:24:59.657 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299657000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.657Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.661 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299661000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.661Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.664 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299664000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.664Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.670 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299670000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.670Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.672 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299672000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.672Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.673 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299673000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.673Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.674 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299674000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.674Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.675 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299675000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.675Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.680 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.680 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.683 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:24:59.683 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.689 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.689 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.690 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:24:59.690 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.694 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.694 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.698 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.698 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.700 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.700 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.702 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:24:59.702 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.703 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.703 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.705 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 58 2024-09-11 17:24:59.705 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.705 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.705 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.705 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.705 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.707 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.707 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.708 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.708 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.709 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.709 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.712 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.712 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.715 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.715 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.715 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.715 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.718 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.718 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.718 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.718 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.719 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.719 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.720 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.720 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.720 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.720 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.721 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.721 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.721 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.721 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.721 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.722 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.725 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.725 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.726 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.726 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.728 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.728 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.728 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.731 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.657, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.732 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.732 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.732 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.732 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.673, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.735 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.735 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.735 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.735 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.735 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.737 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.737 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.661, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.739 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.739 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.67, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.742 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.742 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.744 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.672, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.745 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.745 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.675, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.746 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.664, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.753 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.674, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.769 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299769000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.769Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.774 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299774000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.774Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.775 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299775000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.775Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.779 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:24:59.779 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.779 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299779000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.779Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.785 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299785000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.785Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.788 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:24:59.788 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.791 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.791 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.792 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.792 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.793 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.793 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.796 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299796000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.796Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.797 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299797000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=652DD95F8EA2C33B685ED2AC08D4CAF5, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.797Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.799 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.799 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.800 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.800 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.801 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299801000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.801Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.807 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.807 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.807 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.807 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.810 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.810 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.811 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.811 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.811 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.812 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.812 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.813 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.813 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.813 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.813 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.814 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.814 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.815 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.815 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.818 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.769, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.819 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.819 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.820 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.820 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.824 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.824 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=652DD95F8EA2C33B685ED2AC08D4CAF5, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.825 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.825 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.827 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.827 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.828 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.828 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.828 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.828 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.829 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.829 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.829 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.830 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.832 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.834 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.834 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.834 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.779, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.836 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.836 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.837 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.837 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.840 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.840 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.841 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.841 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.843 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.843 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=652DD95F8EA2C33B685ED2AC08D4CAF5, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.847 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299847000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.847Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.848 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.775, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.850 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.850 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.850 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.851 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.851 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.854 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.855 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.855 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.856 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.774, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.856 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.785, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.858 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.797, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=652DD95F8EA2C33B685ED2AC08D4CAF5, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.858 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:24:59.858 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.860 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.796, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.861 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.801, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.864 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.864 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.866 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.866 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.870 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.870 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.871 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:24:59.874 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.847, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.874 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:24:59.874 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:20:59.189, end=2024-09-11T17:22:59.577},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:24:59.874 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:24:59.874 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:24:59.875 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:24:59.878 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 108 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:20:59.189, end=2024-09-11T17:22:59.577},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:24:59.922 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:24:59.922 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:24:59.932 INFO 2981023 --- [plan_change_detector18,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:24:59.932 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-8344E91CF4F29A8CAD0301EFE628E5A5 2024-09-11 17:24:59.932 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-FF3782D16DEF7A30093E09436047E98C 2024-09-11 17:24:59.932 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:24:59.933 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:20:59.189, end=2024-09-11T17:22:59.577},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:24:59.933 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:24:59.933 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:24:59.934 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:24:59.934 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:20:59.189, end=2024-09-11T17:22:59.577},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:24:59.934 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:24:59.934 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:24:59.935 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:24:59.936 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:20:59.189, end=2024-09-11T17:22:59.577},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:24:59.936 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:24:59.936 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:24:59.946 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:24:59.962 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299962000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.962Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.963 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960299963000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=AD63995D4A6E8C3787D81BDA88E06774, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:24:59.963Z, endTime=2024-09-11T09:22:59.577Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:24:59.964 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:24:59.964 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.972 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.972 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.977 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.977 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.983 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.983 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.984 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 250 2024-09-11 17:24:59.984 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:24:59.985 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:24:59.985 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.962, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:24:59.989 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.989 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.992 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:24:59.992 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:24:59.997 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:24:59.997 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046579577000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 17:24:59.998 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:25:00.002 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:24:59.963, end=2024-09-11T17:22:59.577}, clusterId=2, tenantId=4, sqlId=AD63995D4A6E8C3787D81BDA88E06774, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:25:00.002 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:25:00.716 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:20:00.0, nextRun=2024-09-11 17:25:00.0, currentDate=Wed Sep 11 17:25:00 HKT 2024 2024-09-11 17:25:00.716 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:20:00.0, nextRunTime=2024-09-11 17:25:00.0} 2024-09-11 17:25:00.722 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:25:00.722 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:25:00.722 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:25:00.722+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:25:00.734 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:20:00.0, nextRun=2024-09-11 17:25:00.0, currentDate=Wed Sep 11 17:25:00 HKT 2024 2024-09-11 17:25:00.734 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:20:00.0, nextRunTime=2024-09-11 17:25:00.0} 2024-09-11 17:25:00.746 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2, 1000006] 2024-09-11 17:25:00.752 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:25:00.753 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:25:00.753 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:25:00.753 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1ca1bfc735ea5fde] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2,1000006]]}}, createTime=null, startTime=2024-09-11T17:25:00.753+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:25:01.149 INFO 2981023 --- [ocp-async-8,b0a72bd3753b6b6b,d6ec2a4f090de8cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:01.624 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,4ea3114705029ece] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048160, operation:EXECUTE, state: RUNNING 2024-09-11 17:25:01.637 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,4ea3114705029ece] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048161, operation:EXECUTE, state: RUNNING 2024-09-11 17:25:01.649 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048160, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048180, task_operation=execute, latest_execution_start_time=2024-09-11T17:25:01.617+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048160}, listMap={}}, executor=10.31.35.10 2024-09-11 17:25:01.657 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048161, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048181, task_operation=execute, latest_execution_start_time=2024-09-11T17:25:01.635+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048161}, listMap={cluster_ids=[[1,2,1000006]]}}, executor=10.31.35.10 2024-09-11 17:25:01.661 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:25:01.668 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:25:01.685 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:25:01.688 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:25:01.699 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:25:01.699 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:25:01.699 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:25:01.710 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:25:01.716 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:25:01.724 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:25:01.724 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1000006 2024-09-11 17:25:01.747 INFO 2981023 --- [ocp-schedules-30,291a45cf670f9b5d,39b5546f95d2e2d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:25:01.755 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,3aa47bb6925c6548] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048161, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:25:01.793 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:25:01.812 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:25:01.838 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:25:01.849 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:25:01.849 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:25:01.853 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:25:01.886 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048181, status:SUCCESS 2024-09-11 17:25:01.888 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048181, status:SUCCESS 2024-09-11 17:25:01.958 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:25:02.004 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:02.004 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:02.008 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:25:02.029 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:25:02.049 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:25:02.084 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:25:02.097 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:25:02.112 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:25:02.119 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:25:02.119 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:25:02.123 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:25:02.193 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:25:02.225 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:25:02.237 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:25:02.245 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:25:02.254 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:25:02.260 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:25:02.266 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:25:02.268 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:25:02.269 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:25:02.282 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:25:02.291 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:25:02.304 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5d356f4c652b9ce5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048160, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:25:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,867e16e349466982,4910225741e325c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:02.315 INFO 2981023 --- [prometheus-minute-schedule-1,867e16e349466982,4910225741e325c0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:02.908 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048180, status:SUCCESS 2024-09-11 17:25:02.911 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048180, status:SUCCESS 2024-09-11 17:25:03.149 INFO 2981023 --- [ocp-async-1,e4f8cb988d13a8c0,a7d683ea6483123e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:03.655 INFO 2981023 --- [ocp-schedules-5,5507f042c1756fb8,f3dc262f05fe7345] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:25:03.706 WARN 2981023 --- [scanner-13477934-1,5507f042c1756fb8,a3808a0ab11d582c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:25:03.765 INFO 2981023 --- [alarm-detect-11,5507f042c1756fb8,5c3822a90f431c52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643730, data is empty. 2024-09-11 17:25:03.770 INFO 2981023 --- [alarm-detect-11,5507f042c1756fb8,5c3822a90f431c52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643751, data is empty. 2024-09-11 17:25:03.773 INFO 2981023 --- [alarm-detect-19,5507f042c1756fb8,8bceaf1cacc24a94] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643730, data is empty. 2024-09-11 17:25:03.776 INFO 2981023 --- [alarm-detect-16,5507f042c1756fb8,195b47d23f55fb06] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643730, data is empty. 2024-09-11 17:25:03.777 INFO 2981023 --- [alarm-detect-19,5507f042c1756fb8,8bceaf1cacc24a94] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643751, data is empty. 2024-09-11 17:25:03.779 INFO 2981023 --- [alarm-detect-11,5507f042c1756fb8,5c3822a90f431c52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643751, data is empty. 2024-09-11 17:25:03.779 INFO 2981023 --- [alarm-detect-16,5507f042c1756fb8,195b47d23f55fb06] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643751, data is empty. 2024-09-11 17:25:03.782 INFO 2981023 --- [alarm-detect-19,5507f042c1756fb8,8bceaf1cacc24a94] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643751, data is empty. 2024-09-11 17:25:03.783 INFO 2981023 --- [alarm-detect-16,5507f042c1756fb8,195b47d23f55fb06] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643751, data is empty. 2024-09-11 17:25:03.920 INFO 2981023 --- [alarm-detect-4,5507f042c1756fb8,f240a51efa783e6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643730, data is empty. 2024-09-11 17:25:03.923 INFO 2981023 --- [alarm-detect-4,5507f042c1756fb8,f240a51efa783e6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643751, data is empty. 2024-09-11 17:25:03.929 INFO 2981023 --- [alarm-detect-4,5507f042c1756fb8,f240a51efa783e6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643751, data is empty. 2024-09-11 17:25:05.004 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:05.006 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:05.149 INFO 2981023 --- [ocp-async-7,e04f36c86eda451d,0466d934b20538d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,9eb823c24dd0482a,db9957196bfad0d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,9eb823c24dd0482a,db9957196bfad0d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:25:06.352673+08:00, lastProcessTime=2024-09-11T17:24:56.351536+08:00 2024-09-11 17:25:06.435 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2009a45564265c1f,1954acc98a78b21b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=2009a45564265c1f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:25:06.435 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,8c47b27713185a24,a6bba928872dc468] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=8c47b27713185a24, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:25:06.436 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2009a45564265c1f,1954acc98a78b21b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=2009a45564265c1f, duration=6 ms] 2024-09-11 17:25:06.439 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,8c47b27713185a24,a6bba928872dc468] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=8c47b27713185a24, duration=9 ms] 2024-09-11 17:25:06.446 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,0caf258e60eed194,15c4ba72800f3d24] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=0caf258e60eed194, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:25:06.446 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,0caf258e60eed194,15c4ba72800f3d24] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=0caf258e60eed194, duration=6 ms] 2024-09-11 17:25:06.500 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:25:06.352673+08:00 2024-09-11 17:25:07.150 INFO 2981023 --- [ocp-async-6,a708c44d06ef5b17,573cc27be8b13b17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:07.172 INFO 2981023 --- [ocp-schedules-1,e7835d122f10be79,3a0129659a820acb] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:25:07.172+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:25:07.173 INFO 2981023 --- [ocp-schedules-23,c8a18453cc0e1cb0,ad1bba47490edad0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:25:07.173 INFO 2981023 --- [ocp-schedules-1,e7835d122f10be79,3a0129659a820acb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:25:07.175 INFO 2981023 --- [ocp-schedules-1,e7835d122f10be79,3a0129659a820acb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:25:07.175 INFO 2981023 --- [ocp-schedules-1,e7835d122f10be79,3a0129659a820acb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:25:07.738 INFO 2981023 --- [ocp-async-5,88fb874745ba49ae,f4931ef308f88e09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:25:08.006 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:08.008 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,20147ec91de238a2,be6c7345a5bd8375] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:08.318 INFO 2981023 --- [prometheus-minute-schedule-1,20147ec91de238a2,be6c7345a5bd8375] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:09.150 INFO 2981023 --- [ocp-async-2,5605a2914971db4c,8672346709351ade] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:10.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:25:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:25:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:25:10.920 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:25:11.008 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:11.008 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:11.150 INFO 2981023 --- [ocp-async-7,c287c3ddad4923c3,df01a21b17260e4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,9572c9b1cb87803f,ec85541f814a9c86] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,9572c9b1cb87803f,ec85541f814a9c86] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:11.755 INFO 2981023 --- [ocp-schedules-7,f4c0abbccd890710,9abffd6d1c194e89] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:25:12.947 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046640, nodeCount=1000, affectRows=1000 2024-09-11 17:25:12.950 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046640, nodeCount=999, validCount=999 2024-09-11 17:25:13.033 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046640, nodeCount=1000, affectRows=1000 2024-09-11 17:25:13.035 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046640, nodeCount=1000, validCount=1000 2024-09-11 17:25:13.043 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046640, nodeCount=49, affectRows=49 2024-09-11 17:25:13.043 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046640, nodeCount=49, validCount=49 2024-09-11 17:25:13.150 INFO 2981023 --- [ocp-async-6,9bcab954e6a8a239,84cbf86506c55275] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:13.508 INFO 2981023 --- [ocp-schedules-3,bf5987fdd2072975,e8aab152cacb0950] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:25:13.517 INFO 2981023 --- [ocp-schedules-3,bf5987fdd2072975,e8aab152cacb0950] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:25:13.517 INFO 2981023 --- [ocp-schedules-3,bf5987fdd2072975,e8aab152cacb0950] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:25:13.976 INFO 2981023 --- [ocp-schedules-9,595f6b48f0fc3fbf,d3c9eaca07068e58] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:25:14.009 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:14.011 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:14.029 WARN 2981023 --- [scanner-13479761-1,595f6b48f0fc3fbf,379e931277ebeb1f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:25:14.241 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046640, nodeCount=480, affectRows=480 2024-09-11 17:25:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,74dc207dfed67cbd,52b66d1d010d97c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,9e10a48bb12db512,2a15882a06653763] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:25:14.318 INFO 2981023 --- [prometheus-minute-schedule-1,74dc207dfed67cbd,52b66d1d010d97c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:15.150 INFO 2981023 --- [ocp-async-1,95c613f5d29a57aa,fa3b8468dcce4108] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:16.029 INFO 2981023 --- [ocp-schedules-9,97784a79d40583da,2cc67f5dd2417230] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:25:16.029 INFO 2981023 --- [ocp-schedules-6,9db0ec3bc6c1eaeb,575924a9df422101] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:25:16.029 INFO 2981023 --- [host-check-clockdiff1,97784a79d40583da,49cb27f52df27634] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:25:16.029 INFO 2981023 --- [host-check-agent-status1,9db0ec3bc6c1eaeb,373c3f7d63436a3d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:25:16.030 INFO 2981023 --- [host-check-agent-status2,9db0ec3bc6c1eaeb,b3e9c30b4b75b7f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:25:16.031 INFO 2981023 --- [host-check-agent-status3,9db0ec3bc6c1eaeb,20d406588b4731c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:25:16.032 INFO 2981023 --- [host-check-agent-status4,9db0ec3bc6c1eaeb,699d7a1146fa9eea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:25:16.032 INFO 2981023 --- [host-check-clockdiff2,97784a79d40583da,036725da91f0450b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:25:16.033 INFO 2981023 --- [host-check-clockdiff3,97784a79d40583da,57c0da9102a15c0a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:25:16.033 INFO 2981023 --- [host-check-clockdiff4,97784a79d40583da,48a6959a06cccb2c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:25:16.056 INFO 2981023 --- [ocp-schedules-24,6cd13bc95edb0bb5,54e6bb5f328020cb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:25:16.056 INFO 2981023 --- [host-check-agent-status1,9db0ec3bc6c1eaeb,373c3f7d63436a3d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:25:16.057 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:25:16.057 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:25:16.057 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:25:16.058 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:25:16.063 INFO 2981023 --- [host-check-clockdiff4,97784a79d40583da,48a6959a06cccb2c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046716 0 0 , error= 2024-09-11 17:25:16.061 INFO 2981023 --- [host-check-agent-status1,9db0ec3bc6c1eaeb,373c3f7d63436a3d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:25:16.066 INFO 2981023 --- [host-check-agent-status3,9db0ec3bc6c1eaeb,20d406588b4731c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:25:16.067 INFO 2981023 --- [host-check-agent-status3,9db0ec3bc6c1eaeb,20d406588b4731c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:25:16.067 INFO 2981023 --- [host-check-agent-status2,9db0ec3bc6c1eaeb,b3e9c30b4b75b7f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:25:16.068 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:25:16.068 INFO 2981023 --- [host-check-agent-status4,9db0ec3bc6c1eaeb,699d7a1146fa9eea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:25:16.069 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:25:16.070 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:25:16.070 INFO 2981023 --- [host-check-agent-status2,9db0ec3bc6c1eaeb,b3e9c30b4b75b7f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:25:16.070 INFO 2981023 --- [host-check-agent-status4,9db0ec3bc6c1eaeb,699d7a1146fa9eea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:25:16.072 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046640, nodeCount=480, validCount=480 2024-09-11 17:25:16.074 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:25:16.105 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:25:16.121 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:25:16.163 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:25:16.164 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:25:16.188 INFO 2981023 --- [ocp-schedules-10,89f2eb282a3596b9,f16f2f937af288be] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:25:16.188 INFO 2981023 --- [ocp-schedules-28,fd58ab8e75d996d0,7d4a7829897937e9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:25:16.188 INFO 2981023 --- [ocp-schedules-20,6d65814d0fdc8036,1acb234c18a5278a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:25:16.188 INFO 2981023 --- [ocp-schedules-19,88cdc425bb4aa912,87739502b3ef988f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:25:16.219 INFO 2981023 --- [ocp-schedules-10,89f2eb282a3596b9,f16f2f937af288be] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:25:16.220 INFO 2981023 --- [ocp-schedules-20,6d65814d0fdc8036,1acb234c18a5278a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:25:16.266 INFO 2981023 --- [ocp-schedules-28,fd58ab8e75d996d0,7d4a7829897937e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:25:16.277 INFO 2981023 --- [ocp-schedules-28,fd58ab8e75d996d0,7d4a7829897937e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:25:16.323 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:25:16.324 INFO 2981023 --- [ocp-schedules-15,a9d2cf795ccbbc17,017c8adfcf917ca0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:25:16.326 INFO 2981023 --- [host-check-agent-status3,9db0ec3bc6c1eaeb,20d406588b4731c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:25:16.327 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:25:16.327544130+08:00 2024-09-11 17:25:16.327 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:25:16.327+08:00 2024-09-11 17:25:16.330 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:25:16.327+08:00 and status AVAILABLE 2024-09-11 17:25:16.331 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:25:16.331 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:25:16.333 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:25:16.333170120+08:00 2024-09-11 17:25:16.333 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:25:16.333262473+08:00 2024-09-11 17:25:16.334 INFO 2981023 --- [host-check-agent-status3,9db0ec3bc6c1eaeb,20d406588b4731c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:25:16.334 INFO 2981023 --- [host-check-agent-status3,9db0ec3bc6c1eaeb,20d406588b4731c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:25:16.334 INFO 2981023 --- [host-check-agent-status3,9db0ec3bc6c1eaeb,20d406588b4731c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:25:16.334 INFO 2981023 --- [host-check-agent-status1,9db0ec3bc6c1eaeb,373c3f7d63436a3d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:25:16.336 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:25:16.333+08:00 2024-09-11 17:25:16.348 INFO 2981023 --- [host-check-agent-status1,9db0ec3bc6c1eaeb,373c3f7d63436a3d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:25:16.348 INFO 2981023 --- [host-check-agent-status1,9db0ec3bc6c1eaeb,373c3f7d63436a3d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:25:16.348 INFO 2981023 --- [host-check-agent-status1,9db0ec3bc6c1eaeb,373c3f7d63436a3d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:25:16.356 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:25:16.357 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:25:16.333+08:00 2024-09-11 17:25:16.357 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:25:16.357019550+08:00 2024-09-11 17:25:16.358 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:25:16.357+08:00 2024-09-11 17:25:16.358 INFO 2981023 --- [host-check-agent-status2,9db0ec3bc6c1eaeb,b3e9c30b4b75b7f6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:25:16.358 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:25:16.358 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:25:16.333+08:00 and status AVAILABLE 2024-09-11 17:25:16.361 INFO 2981023 --- [host-check-agent-status2,9db0ec3bc6c1eaeb,b3e9c30b4b75b7f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:25:16.361 INFO 2981023 --- [host-check-agent-status2,9db0ec3bc6c1eaeb,b3e9c30b4b75b7f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:25:16.361 INFO 2981023 --- [host-check-agent-status2,9db0ec3bc6c1eaeb,b3e9c30b4b75b7f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:25:16.362 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:25:16.333+08:00 and status AVAILABLE 2024-09-11 17:25:16.366 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:25:16.357+08:00 and status AVAILABLE 2024-09-11 17:25:16.369 INFO 2981023 --- [host-check-agent-status4,9db0ec3bc6c1eaeb,699d7a1146fa9eea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:25:16.372 INFO 2981023 --- [host-check-agent-status4,9db0ec3bc6c1eaeb,699d7a1146fa9eea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:25:16.372 INFO 2981023 --- [host-check-agent-status4,9db0ec3bc6c1eaeb,699d7a1146fa9eea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:25:16.372 INFO 2981023 --- [host-check-agent-status4,9db0ec3bc6c1eaeb,699d7a1146fa9eea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:25:16.387 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.388 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:16.389 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:25:16.389 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:25:16.391 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:25:16.393 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.394 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:16.394 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:25:16.395 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:25:16.397 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.397 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.397 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:25:16.396960+08:00, lastProcessTime=2024-09-11T17:25:06.352673+08:00 2024-09-11 17:25:16.398 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:16.398 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:16.399 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:25:16.399 INFO 2981023 --- [ocp-schedules-10,89f2eb282a3596b9,f16f2f937af288be] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:25:16.400 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:25:16.401 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:25:16.402 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:25:16.402 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:25:16.403 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:25:16.406 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:25:16.413 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:25:16.414 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:25:16.415 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.415 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:16.419 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.420 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:16.460 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,097f7bb806063e76,4c25ba0bfcd293cf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=097f7bb806063e76, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:25:16.460 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6d1cdc707f6db474,04caf224d6584e32] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=6d1cdc707f6db474, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:25:16.462 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,936dab8a7b1cd818,148f38cac964cc53] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=936dab8a7b1cd818, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:25:16.464 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:25:16.469 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,936dab8a7b1cd818,148f38cac964cc53] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=936dab8a7b1cd818, duration=33 ms] 2024-09-11 17:25:16.470 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6d1cdc707f6db474,04caf224d6584e32] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=6d1cdc707f6db474, duration=45 ms] 2024-09-11 17:25:16.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,097f7bb806063e76,4c25ba0bfcd293cf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=097f7bb806063e76, duration=46 ms] 2024-09-11 17:25:16.481 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:25:16.486 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:25:16.493 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:25:16.498 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:25:16.507 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:25:16.513 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:25:16.519 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:25:16.522 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:25:16.527 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.530 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.531 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:25:16.531 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:25:16.534 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:25:16.536 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:25:16.567 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:25:16.577 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:25:16.579 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.581 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:25:16.582 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.582 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:16.584 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:25:16.584 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.586 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:25:16.587 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:25:16.587 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:16.588 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.591 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:25:16.592 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:25:16.592 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:25:16.593 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:25:16.602 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:25:16.604 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:16.604 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:25:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:25:16.610 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:25:16.624 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.625 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.626 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:25:16.626 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:16.626 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:25:16.627 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:25:16.628 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:25:16.628 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:25:16.634 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:16.645 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.646 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.646 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.646 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.646 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:16.647 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:16.647 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:25:16.649 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:25:16.651 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:25:16.652 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:25:16.654 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:25:16.655 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.657 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:25:16.657 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:25:16.659 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:16.660 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:25:16.660 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:25:16.661 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.664 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:25:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:25:16.672 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:25:16.674 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:25:16.674 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:25:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:25:16.686 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:25:16.686 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:25:16.694 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:25:16.703 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:25:16.704 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:25:16.704 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:25:16.729 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:25:16.730 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:16.737 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:25:16.740 INFO 2981023 --- [host-check-host-status1,6cd13bc95edb0bb5,b8457e2c63e07aee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:25:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.744 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.745 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:16.745 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:25:16.748 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:25:16.749 INFO 2981023 --- [host-check-host-status4,6cd13bc95edb0bb5,ee509b44c9c175a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:25:16.752 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:25:16.756 INFO 2981023 --- [host-check-host-status3,6cd13bc95edb0bb5,c08614e8634c830d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:25:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:25:16.761 INFO 2981023 --- [host-check-host-status2,6cd13bc95edb0bb5,6a885f6e54365022] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:25:16.762 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:25:16.777 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:25:16.782 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:25:16.786 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.786 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.786 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:25:16.787 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:25:16.794 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:25:16.794 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:25:16.795 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.796 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.797 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:25:16.803 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:25:16.803 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:25:16.811 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:25:16.817 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:16.818 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:16.818 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:25:16.825 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:25:16.825 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:25:16.825 INFO 2981023 --- [ocp-schedules-20,6d65814d0fdc8036,1acb234c18a5278a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:25:16.829 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:25:16.396960+08:00 2024-09-11 17:25:17.002 INFO 2981023 --- [host-check-clockdiff2,97784a79d40583da,036725da91f0450b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046716 0 0 , error= 2024-09-11 17:25:17.011 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:17.011 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:17.150 INFO 2981023 --- [ocp-async-3,bbe5ed02a963fff3,ef31b5464560a690] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,386d082f004f765b,19764b015e5e1117] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,386d082f004f765b,19764b015e5e1117] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:17.738 INFO 2981023 --- [ocp-async-2,73637dec1a57b4ae,f7a31ae0126c1c3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:25:19.150 INFO 2981023 --- [ocp-async-9,b280da76c8c18c3d,b6ad3d3e78653a76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:19.734 INFO 2981023 --- [host-check-clockdiff1,97784a79d40583da,49cb27f52df27634] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046719 0 0 , error= 2024-09-11 17:25:20.011 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:20.012 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,0fba6352330f36d9,9db947d52a7c10d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:20.315 INFO 2981023 --- [prometheus-minute-schedule-1,0fba6352330f36d9,9db947d52a7c10d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:20.884 INFO 2981023 --- [host-check-clockdiff3,97784a79d40583da,57c0da9102a15c0a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046720 1 1 , error= 2024-09-11 17:25:21.150 INFO 2981023 --- [ocp-async-1,ab7b42df02e116e6,8dc572a119f435bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:21.760 INFO 2981023 --- [ocp-schedules-3,c5353b13d82a083e,b09cb3a79df87239] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:25:22.310 INFO 2981023 --- [ocp-schedules-20,a75d9198c32ef032,bdc960bd97c82a4c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:25:22.310 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:25:22.310 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:25:22.310 INFO 2981023 --- [ocp-schedules-20,a75d9198c32ef032,bdc960bd97c82a4c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:25:22.310 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:25:22.314 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:25:22.314 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:25:22.315 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:25:22.315 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:25:22.318 INFO 2981023 --- [ocp-schedules-20,a75d9198c32ef032,bdc960bd97c82a4c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:25:22.319 INFO 2981023 --- [ocp-schedules-20,a75d9198c32ef032,bdc960bd97c82a4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:25:22.319 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:25:22.320 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:25:22.330 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:25:22.330 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:25:22.335 INFO 2981023 --- [ocp-schedules-20,a75d9198c32ef032,bdc960bd97c82a4c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:25:22.337 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:25:22.339 INFO 2981023 --- [ocp-schedules-20,a75d9198c32ef032,bdc960bd97c82a4c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:25:23.012 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:23.012 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:23.150 INFO 2981023 --- [ocp-async-10,16d2b2955560bc6a,662d25bf983328e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,1ed168e8a81cd76c,3840cab0282a5bb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,1ed168e8a81cd76c,3840cab0282a5bb9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:24.165 INFO 2981023 --- [ocp-schedules-21,758fec31e3af4d7a,546d61d30cc2fd2d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:25:24.201 INFO 2981023 --- [ocp-schedules-23,df8be34f10da738c,305559219acd9b5e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:25:24.256 WARN 2981023 --- [scanner-13479991-1,df8be34f10da738c,1a9f9bb310568ae0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:25:24.276 INFO 2981023 --- [alarm-detect-14,df8be34f10da738c,ad179e9fbdc8eada] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643751, data is empty. 2024-09-11 17:25:24.281 INFO 2981023 --- [alarm-detect-14,df8be34f10da738c,ad179e9fbdc8eada] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643751, data is empty. 2024-09-11 17:25:24.286 INFO 2981023 --- [alarm-detect-14,df8be34f10da738c,ad179e9fbdc8eada] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643751, data is empty. 2024-09-11 17:25:24.300 INFO 2981023 --- [alarm-detect-9,df8be34f10da738c,1ed5b97d38d85e01] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643751, data is empty. 2024-09-11 17:25:24.300 INFO 2981023 --- [alarm-detect-14,df8be34f10da738c,255379c3f2c44eab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643751, data is empty. 2024-09-11 17:25:24.305 INFO 2981023 --- [alarm-detect-9,df8be34f10da738c,1ed5b97d38d85e01] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643751, data is empty. 2024-09-11 17:25:24.306 INFO 2981023 --- [alarm-detect-14,df8be34f10da738c,255379c3f2c44eab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643751, data is empty. 2024-09-11 17:25:24.311 INFO 2981023 --- [alarm-detect-9,df8be34f10da738c,1ed5b97d38d85e01] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643751, data is empty. 2024-09-11 17:25:24.311 INFO 2981023 --- [alarm-detect-14,df8be34f10da738c,255379c3f2c44eab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643751, data is empty. 2024-09-11 17:25:24.409 INFO 2981023 --- [alarm-detect-17,df8be34f10da738c,6b1dd211c217137b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643751, data is empty. 2024-09-11 17:25:24.413 INFO 2981023 --- [alarm-detect-17,df8be34f10da738c,6b1dd211c217137b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643751, data is empty. 2024-09-11 17:25:24.420 INFO 2981023 --- [alarm-detect-17,df8be34f10da738c,6b1dd211c217137b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643751, data is empty. 2024-09-11 17:25:25.150 INFO 2981023 --- [ocp-async-6,0e45a0789e439e39,ccdd238954d62a87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:26.012 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:26.013 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,26c70268e4ada54e,a055726ccbbbfce9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,26c70268e4ada54e,a055726ccbbbfce9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:25:26.351218+08:00, lastProcessTime=2024-09-11T17:25:16.396960+08:00 2024-09-11 17:25:26.467 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,9c2ef9d99d77e405,246bba5f9d95a9cc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=9c2ef9d99d77e405, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:25:26.467 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f4998658b994759b,97040ec6d7a04640] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=f4998658b994759b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:25:26.469 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,9c2ef9d99d77e405,246bba5f9d95a9cc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=9c2ef9d99d77e405, duration=7 ms] 2024-09-11 17:25:26.475 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,03ff38872677b2fc,9c8f982e495c6a05] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=03ff38872677b2fc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:25:26.476 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,03ff38872677b2fc,9c8f982e495c6a05] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=03ff38872677b2fc, duration=3 ms] 2024-09-11 17:25:26.478 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:25:26.351218+08:00 2024-09-11 17:25:26.482 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f4998658b994759b,97040ec6d7a04640] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=f4998658b994759b, duration=20 ms] 2024-09-11 17:25:27.151 INFO 2981023 --- [ocp-async-8,777cef857b991e61,d62fe457f5f2d2f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:27.738 INFO 2981023 --- [ocp-async-9,75b173ed472ec9f4,9d9352f77c3fce40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:25:29.013 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:29.013 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:29.151 INFO 2981023 --- [ocp-async-10,4e9b8dbefae1c24b,ba3786a1ce6fd234] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,4acdc4b489e565d5,d3db860e5357113a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,4acdc4b489e565d5,d3db860e5357113a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:31.151 INFO 2981023 --- [ocp-async-5,74b7cbb0c259589c,c8021fa54a83209f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:31.762 INFO 2981023 --- [ocp-schedules-22,5510deaeb2ae57f8,85b6b618ede575c5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:25:32.013 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:32.014 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,73f1d8f211e6f8c5,c9f3ae2193fe187a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,73f1d8f211e6f8c5,c9f3ae2193fe187a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:33.151 INFO 2981023 --- [ocp-async-4,dbfd1120ed25bb2e,7bfc97a0006fabdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:33.520 INFO 2981023 --- [ocp-schedules-15,499cc6deee770aea,d618fe58901a4179] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:25:33.530 INFO 2981023 --- [ocp-schedules-15,499cc6deee770aea,d618fe58901a4179] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:25:33.530 INFO 2981023 --- [ocp-schedules-15,499cc6deee770aea,d618fe58901a4179] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:25:34.462 INFO 2981023 --- [ocp-schedules-7,1d7e24b3fc3b7558,0e18aa4d59ad100c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:25:34.504 WARN 2981023 --- [scanner-13480277-1,1d7e24b3fc3b7558,c6a2809cb4bbb9ac] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:25:35.014 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:35.014 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:35.151 INFO 2981023 --- [ocp-async-3,4bbbe797abde10b1,ce11cc8bfcd44664] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,b898feb87b6c8458,83fa8fda3ebab065] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,b898feb87b6c8458,83fa8fda3ebab065] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:25:36.337681+08:00, lastProcessTime=2024-09-11T17:25:26.351218+08:00 2024-09-11 17:25:36.429 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d857237e2384dbdb,afd03a1eb544ac1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=d857237e2384dbdb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:25:36.432 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,df0723f1ebbaac91,a5d034fd913ef9f3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=df0723f1ebbaac91, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:25:36.441 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,163eccbd310486bb,a6961f105b912038] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=163eccbd310486bb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:25:36.444 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,df0723f1ebbaac91,a5d034fd913ef9f3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=df0723f1ebbaac91, duration=18 ms] 2024-09-11 17:25:36.445 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,163eccbd310486bb,a6961f105b912038] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=163eccbd310486bb, duration=6 ms] 2024-09-11 17:25:36.445 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d857237e2384dbdb,afd03a1eb544ac1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=d857237e2384dbdb, duration=18 ms] 2024-09-11 17:25:36.475 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:25:36.337681+08:00 2024-09-11 17:25:37.151 INFO 2981023 --- [ocp-async-10,2fb020fa6f9cd7a0,f3c19577b80315c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:37.314 INFO 2981023 --- [prometheus-cache-monitor1,c62d13427f9ef97a,bf11c5e9de51d60f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2050,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:25:37.315 INFO 2981023 --- [prometheus-cache-monitor1,c62d13427f9ef97a,bf11c5e9de51d60f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:25:37.738 INFO 2981023 --- [ocp-async-3,8dea8bc231b12e5f,e6b22d0559bb84b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:25:37.817 INFO 2981023 --- [ocp-schedules-4,e3b4da568680fe64,64289b1d5484a9a3] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:25:37.835 WARN 2981023 --- [ocp-schedules-4,e3b4da568680fe64,64289b1d5484a9a3] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:25:38.014 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:38.015 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,749f995da277e61c,8b10c05a4fea164a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,749f995da277e61c,8b10c05a4fea164a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:39.151 INFO 2981023 --- [ocp-async-8,65d93019636b1e34,5375747ab742fad4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:40.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:25:40.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:25:41.015 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:41.015 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:41.151 INFO 2981023 --- [ocp-async-2,f903d1b0dc8ffc05,ae137274325de48a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,550a64450da861a9,8ead2f7560a5a89b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,550a64450da861a9,8ead2f7560a5a89b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:41.692 INFO 2981023 --- [ocp-schedules-10,db28311931c08e05,9b2573110e7c2945] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:25:41.764 INFO 2981023 --- [ocp-schedules-7,73d99a38b6d98499,8332789651a119bb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:25:43.152 INFO 2981023 --- [ocp-async-7,3072d275a72f7062,7d90c79cec6c5b9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:44.015 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:44.016 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,b81787f184156408,8584705628484c5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,b81787f184156408,8584705628484c5d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:44.654 INFO 2981023 --- [ocp-schedules-23,81997a964262d988,f1b2a9ba980deba3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:25:44.708 WARN 2981023 --- [scanner-13480497-1,81997a964262d988,246238285e088556] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:25:44.724 INFO 2981023 --- [alarm-detect-17,81997a964262d988,39b07f60e0cb2e64] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643751, data is empty. 2024-09-11 17:25:44.728 INFO 2981023 --- [alarm-detect-17,81997a964262d988,39b07f60e0cb2e64] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643751, data is empty. 2024-09-11 17:25:44.732 INFO 2981023 --- [alarm-detect-17,81997a964262d988,39b07f60e0cb2e64] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643751, data is empty. 2024-09-11 17:25:44.754 INFO 2981023 --- [alarm-detect-11,81997a964262d988,b2bfd3eec32f0b78] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643751, data is empty. 2024-09-11 17:25:44.755 INFO 2981023 --- [alarm-detect-9,81997a964262d988,04926fa323f26675] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643751, data is empty. 2024-09-11 17:25:44.759 INFO 2981023 --- [alarm-detect-11,81997a964262d988,b2bfd3eec32f0b78] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643751, data is empty. 2024-09-11 17:25:44.760 INFO 2981023 --- [alarm-detect-9,81997a964262d988,04926fa323f26675] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643751, data is empty. 2024-09-11 17:25:44.764 INFO 2981023 --- [alarm-detect-11,81997a964262d988,b2bfd3eec32f0b78] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643751, data is empty. 2024-09-11 17:25:44.769 INFO 2981023 --- [alarm-detect-9,81997a964262d988,04926fa323f26675] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643751, data is empty. 2024-09-11 17:25:44.859 INFO 2981023 --- [alarm-detect-11,81997a964262d988,218f51dbc96f22f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643751, data is empty. 2024-09-11 17:25:44.865 INFO 2981023 --- [alarm-detect-11,81997a964262d988,218f51dbc96f22f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643751, data is empty. 2024-09-11 17:25:44.868 INFO 2981023 --- [alarm-detect-11,81997a964262d988,218f51dbc96f22f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643751, data is empty. 2024-09-11 17:25:45.101 INFO 2981023 --- [prometheus-second-schedule-1,e68e52564f5956e6,4a4c84ff0e0b6bf5] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:25:45.152 INFO 2981023 --- [ocp-async-5,2080a439ba72a5f6,32f9094090b1417e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:46.018 INFO 2981023 --- [ocp-schedules-17,86d041d8c5948465,99f608eb5bb39361] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:25:46.019 INFO 2981023 --- [host-check-clockdiff1,86d041d8c5948465,6aec3653812918d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:25:46.019 INFO 2981023 --- [host-check-clockdiff2,86d041d8c5948465,6b875d2c6ee61048] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:25:46.019 INFO 2981023 --- [host-check-clockdiff3,86d041d8c5948465,d6e53f1742c2d033] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:25:46.020 INFO 2981023 --- [host-check-clockdiff4,86d041d8c5948465,042c02a3b9733bdd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:25:46.030 INFO 2981023 --- [ocp-schedules-24,e065029eb0cedb4a,a4feefb6e9f8cabf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:25:46.031 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:25:46.031 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:25:46.031 INFO 2981023 --- [ocp-schedules-21,d733fcd0858022a9,af7a23a3a63cb8e2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:25:46.032 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:25:46.032 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:25:46.033 INFO 2981023 --- [host-check-clockdiff2,86d041d8c5948465,6b875d2c6ee61048] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046746 0 0 , error= 2024-09-11 17:25:46.034 INFO 2981023 --- [host-check-agent-status1,d733fcd0858022a9,34cb91357ddc4630] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:25:46.034 INFO 2981023 --- [host-check-agent-status2,d733fcd0858022a9,fef1ce3d15a73af1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:25:46.035 INFO 2981023 --- [host-check-agent-status3,d733fcd0858022a9,57aa5777c2f4869a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:25:46.035 INFO 2981023 --- [host-check-agent-status4,d733fcd0858022a9,baa8158390d3c4a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:25:46.037 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:25:46.039 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:25:46.039 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:25:46.039 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:25:46.040 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:25:46.040 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:25:46.041 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:25:46.042 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:25:46.043 INFO 2981023 --- [host-check-agent-status1,d733fcd0858022a9,34cb91357ddc4630] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:25:46.044 INFO 2981023 --- [host-check-agent-status3,d733fcd0858022a9,57aa5777c2f4869a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:25:46.044 INFO 2981023 --- [host-check-agent-status4,d733fcd0858022a9,baa8158390d3c4a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:25:46.045 INFO 2981023 --- [host-check-agent-status1,d733fcd0858022a9,34cb91357ddc4630] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:25:46.045 INFO 2981023 --- [host-check-agent-status2,d733fcd0858022a9,fef1ce3d15a73af1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:25:46.046 INFO 2981023 --- [host-check-agent-status4,d733fcd0858022a9,baa8158390d3c4a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:25:46.048 INFO 2981023 --- [host-check-agent-status3,d733fcd0858022a9,57aa5777c2f4869a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:25:46.048 INFO 2981023 --- [host-check-agent-status2,d733fcd0858022a9,fef1ce3d15a73af1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:25:46.068 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:25:46.071 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:25:46.071958838+08:00 2024-09-11 17:25:46.072 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:25:46.072 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:25:46.072+08:00 2024-09-11 17:25:46.073 INFO 2981023 --- [host-check-agent-status1,d733fcd0858022a9,34cb91357ddc4630] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:25:46.073 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:25:46.074 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:25:46.074288087+08:00 2024-09-11 17:25:46.074 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:25:46.072+08:00 and status AVAILABLE 2024-09-11 17:25:46.074 INFO 2981023 --- [host-check-agent-status3,d733fcd0858022a9,57aa5777c2f4869a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:25:46.075 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:25:46.074+08:00 2024-09-11 17:25:46.075 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:25:46.075091949+08:00 2024-09-11 17:25:46.076 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:25:46.075+08:00 2024-09-11 17:25:46.076 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:25:46.074+08:00 and status AVAILABLE 2024-09-11 17:25:46.078 INFO 2981023 --- [host-check-agent-status1,d733fcd0858022a9,34cb91357ddc4630] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:25:46.078 INFO 2981023 --- [host-check-agent-status1,d733fcd0858022a9,34cb91357ddc4630] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:25:46.078 INFO 2981023 --- [host-check-agent-status1,d733fcd0858022a9,34cb91357ddc4630] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:25:46.078 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:25:46.075+08:00 and status AVAILABLE 2024-09-11 17:25:46.079 INFO 2981023 --- [host-check-agent-status3,d733fcd0858022a9,57aa5777c2f4869a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:25:46.079 INFO 2981023 --- [host-check-agent-status3,d733fcd0858022a9,57aa5777c2f4869a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:25:46.079 INFO 2981023 --- [host-check-agent-status3,d733fcd0858022a9,57aa5777c2f4869a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:25:46.080 INFO 2981023 --- [host-check-agent-status4,d733fcd0858022a9,baa8158390d3c4a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:25:46.080 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=http, hostAddress=10.31.35.10, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null) 2024-09-11 17:25:46.096 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=http, hostAddress=10.31.35.10, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null) 2024-09-11 17:25:46.105 INFO 2981023 --- [host-check-agent-status4,d733fcd0858022a9,baa8158390d3c4a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:25:46.105 INFO 2981023 --- [host-check-agent-status4,d733fcd0858022a9,baa8158390d3c4a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:25:46.105 INFO 2981023 --- [host-check-agent-status4,d733fcd0858022a9,baa8158390d3c4a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:25:46.108 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:25:46.115 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:25:46.118 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:25:46.121 INFO 2981023 --- [host-check-agent-status2,d733fcd0858022a9,fef1ce3d15a73af1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:25:46.121 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:25:46.121 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:25:46.120971440+08:00 2024-09-11 17:25:46.122 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:25:46.122 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:25:46.121+08:00 2024-09-11 17:25:46.126 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:25:46.121+08:00 and status AVAILABLE 2024-09-11 17:25:46.129 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:25:46.136 INFO 2981023 --- [host-check-agent-status2,d733fcd0858022a9,fef1ce3d15a73af1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:25:46.136 INFO 2981023 --- [host-check-agent-status2,d733fcd0858022a9,fef1ce3d15a73af1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:25:46.136 INFO 2981023 --- [host-check-agent-status2,d733fcd0858022a9,fef1ce3d15a73af1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:25:46.146 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:25:46.151 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:25:46.155 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:25:46.159 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:25:46.159 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:25:46.173 INFO 2981023 --- [ocp-schedules-11,ffb4263ab8282766,e689d3fb4e264727] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:25:46.175 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:25:46.180 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:25:46.181 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:25:46.184 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:25:46.187 INFO 2981023 --- [ocp-schedules-16,3cf72df93804e73f,af5e74e5124e0fd2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:25:46.187 INFO 2981023 --- [ocp-schedules-13,f67be37cef416ac4,0dc96f1f775c797a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:25:46.188 INFO 2981023 --- [ocp-schedules-10,dd65ba03dbfdd9bf,510b0de7c96aae19] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:25:46.188 INFO 2981023 --- [ocp-schedules-30,fa2a6ebbe192ad4b,b6e1389b0348768d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:25:46.189 INFO 2981023 --- [ocp-schedules-13,f67be37cef416ac4,0dc96f1f775c797a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:25:46.196 INFO 2981023 --- [ocp-schedules-10,dd65ba03dbfdd9bf,510b0de7c96aae19] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:25:46.197 INFO 2981023 --- [ocp-schedules-16,3cf72df93804e73f,af5e74e5124e0fd2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:25:46.214 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:25:46.214 INFO 2981023 --- [ocp-schedules-11,ffb4263ab8282766,e689d3fb4e264727] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:25:46.218 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:25:46.223 INFO 2981023 --- [ocp-schedules-13,f67be37cef416ac4,0dc96f1f775c797a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:25:46.237 INFO 2981023 --- [ocp-schedules-22,af0da8debe4d93ad,17e81878ec678141] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:25:46.240 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:25:46.245 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:25:46.248 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:25:46.263 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.264 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:46.264 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:25:46.267 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.268 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:46.269 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:25:46.270 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:25:46.281 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.282 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:46.284 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.285 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:25:46.285 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:25:46.285 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:25:46.285 INFO 2981023 --- [ocp-schedules-16,3cf72df93804e73f,af5e74e5124e0fd2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:25:46.285 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:25:46.286 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.297 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:25:46.299 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.302 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.304 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:25:46.305 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:25:46.305 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:46.306 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:25:46.307 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.308 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:46.308 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:46.317 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:25:46.319 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:25:46.331 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:25:46.332 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:25:46.347 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:25:46.345172+08:00, lastProcessTime=2024-09-11T17:25:36.337681+08:00 2024-09-11 17:25:46.349 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:25:46.357 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:25:46.360 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.360 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.360 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:25:46.367 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:25:46.376 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:25:46.377 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:25:46.382 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.383 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:25:46.384 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.384 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:25:46.385 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:25:46.385 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.396 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:25:46.399 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:25:46.402 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d88c9d8114352293,576892ca77319d82] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=d88c9d8114352293, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:25:46.405 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d88c9d8114352293,576892ca77319d82] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=d88c9d8114352293, duration=8 ms] 2024-09-11 17:25:46.409 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:25:46.411 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6b879f571ae1f09c,43f82d28bbff529b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=6b879f571ae1f09c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:25:46.412 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:25:46.414 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6b879f571ae1f09c,43f82d28bbff529b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=6b879f571ae1f09c, duration=7 ms] 2024-09-11 17:25:46.416 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:25:46.417 INFO 2981023 --- [host-check-host-status3,e065029eb0cedb4a,dcc62c3c58726d88] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:25:46.420 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:25:46.420 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e25dc9c5ef87b841,8f7d9203ca431322] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=e25dc9c5ef87b841, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:25:46.421 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e25dc9c5ef87b841,8f7d9203ca431322] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=e25dc9c5ef87b841, duration=24 ms] 2024-09-11 17:25:46.424 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:25:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.431 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:25:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:46.433 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:25:46.433 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.435 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:25:46.436 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:25:46.447 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:25:46.455 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.457 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.457 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:25:46.458 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:25:46.465 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:46.465 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:25:46.469 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.472 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:25:46.479 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.479 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.480 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:25:46.480 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:25:46.481 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.481 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:25:46.482 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:25:46.482 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.484 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:25:46.486 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:25:46.489 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:46.490 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:25:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.493 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.493 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:46.494 INFO 2981023 --- [host-check-host-status1,e065029eb0cedb4a,51d0457564e1338a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:25:46.498 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.500 INFO 2981023 --- [host-check-host-status4,e065029eb0cedb4a,d4e2b24d0ebbb6a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:25:46.502 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.502 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:25:46.502 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.503 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:46.503 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:25:46.505 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:25:46.506 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:25:46.508 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:25:46.514 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:25:46.516 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:25:46.516 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:25:46.519 INFO 2981023 --- [host-check-host-status2,e065029eb0cedb4a,1c5e9d1dcb118466] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:25:46.520 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:46.520 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:25:46.522 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.522 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.523 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:25:46.535 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:46.538 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:25:46.541 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.542 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.542 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:25:46.543 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:25:46.543 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.554 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.555 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:25:46.555 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.557 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:46.558 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:25:46.560 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.561 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.561 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:25:46.562 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:25:46.562 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.563 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:25:46.565 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:25:46.565 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:25:46.569 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:25:46.570 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:25:46.574 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:25:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:25:46.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:25:46.592 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:25:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.597 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.597 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:25:46.597 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.597 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.598 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:25:46.598 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:25:46.598 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.599 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:25:46.602 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:25:46.608 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:25:46.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:25:46.641 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:25:46.650 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:25:46.652 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:25:46.659 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.659 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.660 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:25:46.661 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.661 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.662 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:25:46.665 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:25:46.666 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:25:46.345172+08:00 2024-09-11 17:25:46.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:25:46.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:25:46.672 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:25:46.672 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:25:46.674 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:25:46.680 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:46.680 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:46.681 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:25:46.684 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:25:46.689 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:25:46.689 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:25:46.690 INFO 2981023 --- [ocp-schedules-10,dd65ba03dbfdd9bf,510b0de7c96aae19] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:25:46.691 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:25:46.693 INFO 2981023 --- [ocp-schedules-27,cc9925942ceeef33,ed4015a8a59a2ecc] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:25:46.835 INFO 2981023 --- [ocp-schedules-3,97662948f6e0395b,511d2a57a40a9bb2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2050,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:25:46.835 INFO 2981023 --- [ocp-schedules-3,97662948f6e0395b,511d2a57a40a9bb2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:25:46.971 INFO 2981023 --- [host-check-clockdiff4,86d041d8c5948465,042c02a3b9733bdd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046746 0 0 , error= 2024-09-11 17:25:47.016 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:47.016 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:47.152 INFO 2981023 --- [ocp-async-8,cd06a7f0db53e3ab,9adbd6d3b784a545] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,794861f747417ae9,d90709b623e1df02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,794861f747417ae9,d90709b623e1df02] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:47.739 INFO 2981023 --- [ocp-async-6,c316e41bf20814b9,9b1b121994ee3ccf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:25:49.152 INFO 2981023 --- [ocp-async-7,c6031fa83f32a000,0ac6f04005438f0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:49.193 INFO 2981023 --- [host-check-clockdiff1,86d041d8c5948465,6aec3653812918d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046749 0 0 , error= 2024-09-11 17:25:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:25:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:25:50.017 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:50.018 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:50.200 INFO 2981023 --- [ocp-schedules-10,6276146053fdc32b,c0114fb53b85c8e4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:25:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,260ca3c7749ef489,408e0ae1f16ce44a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,260ca3c7749ef489,408e0ae1f16ce44a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:50.980 INFO 2981023 --- [ocp-schedules-19,ea5045784e765dc4,bf3cb3e77069d0c8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:25:50.980+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:25:50.985 INFO 2981023 --- [ocp-schedules-19,ea5045784e765dc4,bf3cb3e77069d0c8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:25:51.051 INFO 2981023 --- [ocp-schedules-19,ea5045784e765dc4,bf3cb3e77069d0c8] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 43 to SqlAggregateData 2024-09-11 17:25:51.052 INFO 2981023 --- [ocp-schedules-19,ea5045784e765dc4,bf3cb3e77069d0c8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:25:51.152 INFO 2981023 --- [ocp-async-5,c7e8669782a5c015,fa38a8468a54162c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:51.479 INFO 2981023 --- [ocp-schedules-4,988ed7aafcbab3db,18e5df0a70842676] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:25:51.480 INFO 2981023 --- [ocp-schedules-4,988ed7aafcbab3db,18e5df0a70842676] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:25:51.506 INFO 2981023 --- [ocp-schedules-7,2b2b0519f82ea196,e29e31358bedd92a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:25:51.507 INFO 2981023 --- [ocp-schedules-7,2b2b0519f82ea196,e29e31358bedd92a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:25:51.507 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:25:51.508 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:25:51.509 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:25:51.509 INFO 2981023 --- [ocp-schedules-7,2b2b0519f82ea196,e29e31358bedd92a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:25:51.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:25:51.514 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:25:51.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.518 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:25:51.518 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:25:51.519 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.519 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:25:51.521 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:25:51.522 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.522 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.523 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.523 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.525 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.526 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.533 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.533 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.535 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.535 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:25:51.539 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.539 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.542 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.543 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:25:51.544 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.544 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.545 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:25:51.545 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1D71D755-0-0] 2024-09-11 17:25:51.545 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:25:51.545 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.546 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:25:51.548 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.548 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:25:51.551 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.551 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.555 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.556 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:25:51.558 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.559 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:25:51.561 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.562 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:25:51.562 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.563 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:25:51.564 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.564 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:25:51.565 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:25:51.569 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:25:51.570 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.571 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:25:51.571 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.572 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:25:51.572 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.573 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:25:51.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:25:51.574 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.574 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:25:51.574 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:25:51.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1D71D75F-0-0] 2024-09-11 17:25:51.575 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:25:51.575 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.576 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:25:51.577 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.577 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:25:51.578 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.578 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:25:51.579 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.579 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:25:51.580 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.581 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:25:51.581 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:25:51.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:25:51.584 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.584 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:51.584 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:25:51.585 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.585 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:25:51.586 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.586 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.586 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.587 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:25:51.587 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.588 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:25:51.588 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.589 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:25:51.590 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.590 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:25:51.590 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:25:51.593 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:25:51.596 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:51.598 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.598 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:25:51.601 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:25:51.605 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:25:51.605 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:25:51.605 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:25:51.606 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:25:51.606 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:25:51.608 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.608 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:25:51.608 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:25:51.608 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.609 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:25:51.615 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:25:51.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:25:51.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.621 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:25:51.622 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:25:51.622 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:51.623 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:25:51.623 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:25:51.624 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:25:51.624 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:25:51.624 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:25:51.633 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:25:51.635 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:25:51.635 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:25:51.635 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:25:51.635 INFO 2981023 --- [ocp-schedules-4,988ed7aafcbab3db,18e5df0a70842676] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:25:51.767 INFO 2981023 --- [ocp-schedules-16,93080d1bbbc79495,62eb085f0e0c74df] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:25:51.993 INFO 2981023 --- [host-check-clockdiff3,86d041d8c5948465,d6e53f1742c2d033] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046751 1 1 , error= 2024-09-11 17:25:52.195 INFO 2981023 --- [ocp-schedules-9,4838d233d98e1c7d,75e9a1b1cbf52601] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:25:52.212 WARN 2981023 --- [ocp-schedules-9,4838d233d98e1c7d,75e9a1b1cbf52601] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:25:53.018 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:53.020 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:53.024 INFO 2981023 --- [ocp-schedules-4,418ef4ea8f05f7c8,1698a834f73c8105] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:25:53.024 INFO 2981023 --- [ocp-schedules-4,418ef4ea8f05f7c8,1698a834f73c8105] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:25:53.152 INFO 2981023 --- [ocp-async-2,69448dd2ac220e91,fed8b02882a4b7f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,9d848a8337bdcea0,6a57e01dcec24a6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:53.318 INFO 2981023 --- [prometheus-minute-schedule-1,9d848a8337bdcea0,6a57e01dcec24a6a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:53.533 INFO 2981023 --- [ocp-schedules-22,2b7379ec08aa1637,cd1b40e9d721b7ae] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:25:53.542 INFO 2981023 --- [ocp-schedules-22,2b7379ec08aa1637,cd1b40e9d721b7ae] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:25:53.542 INFO 2981023 --- [ocp-schedules-22,2b7379ec08aa1637,cd1b40e9d721b7ae] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:25:54.168 INFO 2981023 --- [ocp-schedules-26,21805d33242c29a9,ac64d88e440cbb6f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:25:54.907 INFO 2981023 --- [ocp-schedules-24,e4021037e5ff6624,a6edf99aa6df2c72] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:25:54.964 WARN 2981023 --- [scanner-13480836-1,e4021037e5ff6624,64f69f7fbaef10f9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:25:55.152 INFO 2981023 --- [ocp-async-10,4d8ad90dea6816ff,a538236994eaef6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:55.627 INFO 2981023 --- [ocp-schedules-23,148323a3582ea6d6,0fcebfa3151683ac] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:25:55.627 INFO 2981023 --- [ocp-schedules-23,148323a3582ea6d6,0fcebfa3151683ac] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:25:55.628 INFO 2981023 --- [ocp-schedules-23,148323a3582ea6d6,0fcebfa3151683ac] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:25:55.628 INFO 2981023 --- [ocp-schedules-23,148323a3582ea6d6,0fcebfa3151683ac] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:25:55.629 INFO 2981023 --- [ocp-schedules-23,148323a3582ea6d6,0fcebfa3151683ac] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:25:55.629 INFO 2981023 --- [ocp-schedules-23,148323a3582ea6d6,0fcebfa3151683ac] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:25:55.686 INFO 2981023 --- [ocp-schedules-26,3d04e56e82cb82f6,f1e3058ef19c57ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:25:55.693 INFO 2981023 --- [ocp-schedules-26,3d04e56e82cb82f6,f1e3058ef19c57ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:25:55.693 INFO 2981023 --- [ocp-schedules-26,3d04e56e82cb82f6,f1e3058ef19c57ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:25:55.730 INFO 2981023 --- [ocp-schedules-25,dabc3645eb6d631b,f90f60f04ac1f7fa] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:25:55.730 INFO 2981023 --- [ocp-schedules-25,dabc3645eb6d631b,f90f60f04ac1f7fa] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:25:55.751 INFO 2981023 --- [Collect all tenants major compactions2,dabc3645eb6d631b,1bd58be1dc15b3c4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:25:55.751 INFO 2981023 --- [Collect all tenants major compactions1,dabc3645eb6d631b,e326d90e9fe27d5e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:25:55.752 INFO 2981023 --- [Collect all tenants major compactions3,dabc3645eb6d631b,7ebf0727dd0580cf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:25:55.753 INFO 2981023 --- [Collect all tenants major compactions5,dabc3645eb6d631b,26b77c2132418048] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:25:55.753 INFO 2981023 --- [Collect all tenants major compactions4,dabc3645eb6d631b,5cd54bc3dca89331] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:25:55.756 INFO 2981023 --- [Collect all tenants major compactions6,dabc3645eb6d631b,5989aa94a76b37d1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:25:55.769 INFO 2981023 --- [Collect all tenants major compactions3,dabc3645eb6d631b,7ebf0727dd0580cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:55.769 INFO 2981023 --- [Collect all tenants major compactions2,dabc3645eb6d631b,1bd58be1dc15b3c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:55.769 INFO 2981023 --- [Collect all tenants major compactions4,dabc3645eb6d631b,5cd54bc3dca89331] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:55.769 INFO 2981023 --- [Collect all tenants major compactions5,dabc3645eb6d631b,26b77c2132418048] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:55.769 INFO 2981023 --- [Collect all tenants major compactions1,dabc3645eb6d631b,e326d90e9fe27d5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:55.770 INFO 2981023 --- [Collect all tenants major compactions2,dabc3645eb6d631b,1bd58be1dc15b3c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:25:55.770 INFO 2981023 --- [Collect all tenants major compactions4,dabc3645eb6d631b,5cd54bc3dca89331] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:25:55.770 INFO 2981023 --- [Collect all tenants major compactions3,dabc3645eb6d631b,7ebf0727dd0580cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:25:55.771 INFO 2981023 --- [Collect all tenants major compactions1,dabc3645eb6d631b,e326d90e9fe27d5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:25:55.771 INFO 2981023 --- [Collect all tenants major compactions5,dabc3645eb6d631b,26b77c2132418048] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:25:55.772 INFO 2981023 --- [Collect all tenants major compactions6,dabc3645eb6d631b,5989aa94a76b37d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:25:55.773 INFO 2981023 --- [Collect all tenants major compactions6,dabc3645eb6d631b,5989aa94a76b37d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:25:55.773 INFO 2981023 --- [Collect all tenants major compactions4,dabc3645eb6d631b,5cd54bc3dca89331] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:25:55.774 INFO 2981023 --- [Collect all tenants major compactions4,dabc3645eb6d631b,5cd54bc3dca89331] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:25:55.774 INFO 2981023 --- [Collect all tenants major compactions2,dabc3645eb6d631b,1bd58be1dc15b3c4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:25:55.774 INFO 2981023 --- [Collect all tenants major compactions2,dabc3645eb6d631b,1bd58be1dc15b3c4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:25:55.775 INFO 2981023 --- [Collect all tenants major compactions6,dabc3645eb6d631b,5989aa94a76b37d1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:25:55.775 INFO 2981023 --- [Collect all tenants major compactions6,dabc3645eb6d631b,5989aa94a76b37d1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:25:55.776 INFO 2981023 --- [Collect all tenants major compactions5,dabc3645eb6d631b,26b77c2132418048] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:25:55.776 INFO 2981023 --- [Collect all tenants major compactions5,dabc3645eb6d631b,26b77c2132418048] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:25:55.776 INFO 2981023 --- [Collect all tenants major compactions1,dabc3645eb6d631b,e326d90e9fe27d5e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:25:55.776 INFO 2981023 --- [Collect all tenants major compactions1,dabc3645eb6d631b,e326d90e9fe27d5e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:25:55.777 INFO 2981023 --- [Collect all tenants major compactions3,dabc3645eb6d631b,7ebf0727dd0580cf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:25:55.777 INFO 2981023 --- [Collect all tenants major compactions3,dabc3645eb6d631b,7ebf0727dd0580cf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:25:55.778 INFO 2981023 --- [ocp-schedules-25,dabc3645eb6d631b,f90f60f04ac1f7fa] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:25:56.020 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:56.021 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,5f5b1fd9c20c6af2,c372ab455ecdbda3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,5f5b1fd9c20c6af2,c372ab455ecdbda3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:25:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:25:56.352499+08:00, lastProcessTime=2024-09-11T17:25:46.345172+08:00 2024-09-11 17:25:56.488 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:25:56.352499+08:00 2024-09-11 17:25:56.830 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,cfdecdabb8fe112c,d5aeb93138bc1371] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=cfdecdabb8fe112c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:25:56.831 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ea001c76e0f4f3e0,c848d1a8ba8dbd4f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=ea001c76e0f4f3e0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:25:56.832 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ea001c76e0f4f3e0,c848d1a8ba8dbd4f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=ea001c76e0f4f3e0, duration=5 ms] 2024-09-11 17:25:56.833 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,cfdecdabb8fe112c,d5aeb93138bc1371] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=cfdecdabb8fe112c, duration=8 ms] 2024-09-11 17:25:56.839 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,9bdc9c60cc8f50ee,b66affaa43c028d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=9bdc9c60cc8f50ee, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:25:56.839 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,9bdc9c60cc8f50ee,b66affaa43c028d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=9bdc9c60cc8f50ee, duration=2 ms] 2024-09-11 17:25:57.152 INFO 2981023 --- [ocp-async-1,9c3454a1329ad1bd,6942adadb94332bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:57.739 INFO 2981023 --- [ocp-async-9,0a1882e372e35371,2650458c61a003b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:25:59.021 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:25:59.021 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:25:59.152 INFO 2981023 --- [ocp-async-2,c60c0c984f39704c,198c453a9d660c42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:25:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,3d66100781272601,a1f8752f76d9b57c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046760, interval=60 2024-09-11 17:25:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,3d66100781272601,a1f8752f76d9b57c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:01.153 INFO 2981023 --- [ocp-async-7,fde05da1883cc998,68ecf5af7c5ea9cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:01.770 INFO 2981023 --- [ocp-schedules-11,4ffd326906940885,4fdc6b1cf96596bb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:26:01.973 INFO 2981023 --- [ocp-schedules-9,1f2817ef44a7c6a9,b4a12125301a3834] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:26:01.973 INFO 2981023 --- [ocp-schedules-9,1f2817ef44a7c6a9,b4a12125301a3834] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:26:02.021 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:02.022 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,51dbfc95278761bd,e56f500c4d58e712] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,51dbfc95278761bd,e56f500c4d58e712] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:02.358 INFO 2981023 --- [ocp-schedules-16,0526c811ecb656ed,bec06060629ab6ad] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:26:02.359 INFO 2981023 --- [ocp-schedules-16,0526c811ecb656ed,bec06060629ab6ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:02.371 INFO 2981023 --- [ocp-schedules-16,0526c811ecb656ed,bec06060629ab6ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:26:03.153 INFO 2981023 --- [ocp-async-1,5999e4a1ffc984d2,2b9824858e803fa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:05.022 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:05.024 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:05.114 INFO 2981023 --- [ocp-schedules-18,b8be4e5a7f272d8d,e93a52998413ba28] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:26:05.153 INFO 2981023 --- [ocp-async-3,ad1b86a4176f6a02,2b2d061aaa3f53c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:05.178 WARN 2981023 --- [scanner-13481064-1,b8be4e5a7f272d8d,f30b6dcac5057919] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:26:05.192 INFO 2981023 --- [alarm-detect-10,b8be4e5a7f272d8d,090f9fb37f29707c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643751, data is empty. 2024-09-11 17:26:05.199 INFO 2981023 --- [alarm-detect-10,b8be4e5a7f272d8d,090f9fb37f29707c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643794, data is empty. 2024-09-11 17:26:05.205 INFO 2981023 --- [alarm-detect-10,b8be4e5a7f272d8d,090f9fb37f29707c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643794, data is empty. 2024-09-11 17:26:05.214 INFO 2981023 --- [alarm-detect-10,b8be4e5a7f272d8d,320757083b700f7e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643751, data is empty. 2024-09-11 17:26:05.221 INFO 2981023 --- [alarm-detect-10,b8be4e5a7f272d8d,320757083b700f7e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643794, data is empty. 2024-09-11 17:26:05.223 INFO 2981023 --- [alarm-detect-4,b8be4e5a7f272d8d,9ce667209f264428] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643751, data is empty. 2024-09-11 17:26:05.226 INFO 2981023 --- [alarm-detect-10,b8be4e5a7f272d8d,320757083b700f7e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643794, data is empty. 2024-09-11 17:26:05.230 INFO 2981023 --- [alarm-detect-4,b8be4e5a7f272d8d,9ce667209f264428] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643794, data is empty. 2024-09-11 17:26:05.236 INFO 2981023 --- [alarm-detect-4,b8be4e5a7f272d8d,9ce667209f264428] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643794, data is empty. 2024-09-11 17:26:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,42137a7ca1f587ab,2117880e2cb06cd3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:05.319 INFO 2981023 --- [prometheus-minute-schedule-1,42137a7ca1f587ab,2117880e2cb06cd3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:05.351 INFO 2981023 --- [alarm-detect-3,b8be4e5a7f272d8d,d06dcda08e8a6dfa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643751, data is empty. 2024-09-11 17:26:05.357 INFO 2981023 --- [alarm-detect-3,b8be4e5a7f272d8d,d06dcda08e8a6dfa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643794, data is empty. 2024-09-11 17:26:05.361 INFO 2981023 --- [alarm-detect-3,b8be4e5a7f272d8d,d06dcda08e8a6dfa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643794, data is empty. 2024-09-11 17:26:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:26:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:26:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:26:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:26:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:26:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:26:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:26:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:26:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:26:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:06.010 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:06.010 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:26:06.010 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:26:06.011 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:06.011 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:06.011 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:26:06.011 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:26:06.054 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:23:40, start=2024-09-11T17:20:40, end=2024-09-11T17:23:10, updates=281, granule=120000ms, delay=180000ms 2024-09-11 17:26:06.059 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:23:40, target=2024-09-11T17:23:10, granule=120000ms, delay=30000ms) 2024-09-11 17:26:06.059 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:23:10, target=2024-09-11T17:09:41, granule=600000ms, delay=809000ms) 2024-09-11 17:26:06.134 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:23:41, start=2024-09-11T17:20:11, end=2024-09-11T17:22:41, updates=309, granule=120000ms, delay=210000ms 2024-09-11 17:26:06.139 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:23:41, target=2024-09-11T17:22:41, granule=120000ms, delay=60000ms) 2024-09-11 17:26:06.139 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:22:41, target=2024-09-11T17:10:51, granule=600000ms, delay=710000ms) 2024-09-11 17:26:06.139 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=169ms, period=120000ms 2024-09-11 17:26:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:26:06.352500+08:00, lastProcessTime=2024-09-11T17:25:56.352499+08:00 2024-09-11 17:26:06.421 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c9d873d666c9df90,468fa87eeed0dbcb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=c9d873d666c9df90, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:26:06.421 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,227600d691cf8aca,4aee132ad29c7a2f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=227600d691cf8aca, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:26:06.422 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,227600d691cf8aca,4aee132ad29c7a2f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=227600d691cf8aca, duration=4 ms] 2024-09-11 17:26:06.427 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c9d873d666c9df90,468fa87eeed0dbcb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=c9d873d666c9df90, duration=11 ms] 2024-09-11 17:26:06.430 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e072094d392ccae8,8629298ee0686ff4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=e072094d392ccae8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:26:06.431 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e072094d392ccae8,8629298ee0686ff4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=e072094d392ccae8, duration=3 ms] 2024-09-11 17:26:06.509 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:26:06.352500+08:00 2024-09-11 17:26:07.153 INFO 2981023 --- [ocp-async-4,dbd642b918cd501c,cce5844b4e855075] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:07.188 INFO 2981023 --- [ocp-schedules-6,655a9aea77729622,a7db7f3460f66592] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:26:07.189 INFO 2981023 --- [ocp-schedules-25,0f764d429c50e358,5d881d876e68254d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:26:07.189+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:26:07.190 INFO 2981023 --- [ocp-schedules-25,0f764d429c50e358,5d881d876e68254d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:26:07.191 INFO 2981023 --- [ocp-schedules-25,0f764d429c50e358,5d881d876e68254d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:26:07.192 INFO 2981023 --- [ocp-schedules-25,0f764d429c50e358,5d881d876e68254d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:26:07.739 INFO 2981023 --- [ocp-async-2,c22f754b77456729,17831e1a536a2ff5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:26:08.024 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:08.026 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,ada23e98a41e2015,d7e2de25c9051cd2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,ada23e98a41e2015,d7e2de25c9051cd2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:09.153 INFO 2981023 --- [ocp-async-1,a735be2897c814a2,7e258b927e51d093] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:10.004 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:26:10.004 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:26:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:26:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:26:11.026 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:11.026 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:11.153 INFO 2981023 --- [ocp-async-8,29ddd91dcbe28a8c,a6e423e60031946d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,4b5249e91546d726,94ec0cc8d07ba4ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,4b5249e91546d726,94ec0cc8d07ba4ac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:11.772 INFO 2981023 --- [ocp-schedules-3,b42aee7df83c1044,7242340a8fc4309c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:26:13.153 INFO 2981023 --- [ocp-async-10,d346371dc1067fc9,16d9457ecf1b1748] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:13.197 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046700, nodeCount=1000, affectRows=1000 2024-09-11 17:26:13.200 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046700, nodeCount=999, validCount=999 2024-09-11 17:26:13.262 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046700, nodeCount=1000, affectRows=1000 2024-09-11 17:26:13.265 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046700, nodeCount=1000, validCount=1000 2024-09-11 17:26:13.276 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046700, nodeCount=50, affectRows=50 2024-09-11 17:26:13.276 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046700, nodeCount=50, validCount=50 2024-09-11 17:26:13.544 INFO 2981023 --- [ocp-schedules-16,9ae7c5870fcffc6d,08a3cf131edee427] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:26:13.553 INFO 2981023 --- [ocp-schedules-16,9ae7c5870fcffc6d,08a3cf131edee427] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:26:13.553 INFO 2981023 --- [ocp-schedules-16,9ae7c5870fcffc6d,08a3cf131edee427] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:26:14.026 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:14.027 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,0944e91e9a383683,c593acb8111a9801] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,c52aeed6cffda309,b65a9f6cf55fc46c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:26:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,0944e91e9a383683,c593acb8111a9801] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:15.153 INFO 2981023 --- [ocp-async-5,1bfd5c582da3e335,473854b9b02e606c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:15.400 INFO 2981023 --- [ocp-schedules-1,2e833fdebba8bf8c,a50c61907d036ebf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:26:15.416 WARN 2981023 --- [scanner-13482829-1,2e833fdebba8bf8c,0b1550bd5862bdbb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:26:15.487 INFO 2981023 --- [alarm-detect-20,2e833fdebba8bf8c,ca7dd37a1c9d583d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046745 ~ 1726046765 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726046760, value=2.106)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.106, values={1726046760=2.106}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:26:15.488 INFO 2981023 --- [alarm-detect-20,2e833fdebba8bf8c,ca7dd37a1c9d583d] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Pending 2024-09-11 17:26:15.672 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046700, nodeCount=480, affectRows=480 2024-09-11 17:26:16.015 INFO 2981023 --- [ocp-schedules-24,aedaa9fe704a2ae4,855ba1921e5181c0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:26:16.017 INFO 2981023 --- [ocp-schedules-13,d73f63cf5b55b6ef,ec6228c1e6dbe7b0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:26:16.017 INFO 2981023 --- [host-check-clockdiff1,aedaa9fe704a2ae4,93b5f2ee97ee275e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:26:16.018 INFO 2981023 --- [host-check-clockdiff2,aedaa9fe704a2ae4,d9e2c798ac8e7586] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:26:16.018 INFO 2981023 --- [host-check-agent-status1,d73f63cf5b55b6ef,d291dfd4f0f64d1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:26:16.019 INFO 2981023 --- [host-check-agent-status2,d73f63cf5b55b6ef,d3ae9b3202cdf775] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:26:16.019 INFO 2981023 --- [ocp-schedules-17,bca27cf687ba574e,7ab33ff92ad8fc3a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:26:16.020 INFO 2981023 --- [host-check-agent-status3,d73f63cf5b55b6ef,0a7d89789b64e86a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:26:16.020 INFO 2981023 --- [host-check-agent-status4,d73f63cf5b55b6ef,4606b754c70e8810] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:26:16.022 INFO 2981023 --- [host-check-clockdiff3,aedaa9fe704a2ae4,7d784f28659cfbdb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:26:16.025 INFO 2981023 --- [host-check-clockdiff4,aedaa9fe704a2ae4,c5ef3954e3c01117] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:26:16.033 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:26:16.041 INFO 2981023 --- [host-check-agent-status2,d73f63cf5b55b6ef,d3ae9b3202cdf775] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:26:16.042 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:26:16.042 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:26:16.043 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:26:16.046 INFO 2981023 --- [host-check-agent-status2,d73f63cf5b55b6ef,d3ae9b3202cdf775] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:26:16.050 INFO 2981023 --- [host-check-agent-status4,d73f63cf5b55b6ef,4606b754c70e8810] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:26:16.051 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:26:16.051 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:26:16.051 INFO 2981023 --- [host-check-agent-status4,d73f63cf5b55b6ef,4606b754c70e8810] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:26:16.053 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:26:16.059 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:26:16.062 INFO 2981023 --- [host-check-agent-status1,d73f63cf5b55b6ef,d291dfd4f0f64d1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:26:16.063 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:26:16.063 INFO 2981023 --- [host-check-agent-status1,d73f63cf5b55b6ef,d291dfd4f0f64d1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:26:16.064 INFO 2981023 --- [host-check-clockdiff4,aedaa9fe704a2ae4,c5ef3954e3c01117] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046776 0 0 , error= 2024-09-11 17:26:16.064 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:26:16.065 INFO 2981023 --- [host-check-agent-status3,d73f63cf5b55b6ef,0a7d89789b64e86a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:26:16.072 INFO 2981023 --- [host-check-agent-status3,d73f63cf5b55b6ef,0a7d89789b64e86a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:26:16.093 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:26:16.104 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:26:16.187 INFO 2981023 --- [ocp-schedules-11,fc4da96bb65685d0,903a93187a34c887] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:26:16.187 INFO 2981023 --- [ocp-schedules-19,5d9e7efbbcc649a0,0bee7accd3399ee6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:26:16.191 INFO 2981023 --- [ocp-schedules-19,5d9e7efbbcc649a0,0bee7accd3399ee6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:16.192 INFO 2981023 --- [ocp-schedules-11,fc4da96bb65685d0,903a93187a34c887] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:16.192 INFO 2981023 --- [ocp-schedules-30,6e9a6a3ac7b931e6,ca0811b494c7bfdc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:26:16.192 INFO 2981023 --- [ocp-schedules-23,869f9ddf08f6c7cc,1505fe19a81eaf6f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:26:16.201 INFO 2981023 --- [ocp-schedules-30,6e9a6a3ac7b931e6,ca0811b494c7bfdc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:16.227 INFO 2981023 --- [ocp-schedules-21,84a81325d9c0705c,9ecab26094ac8d7f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:26:16.260 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:26:16.262 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:26:16.262434141+08:00 2024-09-11 17:26:16.263 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:26:16.262+08:00 2024-09-11 17:26:16.265 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:26:16.262+08:00 and status AVAILABLE 2024-09-11 17:26:16.268 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:26:16.273 INFO 2981023 --- [host-check-agent-status2,d73f63cf5b55b6ef,d3ae9b3202cdf775] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:26:16.273 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:26:16.273571540+08:00 2024-09-11 17:26:16.275 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:26:16.273+08:00 2024-09-11 17:26:16.277 INFO 2981023 --- [host-check-agent-status2,d73f63cf5b55b6ef,d3ae9b3202cdf775] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:26:16.277 INFO 2981023 --- [host-check-agent-status2,d73f63cf5b55b6ef,d3ae9b3202cdf775] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:26:16.277 INFO 2981023 --- [host-check-agent-status2,d73f63cf5b55b6ef,d3ae9b3202cdf775] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:26:16.279 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:26:16.273+08:00 and status AVAILABLE 2024-09-11 17:26:16.280 INFO 2981023 --- [host-check-agent-status3,d73f63cf5b55b6ef,0a7d89789b64e86a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:26:16.282 INFO 2981023 --- [host-check-agent-status1,d73f63cf5b55b6ef,d291dfd4f0f64d1a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:26:16.283 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:26:16.283 INFO 2981023 --- [host-check-agent-status3,d73f63cf5b55b6ef,0a7d89789b64e86a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:26:16.283 INFO 2981023 --- [host-check-agent-status3,d73f63cf5b55b6ef,0a7d89789b64e86a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:26:16.283 INFO 2981023 --- [host-check-agent-status3,d73f63cf5b55b6ef,0a7d89789b64e86a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:26:16.283 INFO 2981023 --- [host-check-agent-status4,d73f63cf5b55b6ef,4606b754c70e8810] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:26:16.284 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:26:16.283444341+08:00 2024-09-11 17:26:16.285 INFO 2981023 --- [host-check-agent-status1,d73f63cf5b55b6ef,d291dfd4f0f64d1a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:26:16.285 INFO 2981023 --- [host-check-agent-status1,d73f63cf5b55b6ef,d291dfd4f0f64d1a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:26:16.285 INFO 2981023 --- [host-check-agent-status1,d73f63cf5b55b6ef,d291dfd4f0f64d1a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:26:16.285 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:26:16.284+08:00 2024-09-11 17:26:16.287 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:26:16.284+08:00 and status AVAILABLE 2024-09-11 17:26:16.292 INFO 2981023 --- [host-check-agent-status4,d73f63cf5b55b6ef,4606b754c70e8810] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:26:16.292 INFO 2981023 --- [host-check-agent-status4,d73f63cf5b55b6ef,4606b754c70e8810] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:26:16.292 INFO 2981023 --- [host-check-agent-status4,d73f63cf5b55b6ef,4606b754c70e8810] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:26:16.296 INFO 2981023 --- [ocp-schedules-19,5d9e7efbbcc649a0,0bee7accd3399ee6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:26:16.299 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:26:16.318 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:26:16.320 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]:connector is evicted from holder's cache. cause=EXPIRED, key=ConnectorKey(protocol=ssh, hostAddress=10.31.35.10, httpPort=null, sshPort=22, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=password, passwordAuthConfig=PasswordAuthConfig(username=root), privateKeyAuthConfig=null)), proxy=null) 2024-09-11 17:26:16.320 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] n.schmizz.sshj.transport.TransportImpl : Disconnected - BY_APPLICATION 2024-09-11 17:26:16.322 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:26:16.322236594+08:00 2024-09-11 17:26:16.328 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:26:16.330 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:26:16.322+08:00 2024-09-11 17:26:16.330 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.332 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:16.333 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:26:16.322+08:00 and status AVAILABLE 2024-09-11 17:26:16.333 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:26:16.334 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:26:16.345 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.345 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:16.355 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:26:16.356 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.356 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:16.358 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:26:16.359 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:26:16.359 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.360 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:16.361 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:26:16.362 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:26:16.364 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:26:16.365 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:26:16.366 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:26:16.372 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:26:16.372 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:26:16.372032+08:00, lastProcessTime=2024-09-11T17:26:06.352500+08:00 2024-09-11 17:26:16.378 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:26:16.383 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.384 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:16.393 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:26:16.396 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.397 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:16.397 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:26:16.397 INFO 2981023 --- [ocp-schedules-11,fc4da96bb65685d0,903a93187a34c887] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:26:16.400 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:26:16.400 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:26:16.409 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.410 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.410 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:26:16.415 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:26:16.421 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:26:16.422 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:26:16.429 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:26:16.437 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:26:16.442 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:26:16.459 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:26:16.467 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:26:16.469 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.470 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:26:16.471 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.471 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:26:16.473 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,19475b207d5f21d8,e14a182a2562dbed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=19475b207d5f21d8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:26:16.474 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0d4fe6369c1dbbc8,39319e576d0d759f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=0d4fe6369c1dbbc8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:26:16.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,acacc849ac8b1760,068ae9bb12f11457] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=acacc849ac8b1760, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:26:16.486 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:16.491 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0d4fe6369c1dbbc8,39319e576d0d759f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=0d4fe6369c1dbbc8, duration=22 ms] 2024-09-11 17:26:16.491 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,acacc849ac8b1760,068ae9bb12f11457] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=acacc849ac8b1760, duration=13 ms] 2024-09-11 17:26:16.491 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:26:16.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.495 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:26:16.496 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:26:16.497 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,19475b207d5f21d8,e14a182a2562dbed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=19475b207d5f21d8, duration=31 ms] 2024-09-11 17:26:16.510 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:26:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:26:16.519 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.520 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.520 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:16.522 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.523 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:26:16.525 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:26:16.525 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:26:16.528 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:26:16.531 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:26:16.534 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:16.534 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:26:16.540 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:26:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.543 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:26:16.545 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:16.552 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:26:16.554 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:26:16.557 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:26:16.562 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.562 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.562 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:26:16.563 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:16.563 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:26:16.565 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:26:16.567 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:26:16.569 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:26:16.574 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:26:16.576 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.577 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.577 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:26:16.577 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:26:16.578 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:26:16.579 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:26:16.582 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:26:16.587 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.587 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:26:16.588 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.588 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:26:16.588 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:16.589 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:26:16.589 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.592 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:26:16.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:26:16.594 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:26:16.598 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:26:16.601 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:16.601 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:26:16.602 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:26:16.602 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:26:16.608 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:26:16.610 INFO 2981023 --- [host-check-host-status1,bca27cf687ba574e,045230cdf6ee881b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:26:16.614 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:26:16.615 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:26:16.621 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:16.626 INFO 2981023 --- [host-check-host-status3,bca27cf687ba574e,8f6ae0b1bb5e8aeb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:26:16.635 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:26:16.638 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:26:16.647 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.648 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.648 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:16.649 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:26:16.652 INFO 2981023 --- [host-check-host-status4,bca27cf687ba574e,c395a4a2ee25c253] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:26:16.657 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:26:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:26:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:26:16.677 INFO 2981023 --- [host-check-host-status2,bca27cf687ba574e,097fa2ff5c004c1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:26:16.685 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:26:16.695 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.696 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:26:16.697 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:26:16.704 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:26:16.704 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:26:16.708 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:26:16.716 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:26:16.372032+08:00 2024-09-11 17:26:16.716 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.716 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.717 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:26:16.722 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:26:16.722 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:26:16.725 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:26:16.730 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:16.730 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:26:16.736 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:26:16.736 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:26:16.736 INFO 2981023 --- [ocp-schedules-30,6e9a6a3ac7b931e6,ca0811b494c7bfdc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:26:17.027 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:17.027 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:17.153 INFO 2981023 --- [ocp-async-3,5e1d63dbd1e92bf8,daec7b037d0bec22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,44bf26cb892249c8,babcde36c53824d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,44bf26cb892249c8,babcde36c53824d1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:17.739 INFO 2981023 --- [ocp-async-1,b55dc9a566408f14,5663e06eab2668ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:26:17.896 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046700, nodeCount=480, validCount=480 2024-09-11 17:26:18.549 INFO 2981023 --- [host-check-clockdiff2,aedaa9fe704a2ae4,d9e2c798ac8e7586] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046778 0 0 , error= 2024-09-11 17:26:19.154 INFO 2981023 --- [ocp-async-10,3591fccc95c08944,be4afb2134fa7164] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:20.027 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:20.028 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:20.166 INFO 2981023 --- [host-check-clockdiff1,aedaa9fe704a2ae4,93b5f2ee97ee275e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046780 0 0 , error= 2024-09-11 17:26:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,c538acba810659ed,b9ddd27d420c2232] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,c538acba810659ed,b9ddd27d420c2232] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:21.154 INFO 2981023 --- [ocp-async-9,846632d09eec368c,4ecba06b30fc8171] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:21.775 INFO 2981023 --- [ocp-schedules-27,c3a747de86cb5d6b,9cd9c15525801890] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:26:22.005 INFO 2981023 --- [host-check-clockdiff3,aedaa9fe704a2ae4,7d784f28659cfbdb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046782 1 1 , error= 2024-09-11 17:26:22.342 INFO 2981023 --- [ocp-schedules-29,a53d7ec0ab5727a8,f4894aa4bfca7916] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:26:22.343 INFO 2981023 --- [ocp-schedules-29,a53d7ec0ab5727a8,f4894aa4bfca7916] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:26:22.343 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:26:22.343 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:26:22.343 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:26:22.348 INFO 2981023 --- [ocp-schedules-29,a53d7ec0ab5727a8,f4894aa4bfca7916] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:26:22.348 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:26:22.349 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:26:22.349 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:26:22.349 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:26:22.349 INFO 2981023 --- [ocp-schedules-29,a53d7ec0ab5727a8,f4894aa4bfca7916] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:26:22.350 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:26:22.351 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:26:22.367 INFO 2981023 --- [ocp-schedules-29,a53d7ec0ab5727a8,f4894aa4bfca7916] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:26:22.367 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:26:22.367 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:26:22.370 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:26:22.372 INFO 2981023 --- [ocp-schedules-29,a53d7ec0ab5727a8,f4894aa4bfca7916] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:26:23.028 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:23.029 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:23.154 INFO 2981023 --- [ocp-async-3,b648649bf9792f15,4e40683db1373857] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,c47890d4fc7ab864,52bb6e790ba94649] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:23.318 INFO 2981023 --- [prometheus-minute-schedule-1,c47890d4fc7ab864,52bb6e790ba94649] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:24.170 INFO 2981023 --- [ocp-schedules-5,45956b110769714c,a04fffab62705ed3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:26:25.154 INFO 2981023 --- [ocp-async-4,fb307611d2b5dadc,1cebc97eb8d07580] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:25.598 INFO 2981023 --- [ocp-schedules-10,622e82dc42ebaea1,747e3ba0c28d39af] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:26:25.631 WARN 2981023 --- [scanner-13483063-1,622e82dc42ebaea1,372d220b0da7a177] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:26:25.670 INFO 2981023 --- [alarm-detect-9,622e82dc42ebaea1,7d0a3a092e71a2db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643794, data is empty. 2024-09-11 17:26:25.673 INFO 2981023 --- [alarm-detect-9,622e82dc42ebaea1,7d0a3a092e71a2db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643794, data is empty. 2024-09-11 17:26:25.677 INFO 2981023 --- [alarm-detect-9,622e82dc42ebaea1,7d0a3a092e71a2db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643794, data is empty. 2024-09-11 17:26:25.693 INFO 2981023 --- [alarm-detect-13,622e82dc42ebaea1,d7f6a485c129c9b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643794, data is empty. 2024-09-11 17:26:25.693 INFO 2981023 --- [alarm-detect-9,622e82dc42ebaea1,d612814357be00c2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643794, data is empty. 2024-09-11 17:26:25.696 INFO 2981023 --- [alarm-detect-13,622e82dc42ebaea1,d7f6a485c129c9b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643794, data is empty. 2024-09-11 17:26:25.700 INFO 2981023 --- [alarm-detect-13,622e82dc42ebaea1,d7f6a485c129c9b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643794, data is empty. 2024-09-11 17:26:25.702 INFO 2981023 --- [alarm-detect-9,622e82dc42ebaea1,d612814357be00c2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643794, data is empty. 2024-09-11 17:26:25.704 INFO 2981023 --- [alarm-detect-9,622e82dc42ebaea1,d612814357be00c2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643794, data is empty. 2024-09-11 17:26:25.813 INFO 2981023 --- [alarm-detect-9,622e82dc42ebaea1,97a0f6bdeee1c826] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643794, data is empty. 2024-09-11 17:26:25.816 INFO 2981023 --- [alarm-detect-9,622e82dc42ebaea1,97a0f6bdeee1c826] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643794, data is empty. 2024-09-11 17:26:25.819 INFO 2981023 --- [alarm-detect-9,622e82dc42ebaea1,97a0f6bdeee1c826] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643794, data is empty. 2024-09-11 17:26:26.029 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:26.029 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,8ca7e28c021d694e,8adefea4d6be0781] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,8ca7e28c021d694e,8adefea4d6be0781] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:26.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:26:26.350580+08:00, lastProcessTime=2024-09-11T17:26:16.372032+08:00 2024-09-11 17:26:26.472 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:26:26.350580+08:00 2024-09-11 17:26:26.512 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6355091791110308,f03835ce20f431ac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=6355091791110308, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:26:26.512 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71a34b7c8793136d,219bcef5f2533f37] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=71a34b7c8793136d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:26:26.515 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6355091791110308,f03835ce20f431ac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=6355091791110308, duration=8 ms] 2024-09-11 17:26:26.519 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71a34b7c8793136d,219bcef5f2533f37] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=71a34b7c8793136d, duration=9 ms] 2024-09-11 17:26:26.521 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b78d512f8db9c9f7,7928c6fad31cebfc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=b78d512f8db9c9f7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:26:26.522 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b78d512f8db9c9f7,7928c6fad31cebfc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=b78d512f8db9c9f7, duration=3 ms] 2024-09-11 17:26:27.154 INFO 2981023 --- [ocp-async-5,281fa894e0234425,fa280766920407d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:27.739 INFO 2981023 --- [ocp-async-4,d281a214d583cb37,9b24befe60b67572] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:26:29.030 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:29.030 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:29.154 INFO 2981023 --- [ocp-async-6,e7acdc8ba2243cae,fade66b450e9b5ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,d69c64b85b12d871,eb601429ba4af5d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,d69c64b85b12d871,eb601429ba4af5d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:31.154 INFO 2981023 --- [ocp-async-10,5c29dc0b9492bbd9,5004492f38ac89f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:31.779 INFO 2981023 --- [ocp-schedules-24,39823fb045191cb6,2c9d6f0131996796] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:26:32.030 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:32.030 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,9f23f7d1e2d5a204,d9276461cc0005c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,9f23f7d1e2d5a204,d9276461cc0005c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:33.154 INFO 2981023 --- [ocp-async-5,0baf2a60a8ae7c17,7219e715991fc043] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:33.556 INFO 2981023 --- [ocp-schedules-18,4e986f197b6d509a,6acf1e26d061af40] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:26:33.568 INFO 2981023 --- [ocp-schedules-18,4e986f197b6d509a,6acf1e26d061af40] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:26:33.568 INFO 2981023 --- [ocp-schedules-18,4e986f197b6d509a,6acf1e26d061af40] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:26:35.031 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:35.031 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:35.154 INFO 2981023 --- [ocp-async-1,9a084b73342c9d89,e04adec24bfbb79d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,d0732d13bf623735,4e2a1013191dc927] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,d0732d13bf623735,4e2a1013191dc927] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:35.887 INFO 2981023 --- [ocp-schedules-26,05086c4b1f63b8d2,55f8f44afc40da66] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:26:35.916 WARN 2981023 --- [scanner-13483372-1,05086c4b1f63b8d2,f761a54c2fef615f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:26:35.967 INFO 2981023 --- [alarm-detect-8,05086c4b1f63b8d2,d83a9a79487fad14] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726046760 ~ 1726046785 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726046775, value=1.66)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.66, values={1726046775=1.66}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:26:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:26:36.337538+08:00, lastProcessTime=2024-09-11T17:26:26.350580+08:00 2024-09-11 17:26:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:26:36.337538+08:00 2024-09-11 17:26:37.155 INFO 2981023 --- [ocp-async-6,beeebcbbb78cecf7,4dbb59aaf3b3e622] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:37.314 INFO 2981023 --- [prometheus-cache-monitor1,8562420f1fc930b9,31527bd941c2a3fe] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:26:37.315 INFO 2981023 --- [prometheus-cache-monitor1,8562420f1fc930b9,31527bd941c2a3fe] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:26:37.335 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,8d254a74f5848b37,b1375174257961ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=8d254a74f5848b37, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:26:37.335 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,566c7d1e7ef0496c,98dbff5c12289176] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=566c7d1e7ef0496c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:26:37.336 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,566c7d1e7ef0496c,98dbff5c12289176] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=566c7d1e7ef0496c, duration=2 ms] 2024-09-11 17:26:37.350 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,8d254a74f5848b37,b1375174257961ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=8d254a74f5848b37, duration=17 ms] 2024-09-11 17:26:37.352 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,d43ca13da91d9709,5f585e822914f9e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=d43ca13da91d9709, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:26:37.353 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,d43ca13da91d9709,5f585e822914f9e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=d43ca13da91d9709, duration=8 ms] 2024-09-11 17:26:37.739 INFO 2981023 --- [ocp-async-1,8c7258a52cf2b02f,89aff6ca52c91fbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:26:37.967 INFO 2981023 --- [ocp-schedules-29,42e6dc321cb1154f,81cee11bea925efc] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:26:37.985 WARN 2981023 --- [ocp-schedules-29,42e6dc321cb1154f,81cee11bea925efc] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:26:38.031 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:38.031 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,d476cf78ea8767d0,02d2d3e0190245c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,d476cf78ea8767d0,02d2d3e0190245c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:39.155 INFO 2981023 --- [ocp-async-7,ef40188da8a31127,565271e712fcf156] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:40.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:26:40.003 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:26:41.031 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:41.032 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:41.155 INFO 2981023 --- [ocp-async-3,75711e23647e2e01,64fb58e14253f135] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,0ea599922fe20612,c7aab65dc82728ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,0ea599922fe20612,c7aab65dc82728ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:41.781 INFO 2981023 --- [ocp-schedules-22,91210f6ed53c4e56,d9a9b57ea614cd7f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:26:42.148 INFO 2981023 --- [ocp-schedules-5,37a6c717507722bc,53c17d23c5e2e019] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:26:43.155 INFO 2981023 --- [ocp-async-1,726aa79ff79e1292,946074e3a68d7e7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:44.032 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:44.032 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,09175609430d5416,2796f364474e40ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,09175609430d5416,2796f364474e40ca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:45.101 INFO 2981023 --- [prometheus-second-schedule-1,c6a072ccb45b6698,47cfd7f75feff493] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:26:45.155 INFO 2981023 --- [ocp-async-5,69c7f1b8a539621d,781cce7a6485dc0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:46.020 INFO 2981023 --- [ocp-schedules-12,b99d5d420260ee8f,5c519bd2bcbdc96e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:26:46.020 INFO 2981023 --- [ocp-schedules-26,8476062b4ebed186,03eee9a501a3e07f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:26:46.021 INFO 2981023 --- [host-check-clockdiff2,b99d5d420260ee8f,f98d99bbd68717c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:26:46.021 INFO 2981023 --- [host-check-clockdiff1,b99d5d420260ee8f,84a2e1259a6540bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:26:46.021 INFO 2981023 --- [host-check-agent-status1,8476062b4ebed186,aed8ad864d0fce1e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:26:46.023 INFO 2981023 --- [host-check-agent-status2,8476062b4ebed186,8736763719ea8a9b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:26:46.023 INFO 2981023 --- [ocp-schedules-9,0daccedc779c2ec3,3f3b3f01c5d7616f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:26:46.023 INFO 2981023 --- [host-check-agent-status3,8476062b4ebed186,b23fadfe3d9443a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:26:46.024 INFO 2981023 --- [host-check-agent-status4,8476062b4ebed186,cdd98e5d61737517] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:26:46.024 INFO 2981023 --- [host-check-clockdiff3,b99d5d420260ee8f,5169239c0ce1a866] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:26:46.025 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:26:46.029 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:26:46.029 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:26:46.031 INFO 2981023 --- [host-check-agent-status1,8476062b4ebed186,aed8ad864d0fce1e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:26:46.033 INFO 2981023 --- [host-check-clockdiff4,b99d5d420260ee8f,88c52e857f02b41f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:26:46.035 INFO 2981023 --- [host-check-agent-status3,8476062b4ebed186,b23fadfe3d9443a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:26:46.037 INFO 2981023 --- [host-check-agent-status2,8476062b4ebed186,8736763719ea8a9b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:26:46.038 INFO 2981023 --- [host-check-agent-status1,8476062b4ebed186,aed8ad864d0fce1e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:26:46.038 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:26:46.042 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:26:46.042 INFO 2981023 --- [host-check-agent-status2,8476062b4ebed186,8736763719ea8a9b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:26:46.042 INFO 2981023 --- [host-check-agent-status3,8476062b4ebed186,b23fadfe3d9443a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:26:46.043 INFO 2981023 --- [host-check-clockdiff1,b99d5d420260ee8f,84a2e1259a6540bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046806 0 0 , error= 2024-09-11 17:26:46.044 INFO 2981023 --- [host-check-agent-status4,8476062b4ebed186,cdd98e5d61737517] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:26:46.045 INFO 2981023 --- [host-check-clockdiff4,b99d5d420260ee8f,88c52e857f02b41f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046806 0 0 , error= 2024-09-11 17:26:46.045 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:26:46.045 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:26:46.048 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:26:46.049 INFO 2981023 --- [host-check-agent-status4,8476062b4ebed186,cdd98e5d61737517] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:26:46.053 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:26:46.054 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:26:46.056 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:26:46.056 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:26:46.069 INFO 2981023 --- [ocp-schedules-25,3a206b0eee6a0c2d,1574988e34dc6b54] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:26:46.088 INFO 2981023 --- [host-check-agent-status4,8476062b4ebed186,cdd98e5d61737517] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:26:46.088 INFO 2981023 --- [host-check-agent-status3,8476062b4ebed186,b23fadfe3d9443a8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:26:46.092 INFO 2981023 --- [host-check-agent-status1,8476062b4ebed186,aed8ad864d0fce1e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:26:46.093 INFO 2981023 --- [host-check-agent-status2,8476062b4ebed186,8736763719ea8a9b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:26:46.093 INFO 2981023 --- [host-check-agent-status3,8476062b4ebed186,b23fadfe3d9443a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:26:46.093 INFO 2981023 --- [host-check-agent-status3,8476062b4ebed186,b23fadfe3d9443a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:26:46.093 INFO 2981023 --- [host-check-agent-status3,8476062b4ebed186,b23fadfe3d9443a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:26:46.094 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:26:46.096 INFO 2981023 --- [host-check-agent-status1,8476062b4ebed186,aed8ad864d0fce1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:26:46.096 INFO 2981023 --- [host-check-agent-status1,8476062b4ebed186,aed8ad864d0fce1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:26:46.096 INFO 2981023 --- [host-check-agent-status1,8476062b4ebed186,aed8ad864d0fce1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:26:46.096 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:26:46.096127212+08:00 2024-09-11 17:26:46.097 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:26:46.096+08:00 2024-09-11 17:26:46.099 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:26:46.096+08:00 and status AVAILABLE 2024-09-11 17:26:46.105 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:26:46.125 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:26:46.125 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:26:46.127 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:26:46.126864238+08:00 2024-09-11 17:26:46.128 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:26:46.127+08:00 2024-09-11 17:26:46.129 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:26:46.129 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:26:46.127+08:00 and status AVAILABLE 2024-09-11 17:26:46.130 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:26:46.131 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:26:46.130988042+08:00 2024-09-11 17:26:46.131 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:26:46.131+08:00 2024-09-11 17:26:46.131 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:26:46.133 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:26:46.132750765+08:00 2024-09-11 17:26:46.133 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:26:46.131+08:00 and status AVAILABLE 2024-09-11 17:26:46.134 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:26:46.133+08:00 2024-09-11 17:26:46.136 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:26:46.133+08:00 and status AVAILABLE 2024-09-11 17:26:46.137 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:26:46.143 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:26:46.147 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:26:46.153 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:26:46.157 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:26:46.159 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:26:46.159 INFO 2981023 --- [ocp-schedules-20,a7bef405c5080a22,c8afe3a204221d38] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:26:46.159 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:26:46.161 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:26:46.168 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:26:46.178 INFO 2981023 --- [ocp-schedules-20,a7bef405c5080a22,c8afe3a204221d38] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:26:46.179 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:26:46.182 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:26:46.183 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:26:46.187 INFO 2981023 --- [ocp-schedules-2,b81a0a96ec488828,12197ed5d2a6c04c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:26:46.187 INFO 2981023 --- [ocp-schedules-11,e5f775b57bc4e131,cb21d3cefa6c843b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:26:46.188 INFO 2981023 --- [ocp-schedules-23,8777dd331369aa6f,2b94db9a989b8168] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:26:46.188 INFO 2981023 --- [ocp-schedules-5,9d825f6b03154f3d,e0cf7d5873069538] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:26:46.189 INFO 2981023 --- [ocp-schedules-2,b81a0a96ec488828,12197ed5d2a6c04c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:46.191 INFO 2981023 --- [ocp-schedules-23,8777dd331369aa6f,2b94db9a989b8168] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:46.194 INFO 2981023 --- [ocp-schedules-11,e5f775b57bc4e131,cb21d3cefa6c843b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:46.204 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:26:46.207 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:26:46.220 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:26:46.224 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:26:46.227 INFO 2981023 --- [ocp-schedules-6,e69f64dd1f1b4d68,e11f818b5cbb671a] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:26:46.227 INFO 2981023 --- [ocp-schedules-6,e69f64dd1f1b4d68,e11f818b5cbb671a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:26:46.227 INFO 2981023 --- [ocp-schedules-18,2da2d2cae8448407,ebc2ca9d5ac966c5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:26:46.229 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:26:46.237 INFO 2981023 --- [ocp-schedules-14,a76e9de50e69e233,10aafd080ed64253] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:26:46.238 INFO 2981023 --- [ocp-schedules-14,a76e9de50e69e233,10aafd080ed64253] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:46.245 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:26:46.257 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:26:46.258 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:26:46.276 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:26:46.281 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:26:46.283 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:26:46.294 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:26:46.296 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:26:46.304 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:26:46.310 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:26:46.314 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.314 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:26:46.315 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:26:46.316 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.330 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:26:46.332 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:26:46.340 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:26:46.340 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:26:46.339021+08:00, lastProcessTime=2024-09-11T17:26:36.337538+08:00 2024-09-11 17:26:46.343 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:26:46.353 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:26:46.363 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:26:46.368 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:26:46.376 INFO 2981023 --- [host-check-host-status1,0daccedc779c2ec3,77f50060a8343efe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:26:46.381 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:26:46.395 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:26:46.398 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:26:46.401 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:26:46.422 INFO 2981023 --- [host-check-agent-status2,8476062b4ebed186,8736763719ea8a9b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:26:46.423 INFO 2981023 --- [host-check-agent-status2,8476062b4ebed186,8736763719ea8a9b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:26:46.423 INFO 2981023 --- [host-check-agent-status2,8476062b4ebed186,8736763719ea8a9b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:26:46.424 INFO 2981023 --- [host-check-agent-status4,8476062b4ebed186,cdd98e5d61737517] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:26:46.424 INFO 2981023 --- [host-check-agent-status4,8476062b4ebed186,cdd98e5d61737517] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:26:46.424 INFO 2981023 --- [host-check-agent-status4,8476062b4ebed186,cdd98e5d61737517] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:26:46.429 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:46.430 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.431 WARN 2981023 --- [scanner-13483617-1,3a206b0eee6a0c2d,54307e05a8dbe484] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:26:46.432 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:26:46.434 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:26:46.434 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:26:46.434 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.439 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:26:46.443 INFO 2981023 --- [host-check-host-status3,0daccedc779c2ec3,f6361561c7645f33] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:26:46.449 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.449 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:26:46.456 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,8f604364bcacb199,7bc82ac45f2ca4f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=8f604364bcacb199, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:26:46.456 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,8f604364bcacb199,7bc82ac45f2ca4f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=8f604364bcacb199, duration=1 ms] 2024-09-11 17:26:46.458 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.458 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:46.459 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:26:46.460 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:26:46.461 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.461 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:46.462 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:26:46.472 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:26:46.474 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.474 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:26:46.475 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.476 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.477 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:26:46.504 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.504 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:46.508 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:46.509 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:26:46.509 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.510 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:46.510 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:26:46.516 INFO 2981023 --- [alarm-detect-3,3a206b0eee6a0c2d,5a9e9d337d32f8c4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643794, data is empty. 2024-09-11 17:26:46.525 INFO 2981023 --- [ocp-schedules-11,e5f775b57bc4e131,cb21d3cefa6c843b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:26:46.525 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.526 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:26:46.528 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:26:46.531 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.531 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.531 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:46.533 INFO 2981023 --- [alarm-detect-3,3a206b0eee6a0c2d,5a9e9d337d32f8c4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643794, data is empty. 2024-09-11 17:26:46.534 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:26:46.538 INFO 2981023 --- [pool-10-thread-14,cdf48e987c573b7f,8c01019fab9e6ffd] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:26:46.540 INFO 2981023 --- [alarm-detect-3,3a206b0eee6a0c2d,5a9e9d337d32f8c4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643794, data is empty. 2024-09-11 17:26:46.544 INFO 2981023 --- [pool-10-thread-15,cdf48e987c573b7f,d8e0e413ae473e76] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:26:46.544 INFO 2981023 --- [pool-10-thread-13,cdf48e987c573b7f,c32050c1b32a3990] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:26:46.548 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:46.550 INFO 2981023 --- [pool-10-thread-16,cdf48e987c573b7f,8977636369b62d76] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:26:46.550 INFO 2981023 --- [pool-10-thread-16,cdf48e987c573b7f,8977636369b62d76] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:26:46.561 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:26:46.579 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e7935726aebc5f1a,c6fe0fac9ca2eb14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=e7935726aebc5f1a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:26:46.579 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e7935726aebc5f1a,c6fe0fac9ca2eb14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=e7935726aebc5f1a, duration=0 ms] 2024-09-11 17:26:46.580 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.581 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:26:46.589 INFO 2981023 --- [pool-10-thread-15,cdf48e987c573b7f,d8e0e413ae473e76] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:26:46.589 INFO 2981023 --- [pool-10-thread-15,cdf48e987c573b7f,d8e0e413ae473e76] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:26:46.595 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b2c29bacb812980b,1665eeb6cbd9358b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&ObClusterId=1719994443&version=2, client=10.31.35.10, traceId=b2c29bacb812980b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkrtd,1719994443,false,2,] 2024-09-11 17:26:46.595 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b2c29bacb812980b,1665eeb6cbd9358b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=b2c29bacb812980b, duration=1 ms] 2024-09-11 17:26:46.601 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:26:46.606 INFO 2981023 --- [alarm-detect-12,3a206b0eee6a0c2d,3488d162e8cd0226] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643794, data is empty. 2024-09-11 17:26:46.617 INFO 2981023 --- [alarm-detect-9,3a206b0eee6a0c2d,346a534aacf7ad0d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643794, data is empty. 2024-09-11 17:26:46.620 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:46.621 INFO 2981023 --- [alarm-detect-12,3a206b0eee6a0c2d,3488d162e8cd0226] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643794, data is empty. 2024-09-11 17:26:46.623 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.625 INFO 2981023 --- [alarm-detect-9,3a206b0eee6a0c2d,346a534aacf7ad0d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643794, data is empty. 2024-09-11 17:26:46.628 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:26:46.629 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:26:46.634 INFO 2981023 --- [host-check-host-status2,0daccedc779c2ec3,653dbabd1bfd412b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:26:46.634 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.635 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:26:46.635 INFO 2981023 --- [host-check-host-status4,0daccedc779c2ec3,16759639a0d220f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:26:46.636 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:26:46.636 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.642 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:26:46.644 INFO 2981023 --- [alarm-detect-12,3a206b0eee6a0c2d,3488d162e8cd0226] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643794, data is empty. 2024-09-11 17:26:46.653 INFO 2981023 --- [alarm-detect-9,3a206b0eee6a0c2d,346a534aacf7ad0d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643794, data is empty. 2024-09-11 17:26:46.660 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:46.661 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.662 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:26:46.662 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:26:46.662 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:26:46.664 INFO 2981023 --- [ForkJoinPool-6-worker-7,cdf48e987c573b7f,83e979226fd6c1ac] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:26:46.664 INFO 2981023 --- [ForkJoinPool-6-worker-7,cdf48e987c573b7f,83e979226fd6c1ac] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:26:46.698 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:26:46.699 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.699 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:26:46.700 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.702 INFO 2981023 --- [ocp-schedules-2,b81a0a96ec488828,12197ed5d2a6c04c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:26:46.705 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:26:46.706 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:46.709 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:26:46.711 INFO 2981023 --- [ocp-schedules-14,a76e9de50e69e233,10aafd080ed64253] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:26:46.930 INFO 2981023 --- [pool-10-thread-16,cdf48e987c573b7f,8977636369b62d76] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:26:46.933 INFO 2981023 --- [pool-10-thread-16,cdf48e987c573b7f,8977636369b62d76] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:26:47.032 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:47.039 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:47.067 WARN 2981023 --- [ForkJoinPool-6-worker-7,cdf48e987c573b7f,83e979226fd6c1ac] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:26:47.155 INFO 2981023 --- [ocp-async-3,e2504ef5048a55b7,e278542f57227499] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,4d67cc5a36e41f6e,fb55d109c4997e1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:47.359 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.361 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:26:47.396 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:26:47.399 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.401 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:26:47.412 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.414 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:26:47.422 INFO 2981023 --- [prometheus-minute-schedule-1,4d67cc5a36e41f6e,fb55d109c4997e1f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:47.480 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:47.482 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:26:47.489 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:26:47.489 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:26:47.493 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.498 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:26:47.616 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:47.623 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.624 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:47.629 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.633 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:26:47.663 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:26:47.663 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.663 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:26:47.670 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.671 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.672 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:26:47.675 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:26:47.677 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.679 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:26:47.680 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.681 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:26:47.684 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:26:47.684 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:47.697 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.698 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:47.700 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:47.700 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:26:47.705 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.709 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:26:47.728 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:26:47.729 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:26:47.746 INFO 2981023 --- [ocp-async-9,7414541e3be87078,54486f75e8f39556] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:26:47.747 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:26:47.749 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.751 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:26:47.761 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.762 INFO 2981023 --- [ForkJoinPool-6-worker-5,cdf48e987c573b7f,9c5f8476ac2bfb39] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:26:47.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:47.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:26:47.775 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.777 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:26:47.780 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.784 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:26:47.785 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:26:47.835 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:26:47.896 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:47.897 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:47.897 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:26:47.925 INFO 2981023 --- [ForkJoinPool-6-worker-9,cdf48e987c573b7f,4ddcb9674ba2823f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.928 INFO 2981023 --- [ForkJoinPool-6-worker-9,cdf48e987c573b7f,4ddcb9674ba2823f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:26:47.933 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:47.933 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.934 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:47.935 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:26:47.942 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.942 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.943 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.943 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:26:47, 2024-09-11 17:26:47] 2024-09-11 17:26:47.944 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:26:47, 2024-09-11 17:26:47] 2024-09-11 17:26:47.946 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:26:47.946 INFO 2981023 --- [ForkJoinPool-6-worker-0,cdf48e987c573b7f,483a5ffacfcae9dd] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:26:47.946 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:26:47.954 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:26:47.954 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.955 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:26:47, 2024-09-11 17:26:47] 2024-09-11 17:26:47.956 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:26:47.956 INFO 2981023 --- [ForkJoinPool-6-worker-12,cdf48e987c573b7f,9857b4be0afccd46] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:26:47.964 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:47.964 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.965 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:26:47.965 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:26:47.966 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.982 INFO 2981023 --- [ForkJoinPool-6-worker-9,cdf48e987c573b7f,4ddcb9674ba2823f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:26:47.983 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:26:47.983 INFO 2981023 --- [ForkJoinPool-6-worker-9,cdf48e987c573b7f,4ddcb9674ba2823f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.983 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:26:47.983 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:47.985 INFO 2981023 --- [ForkJoinPool-6-worker-9,cdf48e987c573b7f,4ddcb9674ba2823f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:26:47.985 INFO 2981023 --- [ForkJoinPool-6-worker-9,cdf48e987c573b7f,4ddcb9674ba2823f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.986 INFO 2981023 --- [ForkJoinPool-6-worker-9,cdf48e987c573b7f,4ddcb9674ba2823f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:26:47.988 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:47.988 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:47.988 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:26:47.989 INFO 2981023 --- [pool-10-thread-13,cdf48e987c573b7f,c32050c1b32a3990] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:26:47.995 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:26:48.000 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:26:48.001 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:26:46.339021+08:00 2024-09-11 17:26:48.003 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:48.003 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:48.004 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:48.005 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:26:48.011 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:26:48.019 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:26:48.019 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:26:48.023 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:48.024 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:48.024 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:26:48.025 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:26:48.025 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:48.027 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:26:48.027 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:26:48.029 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:26:48.037 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:48.037 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:26:48.042 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:26:48.051 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:26:48.056 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:26:48.063 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:48.064 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:48.064 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:26:48.065 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:26:48.065 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:48.065 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:48.065 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:48.066 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:26:48.067 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:26:48.071 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:26:48.075 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:26:48.076 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:26:48.082 INFO 2981023 --- [pool-10-thread-14,cdf48e987c573b7f,8c01019fab9e6ffd] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:26:48.105 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:26:48.109 INFO 2981023 --- [alarm-detect-15,3a206b0eee6a0c2d,5418066184725920] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643794, data is empty. 2024-09-11 17:26:48.113 INFO 2981023 --- [alarm-detect-15,3a206b0eee6a0c2d,5418066184725920] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643794, data is empty. 2024-09-11 17:26:48.122 INFO 2981023 --- [alarm-detect-15,3a206b0eee6a0c2d,5418066184725920] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643794, data is empty. 2024-09-11 17:26:48.122 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:26:48.129 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:48.129 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:26:48.129 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:48.130 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:26:48.133 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:26:48.141 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:26:48.141 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:26:48.146 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:48.146 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:48.147 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:26:48.163 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:26:48.168 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:26:48.168 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:26:48.169 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:26:48.177 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:48.177 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:48.178 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:26:48.183 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:26:48.189 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:26:48.189 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:26:48.195 INFO 2981023 --- [ocp-schedules-23,8777dd331369aa6f,2b94db9a989b8168] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:26:48.196 INFO 2981023 --- [ocp-schedules-10,6e6edf5e6a8057f2,fcfbebd93cb78c15] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:26:48.547 INFO 2981023 --- [host-check-clockdiff2,b99d5d420260ee8f,f98d99bbd68717c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046808 0 0 , error= 2024-09-11 17:26:49.155 INFO 2981023 --- [ocp-async-4,b256a0636a05624f,87117d1abdca6e55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:26:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:26:50.040 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:50.041 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:50.193 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.198 INFO 2981023 --- [ocp-schedules-16,ec905f2efc416ac1,00c6ac409d38fa0d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:26:50.203 INFO 2981023 --- [ocp-schedules-1,d6860aa34272c8f0,70399d8830e5d98d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:26:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,0b01e0a893c3c619,2f8df63299710d8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,0b01e0a893c3c619,2f8df63299710d8d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:51.065 INFO 2981023 --- [ocp-schedules-20,93b083918a473ed6,705f74113ee14989] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:26:51.065+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:26:51.070 INFO 2981023 --- [ocp-schedules-20,93b083918a473ed6,705f74113ee14989] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:26:51.155 INFO 2981023 --- [ocp-async-6,03bb9c6c0c5bcd24,34af5a194f3bfca9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:51.203 INFO 2981023 --- [ocp-schedules-20,93b083918a473ed6,705f74113ee14989] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 49 to SqlAggregateData 2024-09-11 17:26:51.205 INFO 2981023 --- [ocp-schedules-20,93b083918a473ed6,705f74113ee14989] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:26:51.509 INFO 2981023 --- [ocp-schedules-8,9317ac1b658aab0f,d3ecba589f2989f4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:26:51.511 INFO 2981023 --- [ocp-schedules-8,9317ac1b658aab0f,d3ecba589f2989f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:51.511 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:26:51.511 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:26:51.511 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:26:51.511 INFO 2981023 --- [ocp-schedules-8,9317ac1b658aab0f,d3ecba589f2989f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:26:51.635 INFO 2981023 --- [ocp-schedules-2,4b94edb243608758,3f8bd16b1689ff10] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:26:51.636 INFO 2981023 --- [ocp-schedules-2,4b94edb243608758,3f8bd16b1689ff10] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:26:51.663 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:26:51.663 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.664 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:26:51.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.667 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:26:51.668 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:26:51.671 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.671 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.672 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.673 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.674 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:26:51.678 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.678 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.679 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.682 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.682 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:26:51.682 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:26:51.687 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.688 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.692 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.692 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:26:51.694 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.695 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.698 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.699 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:26:51.699 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.700 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.702 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.703 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:26:51.703 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.704 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:26:51.705 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.705 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:26:51.706 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:26:51.706 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.706 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:26:51.710 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:26:51.713 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.713 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.714 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:26:51.714 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1E91D6DA-0-0] 2024-09-11 17:26:51.714 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:26:51.714 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.714 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:26:51.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:26:51.722 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.722 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:26:51.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:26:51.724 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.724 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:26:51.725 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.725 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:26:51.725 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:51.725 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.726 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:26:51.728 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.728 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.728 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.728 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:26:51.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:26:51.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:26:51.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:26:51.731 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:26:51.732 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:26:51.742 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.742 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:26:51.742 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:26:51.743 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1E91D6E4-0-0] 2024-09-11 17:26:51.743 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:26:51.744 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:26:51.745 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:51.747 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.747 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:26:51.747 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:26:51.747 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:26:51.747 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:26:51.749 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.749 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:26:51.752 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:26:51.753 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.753 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:26:51.754 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.754 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:26:51.754 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.754 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:26:51.754 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:26:51.755 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.755 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:26:51.759 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:26:51.760 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.760 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:26:51.761 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:26:51.761 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.761 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.761 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.761 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:26:51.762 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:26:51.763 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:26:51.766 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:26:51.768 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:26:51.768 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:26:51.768 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:26:51.774 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:26:51.775 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.776 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:26:51.784 INFO 2981023 --- [ocp-schedules-5,fabaa2351672ef94,7c2aed4cc4763523] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:26:51.784 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:26:51.786 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:26:51.786 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:51.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:26:51.797 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:26:51.798 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:26:51.798 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:26:51.798 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:26:51.798 INFO 2981023 --- [ocp-schedules-2,4b94edb243608758,3f8bd16b1689ff10] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:26:52.016 INFO 2981023 --- [host-check-clockdiff3,b99d5d420260ee8f,5169239c0ce1a866] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046812 0 0 , error= 2024-09-11 17:26:52.240 INFO 2981023 --- [ocp-schedules-6,c9cdc971ab65723a,d951c0ebbc24f8b7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:26:52.259 WARN 2981023 --- [ocp-schedules-6,c9cdc971ab65723a,d951c0ebbc24f8b7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:26:53.029 INFO 2981023 --- [ocp-schedules-3,554639f2836a9f14,b0069b6e0844c91d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:26:53.030 INFO 2981023 --- [ocp-schedules-3,554639f2836a9f14,b0069b6e0844c91d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:26:53.041 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:53.043 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:53.155 INFO 2981023 --- [ocp-async-9,f364ed4e43917194,9a01d916b0346b9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,abfe942b676091da,1b46e9e9332e9619] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,abfe942b676091da,1b46e9e9332e9619] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:53.571 INFO 2981023 --- [ocp-schedules-22,3cd73dd4b7a6d55f,21af9e9d062e5323] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:26:53.579 INFO 2981023 --- [ocp-schedules-22,3cd73dd4b7a6d55f,21af9e9d062e5323] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:26:53.579 INFO 2981023 --- [ocp-schedules-22,3cd73dd4b7a6d55f,21af9e9d062e5323] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:26:54.174 INFO 2981023 --- [ocp-schedules-19,5497193024f68feb,579f6f5b012916bd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:26:55.156 INFO 2981023 --- [ocp-async-10,ee6edbc7a05d5cef,ad98b1d931a23a70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:55.629 INFO 2981023 --- [ocp-schedules-19,b1bed2ea3747e94b,7331d6d851fba262] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:26:55.629 INFO 2981023 --- [ocp-schedules-19,b1bed2ea3747e94b,7331d6d851fba262] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:26:55.630 INFO 2981023 --- [ocp-schedules-19,b1bed2ea3747e94b,7331d6d851fba262] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:26:55.630 INFO 2981023 --- [ocp-schedules-19,b1bed2ea3747e94b,7331d6d851fba262] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:26:55.631 INFO 2981023 --- [ocp-schedules-19,b1bed2ea3747e94b,7331d6d851fba262] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:26:55.631 INFO 2981023 --- [ocp-schedules-19,b1bed2ea3747e94b,7331d6d851fba262] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:26:55.693 INFO 2981023 --- [ocp-schedules-8,2131a6aab81323e6,197617809a0ec90a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:26:55.699 INFO 2981023 --- [ocp-schedules-8,2131a6aab81323e6,197617809a0ec90a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:26:55.699 INFO 2981023 --- [ocp-schedules-8,2131a6aab81323e6,197617809a0ec90a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:26:55.778 INFO 2981023 --- [ocp-schedules-27,186791e284127fb8,337af66b009652b9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:26:55.778 INFO 2981023 --- [ocp-schedules-27,186791e284127fb8,337af66b009652b9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:26:55.798 INFO 2981023 --- [Collect all tenants major compactions1,186791e284127fb8,ddd28383818f81be] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:26:55.798 INFO 2981023 --- [Collect all tenants major compactions2,186791e284127fb8,741d93e171bb3e31] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:26:55.800 INFO 2981023 --- [Collect all tenants major compactions4,186791e284127fb8,41c16a8b40d41a3b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:26:55.800 INFO 2981023 --- [Collect all tenants major compactions3,186791e284127fb8,ae2b383d3d2f125a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:26:55.801 INFO 2981023 --- [Collect all tenants major compactions5,186791e284127fb8,baf43a3523717dc3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:26:55.802 INFO 2981023 --- [Collect all tenants major compactions6,186791e284127fb8,e727c49ed2eda413] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:26:55.816 INFO 2981023 --- [Collect all tenants major compactions2,186791e284127fb8,741d93e171bb3e31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:55.817 INFO 2981023 --- [Collect all tenants major compactions4,186791e284127fb8,41c16a8b40d41a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:55.817 INFO 2981023 --- [Collect all tenants major compactions3,186791e284127fb8,ae2b383d3d2f125a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:55.817 INFO 2981023 --- [Collect all tenants major compactions5,186791e284127fb8,baf43a3523717dc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:55.818 INFO 2981023 --- [Collect all tenants major compactions1,186791e284127fb8,ddd28383818f81be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:55.818 INFO 2981023 --- [Collect all tenants major compactions4,186791e284127fb8,41c16a8b40d41a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:26:55.818 INFO 2981023 --- [Collect all tenants major compactions2,186791e284127fb8,741d93e171bb3e31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:26:55.818 INFO 2981023 --- [Collect all tenants major compactions3,186791e284127fb8,ae2b383d3d2f125a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:26:55.819 INFO 2981023 --- [Collect all tenants major compactions1,186791e284127fb8,ddd28383818f81be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:26:55.819 INFO 2981023 --- [Collect all tenants major compactions5,186791e284127fb8,baf43a3523717dc3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:26:55.821 INFO 2981023 --- [Collect all tenants major compactions6,186791e284127fb8,e727c49ed2eda413] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:26:55.822 INFO 2981023 --- [Collect all tenants major compactions5,186791e284127fb8,baf43a3523717dc3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:26:55.822 INFO 2981023 --- [Collect all tenants major compactions5,186791e284127fb8,baf43a3523717dc3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:26:55.822 INFO 2981023 --- [Collect all tenants major compactions1,186791e284127fb8,ddd28383818f81be] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:26:55.822 INFO 2981023 --- [Collect all tenants major compactions1,186791e284127fb8,ddd28383818f81be] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:26:55.823 INFO 2981023 --- [Collect all tenants major compactions2,186791e284127fb8,741d93e171bb3e31] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:26:55.823 INFO 2981023 --- [Collect all tenants major compactions3,186791e284127fb8,ae2b383d3d2f125a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:26:55.823 INFO 2981023 --- [Collect all tenants major compactions3,186791e284127fb8,ae2b383d3d2f125a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:26:55.823 INFO 2981023 --- [Collect all tenants major compactions2,186791e284127fb8,741d93e171bb3e31] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:26:55.823 INFO 2981023 --- [Collect all tenants major compactions6,186791e284127fb8,e727c49ed2eda413] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:26:55.823 INFO 2981023 --- [Collect all tenants major compactions4,186791e284127fb8,41c16a8b40d41a3b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:26:55.823 INFO 2981023 --- [Collect all tenants major compactions4,186791e284127fb8,41c16a8b40d41a3b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:26:55.825 INFO 2981023 --- [Collect all tenants major compactions6,186791e284127fb8,e727c49ed2eda413] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:26:55.825 INFO 2981023 --- [Collect all tenants major compactions6,186791e284127fb8,e727c49ed2eda413] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:26:55.826 INFO 2981023 --- [ocp-schedules-27,186791e284127fb8,337af66b009652b9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:26:56.043 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:56.043 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,6b657be53789c12b,743119419ba58e19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,6b657be53789c12b,743119419ba58e19] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:26:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:26:56.352250+08:00, lastProcessTime=2024-09-11T17:26:46.339021+08:00 2024-09-11 17:26:56.494 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:26:56.352250+08:00 2024-09-11 17:26:57.156 INFO 2981023 --- [ocp-async-6,d4f9ef98dbaf4d1a,0b31035c3903a4bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:57.746 INFO 2981023 --- [ocp-async-2,65c0349dd8041286,cce305177f028ab4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:26:58.200 INFO 2981023 --- [ocp-schedules-30,b7f8217cf012c53f,081cd2b5153abb33] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:26:58.280 WARN 2981023 --- [scanner-13483922-1,b7f8217cf012c53f,e9d54e8a2434429a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:26:59.044 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:26:59.044 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:26:59.156 INFO 2981023 --- [ocp-async-1,f69acdd81a23b44d,47617900e6737b5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:26:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,8ced2754d6776532,cc4b5fadb4d788e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046820, interval=60 2024-09-11 17:26:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,8ced2754d6776532,cc4b5fadb4d788e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:00.032 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:22:59.577, end=2024-09-11T17:25:00.026} 2024-09-11 17:27:00.034 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, 2024-09-11 17:27:00.035 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:27:00.036 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:59.577, end=2024-09-11T17:25:00.026},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:27:00.036 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:27:00.036 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:27:00.037 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:27:00.038 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:59.577, end=2024-09-11T17:25:00.026},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:27:00.038 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:27:00.038 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:27:00.038 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:27:00.040 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:59.577, end=2024-09-11T17:25:00.026},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:27:00.040 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:27:00.040 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:27:00.049 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:27:00.068 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420068000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.068Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.071 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:27:00.071 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.076 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.076 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.078 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.078 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.083 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.083 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.084 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:27:00.084 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.068, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.085 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:27:00.086 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:59.577, end=2024-09-11T17:25:00.026},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:27:00.086 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:27:00.086 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:27:00.095 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:27:00.095 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:27:00.095 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:59.577, end=2024-09-11T17:25:00.026},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:27:00.095 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:27:00.095 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:27:00.096 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:27:00.098 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:59.577, end=2024-09-11T17:25:00.026},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:27:00.098 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:27:00.098 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:27:00.144 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420144000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.144Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.147 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420147000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.147Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.150 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420150000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.150Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.155 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420155000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.155Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.160 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420160000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.160Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.160 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420160000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.160Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.163 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420163000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.163Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.165 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:27:00.165 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.166 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:27:00.166 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.168 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420168000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.168Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.173 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.173 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.173 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.173 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.207 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:27:00.207 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.210 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:27:00.210 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.210 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:27:00.210 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.214 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:27:00.214 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.216 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 59 2024-09-11 17:27:00.216 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.217 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.217 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.219 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:27:00.219 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.219 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.219 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.221 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.221 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.226 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.226 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.227 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.227 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.228 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.228 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.229 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.229 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.229 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.229 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.230 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.231 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.231 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.232 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.232 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.233 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.233 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.233 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.233 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.234 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.234 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.235 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.235 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.236 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.236 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.237 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.237 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.240 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.241 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.241 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.245 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.245 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.245 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.246 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.144, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.246 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.246 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.247 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.247 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.247 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.248 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.147, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.249 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.249 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.251 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.251 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.252 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.252 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.253 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.16, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.255 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.257 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.16, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.257 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.260 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.15, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.262 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.155, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.263 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.163, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.274 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.168, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.290 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420290000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.290Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.296 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420296000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.296Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.300 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420300000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84D9CB28179A8A6A515E8623A68C21C2, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.300Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.301 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:27:00.301 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.304 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420304000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.304Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.306 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420306000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.306Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.310 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420310000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.310Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.310 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.310 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.312 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:27:00.312 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.312 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:27:00.312 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.313 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 19 2024-09-11 17:27:00.313 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.316 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420316000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=2E0351B66C4102B9EDA81C0E5D431120, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.316Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.317 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.317 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.321 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:27:00.321 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.322 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 76 2024-09-11 17:27:00.322 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.324 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.324 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.325 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.325 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.326 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.326 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.328 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.328 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.328 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:27:00.328 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.328 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.328 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84D9CB28179A8A6A515E8623A68C21C2, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.331 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.331 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=2E0351B66C4102B9EDA81C0E5D431120, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.334 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.334 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.334 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.334 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.335 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.335 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.336 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.338 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.338 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.340 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.341 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.342 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.344 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420344000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.344Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.345 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.345 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.345 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.29, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.345 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.345 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.346 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.346 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.346 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.296, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.347 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.347 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=2E0351B66C4102B9EDA81C0E5D431120, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.348 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.348 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84D9CB28179A8A6A515E8623A68C21C2, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.350 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.351 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.351 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.351 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.352 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.354 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:27:00.354 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.356 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.304, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.357 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.357 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.359 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.316, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=2E0351B66C4102B9EDA81C0E5D431120, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.360 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.3, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=84D9CB28179A8A6A515E8623A68C21C2, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.364 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.364 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.365 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.365 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.366 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.367 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.368 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.368 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.371 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.306, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.371 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.31, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.372 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960420372000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:27:00.372Z, endTime=2024-09-11T09:25:00.026Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:27:00.376 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.376 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.380 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.382 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:27:00.382 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:27:00.383 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.344, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.388 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.388 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.389 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:27:00.389 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:27:00.393 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:27:00.393 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046700026000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:27:00.394 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:27:00.397 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:27:00.372, end=2024-09-11T17:25:00.026}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:27:00.397 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:27:01.156 INFO 2981023 --- [ocp-async-10,cc074311d85608c6,93628125359da400] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:01.787 INFO 2981023 --- [ocp-schedules-9,8a68b2930348a2ba,721084fa39199fd2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:27:02.044 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:02.045 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,cfd092288bd2aea0,e70b36b4caf42343] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,cfd092288bd2aea0,e70b36b4caf42343] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:03.156 INFO 2981023 --- [ocp-async-8,6c7ce08c6d56570b,e33854328a95378b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:05.045 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:05.046 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:05.156 INFO 2981023 --- [ocp-async-9,e2193263187d12a9,4011ff354264817b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,fbff2983a328cdac,6b705bc04b002e49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,fbff2983a328cdac,6b705bc04b002e49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:27:06.336763+08:00, lastProcessTime=2024-09-11T17:26:56.352250+08:00 2024-09-11 17:27:06.470 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:27:06.336763+08:00 2024-09-11 17:27:06.755 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6a2ed10bd4854db4,a6c43a815be20634] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=6a2ed10bd4854db4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:27:06.755 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ec947669e82172aa,f15244402074f694] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=ec947669e82172aa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:27:06.756 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6a2ed10bd4854db4,a6c43a815be20634] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=6a2ed10bd4854db4, duration=5 ms] 2024-09-11 17:27:06.760 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ec947669e82172aa,f15244402074f694] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=ec947669e82172aa, duration=10 ms] 2024-09-11 17:27:06.773 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,aef4c39e50795a5b,583d684fa1a31296] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=aef4c39e50795a5b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:27:06.774 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,aef4c39e50795a5b,583d684fa1a31296] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=aef4c39e50795a5b, duration=12 ms] 2024-09-11 17:27:07.156 INFO 2981023 --- [ocp-async-4,0f82bd8690dd7ef4,9021b670467d1f13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:07.201 INFO 2981023 --- [ocp-schedules-11,baf4551ac250e6d3,27df2c186d127e11] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:27:07.204 INFO 2981023 --- [ocp-schedules-9,d74ef42d55a826e2,f2011f9bd388b430] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:27:07.204+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:27:07.205 INFO 2981023 --- [ocp-schedules-9,d74ef42d55a826e2,f2011f9bd388b430] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:27:07.206 INFO 2981023 --- [ocp-schedules-9,d74ef42d55a826e2,f2011f9bd388b430] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:27:07.207 INFO 2981023 --- [ocp-schedules-9,d74ef42d55a826e2,f2011f9bd388b430] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:27:07.746 INFO 2981023 --- [ocp-async-1,31f59fcc5dfbad98,bdcb17579993b7c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:27:08.046 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:08.046 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,43daa09e4bcf2c6d,be8bdbb47fe2c6a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,43daa09e4bcf2c6d,be8bdbb47fe2c6a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:08.411 INFO 2981023 --- [ocp-schedules-28,23b94cbf2ef0c419,f4ca8de32a641eaf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:27:08.466 WARN 2981023 --- [scanner-13484115-1,23b94cbf2ef0c419,e9b091eeeb2f16b4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:27:08.478 INFO 2981023 --- [alarm-detect-1,23b94cbf2ef0c419,1e42f81ccbc58598] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643794, data is empty. 2024-09-11 17:27:08.482 INFO 2981023 --- [alarm-detect-1,23b94cbf2ef0c419,1e42f81ccbc58598] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643843, data is empty. 2024-09-11 17:27:08.486 INFO 2981023 --- [alarm-detect-1,23b94cbf2ef0c419,1e42f81ccbc58598] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643843, data is empty. 2024-09-11 17:27:08.514 INFO 2981023 --- [alarm-detect-7,23b94cbf2ef0c419,d43d34839c03336c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643794, data is empty. 2024-09-11 17:27:08.518 INFO 2981023 --- [alarm-detect-7,23b94cbf2ef0c419,d43d34839c03336c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643843, data is empty. 2024-09-11 17:27:08.520 INFO 2981023 --- [alarm-detect-14,23b94cbf2ef0c419,f0429ff2b73721a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643794, data is empty. 2024-09-11 17:27:08.522 INFO 2981023 --- [alarm-detect-7,23b94cbf2ef0c419,d43d34839c03336c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643843, data is empty. 2024-09-11 17:27:08.528 INFO 2981023 --- [alarm-detect-14,23b94cbf2ef0c419,f0429ff2b73721a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643843, data is empty. 2024-09-11 17:27:08.531 INFO 2981023 --- [alarm-detect-14,23b94cbf2ef0c419,f0429ff2b73721a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643843, data is empty. 2024-09-11 17:27:08.634 INFO 2981023 --- [alarm-detect-18,23b94cbf2ef0c419,393e31042e51f64b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643794, data is empty. 2024-09-11 17:27:08.638 INFO 2981023 --- [alarm-detect-18,23b94cbf2ef0c419,393e31042e51f64b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643843, data is empty. 2024-09-11 17:27:08.643 INFO 2981023 --- [alarm-detect-18,23b94cbf2ef0c419,393e31042e51f64b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643843, data is empty. 2024-09-11 17:27:09.156 INFO 2981023 --- [ocp-async-8,7e86c94bc56456ef,056c0d10b3b4761a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:10.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:27:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:27:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:27:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:27:11.046 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:11.047 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:11.156 INFO 2981023 --- [ocp-async-9,f695e5319e34b2c2,009709450ff80c14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,e51eee3e18000eff,34015dcf83eae9d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,e51eee3e18000eff,34015dcf83eae9d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:11.789 INFO 2981023 --- [ocp-schedules-15,739e08922083b1fe,b5b3a0af6c9c25d7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:27:12.455 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:27:12.466 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:12.470 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:12.527 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:12.528 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:12.539 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:27:12.539 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:27:12.550 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:12.551 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:12.560 INFO 2981023 --- [ocp-schedules-12,37a4d7be55e5999d,bbd05615777a9c64] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=5 2024-09-11 17:27:12.972 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046760, nodeCount=480, affectRows=480 2024-09-11 17:27:13.156 INFO 2981023 --- [ocp-async-4,e013d7ab68da7bca,9dc8d38918ee91b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:13.410 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046760, nodeCount=1000, affectRows=1000 2024-09-11 17:27:13.412 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046760, nodeCount=999, validCount=999 2024-09-11 17:27:13.507 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046760, nodeCount=1000, affectRows=1000 2024-09-11 17:27:13.509 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046760, nodeCount=1000, validCount=1000 2024-09-11 17:27:13.518 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046760, nodeCount=49, affectRows=49 2024-09-11 17:27:13.518 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046760, nodeCount=49, validCount=49 2024-09-11 17:27:13.581 INFO 2981023 --- [ocp-schedules-17,d70466e6b796481f,9e3d7add9c943808] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:27:13.586 INFO 2981023 --- [ocp-schedules-17,d70466e6b796481f,9e3d7add9c943808] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:27:13.586 INFO 2981023 --- [ocp-schedules-17,d70466e6b796481f,9e3d7add9c943808] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:27:14.047 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:14.048 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,a9994168f4b55fd2,7cc31de5b5457573] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,daab2d0e3aef9a32,60d5b93812f1dddd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:27:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,a9994168f4b55fd2,7cc31de5b5457573] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:14.754 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046760, nodeCount=480, validCount=480 2024-09-11 17:27:15.157 INFO 2981023 --- [ocp-async-6,12b1e77f93a934ca,b8b97d65f4ffe4d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:16.020 INFO 2981023 --- [ocp-schedules-25,ded80cbb226df93d,e968afe38ea6e968] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:27:16.021 INFO 2981023 --- [host-check-agent-status1,ded80cbb226df93d,fd0bb9d9976120c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:27:16.021 INFO 2981023 --- [host-check-agent-status2,ded80cbb226df93d,bc85b05e4ca3e4ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:27:16.021 INFO 2981023 --- [host-check-agent-status3,ded80cbb226df93d,25e5d5dfd8408a50] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:27:16.022 INFO 2981023 --- [ocp-schedules-15,0eddc6556c18fb29,1724afa168aac966] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:27:16.022 INFO 2981023 --- [host-check-agent-status4,ded80cbb226df93d,f799d81ca9f00d28] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:27:16.023 INFO 2981023 --- [host-check-clockdiff1,0eddc6556c18fb29,30ccd3aa629f23a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:27:16.023 INFO 2981023 --- [host-check-clockdiff2,0eddc6556c18fb29,a9b112a70c00601a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:27:16.023 INFO 2981023 --- [ocp-schedules-22,ff441fd5e8ce3eba,13c40e0e93608b0a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:27:16.024 INFO 2981023 --- [host-check-clockdiff3,0eddc6556c18fb29,13f8c63e457daf07] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:27:16.033 INFO 2981023 --- [host-check-clockdiff4,0eddc6556c18fb29,60863a10b93ddf16] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:27:16.040 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:27:16.049 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:27:16.049 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:27:16.049 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:27:16.052 INFO 2981023 --- [host-check-clockdiff2,0eddc6556c18fb29,a9b112a70c00601a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046836 0 0 , error= 2024-09-11 17:27:16.072 INFO 2981023 --- [host-check-clockdiff4,0eddc6556c18fb29,60863a10b93ddf16] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046836 0 0 , error= 2024-09-11 17:27:16.076 INFO 2981023 --- [host-check-agent-status4,ded80cbb226df93d,f799d81ca9f00d28] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:16.077 INFO 2981023 --- [host-check-agent-status3,ded80cbb226df93d,25e5d5dfd8408a50] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:27:16.078 INFO 2981023 --- [host-check-agent-status2,ded80cbb226df93d,bc85b05e4ca3e4ee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:16.078 INFO 2981023 --- [host-check-agent-status4,ded80cbb226df93d,f799d81ca9f00d28] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:16.078 INFO 2981023 --- [host-check-agent-status3,ded80cbb226df93d,25e5d5dfd8408a50] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:27:16.079 INFO 2981023 --- [host-check-agent-status1,ded80cbb226df93d,fd0bb9d9976120c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:16.079 INFO 2981023 --- [host-check-agent-status2,ded80cbb226df93d,bc85b05e4ca3e4ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:16.080 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:16.081 INFO 2981023 --- [host-check-agent-status1,ded80cbb226df93d,fd0bb9d9976120c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:16.082 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:16.082 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:27:16.082 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:16.085 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:16.085 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:16.085 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:27:16.088 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:16.121 INFO 2981023 --- [host-check-agent-status1,ded80cbb226df93d,fd0bb9d9976120c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:27:16.121 INFO 2981023 --- [host-check-agent-status3,ded80cbb226df93d,25e5d5dfd8408a50] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:27:16.121 INFO 2981023 --- [host-check-agent-status4,ded80cbb226df93d,f799d81ca9f00d28] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:27:16.123 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:27:16.125 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:27:16.125938397+08:00 2024-09-11 17:27:16.125 INFO 2981023 --- [host-check-agent-status1,ded80cbb226df93d,fd0bb9d9976120c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:27:16.125 INFO 2981023 --- [host-check-agent-status1,ded80cbb226df93d,fd0bb9d9976120c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:27:16.125 INFO 2981023 --- [host-check-agent-status1,ded80cbb226df93d,fd0bb9d9976120c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:27:16.126 INFO 2981023 --- [host-check-agent-status3,ded80cbb226df93d,25e5d5dfd8408a50] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:27:16.126 INFO 2981023 --- [host-check-agent-status3,ded80cbb226df93d,25e5d5dfd8408a50] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:27:16.126 INFO 2981023 --- [host-check-agent-status3,ded80cbb226df93d,25e5d5dfd8408a50] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:27:16.126 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:27:16.125+08:00 2024-09-11 17:27:16.128 INFO 2981023 --- [host-check-agent-status2,ded80cbb226df93d,bc85b05e4ca3e4ee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:27:16.128 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:27:16.125+08:00 and status AVAILABLE 2024-09-11 17:27:16.129 INFO 2981023 --- [host-check-agent-status4,ded80cbb226df93d,f799d81ca9f00d28] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:27:16.129 INFO 2981023 --- [host-check-agent-status4,ded80cbb226df93d,f799d81ca9f00d28] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:27:16.129 INFO 2981023 --- [host-check-agent-status4,ded80cbb226df93d,f799d81ca9f00d28] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:27:16.131 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:27:16.132 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:27:16.132044530+08:00 2024-09-11 17:27:16.132 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:27:16.132+08:00 2024-09-11 17:27:16.133 INFO 2981023 --- [host-check-agent-status2,ded80cbb226df93d,bc85b05e4ca3e4ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:27:16.133 INFO 2981023 --- [host-check-agent-status2,ded80cbb226df93d,bc85b05e4ca3e4ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:27:16.133 INFO 2981023 --- [host-check-agent-status2,ded80cbb226df93d,bc85b05e4ca3e4ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:27:16.133 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:27:16.134 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:27:16.134054442+08:00 2024-09-11 17:27:16.134 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:27:16.134+08:00 2024-09-11 17:27:16.135 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:27:16.132+08:00 and status AVAILABLE 2024-09-11 17:27:16.136 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:27:16.134+08:00 and status AVAILABLE 2024-09-11 17:27:16.140 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:27:16.149 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:27:16.161 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:27:16.164 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:27:16.165 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:27:16.187 INFO 2981023 --- [ocp-schedules-24,03358e6d1da28594,1affb882d1f75b45] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:27:16.188 INFO 2981023 --- [ocp-schedules-27,88df5095c5af7d3b,7e55c3fed3ba7800] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:27:16.188 INFO 2981023 --- [ocp-schedules-16,0df03918250eb197,496a4b0704cd9a91] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:27:16.188 INFO 2981023 --- [ocp-schedules-30,d5a7e99c7dc6d619,64fccc2392b60b80] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:27:16.190 INFO 2981023 --- [ocp-schedules-16,0df03918250eb197,496a4b0704cd9a91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:27:16.193 INFO 2981023 --- [ocp-schedules-24,03358e6d1da28594,1affb882d1f75b45] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:27:16.195 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:16.196 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:16.197 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:16.199 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:16.200 INFO 2981023 --- [ocp-schedules-27,88df5095c5af7d3b,7e55c3fed3ba7800] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:27:16.204 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:27:16.208 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:27:16.211 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:27:16.208462868+08:00 2024-09-11 17:27:16.220 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:27:16.211+08:00 2024-09-11 17:27:16.224 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:27:16.211+08:00 and status AVAILABLE 2024-09-11 17:27:16.225 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:27:16.229 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:27:16.253 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:27:16.255 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:27:16.266 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:27:16.270 INFO 2981023 --- [ocp-schedules-27,88df5095c5af7d3b,7e55c3fed3ba7800] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:27:16.272 INFO 2981023 --- [ocp-schedules-11,636ca74bf48da71a,5d9d65c58e9fe6e2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:27:16.282 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:27:16.292 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.294 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:16.297 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:16.300 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:16.304 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.304 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:16.305 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:27:16.305 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:27:16.307 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.308 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:16.309 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:27:16.319 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.319 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:16.320 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.321 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:16.322 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:27:16.322 INFO 2981023 --- [ocp-schedules-24,03358e6d1da28594,1affb882d1f75b45] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:27:16.327 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.327 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:16.336 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:27:16.341 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.341 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:27:16.341520+08:00, lastProcessTime=2024-09-11T17:27:06.336763+08:00 2024-09-11 17:27:16.342 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:27:16.342 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.343 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:27:16.347 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:27:16.360 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:27:16.362 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:27:16.370 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:27:16.370 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:27:16.375 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:27:16.385 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:27:16.385 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:27:16.387 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.388 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.388 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:27:16.392 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.393 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.393 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:16.394 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:27:16.394 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.397 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:27:16.397 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:27:16.401 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:27:16.413 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:16.413 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:27:16.415 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:27:16.416 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:27:16.418 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:27:16.425 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.426 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:27:16.426 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.426 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:16.427 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.430 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:27:16.430 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:27:16.436 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:16.440 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:16.440 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:27:16.440 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:27:16.441 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:27:16.447 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.450 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:27:16.452 INFO 2981023 --- [host-check-host-status3,ff441fd5e8ce3eba,bb5dd8716fd5a920] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:27:16.454 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.454 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:27:16.457 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:16.458 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:27:16.464 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:27:16.464 INFO 2981023 --- [host-check-host-status2,ff441fd5e8ce3eba,02d15f690455f476] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:27:16.464 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:27:16.467 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.468 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:16.470 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:27:16.470 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:16.471 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:16.473 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.474 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:27:16.474 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:27:16.475 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.477 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:27:16.480 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:16.481 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:16.490 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:27:16.497 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:27:16.501 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:27:16.512 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.512 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:27:16.512 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.513 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:16.515 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:27:16.520 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:27:16.521 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.521 INFO 2981023 --- [host-check-host-status1,ff441fd5e8ce3eba,5de03c3a13e3d692] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:27:16.523 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:27:16.524 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:27:16.531 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:16.531 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:27:16.538 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:27:16.548 INFO 2981023 --- [host-check-host-status4,ff441fd5e8ce3eba,a77a482e0ba4180c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:27:16.552 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:16.562 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:27:16.567 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:27:16.570 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:16.572 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:27:16.580 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:27:16.582 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.583 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.583 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:27:16.589 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:27:16.590 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:27:16.590 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:27:16.591 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:27:16.598 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:27:16.598 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:27:16.617 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:27:16.341520+08:00 2024-09-11 17:27:16.619 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:27:16.623 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:16.624 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:16.624 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:27:16.630 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:27:16.630 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:27:16.631 INFO 2981023 --- [ocp-schedules-16,0df03918250eb197,496a4b0704cd9a91] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:27:16.970 INFO 2981023 --- [host-check-clockdiff1,0eddc6556c18fb29,30ccd3aa629f23a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046836 0 0 , error= 2024-09-11 17:27:17.048 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:17.049 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:17.157 INFO 2981023 --- [ocp-async-7,d5c732f60c1871ff,e7d2042c97eccf24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,7992ca54c30decd4,771f76ac007b586e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,7992ca54c30decd4,771f76ac007b586e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:17.747 INFO 2981023 --- [ocp-async-8,a1bc29ebe915a554,2ce90cecac47b404] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:27:18.679 INFO 2981023 --- [ocp-schedules-21,43699885ac8acfea,b7f960509cbc94f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:27:18.731 WARN 2981023 --- [scanner-13485984-1,43699885ac8acfea,3bac11625a9eae4c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:27:19.157 INFO 2981023 --- [ocp-async-5,9bf79f8eb8fa5646,5e5ef41661f9fb10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:20.049 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:20.051 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,723592c41e64b024,28b0905a848eb2e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,723592c41e64b024,28b0905a848eb2e1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:21.157 INFO 2981023 --- [ocp-async-2,84ad325dfd4d13b1,f67cbf76dca7a353] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:21.794 INFO 2981023 --- [ocp-schedules-6,0843aea87ab1695c,74a333d5c389f156] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:27:21.968 INFO 2981023 --- [host-check-clockdiff3,0eddc6556c18fb29,13f8c63e457daf07] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046841 1 1 , error= 2024-09-11 17:27:22.375 INFO 2981023 --- [ocp-schedules-12,17db804e12e9105d,5085cbd1fbd58754] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:27:22.375 INFO 2981023 --- [ocp-schedules-12,17db804e12e9105d,5085cbd1fbd58754] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:27:22.376 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:27:22.376 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:27:22.376 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:27:22.380 INFO 2981023 --- [ocp-schedules-12,17db804e12e9105d,5085cbd1fbd58754] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:27:22.380 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:22.380 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:22.380 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:22.384 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:22.384 INFO 2981023 --- [ocp-schedules-12,17db804e12e9105d,5085cbd1fbd58754] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:27:22.384 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:22.386 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:22.409 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:27:22.410 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:27:22.411 INFO 2981023 --- [ocp-schedules-12,17db804e12e9105d,5085cbd1fbd58754] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:27:22.412 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:27:22.414 INFO 2981023 --- [ocp-schedules-12,17db804e12e9105d,5085cbd1fbd58754] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:27:23.051 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:23.051 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:23.157 INFO 2981023 --- [ocp-async-3,7dcd178626e635a4,3da1505a3234939c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,fee0538dc51ab560,40f16040ad235313] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,fee0538dc51ab560,40f16040ad235313] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:24.176 INFO 2981023 --- [ocp-schedules-3,97258b91fa05b440,a35e09984b1c0bae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:27:25.157 INFO 2981023 --- [ocp-async-1,504b7391c9d30518,0a50cddada7e9e11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:25.459 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:27:25.459 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:27:25.463 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:27:25.463 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:27:25.463 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:27:25.463 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:27:25.466 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:27:25.466 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:27:25.466 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:27:25.466 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:27:25.492 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:27:25.496 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:27:25.496 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:27:25.496 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:27:25.496 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:27:25.496 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:27:25.501 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:27:25.759 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:27:26.043 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.576910521S 2024-09-11 17:27:26.043 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:27:26.051 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:26.052 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:26.094 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:27:26.094 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:27:26.109 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:27:26.147 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:27:26.147 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:27:26.147 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:27:26.147 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:27:26.147 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:27:26.147 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:27:26.153 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:27:26.243 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] o.s.boot.SpringApplication : Started application in 0.779 seconds (JVM running for 82646.115) 2024-09-11 17:27:26.243 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.78010872S 2024-09-11 17:27:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,284cc0dc169276d1,2147d8c9e11c89ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:26.318 INFO 2981023 --- [prometheus-minute-schedule-1,284cc0dc169276d1,2147d8c9e11c89ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:26.360 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:27:26.360435+08:00, lastProcessTime=2024-09-11T17:27:16.341520+08:00 2024-09-11 17:27:26.501 INFO 2981023 --- [ocp-schedules-16,b6b6413acf185299,f943db1b6f95eb2c] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:27:26.509 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:27:26.360435+08:00 2024-09-11 17:27:27.157 INFO 2981023 --- [ocp-async-6,e3d22618a4ccd4bd,c13ade304c920dfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:27.747 INFO 2981023 --- [ocp-async-1,ba989cddd470145b,12560518d26b0a71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:27:28.872 INFO 2981023 --- [ocp-schedules-11,da5917750bca4c06,b86a47743ce0b338] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:27:28.928 WARN 2981023 --- [scanner-13486204-1,da5917750bca4c06,dca369dccaa4c29b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:27:28.941 INFO 2981023 --- [alarm-detect-16,da5917750bca4c06,d3394c8720aed37e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643843, data is empty. 2024-09-11 17:27:28.945 INFO 2981023 --- [alarm-detect-16,da5917750bca4c06,d3394c8720aed37e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643843, data is empty. 2024-09-11 17:27:28.949 INFO 2981023 --- [alarm-detect-16,da5917750bca4c06,d3394c8720aed37e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643843, data is empty. 2024-09-11 17:27:28.970 INFO 2981023 --- [alarm-detect-1,da5917750bca4c06,2497b73acfee2652] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643843, data is empty. 2024-09-11 17:27:28.972 INFO 2981023 --- [alarm-detect-1,da5917750bca4c06,2497b73acfee2652] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643843, data is empty. 2024-09-11 17:27:28.975 INFO 2981023 --- [alarm-detect-3,da5917750bca4c06,500cdd53d71f0830] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643843, data is empty. 2024-09-11 17:27:28.976 INFO 2981023 --- [alarm-detect-1,da5917750bca4c06,2497b73acfee2652] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643843, data is empty. 2024-09-11 17:27:28.981 INFO 2981023 --- [alarm-detect-3,da5917750bca4c06,500cdd53d71f0830] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643843, data is empty. 2024-09-11 17:27:28.985 INFO 2981023 --- [alarm-detect-3,da5917750bca4c06,500cdd53d71f0830] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643843, data is empty. 2024-09-11 17:27:29.052 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:29.053 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:29.093 INFO 2981023 --- [alarm-detect-4,da5917750bca4c06,bf5dbbda94f027df] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643843, data is empty. 2024-09-11 17:27:29.097 INFO 2981023 --- [alarm-detect-4,da5917750bca4c06,bf5dbbda94f027df] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643843, data is empty. 2024-09-11 17:27:29.101 INFO 2981023 --- [alarm-detect-4,da5917750bca4c06,bf5dbbda94f027df] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643843, data is empty. 2024-09-11 17:27:29.157 INFO 2981023 --- [ocp-async-3,03879ac3f05f21a3,5c3eea82b2c3799f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,ade030fa1ec7a89d,7d4be2b3dd3f5ee3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:29.315 INFO 2981023 --- [prometheus-minute-schedule-1,ade030fa1ec7a89d,7d4be2b3dd3f5ee3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:31.157 INFO 2981023 --- [ocp-async-4,aff2f427ecef1202,1079191759cfe6c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:31.796 INFO 2981023 --- [ocp-schedules-1,23caf890edabed76,c20f6a526f93928f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:27:32.053 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:32.053 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,9cdea6dde625e84c,9ccd44089a287b92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,9cdea6dde625e84c,9ccd44089a287b92] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:33.157 INFO 2981023 --- [ocp-async-1,545f4aeae8bef740,870514a05845fef3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:33.589 INFO 2981023 --- [ocp-schedules-24,371436cbb5fcff1e,09c9e8cc42dcf105] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:27:33.597 INFO 2981023 --- [ocp-schedules-24,371436cbb5fcff1e,09c9e8cc42dcf105] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:27:33.597 INFO 2981023 --- [ocp-schedules-24,371436cbb5fcff1e,09c9e8cc42dcf105] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:27:35.054 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:35.054 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:35.158 INFO 2981023 --- [ocp-async-8,35193359d81e7444,ac33f7e3a281fca4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,aeb404be561f0650,890df969395e76c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,aeb404be561f0650,890df969395e76c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:27:36.336128+08:00, lastProcessTime=2024-09-11T17:27:26.360435+08:00 2024-09-11 17:27:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:27:36.336128+08:00 2024-09-11 17:27:37.158 INFO 2981023 --- [ocp-async-3,045517a71a02c895,815bf76573d8d359] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:37.314 INFO 2981023 --- [prometheus-cache-monitor1,2606335541023663,97093e494d297a44] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:27:37.315 INFO 2981023 --- [prometheus-cache-monitor1,2606335541023663,97093e494d297a44] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:27:37.747 INFO 2981023 --- [ocp-async-8,c3a1f71f28fde4d9,632ddc3d7f7b6747] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:27:38.054 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:38.054 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:38.116 INFO 2981023 --- [ocp-schedules-16,319c716a20ea91ad,09b77e9b21e40ad9] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:27:38.134 WARN 2981023 --- [ocp-schedules-16,319c716a20ea91ad,09b77e9b21e40ad9] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:27:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,77b956aa17916a7d,b619b3d06d3413f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,77b956aa17916a7d,b619b3d06d3413f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:39.143 INFO 2981023 --- [ocp-schedules-2,6ed24ee4b20aa861,192235c958ab94ff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:27:39.158 INFO 2981023 --- [ocp-async-10,0915e8a415a57096,aa9bca42652d3996] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:39.194 WARN 2981023 --- [scanner-13486514-1,6ed24ee4b20aa861,6c1bb5037815b76d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:27:40.003 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:27:40.004 INFO 2981023 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:27:41.054 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:41.055 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:41.158 INFO 2981023 --- [ocp-async-2,456549fa2b8cc1ed,d25230dd84274087] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,157dabef84884719,53f1bd6aef0e1b76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,157dabef84884719,53f1bd6aef0e1b76] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:41.799 INFO 2981023 --- [ocp-schedules-18,17ccad233d82a504,f55364bb78e44e86] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:27:42.605 INFO 2981023 --- [ocp-schedules-15,20758edc78f4ab55,696b20d9e0a224da] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:27:43.158 INFO 2981023 --- [ocp-async-9,317bedfde594fe75,f721e8b73f4ad6de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:44.055 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:44.055 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,89b26fd26840cf8d,d9d39cba0d50258b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,89b26fd26840cf8d,d9d39cba0d50258b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:45.101 INFO 2981023 --- [prometheus-second-schedule-1,1c850ae8f1b0bf84,feecae61a02c93ca] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:27:45.158 INFO 2981023 --- [ocp-async-1,f2b587a3fff78967,03b5b8732cdf31ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:46.018 INFO 2981023 --- [ocp-schedules-30,939e5dbe5881516f,22d430b7a0cf5a75] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:27:46.018 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:27:46.018 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:27:46.019 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:27:46.019 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:27:46.023 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:46.024 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:46.030 INFO 2981023 --- [ocp-schedules-23,bc0e7198801684be,20ca70e705811164] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:27:46.030 INFO 2981023 --- [ocp-schedules-27,bdeb8fcb7e52a803,abb3c40ee9fcf0d9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:27:46.030 INFO 2981023 --- [host-check-agent-status1,bdeb8fcb7e52a803,c325c0418837e03a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:27:46.031 INFO 2981023 --- [host-check-agent-status2,bdeb8fcb7e52a803,24907e2b1d9ad9a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:27:46.031 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:46.031 INFO 2981023 --- [host-check-agent-status3,bdeb8fcb7e52a803,ae12e70ab9a44857] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:27:46.033 INFO 2981023 --- [host-check-agent-status4,bdeb8fcb7e52a803,6a7598a25a0ec6e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:27:46.033 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:27:46.034 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:46.034 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:46.034 INFO 2981023 --- [host-check-clockdiff1,bc0e7198801684be,705bb212106f6979] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:27:46.035 INFO 2981023 --- [host-check-clockdiff3,bc0e7198801684be,87b5e46ac4f62913] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:27:46.036 INFO 2981023 --- [host-check-clockdiff4,bc0e7198801684be,05a87e8f9e1b69de] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:27:46.036 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:27:46.036 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:46.038 INFO 2981023 --- [host-check-clockdiff2,bc0e7198801684be,ab9c7242d39d8e18] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:27:46.041 INFO 2981023 --- [host-check-agent-status1,bdeb8fcb7e52a803,c325c0418837e03a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:46.043 INFO 2981023 --- [host-check-agent-status3,bdeb8fcb7e52a803,ae12e70ab9a44857] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:27:46.043 INFO 2981023 --- [host-check-agent-status1,bdeb8fcb7e52a803,c325c0418837e03a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:46.045 INFO 2981023 --- [host-check-agent-status4,bdeb8fcb7e52a803,6a7598a25a0ec6e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:46.045 INFO 2981023 --- [host-check-agent-status3,bdeb8fcb7e52a803,ae12e70ab9a44857] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:27:46.047 INFO 2981023 --- [host-check-agent-status4,bdeb8fcb7e52a803,6a7598a25a0ec6e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:46.050 INFO 2981023 --- [host-check-agent-status2,bdeb8fcb7e52a803,24907e2b1d9ad9a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:46.058 INFO 2981023 --- [host-check-clockdiff4,bc0e7198801684be,05a87e8f9e1b69de] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046866 0 0 , error= 2024-09-11 17:27:46.058 INFO 2981023 --- [host-check-agent-status2,bdeb8fcb7e52a803,24907e2b1d9ad9a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:46.058 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:27:46.061 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:27:46.060983522+08:00 2024-09-11 17:27:46.062 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:27:46.061+08:00 2024-09-11 17:27:46.067 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:27:46.061+08:00 and status AVAILABLE 2024-09-11 17:27:46.079 INFO 2981023 --- [host-check-agent-status1,bdeb8fcb7e52a803,c325c0418837e03a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:27:46.080 INFO 2981023 --- [host-check-agent-status3,bdeb8fcb7e52a803,ae12e70ab9a44857] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:27:46.080 INFO 2981023 --- [host-check-agent-status4,bdeb8fcb7e52a803,6a7598a25a0ec6e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:27:46.084 INFO 2981023 --- [host-check-agent-status3,bdeb8fcb7e52a803,ae12e70ab9a44857] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:27:46.084 INFO 2981023 --- [host-check-agent-status3,bdeb8fcb7e52a803,ae12e70ab9a44857] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:27:46.084 INFO 2981023 --- [host-check-agent-status3,bdeb8fcb7e52a803,ae12e70ab9a44857] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:27:46.086 INFO 2981023 --- [host-check-agent-status4,bdeb8fcb7e52a803,6a7598a25a0ec6e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:27:46.086 INFO 2981023 --- [host-check-agent-status4,bdeb8fcb7e52a803,6a7598a25a0ec6e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:27:46.086 INFO 2981023 --- [host-check-agent-status4,bdeb8fcb7e52a803,6a7598a25a0ec6e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:27:46.087 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:27:46.087 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:27:46.088 INFO 2981023 --- [host-check-agent-status1,bdeb8fcb7e52a803,c325c0418837e03a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:27:46.088 INFO 2981023 --- [host-check-agent-status1,bdeb8fcb7e52a803,c325c0418837e03a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:27:46.088 INFO 2981023 --- [host-check-agent-status1,bdeb8fcb7e52a803,c325c0418837e03a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:27:46.089 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:27:46.088857253+08:00 2024-09-11 17:27:46.089 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:27:46.089+08:00 2024-09-11 17:27:46.091 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:27:46.089+08:00 and status AVAILABLE 2024-09-11 17:27:46.093 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:27:46.093 INFO 2981023 --- [host-check-agent-status2,bdeb8fcb7e52a803,24907e2b1d9ad9a9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:27:46.107 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:27:46.109 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:27:46.109381614+08:00 2024-09-11 17:27:46.109 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:27:46.109 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:27:46.103987660+08:00 2024-09-11 17:27:46.110 INFO 2981023 --- [host-check-agent-status2,bdeb8fcb7e52a803,24907e2b1d9ad9a9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:27:46.110 INFO 2981023 --- [host-check-agent-status2,bdeb8fcb7e52a803,24907e2b1d9ad9a9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:27:46.110 INFO 2981023 --- [host-check-agent-status2,bdeb8fcb7e52a803,24907e2b1d9ad9a9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:27:46.111 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:27:46.109+08:00 2024-09-11 17:27:46.112 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:27:46.109+08:00 and status AVAILABLE 2024-09-11 17:27:46.113 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:27:46.110+08:00 2024-09-11 17:27:46.115 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:27:46.110+08:00 and status AVAILABLE 2024-09-11 17:27:46.116 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:46.117 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:46.120 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:46.121 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:46.121 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:27:46.125 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:27:46.131 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:27:46.134 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:27:46.138 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:46.139 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:46.146 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:27:46.146 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:27:46.155 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:27:46.157 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:27:46.158 INFO 2981023 --- [ocp-schedules-22,31619caf3fb868a4,2b481a1d137b8fc5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:27:46.163 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:27:46.165 INFO 2981023 --- [ocp-schedules-22,31619caf3fb868a4,2b481a1d137b8fc5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:27:46.175 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:27:46.176 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:27:46.182 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:27:46.188 INFO 2981023 --- [ocp-schedules-22,7fb012bf4f4e991d,2103d94e21e45711] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:27:46.189 INFO 2981023 --- [ocp-schedules-22,7fb012bf4f4e991d,2103d94e21e45711] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:27:46.191 INFO 2981023 --- [ocp-schedules-11,9ae9a32cd5e4bf56,c1756fa2e048be4d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:27:46.191 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:27:46.191 INFO 2981023 --- [ocp-schedules-7,9da64a57c4be42dc,d57694080f9ce401] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:27:46.192 INFO 2981023 --- [ocp-schedules-8,6717eaeb5f22f335,b0612d9c47151e63] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:27:46.192 INFO 2981023 --- [ocp-schedules-7,9da64a57c4be42dc,d57694080f9ce401] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:27:46.197 INFO 2981023 --- [ocp-schedules-11,9ae9a32cd5e4bf56,c1756fa2e048be4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:27:46.218 INFO 2981023 --- [ocp-schedules-11,9ae9a32cd5e4bf56,c1756fa2e048be4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:27:46.219 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:27:46.221 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:27:46.224 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:27:46.226 INFO 2981023 --- [ocp-schedules-5,8e6379be6188710f,31ca82bfc1dbedfd] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:27:46.264 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:27:46.271 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:27:46.288 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.288 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:46.291 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:27:46.303 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.303 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.303 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:46.303 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:46.304 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:27:46.305 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:27:46.305 INFO 2981023 --- [ocp-schedules-22,7fb012bf4f4e991d,2103d94e21e45711] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:27:46.312 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.313 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:46.314 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:27:46.314 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.314 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:46.324 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:27:46.327 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.328 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:27:46.328 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:27:46.328 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.335 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.336 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:46.341 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:27:46.341 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:27:46.340612+08:00, lastProcessTime=2024-09-11T17:27:36.336128+08:00 2024-09-11 17:27:46.344 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:27:46.348 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:27:46.349 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:27:46.361 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:27:46.374 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:27:46.384 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.388 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:27:46.389 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.389 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:27:46.397 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:27:46.411 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:27:46.413 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:27:46.416 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:27:46.418 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:27:46.418 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.419 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.419 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:27:46.419 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.423 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:27:46.423 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.424 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.424 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:27:46.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:46.425 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:27:46.425 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:27:46.425 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.434 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:27:46.435 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:27:46.437 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:27:46.447 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:27:46.449 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:46.449 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:27:46.451 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:27:46.456 INFO 2981023 --- [host-check-host-status3,939e5dbe5881516f,a62570a93f84e6f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:27:46.463 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:27:46.470 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:27:46.473 INFO 2981023 --- [host-check-host-status2,939e5dbe5881516f,d44f3e0adc6e2c52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:27:46.475 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.475 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.476 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:46.479 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.482 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:27:46.482 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:27:46.483 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:27:46.485 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.493 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:46.493 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:27:46.500 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:27:46.504 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:46.504 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:27:46.505 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:27:46.506 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:27:46.507 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:27:46.510 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.511 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:27:46.511 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.512 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:27:46.514 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.516 INFO 2981023 --- [host-check-host-status4,939e5dbe5881516f,471b20f25ff7c6b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:27:46.522 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:46.522 INFO 2981023 --- [host-check-host-status1,939e5dbe5881516f,7c88edc47458c2c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:27:46.524 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.524 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.524 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:27:46.525 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:27:46.526 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:27:46.528 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.529 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:27:46.529 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:27:46.530 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.536 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:27:46.540 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.541 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.542 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:46.543 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:27:46.547 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:27:46.555 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:27:46.557 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.557 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:27:46.558 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.558 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:27:46.559 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:27:46.559 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.559 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:27:46.564 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:27:46.565 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:27:46.572 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:46.572 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:27:46.581 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:27:46.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:27:46.590 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.591 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.591 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:27:46.592 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:27:46.592 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.596 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:27:46.600 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.601 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.601 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:27:46.602 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:27:46.606 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:27:46.607 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:27:46.610 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:27:46.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:27:46.626 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:27:46.626 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:27:46.629 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.632 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.632 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:27:46.639 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:27:46.639 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:27:46.643 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:27:46.644 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.644 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.645 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:27:46.645 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:27:46.645 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.654 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:27:46.657 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.658 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.658 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:27:46.663 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:46.664 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:46.664 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:27:46.666 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:27:46.666 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:27:46.669 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:27:46.340612+08:00 2024-09-11 17:27:46.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:27:46.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:27:46.671 INFO 2981023 --- [ocp-schedules-7,9da64a57c4be42dc,d57694080f9ce401] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:27:46.692 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:27:46.709 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:27:46.714 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:27:46.717 INFO 2981023 --- [ocp-schedules-21,0b022c608d6097cb,d560db6c9d127bf4] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:27:46.836 INFO 2981023 --- [ocp-schedules-25,025fc4ca555c1525,4d5080c19c0def2f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:27:46.836 INFO 2981023 --- [ocp-schedules-25,025fc4ca555c1525,4d5080c19c0def2f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:27:46.983 INFO 2981023 --- [host-check-clockdiff1,bc0e7198801684be,705bb212106f6979] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046866 0 0 , error= 2024-09-11 17:27:47.055 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:47.056 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:47.158 INFO 2981023 --- [ocp-async-7,4d26fa18b438b203,85930ae4e86693c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,8b7e15e3f9f641f4,1846ccb2d1694bfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,8b7e15e3f9f641f4,1846ccb2d1694bfe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:47.747 INFO 2981023 --- [ocp-async-8,f88ca5dba17a98a2,ab12be8d4a1d1706] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:27:47.834 INFO 2981023 --- [host-check-clockdiff2,bc0e7198801684be,ab9c7242d39d8e18] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046867 0 0 , error= 2024-09-11 17:27:49.158 INFO 2981023 --- [ocp-async-4,cf5745fe10d157bd,9a73747a3bedec8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:49.332 INFO 2981023 --- [ocp-schedules-17,4dce2a5fb6b8c16b,77c49a6d8f178d3c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:27:49.374 WARN 2981023 --- [scanner-13486730-1,4dce2a5fb6b8c16b,4d79793f3ebf4024] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:27:49.398 INFO 2981023 --- [alarm-detect-14,4dce2a5fb6b8c16b,0aacba03111c8d1d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643843, data is empty. 2024-09-11 17:27:49.400 INFO 2981023 --- [alarm-detect-14,4dce2a5fb6b8c16b,0aacba03111c8d1d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643843, data is empty. 2024-09-11 17:27:49.403 INFO 2981023 --- [alarm-detect-14,4dce2a5fb6b8c16b,0aacba03111c8d1d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643843, data is empty. 2024-09-11 17:27:49.429 INFO 2981023 --- [alarm-detect-12,4dce2a5fb6b8c16b,f8334b4cac89bb3f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643843, data is empty. 2024-09-11 17:27:49.431 INFO 2981023 --- [alarm-detect-11,4dce2a5fb6b8c16b,5bad79fd9d4c292e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643843, data is empty. 2024-09-11 17:27:49.433 INFO 2981023 --- [alarm-detect-12,4dce2a5fb6b8c16b,f8334b4cac89bb3f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643843, data is empty. 2024-09-11 17:27:49.435 INFO 2981023 --- [alarm-detect-11,4dce2a5fb6b8c16b,5bad79fd9d4c292e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643843, data is empty. 2024-09-11 17:27:49.436 INFO 2981023 --- [alarm-detect-12,4dce2a5fb6b8c16b,f8334b4cac89bb3f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643843, data is empty. 2024-09-11 17:27:49.440 INFO 2981023 --- [alarm-detect-11,4dce2a5fb6b8c16b,5bad79fd9d4c292e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643843, data is empty. 2024-09-11 17:27:49.554 INFO 2981023 --- [alarm-detect-5,4dce2a5fb6b8c16b,9b4fc9413f4d4045] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643843, data is empty. 2024-09-11 17:27:49.559 INFO 2981023 --- [alarm-detect-5,4dce2a5fb6b8c16b,9b4fc9413f4d4045] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643843, data is empty. 2024-09-11 17:27:49.562 INFO 2981023 --- [alarm-detect-5,4dce2a5fb6b8c16b,9b4fc9413f4d4045] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643843, data is empty. 2024-09-11 17:27:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:27:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:27:50.056 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:50.056 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:50.205 INFO 2981023 --- [ocp-schedules-4,2f09a86907def07a,23cc737f85b071cb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:27:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,0752259669bd9222,806fb9265c4c9cb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,0752259669bd9222,806fb9265c4c9cb7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:51.158 INFO 2981023 --- [ocp-async-6,3b6d6723a8f7b41a,13c258cf74aac95f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:51.219 INFO 2981023 --- [ocp-schedules-24,84f4ea783fd6bfbf,1e3e7cd1bbd41f58] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:27:51.219+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:27:51.223 INFO 2981023 --- [ocp-schedules-24,84f4ea783fd6bfbf,1e3e7cd1bbd41f58] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:27:51.304 INFO 2981023 --- [ocp-schedules-24,84f4ea783fd6bfbf,1e3e7cd1bbd41f58] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 45 to SqlAggregateData 2024-09-11 17:27:51.306 INFO 2981023 --- [ocp-schedules-24,84f4ea783fd6bfbf,1e3e7cd1bbd41f58] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:27:51.511 INFO 2981023 --- [ocp-schedules-30,ec678da22a78a0a2,5e366918d5d5d98f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:27:51.512 INFO 2981023 --- [ocp-schedules-30,ec678da22a78a0a2,5e366918d5d5d98f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:27:51.513 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:27:51.513 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:27:51.513 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:27:51.513 INFO 2981023 --- [ocp-schedules-30,ec678da22a78a0a2,5e366918d5d5d98f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:27:51.798 INFO 2981023 --- [ocp-schedules-29,29548f593f1e2260,b58da10da9b50272] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:27:51.799 INFO 2981023 --- [ocp-schedules-29,29548f593f1e2260,b58da10da9b50272] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:27:51.802 INFO 2981023 --- [ocp-schedules-2,d668cab539d3f093,316f63adb166b251] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:27:51.830 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:27:51.830 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.830 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:27:51.831 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.831 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:27:51.832 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.833 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:27:51.835 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:27:51.835 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:27:51.839 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.839 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.839 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.839 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.840 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.840 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.845 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.845 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.851 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.851 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.851 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.852 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.852 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.852 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:27:51.866 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.867 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.867 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:27:51.867 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.867 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:27:51.868 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1F51D6E1-0-0] 2024-09-11 17:27:51.868 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:27:51.868 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.869 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:27:51.869 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.869 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.873 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.873 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:27:51.880 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.880 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:27:51.884 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.884 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:27:51.890 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.891 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:27:51.892 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.892 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:27:51.893 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.893 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.894 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:27:51.894 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.894 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:27:51.895 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:27:51.895 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:27:51.895 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.896 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:27:51.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.897 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.897 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:27:51.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:27:51.898 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.898 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:27:51.898 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE1F51D6EB-0-0] 2024-09-11 17:27:51.898 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:27:51.898 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:27:51.899 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.900 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:27:51.900 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.901 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:27:51.901 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.902 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:27:51.903 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:27:51.903 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.904 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:27:51.904 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:27:51.905 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.905 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:27:51.906 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.906 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:27:51.907 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:27:51.908 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.908 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:27:51.909 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.910 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:27:51.910 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.911 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:27:51.912 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.913 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:27:51.913 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:27:51.915 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:27:51.915 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:51.917 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.918 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.919 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:51.920 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:27:51.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:27:51.933 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:27:51.934 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:27:51.934 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:27:51.934 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.936 INFO 2981023 --- [host-check-clockdiff3,bc0e7198801684be,87b5e46ac4f62913] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046871 1 1 , error= 2024-09-11 17:27:51.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:27:51.939 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.939 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:27:51.940 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:27:51.940 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:27:51.940 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:27:51.942 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:27:51.943 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:27:51.944 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.944 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.950 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:27:51.952 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:27:51.952 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:27:51.952 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:27:51.952 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:27:51.953 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:27:51.953 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:51.954 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:27:51.965 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:27:51.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:27:51.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:27:51.966 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:27:51.966 INFO 2981023 --- [ocp-schedules-29,29548f593f1e2260,b58da10da9b50272] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:27:52.285 INFO 2981023 --- [ocp-schedules-20,548dba02c6713b61,d9afe7b5b80a9019] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:27:52.301 WARN 2981023 --- [ocp-schedules-20,548dba02c6713b61,d9afe7b5b80a9019] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:27:53.036 INFO 2981023 --- [ocp-schedules-7,e10b428b0f31fd93,78b8cc0ff5eab07b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:27:53.036 INFO 2981023 --- [ocp-schedules-7,e10b428b0f31fd93,78b8cc0ff5eab07b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:27:53.057 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:53.058 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:53.159 INFO 2981023 --- [ocp-async-2,e991cbe9e2c6ac37,ab6763accf975d0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,b95e3567720ed37f,14b5772b7d71787d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,b95e3567720ed37f,14b5772b7d71787d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:53.601 INFO 2981023 --- [ocp-schedules-5,93c784db1e2c55ea,3b32922cfd514b67] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:27:53.611 INFO 2981023 --- [ocp-schedules-5,93c784db1e2c55ea,3b32922cfd514b67] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:27:53.611 INFO 2981023 --- [ocp-schedules-5,93c784db1e2c55ea,3b32922cfd514b67] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:27:54.180 INFO 2981023 --- [ocp-schedules-8,c08fc5f10a456b5b,8696d11763a170f7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:27:54.699 INFO 2981023 --- [ocp-schedules-3,952cd135f5dc7345,24eee135c595e6fc] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:27:54.699 INFO 2981023 --- [ocp-schedules-3,952cd135f5dc7345,24eee135c595e6fc] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:27:54.704 INFO 2981023 --- [ocp-schedules-3,952cd135f5dc7345,24eee135c595e6fc] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:27:54.704 INFO 2981023 --- [ocp-schedules-3,952cd135f5dc7345,24eee135c595e6fc] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:27:54.704 INFO 2981023 --- [ocp-schedules-3,952cd135f5dc7345,24eee135c595e6fc] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:27:54.706 INFO 2981023 --- [ocp-schedules-3,952cd135f5dc7345,24eee135c595e6fc] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:27:54.706 INFO 2981023 --- [ocp-schedules-3,952cd135f5dc7345,24eee135c595e6fc] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:27:54.706 INFO 2981023 --- [ocp-schedules-3,952cd135f5dc7345,24eee135c595e6fc] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:27:54.717 INFO 2981023 --- [ocp-schedules-3,952cd135f5dc7345,24eee135c595e6fc] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:27:55.159 INFO 2981023 --- [ocp-async-3,48a7fd3c63bbeda0,d383b1d5c72223ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:55.631 INFO 2981023 --- [ocp-schedules-30,d9df10c3e0b28d2f,f4d2c7e8d800f633] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:27:55.631 INFO 2981023 --- [ocp-schedules-30,d9df10c3e0b28d2f,f4d2c7e8d800f633] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:27:55.634 INFO 2981023 --- [ocp-schedules-30,d9df10c3e0b28d2f,f4d2c7e8d800f633] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:27:55.634 INFO 2981023 --- [ocp-schedules-30,d9df10c3e0b28d2f,f4d2c7e8d800f633] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:27:55.636 INFO 2981023 --- [ocp-schedules-30,d9df10c3e0b28d2f,f4d2c7e8d800f633] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:27:55.636 INFO 2981023 --- [ocp-schedules-30,d9df10c3e0b28d2f,f4d2c7e8d800f633] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:27:55.700 INFO 2981023 --- [ocp-schedules-8,eb46d367f9d5bf4e,2044d4d4773a9f79] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:27:55.706 INFO 2981023 --- [ocp-schedules-8,eb46d367f9d5bf4e,2044d4d4773a9f79] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:27:55.706 INFO 2981023 --- [ocp-schedules-8,eb46d367f9d5bf4e,2044d4d4773a9f79] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:27:55.826 INFO 2981023 --- [ocp-schedules-22,f52ef098824abce9,cfa12f7f67389ee6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:27:55.826 INFO 2981023 --- [ocp-schedules-22,f52ef098824abce9,cfa12f7f67389ee6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:27:55.845 INFO 2981023 --- [Collect all tenants major compactions2,f52ef098824abce9,e776a44b8ca86156] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:27:55.845 INFO 2981023 --- [Collect all tenants major compactions3,f52ef098824abce9,de9adbd0ccfdc1a5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:27:55.848 INFO 2981023 --- [Collect all tenants major compactions4,f52ef098824abce9,57601dd6883c1ae5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:27:55.848 INFO 2981023 --- [Collect all tenants major compactions1,f52ef098824abce9,64ed77985938ccf1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:27:55.848 INFO 2981023 --- [Collect all tenants major compactions6,f52ef098824abce9,1a1a1d29b4b949ce] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:27:55.849 INFO 2981023 --- [Collect all tenants major compactions5,f52ef098824abce9,4ec0b1500fa726c1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:27:55.867 INFO 2981023 --- [Collect all tenants major compactions4,f52ef098824abce9,57601dd6883c1ae5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:55.868 INFO 2981023 --- [Collect all tenants major compactions4,f52ef098824abce9,57601dd6883c1ae5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:27:55.869 INFO 2981023 --- [Collect all tenants major compactions3,f52ef098824abce9,de9adbd0ccfdc1a5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:55.870 INFO 2981023 --- [Collect all tenants major compactions2,f52ef098824abce9,e776a44b8ca86156] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:55.871 INFO 2981023 --- [Collect all tenants major compactions1,f52ef098824abce9,64ed77985938ccf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:55.871 INFO 2981023 --- [Collect all tenants major compactions3,f52ef098824abce9,de9adbd0ccfdc1a5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:27:55.871 INFO 2981023 --- [Collect all tenants major compactions2,f52ef098824abce9,e776a44b8ca86156] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:27:55.873 INFO 2981023 --- [Collect all tenants major compactions4,f52ef098824abce9,57601dd6883c1ae5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:27:55.874 INFO 2981023 --- [Collect all tenants major compactions3,f52ef098824abce9,de9adbd0ccfdc1a5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:27:55.874 INFO 2981023 --- [Collect all tenants major compactions2,f52ef098824abce9,e776a44b8ca86156] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:27:55.874 INFO 2981023 --- [Collect all tenants major compactions5,f52ef098824abce9,4ec0b1500fa726c1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:55.875 INFO 2981023 --- [Collect all tenants major compactions5,f52ef098824abce9,4ec0b1500fa726c1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:27:55.875 INFO 2981023 --- [Collect all tenants major compactions1,f52ef098824abce9,64ed77985938ccf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:27:55.876 INFO 2981023 --- [Collect all tenants major compactions6,f52ef098824abce9,1a1a1d29b4b949ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:27:55.877 INFO 2981023 --- [Collect all tenants major compactions6,f52ef098824abce9,1a1a1d29b4b949ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:27:55.877 INFO 2981023 --- [Collect all tenants major compactions4,f52ef098824abce9,57601dd6883c1ae5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:27:55.878 INFO 2981023 --- [Collect all tenants major compactions1,f52ef098824abce9,64ed77985938ccf1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:27:55.878 INFO 2981023 --- [Collect all tenants major compactions1,f52ef098824abce9,64ed77985938ccf1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:27:55.878 INFO 2981023 --- [Collect all tenants major compactions5,f52ef098824abce9,4ec0b1500fa726c1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:27:55.878 INFO 2981023 --- [Collect all tenants major compactions5,f52ef098824abce9,4ec0b1500fa726c1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:27:55.879 INFO 2981023 --- [Collect all tenants major compactions2,f52ef098824abce9,e776a44b8ca86156] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:27:55.879 INFO 2981023 --- [Collect all tenants major compactions6,f52ef098824abce9,1a1a1d29b4b949ce] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:27:55.880 INFO 2981023 --- [Collect all tenants major compactions6,f52ef098824abce9,1a1a1d29b4b949ce] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:27:55.880 INFO 2981023 --- [Collect all tenants major compactions3,f52ef098824abce9,de9adbd0ccfdc1a5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:27:55.881 INFO 2981023 --- [ocp-schedules-22,f52ef098824abce9,cfa12f7f67389ee6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:27:56.058 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:56.058 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,18304bdf5dfbc29f,c8a85ee5d235216c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,18304bdf5dfbc29f,c8a85ee5d235216c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:27:56.351542+08:00, lastProcessTime=2024-09-11T17:27:46.340612+08:00 2024-09-11 17:27:56.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:27:56.351542+08:00 2024-09-11 17:27:57.159 INFO 2981023 --- [ocp-async-10,c5e12f78021bd15e,f9293941f53c07e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:57.747 INFO 2981023 --- [ocp-async-1,35d38a9cf9bbe909,5a32f03c1fd4b74e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:27:58.206 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.214 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdoc 2024-09-11 17:27:58.215 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.216 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.217 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.218 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.221 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.224 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.227 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.230 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.233 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.236 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.238 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.241 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.242 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.243 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.244 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.244 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.248 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.251 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.254 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.257 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.257 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.258 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.259 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.259 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:21:58.101, end=2024-09-11T17:26:58.202} 2024-09-11 17:27:58.259 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:27:59.059 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:27:59.060 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:27:59.159 INFO 2981023 --- [ocp-async-2,7a98f20a1866bb83,877caa702139f85a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:27:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,7decf5bd0060ed74,59de5bdbb99a3a36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046880, interval=60 2024-09-11 17:27:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,7decf5bd0060ed74,59de5bdbb99a3a36] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:27:59.601 INFO 2981023 --- [ocp-schedules-23,4073333f735c7a73,e34ff75dd4667930] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:27:59.654 WARN 2981023 --- [scanner-13487026-1,4073333f735c7a73,f4e0b685de3daa73] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:28:01.164 INFO 2981023 --- [ocp-async-3,813de315694210f2,b8dc96053d4de86e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:01.806 INFO 2981023 --- [ocp-schedules-24,5755b88db468d114,5b42ace9e146c664] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:28:02.060 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:02.061 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,d11cdcaa85df7460,145a2fa8c033f0c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,d11cdcaa85df7460,145a2fa8c033f0c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:02.371 INFO 2981023 --- [ocp-schedules-14,9956fed814e3f2db,09d7b95a881146ac] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:28:02.375 INFO 2981023 --- [ocp-schedules-14,9956fed814e3f2db,09d7b95a881146ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:28:02.414 INFO 2981023 --- [ocp-schedules-14,9956fed814e3f2db,09d7b95a881146ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:28:03.159 INFO 2981023 --- [ocp-async-6,7d677d08786f863a,9fc176b0ff46b306] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:05.061 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:05.063 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:05.159 INFO 2981023 --- [ocp-async-7,0532cfc0e8c54e6e,e8e2d0d4954234eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f2746801c8783a9d,c9601e1def58e7b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,f2746801c8783a9d,c9601e1def58e7b7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:28:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:28:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:28:05.992 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:28:05.997 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:28:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:28:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:28:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:28:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:28:06.009 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:28:06.009 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:28:06.010 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:06.010 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:06.010 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:28:06.011 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:28:06.036 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:25:41, start=2024-09-11T17:23:10, end=2024-09-11T17:25:40, updates=108, granule=120000ms, delay=151000ms 2024-09-11 17:28:06.051 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:25:41, target=2024-09-11T17:25:40, granule=120000ms, delay=1000ms) 2024-09-11 17:28:06.110 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - source=2024-09-11T17:25:40, start=2024-09-11T17:09:41, end=2024-09-11T17:19:41, updates=157, granule=600000ms, delay=959000ms 2024-09-11 17:28:06.115 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:25:40, target=2024-09-11T17:19:41, granule=600000ms, delay=359000ms) 2024-09-11 17:28:06.135 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:25:41, start=2024-09-11T17:22:41, end=2024-09-11T17:25:11, updates=293, granule=120000ms, delay=180000ms 2024-09-11 17:28:06.140 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:25:41, target=2024-09-11T17:25:11, granule=120000ms, delay=30000ms) 2024-09-11 17:28:06.222 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T17:25:11, start=2024-09-11T17:10:51, end=2024-09-11T17:20:51, updates=370, granule=600000ms, delay=860000ms 2024-09-11 17:28:06.236 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:25:11, target=2024-09-11T17:20:51, granule=600000ms, delay=260000ms) 2024-09-11 17:28:06.236 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=266ms, period=120000ms 2024-09-11 17:28:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:28:06.352105+08:00, lastProcessTime=2024-09-11T17:27:56.351542+08:00 2024-09-11 17:28:06.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:28:06.352105+08:00 2024-09-11 17:28:07.159 INFO 2981023 --- [ocp-async-9,17d79dfc41a1423f,df0bd2380c1cbad7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:07.215 INFO 2981023 --- [ocp-schedules-20,e1238c10f3454f4d,00a75b625247743c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:28:07.219 INFO 2981023 --- [ocp-schedules-5,ec753484eefd02eb,28cf1c27ff560a59] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:28:07.219+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:28:07.220 INFO 2981023 --- [ocp-schedules-5,ec753484eefd02eb,28cf1c27ff560a59] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:28:07.221 INFO 2981023 --- [ocp-schedules-5,ec753484eefd02eb,28cf1c27ff560a59] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:28:07.222 INFO 2981023 --- [ocp-schedules-5,ec753484eefd02eb,28cf1c27ff560a59] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:28:07.402 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,122256f26fc0c8e3,18631bec1dcebce8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=122256f26fc0c8e3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:28:07.402 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,29fb76b8543c69c6,b183127a7069a9e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=29fb76b8543c69c6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:28:07.404 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,122256f26fc0c8e3,18631bec1dcebce8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=122256f26fc0c8e3, duration=8 ms] 2024-09-11 17:28:07.406 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,29fb76b8543c69c6,b183127a7069a9e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=29fb76b8543c69c6, duration=9 ms] 2024-09-11 17:28:07.414 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0f64ac32ad1cb267,26c12da4cad23663] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=0f64ac32ad1cb267, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:28:07.415 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0f64ac32ad1cb267,26c12da4cad23663] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=0f64ac32ad1cb267, duration=7 ms] 2024-09-11 17:28:07.747 INFO 2981023 --- [ocp-async-7,e4e32d9423ed6ea3,579c7952c4274e18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:28:08.063 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:08.064 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,24001bf4ee4de53b,e7456341a6fef73d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,24001bf4ee4de53b,e7456341a6fef73d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:09.159 INFO 2981023 --- [ocp-async-6,d47ad371b06ffeaf,63cf263609f30eb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:09.797 INFO 2981023 --- [ocp-schedules-30,23e3bc244ca4e7dd,dc8d0a2b8e3529aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:28:09.821 WARN 2981023 --- [scanner-13487232-1,23e3bc244ca4e7dd,70d3e81dd23e72d9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:28:09.872 INFO 2981023 --- [alarm-detect-4,23e3bc244ca4e7dd,295301ced12c5752] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643843, data is empty. 2024-09-11 17:28:09.879 INFO 2981023 --- [alarm-detect-4,23e3bc244ca4e7dd,295301ced12c5752] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643888, data is empty. 2024-09-11 17:28:09.883 INFO 2981023 --- [alarm-detect-4,23e3bc244ca4e7dd,295301ced12c5752] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643888, data is empty. 2024-09-11 17:28:09.897 INFO 2981023 --- [alarm-detect-15,23e3bc244ca4e7dd,8567c4acd4b4816c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643843, data is empty. 2024-09-11 17:28:09.897 INFO 2981023 --- [alarm-detect-1,23e3bc244ca4e7dd,c8c8471b967327a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643843, data is empty. 2024-09-11 17:28:09.903 INFO 2981023 --- [alarm-detect-1,23e3bc244ca4e7dd,c8c8471b967327a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643888, data is empty. 2024-09-11 17:28:09.905 INFO 2981023 --- [alarm-detect-15,23e3bc244ca4e7dd,8567c4acd4b4816c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643888, data is empty. 2024-09-11 17:28:09.908 INFO 2981023 --- [alarm-detect-1,23e3bc244ca4e7dd,c8c8471b967327a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643888, data is empty. 2024-09-11 17:28:09.911 INFO 2981023 --- [alarm-detect-15,23e3bc244ca4e7dd,8567c4acd4b4816c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643888, data is empty. 2024-09-11 17:28:10.008 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:28:10.008 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:28:10.051 INFO 2981023 --- [alarm-detect-2,23e3bc244ca4e7dd,c31fe0df1bdb328d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643843, data is empty. 2024-09-11 17:28:10.063 INFO 2981023 --- [alarm-detect-2,23e3bc244ca4e7dd,c31fe0df1bdb328d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643888, data is empty. 2024-09-11 17:28:10.068 INFO 2981023 --- [alarm-detect-2,23e3bc244ca4e7dd,c31fe0df1bdb328d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643888, data is empty. 2024-09-11 17:28:10.919 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:28:10.920 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:28:11.064 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:11.065 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:11.160 INFO 2981023 --- [ocp-async-5,1085ce41225611e4,09c303940b315614] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,75ccf16ff723f48b,dcabc966560c4fa9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,75ccf16ff723f48b,dcabc966560c4fa9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:11.809 INFO 2981023 --- [ocp-schedules-17,0670df27be1902e7,5e2432eaa7eb6475] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:28:13.160 INFO 2981023 --- [ocp-async-9,a52419dc5c74970d,768300f455cb93fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:13.615 INFO 2981023 --- [ocp-schedules-28,e279c0345c6aad86,dcd1fbf336b977c6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:28:13.624 INFO 2981023 --- [ocp-schedules-28,e279c0345c6aad86,dcd1fbf336b977c6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:28:13.624 INFO 2981023 --- [ocp-schedules-28,e279c0345c6aad86,dcd1fbf336b977c6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:28:13.636 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046820, nodeCount=1000, affectRows=1000 2024-09-11 17:28:13.638 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046820, nodeCount=999, validCount=999 2024-09-11 17:28:13.730 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046820, nodeCount=1000, affectRows=1000 2024-09-11 17:28:13.732 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046820, nodeCount=1000, validCount=1000 2024-09-11 17:28:13.740 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046820, nodeCount=48, affectRows=48 2024-09-11 17:28:13.740 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046820, nodeCount=48, validCount=48 2024-09-11 17:28:14.065 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:14.066 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,6365c6e964e33f8f,e0a4594b96b604d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,9f65728ae83c71fe,6f8e41bdbb8e55f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:28:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,6365c6e964e33f8f,e0a4594b96b604d1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:14.841 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046820, nodeCount=480, affectRows=480 2024-09-11 17:28:15.160 INFO 2981023 --- [ocp-async-3,0158793f0192b61f,9ff56fb8ff6f43c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:16.014 INFO 2981023 --- [ocp-schedules-24,f85c4328d3f1ecb9,7833a7424b86faf1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:28:16.014 INFO 2981023 --- [host-check-clockdiff1,f85c4328d3f1ecb9,42a1579cf0d0bdc0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:28:16.015 INFO 2981023 --- [host-check-clockdiff2,f85c4328d3f1ecb9,8603e3f12481fcf9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:28:16.015 INFO 2981023 --- [host-check-clockdiff3,f85c4328d3f1ecb9,105fd33fdc4cf6ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:28:16.016 INFO 2981023 --- [host-check-clockdiff4,f85c4328d3f1ecb9,2f6a2093454166ed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:28:16.027 INFO 2981023 --- [ocp-schedules-16,2b2af496ab54b21e,2d45e1d1ef528d51] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:28:16.027 INFO 2981023 --- [ocp-schedules-1,b113554996756e75,9916301de537c06a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:28:16.027 INFO 2981023 --- [host-check-agent-status1,2b2af496ab54b21e,7dd72ef3f965bec1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:28:16.027 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:28:16.027 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:28:16.028 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:28:16.028 INFO 2981023 --- [host-check-agent-status2,2b2af496ab54b21e,c1ee0398a3af396c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:28:16.029 INFO 2981023 --- [host-check-agent-status3,2b2af496ab54b21e,48da491a00644165] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:28:16.029 INFO 2981023 --- [host-check-agent-status4,2b2af496ab54b21e,69a682f5f5bf1c4b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:28:16.029 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:28:16.062 INFO 2981023 --- [host-check-agent-status4,2b2af496ab54b21e,69a682f5f5bf1c4b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:28:16.066 INFO 2981023 --- [host-check-agent-status4,2b2af496ab54b21e,69a682f5f5bf1c4b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:28:16.086 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:28:16.087 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:28:16.097 INFO 2981023 --- [host-check-agent-status3,2b2af496ab54b21e,48da491a00644165] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:28:16.097 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:28:16.100 INFO 2981023 --- [host-check-agent-status2,2b2af496ab54b21e,c1ee0398a3af396c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:28:16.155 INFO 2981023 --- [host-check-agent-status2,2b2af496ab54b21e,c1ee0398a3af396c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:28:16.172 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:28:16.178 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:28:16.178 INFO 2981023 --- [host-check-agent-status3,2b2af496ab54b21e,48da491a00644165] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:28:16.180 INFO 2981023 --- [host-check-agent-status1,2b2af496ab54b21e,7dd72ef3f965bec1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:28:16.183 INFO 2981023 --- [host-check-agent-status1,2b2af496ab54b21e,7dd72ef3f965bec1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:28:16.196 INFO 2981023 --- [ocp-schedules-22,40f923556e2a6425,1226e25ade1bcd5c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:28:16.196 INFO 2981023 --- [ocp-schedules-12,bbf187e2a74a55b5,06681559cc0fd349] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:28:16.196 INFO 2981023 --- [ocp-schedules-26,e0e802663ff68827,f178d22548013100] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:28:16.196 INFO 2981023 --- [ocp-schedules-25,a807feafeff13fd8,961d34b4784db596] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:28:16.197 INFO 2981023 --- [ocp-schedules-12,bbf187e2a74a55b5,06681559cc0fd349] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:28:16.198 INFO 2981023 --- [ocp-schedules-12,bbf187e2a74a55b5,06681559cc0fd349] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:28:16.201 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:28:16.203 INFO 2981023 --- [ocp-schedules-26,e0e802663ff68827,f178d22548013100] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:28:16.277 INFO 2981023 --- [ocp-schedules-22,40f923556e2a6425,1226e25ade1bcd5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:28:16.278 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:28:16.294 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:28:16.368 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:28:16.362809+08:00, lastProcessTime=2024-09-11T17:28:06.352105+08:00 2024-09-11 17:28:16.458 INFO 2981023 --- [ocp-schedules-7,68d607f90f07184d,6017fd099fa99e2e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:28:16.462 INFO 2981023 --- [host-check-agent-status1,2b2af496ab54b21e,7dd72ef3f965bec1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:28:16.466 INFO 2981023 --- [host-check-agent-status4,2b2af496ab54b21e,69a682f5f5bf1c4b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:28:16.467 INFO 2981023 --- [host-check-agent-status1,2b2af496ab54b21e,7dd72ef3f965bec1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:28:16.467 INFO 2981023 --- [host-check-agent-status1,2b2af496ab54b21e,7dd72ef3f965bec1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:28:16.467 INFO 2981023 --- [host-check-agent-status1,2b2af496ab54b21e,7dd72ef3f965bec1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:28:16.471 INFO 2981023 --- [host-check-agent-status4,2b2af496ab54b21e,69a682f5f5bf1c4b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:28:16.471 INFO 2981023 --- [host-check-agent-status4,2b2af496ab54b21e,69a682f5f5bf1c4b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:28:16.471 INFO 2981023 --- [host-check-agent-status4,2b2af496ab54b21e,69a682f5f5bf1c4b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:28:16.475 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.476 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:16.476 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:28:16.477 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:28:16.479 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:28:16.479076798+08:00 2024-09-11 17:28:16.480 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:28:16.479+08:00 2024-09-11 17:28:16.483 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:28:16.479+08:00 and status AVAILABLE 2024-09-11 17:28:16.483 INFO 2981023 --- [host-check-agent-status2,2b2af496ab54b21e,c1ee0398a3af396c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:28:16.485 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:28:16.487 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:28:16.488 INFO 2981023 --- [host-check-agent-status2,2b2af496ab54b21e,c1ee0398a3af396c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:28:16.488 INFO 2981023 --- [host-check-agent-status2,2b2af496ab54b21e,c1ee0398a3af396c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:28:16.488 INFO 2981023 --- [host-check-agent-status2,2b2af496ab54b21e,c1ee0398a3af396c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:28:16.488 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:28:16.488606+08:00 2024-09-11 17:28:16.489 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:28:16.487783260+08:00 2024-09-11 17:28:16.489 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:28:16.488+08:00 2024-09-11 17:28:16.490 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:28:16.489+08:00 2024-09-11 17:28:16.490 INFO 2981023 --- [host-check-agent-status3,2b2af496ab54b21e,48da491a00644165] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:28:16.491 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.492 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.492 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:28:16.488+08:00 and status AVAILABLE 2024-09-11 17:28:16.492 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:28:16.489+08:00 and status AVAILABLE 2024-09-11 17:28:16.493 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:16.493 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:16.493 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:28:16.494 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:28:16.495 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:28:16.496 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:28:16.495910263+08:00 2024-09-11 17:28:16.496 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:28:16.496+08:00 2024-09-11 17:28:16.497 INFO 2981023 --- [host-check-agent-status3,2b2af496ab54b21e,48da491a00644165] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:28:16.497 INFO 2981023 --- [host-check-agent-status3,2b2af496ab54b21e,48da491a00644165] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:28:16.497 INFO 2981023 --- [host-check-agent-status3,2b2af496ab54b21e,48da491a00644165] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:28:16.498 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:28:16.496+08:00 and status AVAILABLE 2024-09-11 17:28:16.498 INFO 2981023 --- [ocp-schedules-22,40f923556e2a6425,1226e25ade1bcd5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:28:16.499 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:28:16.503 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:28:16.504 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.505 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.505 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:16.507 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:16.509 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.509 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:16.510 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:28:16.521 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:28:16.562 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:28:16.563 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:28:16.564 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:28:16.564 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:28:16.568 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:28:16.569 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:28:16.571 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:28:16.575 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:28:16.591 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:28:16.596 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:28:16.597 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:28:16.603 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:28:16.608 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.610 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.611 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:28:16.613 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.614 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] ob connector is evicted from holder's cache. cause=EXPIRED, key=ObConnectorKey(connectionMode=direct, clusterName=linkprdocp, obClusterId=1719994437, tenantName=sys, username=root, address=10.31.35.10, port=2881, database=oceanbase), size=9 2024-09-11 17:28:16.614 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] com.alibaba.druid.pool.DruidDataSource : {dataSource-26} closing ... 2024-09-11 17:28:16.614 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] com.alibaba.druid.pool.DruidDataSource : {dataSource-26} closed 2024-09-11 17:28:16.614 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.615 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:28:16.617 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:28:16.618 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:28:16.632 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:28:16.632 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:28:16.635 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:28:16.636 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:28:16.662 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:28:16.671 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.671 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.672 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:16.673 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.675 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:28:16.676 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:28:16.679 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:28:16.692 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.692 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:16.693 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:28:16.693 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.693 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:16.696 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:28:16.696 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.696 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:28:16.699 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:28:16.700 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:28:16.704 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:28:16.723 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:28:16.726 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:16.726 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:28:16.728 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:16.728 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:28:16.734 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:28:16.735 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.737 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.737 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:28:16.737 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:16.738 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:28:16.739 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.740 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.740 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:28:16.742 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:28:16.743 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:28:16.746 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:28:16.748 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:28:16.749 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:28:16.752 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:28:16.753 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:16.761 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:28:16.765 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:28:16.766 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.766 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.767 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:16.768 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:28:16.768 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:28:16.770 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:28:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.772 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.772 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:16.773 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:28:16.773 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.774 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:28:16.776 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:28:16.777 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:28:16.778 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:28:16.779 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:28:16.780 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:28:16.781 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:28:16.783 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:28:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:28:16.788 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:28:16.796 INFO 2981023 --- [host-check-host-status3,b113554996756e75,4044339c399575c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:28:16.813 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:28:16.813 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:28:16.814 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:28:16.822 INFO 2981023 --- [host-check-host-status4,b113554996756e75,5c55798b0cb9052f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:28:16.826 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:16.827 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:28:16.827 INFO 2981023 --- [host-check-host-status1,b113554996756e75,f6ef3ee92b77a669] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:28:16.828 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:28:16.838 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:28:16.362809+08:00 2024-09-11 17:28:16.838 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.839 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.840 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:28:16.840 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.844 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.845 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:16.845 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:28:16.848 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:28:16.855 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:28:16.855 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:28:16.855 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:28:16.855 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:28:16.856 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:28:16.863 INFO 2981023 --- [host-check-host-status2,b113554996756e75,3396d657978260fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:28:16.871 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:28:16.879 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:28:16.885 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.886 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.886 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:28:16.891 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:28:16.891 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:28:16.901 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:28:16.907 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:16.908 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:16.908 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:28:16.914 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:28:16.914 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:28:16.914 INFO 2981023 --- [ocp-schedules-26,e0e802663ff68827,f178d22548013100] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:28:16.963 INFO 2981023 --- [host-check-clockdiff4,f85c4328d3f1ecb9,2f6a2093454166ed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046896 0 0 , error= 2024-09-11 17:28:16.972 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046820, nodeCount=480, validCount=480 2024-09-11 17:28:17.066 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:17.066 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:17.160 INFO 2981023 --- [ocp-async-6,b78e1878357aff76,a38f3822a1034020] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,2f8ee855b674cd0c,1623f9f577d94463] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,2f8ee855b674cd0c,1623f9f577d94463] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:17.362 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,809e4003e984fa07,2312f77cb7e6efa0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=809e4003e984fa07, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:28:17.363 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,2c5bc1d184f659aa,0bb95c15cbfd9015] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=2c5bc1d184f659aa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:28:17.363 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,809e4003e984fa07,2312f77cb7e6efa0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=809e4003e984fa07, duration=3 ms] 2024-09-11 17:28:17.366 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,2c5bc1d184f659aa,0bb95c15cbfd9015] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=2c5bc1d184f659aa, duration=5 ms] 2024-09-11 17:28:17.373 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,9746a038e88f7697,1f8e07a85c837f62] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=9746a038e88f7697, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:28:17.374 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,9746a038e88f7697,1f8e07a85c837f62] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=9746a038e88f7697, duration=3 ms] 2024-09-11 17:28:17.747 INFO 2981023 --- [ocp-async-3,929ac94c0168669b,46569467e0210077] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:28:17.811 INFO 2981023 --- [host-check-clockdiff1,f85c4328d3f1ecb9,42a1579cf0d0bdc0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046897 0 0 , error= 2024-09-11 17:28:19.160 INFO 2981023 --- [ocp-async-4,9617af31e5bc564f,8846959403733745] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:19.180 INFO 2981023 --- [host-check-clockdiff2,f85c4328d3f1ecb9,8603e3f12481fcf9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046899 0 0 , error= 2024-09-11 17:28:20.066 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:20.068 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:20.110 INFO 2981023 --- [ocp-schedules-2,607832fac2ad6e0c,998c70e83558915a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:28:20.209 WARN 2981023 --- [scanner-13489092-1,607832fac2ad6e0c,17ebdf104715ee06] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:28:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,280f5f71df18b5f5,e8901d9990a7643e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:20.323 INFO 2981023 --- [prometheus-minute-schedule-1,280f5f71df18b5f5,e8901d9990a7643e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:21.160 INFO 2981023 --- [ocp-async-10,ac404fa02faecbdd,921e631df5495ec9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:21.676 INFO 2981023 --- [ocp-schedules-1,f2cf6535fa6837d9,92f86c919493f0a1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 17:28:21.676 INFO 2981023 --- [ocp-schedules-1,f2cf6535fa6837d9,92f86c919493f0a1] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 17:28:21.699 INFO 2981023 --- [pool-511-thread-1,f2cf6535fa6837d9,586e291602f38ed7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:28:21.699 INFO 2981023 --- [pool-511-thread-2,f2cf6535fa6837d9,b6f661c91bf0f846] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:28:21.700 INFO 2981023 --- [pool-511-thread-3,f2cf6535fa6837d9,1202e6e1bd198e41] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:28:21.721 INFO 2981023 --- [pool-511-thread-2,f2cf6535fa6837d9,b6f661c91bf0f846] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:21.724 INFO 2981023 --- [pool-511-thread-3,f2cf6535fa6837d9,1202e6e1bd198e41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:21.724 INFO 2981023 --- [pool-511-thread-1,f2cf6535fa6837d9,586e291602f38ed7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:21.724 INFO 2981023 --- [pool-511-thread-2,f2cf6535fa6837d9,b6f661c91bf0f846] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046601000000, 1726046901000000] 2024-09-11 17:28:21.726 INFO 2981023 --- [pool-511-thread-3,f2cf6535fa6837d9,1202e6e1bd198e41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046601000000, 1726046901000000] 2024-09-11 17:28:21.727 INFO 2981023 --- [pool-511-thread-1,f2cf6535fa6837d9,586e291602f38ed7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046601000000, 1726046901000000] 2024-09-11 17:28:21.730 INFO 2981023 --- [pool-511-thread-3,f2cf6535fa6837d9,1202e6e1bd198e41] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:28:21.799 INFO 2981023 --- [pool-511-thread-1,f2cf6535fa6837d9,586e291602f38ed7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:28:21.814 INFO 2981023 --- [ocp-schedules-11,0dc95fb767a0d18b,8a06369c261e773c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:28:21.871 INFO 2981023 --- [pool-511-thread-2,f2cf6535fa6837d9,b6f661c91bf0f846] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:28:21.871 INFO 2981023 --- [ocp-schedules-1,f2cf6535fa6837d9,92f86c919493f0a1] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 17:28:22.001 INFO 2981023 --- [host-check-clockdiff3,f85c4328d3f1ecb9,105fd33fdc4cf6ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046901 1 1 , error= 2024-09-11 17:28:22.425 INFO 2981023 --- [ocp-schedules-27,8847796f82bb3625,0997b6419273468e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:28:22.425 INFO 2981023 --- [ocp-schedules-27,8847796f82bb3625,0997b6419273468e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:28:22.425 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:28:22.427 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:28:22.427 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:28:22.430 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:28:22.431 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:28:22.431 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:28:22.431 INFO 2981023 --- [ocp-schedules-27,8847796f82bb3625,0997b6419273468e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:28:22.434 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:28:22.434 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:28:22.434 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:28:22.436 INFO 2981023 --- [ocp-schedules-27,8847796f82bb3625,0997b6419273468e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:28:22.457 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:28:22.458 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:28:22.459 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:28:22.459 INFO 2981023 --- [ocp-schedules-27,8847796f82bb3625,0997b6419273468e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:28:22.463 INFO 2981023 --- [ocp-schedules-27,8847796f82bb3625,0997b6419273468e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:28:23.068 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:23.070 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:23.160 INFO 2981023 --- [ocp-async-2,92a2788eafec6a7f,a65f51c086b98f13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,6167c413c4483cf4,f776bf639c29985b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,6167c413c4483cf4,f776bf639c29985b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:24.182 INFO 2981023 --- [ocp-schedules-17,d017f643a2f020ef,02ae1cc95a95214f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:28:25.160 INFO 2981023 --- [ocp-async-7,a03fd32a35069091,bba440744f5396c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:26.070 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:26.071 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,8b8761829345868b,07e89db3f11dda26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,8b8761829345868b,07e89db3f11dda26] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:26.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:28:26.351664+08:00, lastProcessTime=2024-09-11T17:28:16.362809+08:00 2024-09-11 17:28:26.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:28:26.351664+08:00 2024-09-11 17:28:26.555 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.561 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, ob 2024-09-11 17:28:26.562 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.562 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:28:26.562 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.563 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.564 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.569 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.572 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 40 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.616 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 8 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.624 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 167 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.650 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.652 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.653 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.653 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.654 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.655 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.659 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.659 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:28:26.661 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.664 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 7 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.670 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.671 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.671 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.672 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.673 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.673 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.677 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.677 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:28:26.680 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.683 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.687 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:22:26.295, end=2024-09-11T17:27:26.552} 2024-09-11 17:28:26.687 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 1 dubious sqls 2024-09-11 17:28:27.160 INFO 2981023 --- [ocp-async-1,6dd18d7bd0f10f4a,a1239752e62224cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:27.316 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,32a03b74ebcc5ff7,25238d47f8ffd6f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=32a03b74ebcc5ff7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:28:27.317 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,32a03b74ebcc5ff7,25238d47f8ffd6f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=32a03b74ebcc5ff7, duration=5 ms] 2024-09-11 17:28:27.747 INFO 2981023 --- [ocp-async-4,f27956c26ff659f0,3ec135c70a9d29d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:28:29.071 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:29.073 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:29.160 INFO 2981023 --- [ocp-async-8,4ece464540a5cd8b,d914b495169a0971] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,72ba21d48a945360,5446ab4d65ee02f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:29.318 INFO 2981023 --- [prometheus-minute-schedule-1,72ba21d48a945360,5446ab4d65ee02f9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:30.348 INFO 2981023 --- [ocp-schedules-14,9f0885c035a90811,84ff291a8428dee5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:28:30.401 WARN 2981023 --- [scanner-13489296-1,9f0885c035a90811,37750b6e28ec9acb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:28:30.413 INFO 2981023 --- [alarm-detect-4,9f0885c035a90811,7bf2e92ea8c1a72c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643888, data is empty. 2024-09-11 17:28:30.425 INFO 2981023 --- [alarm-detect-4,9f0885c035a90811,7bf2e92ea8c1a72c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643888, data is empty. 2024-09-11 17:28:30.433 INFO 2981023 --- [alarm-detect-4,9f0885c035a90811,7bf2e92ea8c1a72c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643888, data is empty. 2024-09-11 17:28:30.445 INFO 2981023 --- [alarm-detect-13,9f0885c035a90811,6c8ebed76589b236] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643888, data is empty. 2024-09-11 17:28:30.448 INFO 2981023 --- [alarm-detect-3,9f0885c035a90811,ca67b156a3b19a3f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643888, data is empty. 2024-09-11 17:28:30.450 INFO 2981023 --- [alarm-detect-13,9f0885c035a90811,6c8ebed76589b236] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643888, data is empty. 2024-09-11 17:28:30.452 INFO 2981023 --- [alarm-detect-3,9f0885c035a90811,ca67b156a3b19a3f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643888, data is empty. 2024-09-11 17:28:30.458 INFO 2981023 --- [alarm-detect-3,9f0885c035a90811,ca67b156a3b19a3f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643888, data is empty. 2024-09-11 17:28:30.458 INFO 2981023 --- [alarm-detect-13,9f0885c035a90811,6c8ebed76589b236] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643888, data is empty. 2024-09-11 17:28:30.560 INFO 2981023 --- [alarm-detect-19,9f0885c035a90811,179bd857a475f123] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643888, data is empty. 2024-09-11 17:28:30.564 INFO 2981023 --- [alarm-detect-19,9f0885c035a90811,179bd857a475f123] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643888, data is empty. 2024-09-11 17:28:30.571 INFO 2981023 --- [alarm-detect-19,9f0885c035a90811,179bd857a475f123] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643888, data is empty. 2024-09-11 17:28:31.172 INFO 2981023 --- [ocp-async-7,901d652fafb532ff,bcca4940893babd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:31.817 INFO 2981023 --- [ocp-schedules-15,9bcba77cdb3108ed,a69f2d880318261f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:28:32.073 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:32.074 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,aa6ddba4cef49006,ccc7007d35673d49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,aa6ddba4cef49006,ccc7007d35673d49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:33.161 INFO 2981023 --- [ocp-async-1,ce84a42785215a2b,cec039e25e19755b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:33.627 INFO 2981023 --- [ocp-schedules-2,a310484901a820a1,e0864ffb37c27613] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:28:33.637 INFO 2981023 --- [ocp-schedules-2,a310484901a820a1,e0864ffb37c27613] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:28:33.637 INFO 2981023 --- [ocp-schedules-2,a310484901a820a1,e0864ffb37c27613] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:28:35.074 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:35.075 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:35.161 INFO 2981023 --- [ocp-async-2,5269cc4d7e70ac63,8b231f20e61b6157] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,09a1b6182192efab,f4f96525735a2090] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,09a1b6182192efab,f4f96525735a2090] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:36.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:28:36.351241+08:00, lastProcessTime=2024-09-11T17:28:26.351664+08:00 2024-09-11 17:28:36.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:28:36.351241+08:00 2024-09-11 17:28:37.161 INFO 2981023 --- [ocp-async-5,da967b2ff7c695ea,29238dc752369d77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:37.314 INFO 2981023 --- [prometheus-cache-monitor1,15703e210c63fc10,c58aaedd7272b28d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:28:37.315 INFO 2981023 --- [prometheus-cache-monitor1,15703e210c63fc10,c58aaedd7272b28d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:28:37.747 INFO 2981023 --- [ocp-async-2,3b6549ca50a7122a,0e99bbe01bce7fe7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:28:38.075 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:38.075 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:38.266 INFO 2981023 --- [ocp-schedules-14,ab4ff72d133bef5f,e373753537caebc4] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:28:38.285 WARN 2981023 --- [ocp-schedules-14,ab4ff72d133bef5f,e373753537caebc4] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:28:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,90246f8619c91c6c,4bf65fc0a0fa0808] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:38.315 INFO 2981023 --- [prometheus-minute-schedule-1,90246f8619c91c6c,4bf65fc0a0fa0808] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:39.161 INFO 2981023 --- [ocp-async-9,f421e1a0cb2e4e78,c867740f0767f2c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:40.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:28:40.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:28:40.624 INFO 2981023 --- [ocp-schedules-5,75ea5a8f362e7918,4fc290d85d64e302] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:28:40.663 WARN 2981023 --- [scanner-13489576-1,75ea5a8f362e7918,4b7bb0be8587db63] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:28:41.075 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:41.076 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:41.161 INFO 2981023 --- [ocp-async-8,6fb27f2965306b02,274e62e03399c669] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,fbfbde09606b6b7d,a0b45da8a3387281] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,fbfbde09606b6b7d,a0b45da8a3387281] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:41.328 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,18f23ddc0c4fccc9,84448cfe2d19e741] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=18f23ddc0c4fccc9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:28:41.329 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,18f23ddc0c4fccc9,84448cfe2d19e741] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=18f23ddc0c4fccc9, duration=3 ms] 2024-09-11 17:28:41.819 INFO 2981023 --- [ocp-schedules-25,8c1677c930601675,ab62fafc6fdd7f64] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:28:43.062 INFO 2981023 --- [ocp-schedules-16,3e7fccf3682122db,f4c67501a3d57695] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:28:43.161 INFO 2981023 --- [ocp-async-6,b5946fc4f993cfc5,f9ff705293b9e5cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:44.076 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:44.077 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,41e4b38867294108,24c7b7408b0cdb49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,41e4b38867294108,24c7b7408b0cdb49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:45.101 INFO 2981023 --- [prometheus-second-schedule-1,1a7c22705029c1ed,820b65e51a361fe6] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:28:45.161 INFO 2981023 --- [ocp-async-7,5f0d3ec9f4097063,f5631a0d5f31cce0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:46.018 INFO 2981023 --- [ocp-schedules-27,8faa26686b29e4ce,4e45952420a836ca] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:28:46.019 INFO 2981023 --- [host-check-clockdiff1,8faa26686b29e4ce,1060ead5e5d0b9ef] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:28:46.019 INFO 2981023 --- [host-check-clockdiff2,8faa26686b29e4ce,9319c21669b53e4e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:28:46.019 INFO 2981023 --- [host-check-clockdiff3,8faa26686b29e4ce,6befed396ee95a33] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:28:46.021 INFO 2981023 --- [host-check-clockdiff4,8faa26686b29e4ce,3bbd048746c5ff4b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:28:46.026 INFO 2981023 --- [ocp-schedules-6,cd6f3e7a1c12204e,837fa2d9d981e005] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:28:46.026 INFO 2981023 --- [ocp-schedules-19,323f011d5cff6935,91663f21a0b75b39] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:28:46.027 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:28:46.027 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:28:46.027 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:28:46.027 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:28:46.028 INFO 2981023 --- [host-check-agent-status1,cd6f3e7a1c12204e,981f2928f9818639] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:28:46.029 INFO 2981023 --- [host-check-agent-status2,cd6f3e7a1c12204e,d637ac28a46c1a5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:28:46.029 INFO 2981023 --- [host-check-agent-status3,cd6f3e7a1c12204e,83d1367f98e006c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:28:46.030 INFO 2981023 --- [host-check-agent-status4,cd6f3e7a1c12204e,d7b3eea4bc1c5e1b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:28:46.042 INFO 2981023 --- [host-check-clockdiff1,8faa26686b29e4ce,1060ead5e5d0b9ef] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046926 0 0 , error= 2024-09-11 17:28:46.045 INFO 2981023 --- [host-check-clockdiff4,8faa26686b29e4ce,3bbd048746c5ff4b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046926 0 0 , error= 2024-09-11 17:28:46.046 INFO 2981023 --- [host-check-clockdiff2,8faa26686b29e4ce,9319c21669b53e4e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046926 0 0 , error= 2024-09-11 17:28:46.061 INFO 2981023 --- [host-check-agent-status1,cd6f3e7a1c12204e,981f2928f9818639] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:28:46.063 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:28:46.065 INFO 2981023 --- [host-check-agent-status1,cd6f3e7a1c12204e,981f2928f9818639] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:28:46.067 INFO 2981023 --- [host-check-agent-status3,cd6f3e7a1c12204e,83d1367f98e006c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:28:46.067 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:28:46.067 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:28:46.068 INFO 2981023 --- [host-check-agent-status3,cd6f3e7a1c12204e,83d1367f98e006c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:28:46.071 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:28:46.171 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:28:46.172 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:28:46.194 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:28:46.197 INFO 2981023 --- [ocp-schedules-22,7d7a1ead43b16796,59b0232333ccf1ed] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:28:46.198 INFO 2981023 --- [ocp-schedules-10,a57eb099d42def9d,b3b5c2aba118d0c0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:28:46.203 INFO 2981023 --- [ocp-schedules-22,7d7a1ead43b16796,59b0232333ccf1ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:28:46.205 INFO 2981023 --- [ocp-schedules-11,9f3f40391a772d84,ab56adb2b7416e2d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:28:46.206 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:28:46.207 INFO 2981023 --- [ocp-schedules-17,e919cb524d4232c9,82e48bd5e7428238] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:28:46.207 INFO 2981023 --- [ocp-schedules-2,05773e9ed40cdb3b,308f40b49ff93f0e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:28:46.208 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:28:46.212 INFO 2981023 --- [ocp-schedules-10,a57eb099d42def9d,b3b5c2aba118d0c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:28:46.212 INFO 2981023 --- [ocp-schedules-17,e919cb524d4232c9,82e48bd5e7428238] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:28:46.220 INFO 2981023 --- [host-check-agent-status4,cd6f3e7a1c12204e,d7b3eea4bc1c5e1b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:28:46.260 INFO 2981023 --- [host-check-agent-status2,cd6f3e7a1c12204e,d637ac28a46c1a5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:28:46.261 INFO 2981023 --- [host-check-agent-status4,cd6f3e7a1c12204e,d7b3eea4bc1c5e1b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:28:46.262 INFO 2981023 --- [ocp-schedules-10,a57eb099d42def9d,b3b5c2aba118d0c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:28:46.284 INFO 2981023 --- [host-check-agent-status2,cd6f3e7a1c12204e,d637ac28a46c1a5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:28:46.308 INFO 2981023 --- [ocp-schedules-11,9f3f40391a772d84,ab56adb2b7416e2d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:28:46.329 INFO 2981023 --- [ocp-schedules-21,eb99b69e3daaf9db,3b13b88e3f521335] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:28:46.342 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:28:46.346 INFO 2981023 --- [host-check-agent-status1,cd6f3e7a1c12204e,981f2928f9818639] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:28:46.348 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:28:46.349 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:28:46.350 INFO 2981023 --- [host-check-agent-status1,cd6f3e7a1c12204e,981f2928f9818639] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:28:46.350 INFO 2981023 --- [host-check-agent-status1,cd6f3e7a1c12204e,981f2928f9818639] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:28:46.350 INFO 2981023 --- [host-check-agent-status1,cd6f3e7a1c12204e,981f2928f9818639] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:28:46.351 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:28:46.351 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:28:46.351793223+08:00 2024-09-11 17:28:46.352 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:28:46.351+08:00 2024-09-11 17:28:46.353 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:28:46.351438378+08:00 2024-09-11 17:28:46.353 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:28:46.352919171+08:00 2024-09-11 17:28:46.354 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:28:46.353+08:00 2024-09-11 17:28:46.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:28:46.352768+08:00, lastProcessTime=2024-09-11T17:28:36.351241+08:00 2024-09-11 17:28:46.354 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:28:46.353+08:00 2024-09-11 17:28:46.355 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:28:46.351+08:00 and status AVAILABLE 2024-09-11 17:28:46.357 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:28:46.353+08:00 and status AVAILABLE 2024-09-11 17:28:46.357 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:28:46.353+08:00 and status AVAILABLE 2024-09-11 17:28:46.361 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.362 INFO 2981023 --- [host-check-agent-status3,cd6f3e7a1c12204e,83d1367f98e006c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:28:46.362 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:46.363 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:28:46.363 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.364 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:46.364 INFO 2981023 --- [host-check-agent-status2,cd6f3e7a1c12204e,d637ac28a46c1a5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:28:46.365 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:28:46.365 INFO 2981023 --- [host-check-agent-status3,cd6f3e7a1c12204e,83d1367f98e006c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:28:46.365 INFO 2981023 --- [host-check-agent-status3,cd6f3e7a1c12204e,83d1367f98e006c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:28:46.365 INFO 2981023 --- [host-check-agent-status3,cd6f3e7a1c12204e,83d1367f98e006c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:28:46.366 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:28:46.366 INFO 2981023 --- [host-check-agent-status4,cd6f3e7a1c12204e,d7b3eea4bc1c5e1b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:28:46.367 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:28:46.367 INFO 2981023 --- [host-check-agent-status2,cd6f3e7a1c12204e,d637ac28a46c1a5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:28:46.367 INFO 2981023 --- [host-check-agent-status2,cd6f3e7a1c12204e,d637ac28a46c1a5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:28:46.367 INFO 2981023 --- [host-check-agent-status2,cd6f3e7a1c12204e,d637ac28a46c1a5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:28:46.368 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:28:46.367746949+08:00 2024-09-11 17:28:46.369 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:28:46.371 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:28:46.372 INFO 2981023 --- [host-check-agent-status4,cd6f3e7a1c12204e,d7b3eea4bc1c5e1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:28:46.372 INFO 2981023 --- [host-check-agent-status4,cd6f3e7a1c12204e,d7b3eea4bc1c5e1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:28:46.372 INFO 2981023 --- [host-check-agent-status4,cd6f3e7a1c12204e,d7b3eea4bc1c5e1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:28:46.374 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:28:46.368+08:00 2024-09-11 17:28:46.376 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:28:46.368+08:00 and status AVAILABLE 2024-09-11 17:28:46.376 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.377 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:46.378 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:28:46.379 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.379 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:46.391 INFO 2981023 --- [ocp-schedules-22,7d7a1ead43b16796,59b0232333ccf1ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:28:46.400 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:28:46.400 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:28:46.402 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:28:46.404 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:28:46.408 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:28:46.418 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:28:46.418 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.418 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:28:46.418 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:46.418 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.418 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:46.454 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:28:46.455 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:28:46.465 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.466 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.467 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:28:46.467 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:28:46.467 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:28:46.468 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.469 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:28:46.478 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:28:46.481 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:28:46.483 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:28:46.486 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:28:46.488 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:28:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.494 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.494 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:46.494 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:28:46.495 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.497 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:28:46.501 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:28:46.507 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:28:46.508 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:28:46.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:46.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:28:46.522 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:28:46.528 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:28:46.531 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:28:46.560 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:28:46.563 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.565 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.566 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:28:46.566 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:28:46.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:46.569 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.569 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,632a3031a05ba12c,0576e3178e81e221] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=632a3031a05ba12c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:28:46.570 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.571 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:28:46.571 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:28:46.572 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.576 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.578 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:46.580 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:28:46.581 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,632a3031a05ba12c,0576e3178e81e221] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=632a3031a05ba12c, duration=16 ms] 2024-09-11 17:28:46.582 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:28:46.585 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:28:46.591 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:28:46.593 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:28:46.597 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:28:46.598 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:28:46.599 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:28:46.603 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.604 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.604 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:46.605 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.606 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:28:46.608 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:28:46.609 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:28:46.612 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:28:46.627 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:28:46.627 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:28:46.635 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:28:46.636 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:28:46.642 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:28:46.642 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:46.642 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:28:46.647 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:28:46.647 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:28:46.660 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,3f4839079b6ffee6,1007706d376954e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=3f4839079b6ffee6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:28:46.663 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,3f4839079b6ffee6,1007706d376954e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=3f4839079b6ffee6, duration=7 ms] 2024-09-11 17:28:46.674 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:28:46.676 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:28:46.680 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.684 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:46.686 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:28:46.691 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:28:46.695 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.696 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.696 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:46.697 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:28:46.697 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:28:46.701 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.702 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.702 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:28:46.703 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:28:46.703 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:28:46.703 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.704 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:28:46.704 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:28:46.704 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:28:46.705 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.705 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:28:46.708 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.711 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:28:46.712 INFO 2981023 --- [host-check-host-status2,323f011d5cff6935,a2392ffaee018c62] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:28:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:28:46.716 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:28:46.720 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.720 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.721 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:28:46.724 INFO 2981023 --- [host-check-host-status1,323f011d5cff6935,6d8c07ed1abd0ffe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:28:46.734 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:28:46.734 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:28:46.735 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:28:46.736 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:28:46.739 INFO 2981023 --- [host-check-host-status3,323f011d5cff6935,b1fe6e07425b4a46] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:28:46.750 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:28:46.755 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:28:46.760 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:28:46.764 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.764 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.765 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:28:46.765 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:28:46.766 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.768 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:28:46.769 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:28:46.773 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.774 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.774 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:28:46.775 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:28:46.775 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.782 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:28:46.791 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:28:46.793 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:46.793 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:28:46.808 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:28:46.809 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:28:46.352768+08:00 2024-09-11 17:28:46.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:28:46.814 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:28:46.816 INFO 2981023 --- [host-check-host-status4,323f011d5cff6935,67a38f753778c5e4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:28:46.816 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.817 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.817 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:28:46.818 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:28:46.819 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:28:46.822 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:28:46.825 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:28:46.826 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:28:46.826 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.827 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.827 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:28:46.831 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.831 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.832 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:28:46.832 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:28:46.832 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.835 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:28:46.835 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:28:46.840 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:28:46.869 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:28:46.872 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:28:46.874 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:46.874 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:46.881 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:28:46.888 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:28:46.888 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:28:46.888 INFO 2981023 --- [ocp-schedules-17,e919cb524d4232c9,82e48bd5e7428238] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:28:46.890 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:28:46.896 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:28:46.899 INFO 2981023 --- [ocp-schedules-13,2f88bf60736e57d7,daec4a1c803ca7b1] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:28:47.077 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:47.077 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:47.161 INFO 2981023 --- [ocp-async-9,ae09ced2b0ff09cf,e0612cae1acb2cb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,a3eb6a67b1d9c8f5,2c9ac5495f01b7d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,a3eb6a67b1d9c8f5,2c9ac5495f01b7d4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:47.748 INFO 2981023 --- [ocp-async-8,2e1b7b4e947a6027,c464ef0e3e88f193] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:28:49.161 INFO 2981023 --- [ocp-async-1,60dc047a3b08a89e,3d6aeab44c259621] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:28:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:28:50.077 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:50.079 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:50.207 INFO 2981023 --- [ocp-schedules-30,dcbdd8ff2860237b,e7e39bda3498c499] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:28:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,f160413345b8499d,ae223c35d28de530] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,f160413345b8499d,ae223c35d28de530] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:50.836 INFO 2981023 --- [ocp-schedules-6,5f77ea4b82093d26,22e2280e8eda1b18] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:28:50.877 WARN 2981023 --- [scanner-13489797-1,5f77ea4b82093d26,ae338ce02d658fc7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:28:50.904 INFO 2981023 --- [alarm-detect-17,5f77ea4b82093d26,59ed2e879b7d06fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643888, data is empty. 2024-09-11 17:28:50.907 INFO 2981023 --- [alarm-detect-17,5f77ea4b82093d26,59ed2e879b7d06fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643888, data is empty. 2024-09-11 17:28:50.911 INFO 2981023 --- [alarm-detect-17,5f77ea4b82093d26,59ed2e879b7d06fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643888, data is empty. 2024-09-11 17:28:50.937 INFO 2981023 --- [alarm-detect-7,5f77ea4b82093d26,c99ad9dd54f2e831] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643888, data is empty. 2024-09-11 17:28:50.941 INFO 2981023 --- [alarm-detect-1,5f77ea4b82093d26,b03de33f90b397a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643888, data is empty. 2024-09-11 17:28:50.944 INFO 2981023 --- [alarm-detect-7,5f77ea4b82093d26,c99ad9dd54f2e831] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643888, data is empty. 2024-09-11 17:28:50.946 INFO 2981023 --- [alarm-detect-7,5f77ea4b82093d26,c99ad9dd54f2e831] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643888, data is empty. 2024-09-11 17:28:50.947 INFO 2981023 --- [alarm-detect-1,5f77ea4b82093d26,b03de33f90b397a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643888, data is empty. 2024-09-11 17:28:50.949 INFO 2981023 --- [alarm-detect-1,5f77ea4b82093d26,b03de33f90b397a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643888, data is empty. 2024-09-11 17:28:51.077 INFO 2981023 --- [alarm-detect-7,5f77ea4b82093d26,cf70a3fddc42fb37] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643888, data is empty. 2024-09-11 17:28:51.083 INFO 2981023 --- [alarm-detect-7,5f77ea4b82093d26,cf70a3fddc42fb37] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643888, data is empty. 2024-09-11 17:28:51.088 INFO 2981023 --- [alarm-detect-7,5f77ea4b82093d26,cf70a3fddc42fb37] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643888, data is empty. 2024-09-11 17:28:51.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,b1ea628363f766d8,0ab8e67d2b6cd668] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=b1ea628363f766d8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:28:51.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,99176d52fc6b7e26,bdac712a612e1692] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=99176d52fc6b7e26, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:28:51.131 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,99176d52fc6b7e26,bdac712a612e1692] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=99176d52fc6b7e26, duration=14 ms] 2024-09-11 17:28:51.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,b1ea628363f766d8,0ab8e67d2b6cd668] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=b1ea628363f766d8, duration=28 ms] 2024-09-11 17:28:51.162 INFO 2981023 --- [ocp-async-7,6129f8ed8408408e,d0d78e635c411a6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:51.211 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,28a0b23294b34f52,5fdd7ea2dcf30bd3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=28a0b23294b34f52, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:28:51.211 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,28a0b23294b34f52,5fdd7ea2dcf30bd3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=28a0b23294b34f52, duration=2 ms] 2024-09-11 17:28:51.325 INFO 2981023 --- [ocp-schedules-5,6ff93a8ae9780b38,0e3eaa1e2e2a8039] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:28:51.325+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:28:51.329 INFO 2981023 --- [ocp-schedules-5,6ff93a8ae9780b38,0e3eaa1e2e2a8039] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:28:51.450 INFO 2981023 --- [ocp-schedules-5,6ff93a8ae9780b38,0e3eaa1e2e2a8039] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 46 to SqlAggregateData 2024-09-11 17:28:51.452 INFO 2981023 --- [ocp-schedules-5,6ff93a8ae9780b38,0e3eaa1e2e2a8039] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:28:51.513 INFO 2981023 --- [ocp-schedules-24,0e74531097af3631,f57f5ad6cd43f8d0] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:28:51.514 INFO 2981023 --- [ocp-schedules-24,0e74531097af3631,f57f5ad6cd43f8d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:28:51.514 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:28:51.514 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:28:51.515 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:28:51.515 INFO 2981023 --- [ocp-schedules-24,0e74531097af3631,f57f5ad6cd43f8d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:28:51.822 INFO 2981023 --- [ocp-schedules-30,d58d6d6abcae7684,b9f7e5dba9242c69] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:28:51.966 INFO 2981023 --- [ocp-schedules-16,106d4d064b7753a1,4e10894d88c98f34] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:28:51.967 INFO 2981023 --- [ocp-schedules-16,106d4d064b7753a1,4e10894d88c98f34] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:28:52.001 INFO 2981023 --- [host-check-clockdiff3,8faa26686b29e4ce,6befed396ee95a33] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046931 1 1 , error= 2024-09-11 17:28:52.002 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:28:52.002 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:28:52.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.003 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:28:52.003 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.005 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:28:52.007 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:28:52.007 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:28:52.009 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.010 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.011 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.011 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.011 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.012 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.021 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.021 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:28:52.023 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.024 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.024 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.024 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.028 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.028 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:28:52.034 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.035 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.035 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.036 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:28:52.036 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.037 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.037 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:28:52.037 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE20A1D6ED-0-0] 2024-09-11 17:28:52.037 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:28:52.038 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.038 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:28:52.041 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.041 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:28:52.041 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.042 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:28:52.043 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.043 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:28:52.044 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.044 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:28:52.044 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:28:52.048 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:28:52.049 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.050 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:28:52.061 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:52.061 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.062 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:28:52.062 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.062 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.063 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:28:52.063 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.063 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:28:52.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.065 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:28:52.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.066 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.066 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:28:52.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:28:52.067 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:28:52.067 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.067 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE20A1D6F7-0-0] 2024-09-11 17:28:52.067 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:28:52.068 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:28:52.068 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.069 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:28:52.070 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.070 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:28:52.071 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.071 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:28:52.072 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:28:52.074 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:28:52.075 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.075 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:28:52.076 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.077 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:28:52.077 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.077 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:28:52.078 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:28:52.078 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.079 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:28:52.079 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.080 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:28:52.080 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.081 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:28:52.081 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:28:52.081 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:28:52.081 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:28:52.082 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:28:52.083 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:28:52.086 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:52.088 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.088 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:28:52.095 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:28:52.095 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:28:52.096 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:28:52.097 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.097 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.097 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:28:52.097 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:28:52.103 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:28:52.105 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:28:52.105 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.106 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.109 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:28:52.110 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:28:52.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:52.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:28:52.112 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:28:52.114 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:28:52.114 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:28:52.114 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:28:52.121 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:28:52.122 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:28:52.122 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:28:52.123 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:28:52.123 INFO 2981023 --- [ocp-schedules-16,106d4d064b7753a1,4e10894d88c98f34] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:28:52.327 INFO 2981023 --- [ocp-schedules-18,7953b135b9bae0ff,ced3275219517e00] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:28:52.346 WARN 2981023 --- [ocp-schedules-18,7953b135b9bae0ff,ced3275219517e00] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:28:53.042 INFO 2981023 --- [ocp-schedules-6,1ff7107e549ee21b,781e0b178bff61af] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:28:53.042 INFO 2981023 --- [ocp-schedules-6,1ff7107e549ee21b,781e0b178bff61af] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:28:53.079 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:53.081 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:53.162 INFO 2981023 --- [ocp-async-9,1b641d0cbd2cc3c8,aba88ab2f6fa6037] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,d6be7827b4891829,cec032167108e830] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,d6be7827b4891829,cec032167108e830] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:53.640 INFO 2981023 --- [ocp-schedules-15,c183ba271d3efeba,890fb39f2a4ee70e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:28:53.648 INFO 2981023 --- [ocp-schedules-15,c183ba271d3efeba,890fb39f2a4ee70e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:28:53.648 INFO 2981023 --- [ocp-schedules-15,c183ba271d3efeba,890fb39f2a4ee70e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:28:54.185 INFO 2981023 --- [ocp-schedules-26,0f2c1de56884e3e5,a7374ab4c6dc183f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:28:54.535 INFO 2981023 --- [ocp-schedules-21,fc9214297de4e0e9,56e9324e7f3a0a29] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T17:28:54.535+08:00, tenantSize=6 2024-09-11 17:28:54.537 INFO 2981023 --- [ocp-schedules-21,fc9214297de4e0e9,56e9324e7f3a0a29] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726046934535000 2024-09-11 17:28:54.537 INFO 2981023 --- [ocp-schedules-21,fc9214297de4e0e9,56e9324e7f3a0a29] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726046634510000 2024-09-11 17:28:54.552 INFO 2981023 --- [ocp-schedules-21,fc9214297de4e0e9,56e9324e7f3a0a29] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 17:28:55.162 INFO 2981023 --- [ocp-async-6,c5d2783fc0733bd6,33128c39602fb0a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:55.639 INFO 2981023 --- [ocp-schedules-18,1fdf198857928b1c,ea0db0dd24b1d457] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:28:55.639 INFO 2981023 --- [ocp-schedules-18,1fdf198857928b1c,ea0db0dd24b1d457] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:28:55.640 INFO 2981023 --- [ocp-schedules-18,1fdf198857928b1c,ea0db0dd24b1d457] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:28:55.640 INFO 2981023 --- [ocp-schedules-18,1fdf198857928b1c,ea0db0dd24b1d457] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:28:55.640 INFO 2981023 --- [ocp-schedules-18,1fdf198857928b1c,ea0db0dd24b1d457] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:28:55.640 INFO 2981023 --- [ocp-schedules-18,1fdf198857928b1c,ea0db0dd24b1d457] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:28:55.706 INFO 2981023 --- [ocp-schedules-12,0a925c0172dde192,1fa9c9509c3d3aa3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:28:55.713 INFO 2981023 --- [ocp-schedules-12,0a925c0172dde192,1fa9c9509c3d3aa3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:28:55.713 INFO 2981023 --- [ocp-schedules-12,0a925c0172dde192,1fa9c9509c3d3aa3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:28:55.881 INFO 2981023 --- [ocp-schedules-5,e79e0c36669d180a,b49186e703fa4db4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:28:55.881 INFO 2981023 --- [ocp-schedules-5,e79e0c36669d180a,b49186e703fa4db4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:28:55.896 INFO 2981023 --- [Collect all tenants major compactions1,e79e0c36669d180a,a9983f69fe30a713] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:28:55.896 INFO 2981023 --- [Collect all tenants major compactions5,e79e0c36669d180a,26003f037d399718] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:28:55.897 INFO 2981023 --- [Collect all tenants major compactions3,e79e0c36669d180a,3f8a96189d037b03] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:28:55.897 INFO 2981023 --- [Collect all tenants major compactions4,e79e0c36669d180a,03846c23edfc4f64] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:28:55.898 INFO 2981023 --- [Collect all tenants major compactions2,e79e0c36669d180a,61e7099b0bb0d6a0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:28:55.899 INFO 2981023 --- [Collect all tenants major compactions6,e79e0c36669d180a,c4d324e02895098f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:28:55.908 INFO 2981023 --- [Collect all tenants major compactions5,e79e0c36669d180a,26003f037d399718] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:55.909 INFO 2981023 --- [Collect all tenants major compactions3,e79e0c36669d180a,3f8a96189d037b03] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:55.909 INFO 2981023 --- [Collect all tenants major compactions2,e79e0c36669d180a,61e7099b0bb0d6a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:55.909 INFO 2981023 --- [Collect all tenants major compactions1,e79e0c36669d180a,a9983f69fe30a713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:55.909 INFO 2981023 --- [Collect all tenants major compactions5,e79e0c36669d180a,26003f037d399718] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:28:55.910 INFO 2981023 --- [Collect all tenants major compactions3,e79e0c36669d180a,3f8a96189d037b03] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:28:55.910 INFO 2981023 --- [Collect all tenants major compactions4,e79e0c36669d180a,03846c23edfc4f64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:55.911 INFO 2981023 --- [Collect all tenants major compactions2,e79e0c36669d180a,61e7099b0bb0d6a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:28:55.911 INFO 2981023 --- [Collect all tenants major compactions6,e79e0c36669d180a,c4d324e02895098f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:28:55.912 INFO 2981023 --- [Collect all tenants major compactions4,e79e0c36669d180a,03846c23edfc4f64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:28:55.912 INFO 2981023 --- [Collect all tenants major compactions6,e79e0c36669d180a,c4d324e02895098f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:28:55.912 INFO 2981023 --- [Collect all tenants major compactions1,e79e0c36669d180a,a9983f69fe30a713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:28:55.914 INFO 2981023 --- [Collect all tenants major compactions3,e79e0c36669d180a,3f8a96189d037b03] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:28:55.914 INFO 2981023 --- [Collect all tenants major compactions3,e79e0c36669d180a,3f8a96189d037b03] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:28:55.914 INFO 2981023 --- [Collect all tenants major compactions2,e79e0c36669d180a,61e7099b0bb0d6a0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:28:55.914 INFO 2981023 --- [Collect all tenants major compactions2,e79e0c36669d180a,61e7099b0bb0d6a0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:28:55.914 INFO 2981023 --- [Collect all tenants major compactions5,e79e0c36669d180a,26003f037d399718] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:28:55.914 INFO 2981023 --- [Collect all tenants major compactions5,e79e0c36669d180a,26003f037d399718] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:28:55.915 INFO 2981023 --- [Collect all tenants major compactions4,e79e0c36669d180a,03846c23edfc4f64] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:28:55.915 INFO 2981023 --- [Collect all tenants major compactions4,e79e0c36669d180a,03846c23edfc4f64] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:28:55.915 INFO 2981023 --- [Collect all tenants major compactions1,e79e0c36669d180a,a9983f69fe30a713] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:28:55.915 INFO 2981023 --- [Collect all tenants major compactions6,e79e0c36669d180a,c4d324e02895098f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:28:55.915 INFO 2981023 --- [Collect all tenants major compactions1,e79e0c36669d180a,a9983f69fe30a713] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:28:55.915 INFO 2981023 --- [Collect all tenants major compactions6,e79e0c36669d180a,c4d324e02895098f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:28:55.916 INFO 2981023 --- [ocp-schedules-5,e79e0c36669d180a,b49186e703fa4db4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:28:56.081 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:56.081 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,7c88c7f8773bd9c0,cafbb19f00ba1339] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,7c88c7f8773bd9c0,cafbb19f00ba1339] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:28:56.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:28:56.354520+08:00, lastProcessTime=2024-09-11T17:28:46.352768+08:00 2024-09-11 17:28:56.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:28:56.354520+08:00 2024-09-11 17:28:57.162 INFO 2981023 --- [ocp-async-3,2a7af82ec1d49b4a,fb21090fbff88363] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:57.748 INFO 2981023 --- [ocp-async-4,383e2892ce14cd5c,67d8148db332d9e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:28:59.082 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:28:59.082 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:28:59.162 INFO 2981023 --- [ocp-async-5,f8b040324e8325a0,18d201a1adad991f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:28:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,75aa664fbb1c916c,ed26d1ace6b39197] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726046940, interval=60 2024-09-11 17:28:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,75aa664fbb1c916c,ed26d1ace6b39197] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:00.410 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:25:00.026, end=2024-09-11T17:27:00.406} 2024-09-11 17:29:00.412 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obCl 2024-09-11 17:29:00.413 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:29:00.414 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:25:00.026, end=2024-09-11T17:27:00.406},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:29:00.414 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:29:00.414 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:29:00.415 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:29:00.417 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 27 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:25:00.026, end=2024-09-11T17:27:00.406},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:29:00.439 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:29:00.439 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:29:00.448 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:29:00.483 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540483000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=D41D8CD98F00B204E9800998ECF8427E, obServerId=null, obDbId=201009, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.483Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.485 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540485000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=3A687177DEA9627C4760775163B1E57D, obServerId=null, obDbId=500005, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.485Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.485 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540485000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=895B7E0E45EA4177A2488EF3C1E6FC53, obServerId=null, obDbId=201009, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.485Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.486 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540486000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=8A8CBF7717773217DAD492C063701C64, obServerId=null, obDbId=500005, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.486Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.489 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 17:29:00.489 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.490 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:29:00.490 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.491 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 2 2024-09-11 17:29:00.491 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.491 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540491000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=B8876ACF397B67AAD7AA21573EFF469B, obServerId=null, obDbId=500005, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.491Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.492 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 17:29:00.492 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.493 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540493000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=1D74B418055588B8C94F1A2E4ECEA51D, obServerId=null, obDbId=500005, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.493Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.495 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540495000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=496854B3311EABEF4DF90C1E3BB81741, obServerId=null, obDbId=500005, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.495Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.496 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540496000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=3D78DDC36C3AC3483EF228982C0BDD3D, obServerId=null, obDbId=500005, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.496Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.498 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:29:00.498 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.499 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 17:29:00.499 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.499 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 17:29:00.499 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.499 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 17:29:00.499 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.502 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.502 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201009, sqlId=895B7E0E45EA4177A2488EF3C1E6FC53, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.502 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.502 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201009, sqlId=D41D8CD98F00B204E9800998ECF8427E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.504 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.504 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=3A687177DEA9627C4760775163B1E57D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.507 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.507 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=8A8CBF7717773217DAD492C063701C64, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.508 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.508 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.508 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.508 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.511 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.511 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.512 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.512 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.513 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.513 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=496854B3311EABEF4DF90C1E3BB81741, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.514 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.514 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=1D74B418055588B8C94F1A2E4ECEA51D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.515 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.515 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=3D78DDC36C3AC3483EF228982C0BDD3D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.515 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.515 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=B8876ACF397B67AAD7AA21573EFF469B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.517 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.517 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.517 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.517 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.517 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.517 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.518 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.518 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.519 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.519 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201009, sqlId=D41D8CD98F00B204E9800998ECF8427E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.522 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.522 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201009, sqlId=895B7E0E45EA4177A2488EF3C1E6FC53, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.523 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.523 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=8A8CBF7717773217DAD492C063701C64, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.524 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.525 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.525 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.525 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=3A687177DEA9627C4760775163B1E57D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.526 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.529 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.529 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=496854B3311EABEF4DF90C1E3BB81741, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.531 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.485, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=895B7E0E45EA4177A2488EF3C1E6FC53, serverId=null, dbName=NO_database, obDbId=201009, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.533 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:29:00.533 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.485, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=3A687177DEA9627C4760775163B1E57D, serverId=null, dbName=ocp, obDbId=500005, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.533 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.533 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.483, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=D41D8CD98F00B204E9800998ECF8427E, serverId=null, dbName=NO_database, obDbId=201009, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.533 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.533 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=3D78DDC36C3AC3483EF228982C0BDD3D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.534 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.534 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=1D74B418055588B8C94F1A2E4ECEA51D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.535 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.486, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=8A8CBF7717773217DAD492C063701C64, serverId=null, dbName=ocp, obDbId=500005, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.535 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.535 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=B8876ACF397B67AAD7AA21573EFF469B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.536 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:29:00.536 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.493, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=1D74B418055588B8C94F1A2E4ECEA51D, serverId=null, dbName=ocp, obDbId=500005, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.537 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.537 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.539 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.495, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=496854B3311EABEF4DF90C1E3BB81741, serverId=null, dbName=ocp, obDbId=500005, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.542 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.496, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=3D78DDC36C3AC3483EF228982C0BDD3D, serverId=null, dbName=ocp, obDbId=500005, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.542 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.491, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=B8876ACF397B67AAD7AA21573EFF469B, serverId=null, dbName=ocp, obDbId=500005, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.557 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540557000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=00D7D9798D373B9A5A20042B530A3A2C, obServerId=null, obDbId=500005, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.557Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.557 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540557000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=FDDFD2178ED36D79A3CA07281C82EDEB, obServerId=null, obDbId=500005, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:29:00.557Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.559 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 17:29:00.559 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.559 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 17:29:00.559 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.565 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.565 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=FDDFD2178ED36D79A3CA07281C82EDEB, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.565 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.565 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=00D7D9798D373B9A5A20042B530A3A2C, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.567 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.567 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.567 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.567 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.573 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.573 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.573 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=FDDFD2178ED36D79A3CA07281C82EDEB, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.573 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=500005, sqlId=00D7D9798D373B9A5A20042B530A3A2C, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.575 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.577 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.578 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.557, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=00D7D9798D373B9A5A20042B530A3A2C, serverId=null, dbName=ocp, obDbId=500005, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.579 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.557, end=2024-09-11T17:27:00.406}, clusterId=1000006, tenantId=1000006, sqlId=FDDFD2178ED36D79A3CA07281C82EDEB, serverId=null, dbName=ocp, obDbId=500005, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.580 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:29:00.581 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:25:00.026, end=2024-09-11T17:27:00.406},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:29:00.581 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:29:00.581 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:29:00.590 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:29:00.603 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540603000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.603Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.605 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:29:00.605 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.610 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.610 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.611 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.611 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.616 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.616 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.617 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:29:00.617 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.603, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.617 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:29:00.619 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:25:00.026, end=2024-09-11T17:27:00.406},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:29:00.620 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:29:00.620 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:29:00.656 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540656000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.656Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.670 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540670000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.670Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.671 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540671000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=8CA300BEB4EFD3CF2B6C5EEE281E3957, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.671Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.674 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540674000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.674Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.674 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:29:00.674 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.674 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540674000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.674Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.681 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 8 2024-09-11 17:29:00.681 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.683 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 59 2024-09-11 17:29:00.683 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.684 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:29:00.684 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.684 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540684000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.684Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.686 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540686000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.686Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.689 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540689000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.689Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.692 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.692 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8CA300BEB4EFD3CF2B6C5EEE281E3957, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.694 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:29:00.694 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.694 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.694 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.696 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.696 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.696 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.696 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.701 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.701 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.707 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:29:00.707 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.708 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.708 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.710 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:29:00.710 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.711 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.711 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.712 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.712 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.713 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:29:00.713 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.713 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.713 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.718 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.718 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.719 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.719 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.723 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.724 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.724 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.725 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.725 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.726 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.726 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.727 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.727 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.728 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.728 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.728 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.728 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.729 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.730 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.730 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.732 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.732 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.732 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.735 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.736 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.736 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.738 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.656, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.739 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.739 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8CA300BEB4EFD3CF2B6C5EEE281E3957, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.740 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.684, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.742 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.67, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.742 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.742 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.742 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.743 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.674, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.744 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.746 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.746 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.750 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.752 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.671, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=8CA300BEB4EFD3CF2B6C5EEE281E3957, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.754 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.686, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.755 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.755 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.756 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.689, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.759 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.765 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.674, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.768 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540768000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.768Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.772 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540772000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.772Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.780 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:29:00.780 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.780 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:29:00.780 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.783 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540783000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.783Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.785 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540785000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.785Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.788 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540788000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.788Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.795 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540795000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.795Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.801 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:29:00.801 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.801 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960540801000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:29:00.801Z, endTime=2024-09-11T09:27:00.406Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:29:00.807 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:29:00.807 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.810 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.810 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.811 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:29:00.811 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.812 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.812 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.812 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:29:00.812 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.818 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.818 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.822 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:29:00.822 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:29:00.825 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.825 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.828 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.828 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.830 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.830 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.834 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.834 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.836 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.836 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.838 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.838 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.838 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.838 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.841 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.843 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.843 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.845 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.845 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.847 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.847 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.847 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.851 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.851 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.851 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.851 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.852 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.772, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.854 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:29:00.854 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:29:00.857 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.857 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.859 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.860 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.768, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.863 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.783, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.865 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.865 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.866 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.866 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.867 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.870 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.871 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.871 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.871 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:29:00.871 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046820406000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:29:00.872 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.874 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:29:00.878 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.788, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.878 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.801, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.880 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.795, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.882 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:29:00.785, end=2024-09-11T17:27:00.406}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:29:00.884 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:29:00.885 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:25:00.026, end=2024-09-11T17:27:00.406},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:29:00.885 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:29:00.885 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:29:00.886 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:29:00.886 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:25:00.026, end=2024-09-11T17:27:00.406},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:29:00.886 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:29:00.886 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:29:00.887 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:29:01.138 INFO 2981023 --- [ocp-schedules-3,fa8dd37f1893e5b7,3d459e0b61420714] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:29:01.168 INFO 2981023 --- [ocp-async-1,4bd8a5c307402e39,2cde7c2e6e462366] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:01.533 WARN 2981023 --- [scanner-13490119-1,fa8dd37f1893e5b7,1cfa8074accab5d4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:29:01.825 INFO 2981023 --- [ocp-schedules-11,6eaea3f8ab7683d7,3cf206a73746684b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:29:02.082 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:02.083 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,3935479070220c31,dbc1171b355f25aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,3935479070220c31,dbc1171b355f25aa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:03.162 INFO 2981023 --- [ocp-async-7,6c4c2502d6e29c42,ff01f17f6b901795] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:03.368 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,40d7fcc2b72a58eb,242ca2d7d1f843f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=40d7fcc2b72a58eb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:29:03.368 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c11b8c31f14b6b0b,f8ebc4f824143bfe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=c11b8c31f14b6b0b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:29:03.370 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,40d7fcc2b72a58eb,242ca2d7d1f843f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=40d7fcc2b72a58eb, duration=6 ms] 2024-09-11 17:29:03.374 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c11b8c31f14b6b0b,f8ebc4f824143bfe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=c11b8c31f14b6b0b, duration=10 ms] 2024-09-11 17:29:03.381 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ce3db09d31b5d548,70d3e56ecc74dc6a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=ce3db09d31b5d548, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:29:03.381 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ce3db09d31b5d548,70d3e56ecc74dc6a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=ce3db09d31b5d548, duration=5 ms] 2024-09-11 17:29:05.083 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:05.083 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:05.162 INFO 2981023 --- [ocp-async-10,a214f79d24d70163,10aa8855875d0dd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,85117f57b5b2962c,d56bb8c499bef7c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,85117f57b5b2962c,d56bb8c499bef7c7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:29:06.336917+08:00, lastProcessTime=2024-09-11T17:28:56.354520+08:00 2024-09-11 17:29:06.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:29:06.336917+08:00 2024-09-11 17:29:07.163 INFO 2981023 --- [ocp-async-4,c1aaa1efaaa0cf5a,a29ecd2a35736e45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:07.231 INFO 2981023 --- [ocp-schedules-3,7997152656904ddc,47f9fe0a97f9a3c2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:29:07.235 INFO 2981023 --- [ocp-schedules-11,46a8277adab0ee38,a64201bf0d6fe9f8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:29:07.235+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:29:07.236 INFO 2981023 --- [ocp-schedules-11,46a8277adab0ee38,a64201bf0d6fe9f8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:29:07.237 INFO 2981023 --- [ocp-schedules-11,46a8277adab0ee38,a64201bf0d6fe9f8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:29:07.237 INFO 2981023 --- [ocp-schedules-11,46a8277adab0ee38,a64201bf0d6fe9f8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:29:07.241 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,99dbb73b761b95bc,d3c061e074ed151d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=99dbb73b761b95bc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:29:07.241 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a989853d3c33b1c1,dc6b24f488c58cca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=a989853d3c33b1c1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:29:07.242 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,99dbb73b761b95bc,d3c061e074ed151d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=99dbb73b761b95bc, duration=3 ms] 2024-09-11 17:29:07.246 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a989853d3c33b1c1,dc6b24f488c58cca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=a989853d3c33b1c1, duration=7 ms] 2024-09-11 17:29:07.267 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,10d31b4193333472,d7fdd0849afc9896] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=10d31b4193333472, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:29:07.268 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,10d31b4193333472,d7fdd0849afc9896] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=10d31b4193333472, duration=17 ms] 2024-09-11 17:29:07.748 INFO 2981023 --- [ocp-async-5,fcfca63c462e7750,f49292960264dea6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:29:08.084 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:08.084 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,8beba934781b29a1,560f813552e057a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,8beba934781b29a1,560f813552e057a7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:09.163 INFO 2981023 --- [ocp-async-7,6f4b0729fa7a2d72,3b15ef2bab5b6a0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:10.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:29:10.004 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:29:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:29:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:29:11.084 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:11.085 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:11.163 INFO 2981023 --- [ocp-async-10,e8880b1ea4c13f76,1d0287e61a6d71a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,9063987fb41b6ff9,4643cc2ac3eb2bfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,9063987fb41b6ff9,4643cc2ac3eb2bfe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:11.673 INFO 2981023 --- [ocp-schedules-24,12c5af1bdd24af93,faf06c3f9690a437] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:29:11.741 WARN 2981023 --- [scanner-13491815-1,12c5af1bdd24af93,b9a7f4a3584b37e8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:29:11.749 INFO 2981023 --- [alarm-detect-7,12c5af1bdd24af93,440e26ee9eb91549] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643888, data is empty. 2024-09-11 17:29:11.753 INFO 2981023 --- [alarm-detect-7,12c5af1bdd24af93,440e26ee9eb91549] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643934, data is empty. 2024-09-11 17:29:11.758 INFO 2981023 --- [alarm-detect-7,12c5af1bdd24af93,440e26ee9eb91549] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643934, data is empty. 2024-09-11 17:29:11.782 INFO 2981023 --- [alarm-detect-2,12c5af1bdd24af93,59a19d4c7a46caab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643888, data is empty. 2024-09-11 17:29:11.784 INFO 2981023 --- [alarm-detect-3,12c5af1bdd24af93,b63c9ca1202207a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643888, data is empty. 2024-09-11 17:29:11.789 INFO 2981023 --- [alarm-detect-3,12c5af1bdd24af93,b63c9ca1202207a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643934, data is empty. 2024-09-11 17:29:11.790 INFO 2981023 --- [alarm-detect-2,12c5af1bdd24af93,59a19d4c7a46caab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643934, data is empty. 2024-09-11 17:29:11.792 INFO 2981023 --- [alarm-detect-3,12c5af1bdd24af93,b63c9ca1202207a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643934, data is empty. 2024-09-11 17:29:11.794 INFO 2981023 --- [alarm-detect-2,12c5af1bdd24af93,59a19d4c7a46caab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643934, data is empty. 2024-09-11 17:29:11.827 INFO 2981023 --- [ocp-schedules-25,076f2beec8288377,9b86ce6cfa14786b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:29:11.906 INFO 2981023 --- [alarm-detect-2,12c5af1bdd24af93,132f8f8feda1abb3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643888, data is empty. 2024-09-11 17:29:11.914 INFO 2981023 --- [alarm-detect-2,12c5af1bdd24af93,132f8f8feda1abb3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643934, data is empty. 2024-09-11 17:29:11.918 INFO 2981023 --- [alarm-detect-2,12c5af1bdd24af93,132f8f8feda1abb3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643934, data is empty. 2024-09-11 17:29:12.068 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046880, nodeCount=480, affectRows=480 2024-09-11 17:29:13.163 INFO 2981023 --- [ocp-async-2,a3752b5bddc34483,c22752c2d6ac3d96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:13.650 INFO 2981023 --- [ocp-schedules-26,2043ab1ba50882c5,e5f7aa94675fff42] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:29:13.656 INFO 2981023 --- [ocp-schedules-26,2043ab1ba50882c5,e5f7aa94675fff42] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:29:13.656 INFO 2981023 --- [ocp-schedules-26,2043ab1ba50882c5,e5f7aa94675fff42] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:29:13.874 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046880, nodeCount=1000, affectRows=1000 2024-09-11 17:29:13.876 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046880, nodeCount=999, validCount=999 2024-09-11 17:29:13.914 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046880, nodeCount=480, validCount=480 2024-09-11 17:29:13.971 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046880, nodeCount=1000, affectRows=1000 2024-09-11 17:29:13.974 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046880, nodeCount=1000, validCount=1000 2024-09-11 17:29:13.981 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046880, nodeCount=48, affectRows=48 2024-09-11 17:29:13.981 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046880, nodeCount=48, validCount=48 2024-09-11 17:29:14.085 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:14.086 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,e0c6d7d827eac5ef,6cb44544b618fc8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,c71accc7341be070,e7db51ba64aa8526] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:29:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,e0c6d7d827eac5ef,6cb44544b618fc8b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:15.163 INFO 2981023 --- [ocp-async-3,065d103e0d1bb9d1,e491b7a1d30c4c0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:16.023 INFO 2981023 --- [ocp-schedules-28,39af3d294ba46426,2de44c7b2d4d22c0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:29:16.023 INFO 2981023 --- [host-check-clockdiff1,39af3d294ba46426,2d8835cc0b79879a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:29:16.023 INFO 2981023 --- [host-check-clockdiff2,39af3d294ba46426,b68b701d340d3361] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:29:16.024 INFO 2981023 --- [ocp-schedules-16,41bb4ad26838c03a,c6948f1a85185dda] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:29:16.025 INFO 2981023 --- [host-check-clockdiff3,39af3d294ba46426,70bffe3a5fe201b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:29:16.025 INFO 2981023 --- [host-check-clockdiff4,39af3d294ba46426,1c698183ebe88589] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:29:16.025 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:29:16.029 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:29:16.029 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:29:16.030 INFO 2981023 --- [ocp-schedules-10,1255e23936b63f50,65018bd2fd1ed685] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:29:16.047 INFO 2981023 --- [host-check-agent-status1,1255e23936b63f50,c49329151c8b4b39] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:29:16.067 INFO 2981023 --- [host-check-clockdiff2,39af3d294ba46426,b68b701d340d3361] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046956 0 0 , error= 2024-09-11 17:29:16.067 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:29:16.068 INFO 2981023 --- [host-check-agent-status2,1255e23936b63f50,c81a86d9035da975] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:29:16.068 INFO 2981023 --- [host-check-agent-status3,1255e23936b63f50,06cdc948cb09f8c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:29:16.069 INFO 2981023 --- [host-check-agent-status4,1255e23936b63f50,cacd0254d964997e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:29:16.083 INFO 2981023 --- [host-check-clockdiff4,39af3d294ba46426,1c698183ebe88589] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046956 0 0 , error= 2024-09-11 17:29:16.096 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:29:16.100 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:29:16.104 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:29:16.104 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:29:16.105 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:29:16.107 INFO 2981023 --- [host-check-agent-status1,1255e23936b63f50,c49329151c8b4b39] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:29:16.108 INFO 2981023 --- [host-check-agent-status1,1255e23936b63f50,c49329151c8b4b39] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:29:16.114 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:29:16.116 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:29:16.122 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:29:16.194 INFO 2981023 --- [ocp-schedules-1,b6c4c72dc2d2e242,fdf02ef716e352db] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:29:16.195 INFO 2981023 --- [ocp-schedules-2,c5f10e5ffa95cc61,63895912a198e523] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:29:16.195 INFO 2981023 --- [ocp-schedules-4,476e8978a57b904c,bf7508a302e8b758] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:29:16.195 INFO 2981023 --- [ocp-schedules-24,5128947f0cbc0867,fbc5fa903f516a8c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:29:16.197 INFO 2981023 --- [ocp-schedules-4,476e8978a57b904c,bf7508a302e8b758] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:29:16.205 INFO 2981023 --- [ocp-schedules-1,b6c4c72dc2d2e242,fdf02ef716e352db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:29:16.207 INFO 2981023 --- [host-check-agent-status4,1255e23936b63f50,cacd0254d964997e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:29:16.211 INFO 2981023 --- [host-check-agent-status3,1255e23936b63f50,06cdc948cb09f8c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:29:16.212 INFO 2981023 --- [host-check-agent-status2,1255e23936b63f50,c81a86d9035da975] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:29:16.261 INFO 2981023 --- [host-check-agent-status3,1255e23936b63f50,06cdc948cb09f8c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:29:16.265 INFO 2981023 --- [host-check-agent-status2,1255e23936b63f50,c81a86d9035da975] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:29:16.265 INFO 2981023 --- [host-check-agent-status4,1255e23936b63f50,cacd0254d964997e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:29:16.268 INFO 2981023 --- [ocp-schedules-2,c5f10e5ffa95cc61,63895912a198e523] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:29:16.273 INFO 2981023 --- [ocp-schedules-2,c5f10e5ffa95cc61,63895912a198e523] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:29:16.283 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:29:16.286 INFO 2981023 --- [ocp-schedules-8,eaf138081e316651,37ce104ff93da682] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:29:16.287 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:29:16.286770188+08:00 2024-09-11 17:29:16.290 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:29:16.287+08:00 2024-09-11 17:29:16.300 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:29:16.287+08:00 and status AVAILABLE 2024-09-11 17:29:16.307 INFO 2981023 --- [host-check-agent-status1,1255e23936b63f50,c49329151c8b4b39] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:29:16.311 INFO 2981023 --- [host-check-agent-status1,1255e23936b63f50,c49329151c8b4b39] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:29:16.311 INFO 2981023 --- [host-check-agent-status1,1255e23936b63f50,c49329151c8b4b39] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:29:16.311 INFO 2981023 --- [host-check-agent-status1,1255e23936b63f50,c49329151c8b4b39] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:29:16.318 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:29:16.322 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.322 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:16.327 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.327 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:16.328 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:29:16.328 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:29:16.330 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:29:16.329737528+08:00 2024-09-11 17:29:16.332 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:29:16.330+08:00 2024-09-11 17:29:16.334 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.334 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.334 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.335 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:16.335 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:16.335 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:29:16.335 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:16.355 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:29:16.330+08:00 and status AVAILABLE 2024-09-11 17:29:16.356 INFO 2981023 --- [host-check-agent-status2,1255e23936b63f50,c81a86d9035da975] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:29:16.356 INFO 2981023 --- [host-check-agent-status4,1255e23936b63f50,cacd0254d964997e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:29:16.357 INFO 2981023 --- [host-check-agent-status3,1255e23936b63f50,06cdc948cb09f8c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:29:16.358 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.358 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:16.359 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:29:16.360 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:29:16.360 INFO 2981023 --- [host-check-agent-status4,1255e23936b63f50,cacd0254d964997e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:29:16.360 INFO 2981023 --- [host-check-agent-status4,1255e23936b63f50,cacd0254d964997e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:29:16.360 INFO 2981023 --- [host-check-agent-status4,1255e23936b63f50,cacd0254d964997e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:29:16.360 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:29:16.360 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:29:16.360548184+08:00 2024-09-11 17:29:16.361 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:29:16.360846006+08:00 2024-09-11 17:29:16.361 INFO 2981023 --- [host-check-agent-status3,1255e23936b63f50,06cdc948cb09f8c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:29:16.361 INFO 2981023 --- [host-check-agent-status2,1255e23936b63f50,c81a86d9035da975] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:29:16.361 INFO 2981023 --- [host-check-agent-status3,1255e23936b63f50,06cdc948cb09f8c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:29:16.361 INFO 2981023 --- [host-check-agent-status2,1255e23936b63f50,c81a86d9035da975] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:29:16.361 INFO 2981023 --- [host-check-agent-status3,1255e23936b63f50,06cdc948cb09f8c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:29:16.361 INFO 2981023 --- [host-check-agent-status2,1255e23936b63f50,c81a86d9035da975] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:29:16.361 INFO 2981023 --- [ocp-schedules-1,b6c4c72dc2d2e242,fdf02ef716e352db] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:29:16.362 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:29:16.360+08:00 2024-09-11 17:29:16.362 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:29:16.361+08:00 2024-09-11 17:29:16.364 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:29:16.360+08:00 and status AVAILABLE 2024-09-11 17:29:16.364 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:29:16.361+08:00 and status AVAILABLE 2024-09-11 17:29:16.364 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:29:16.370 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:29:16.368020+08:00, lastProcessTime=2024-09-11T17:29:06.336917+08:00 2024-09-11 17:29:16.371 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:29:16.371 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:29:16.374 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:29:16.376 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:29:16.377 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:29:16.379 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:29:16.385 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:29:16.387 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:29:16.398 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a67cad7eb4cd6817,20f325d2dff842be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=a67cad7eb4cd6817, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:29:16.398 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,7cdd926e9d952fd4,2c1a276f464b9e2c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=7cdd926e9d952fd4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:29:16.399 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:29:16.401 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:29:16.408 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,7cdd926e9d952fd4,2c1a276f464b9e2c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=7cdd926e9d952fd4, duration=14 ms] 2024-09-11 17:29:16.415 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:29:16.417 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,b7e558c3a27e2657,f09c56e466514e26] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=b7e558c3a27e2657, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:29:16.420 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,b7e558c3a27e2657,f09c56e466514e26] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=b7e558c3a27e2657, duration=14 ms] 2024-09-11 17:29:16.423 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.423 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.424 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a67cad7eb4cd6817,20f325d2dff842be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=a67cad7eb4cd6817, duration=30 ms] 2024-09-11 17:29:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:29:16.428 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:29:16.433 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:29:16.440 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.441 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.442 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:29:16.444 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:29:16.449 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:29:16.466 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:29:16.466 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:29:16.468 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:29:16.473 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:29:16.478 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:29:16.479 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.479 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.480 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:16.480 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:29:16.480 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:29:16.481 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:29:16.484 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:29:16.494 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:29:16.504 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.505 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.505 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:16.505 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:29:16.505 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:16.506 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.506 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:29:16.508 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:29:16.509 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:29:16.510 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:29:16.515 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:29:16.522 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.522 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:16.522 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:29:16.523 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.524 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:29:16.528 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:29:16.533 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:16.534 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:29:16.536 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:29:16.539 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:29:16.542 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:16.542 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:29:16.545 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.546 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.546 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:16.547 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:29:16.548 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:29:16.548 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:29:16.553 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:29:16.558 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:29:16.559 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.560 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:16.560 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:16.561 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:29:16.561 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:29:16.561 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.564 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:29:16.565 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:29:16.566 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:29:16.566 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:29:16.567 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:29:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:29:16.573 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:29:16.575 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:29:16.576 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:29:16.578 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:29:16.582 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:16.583 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:29:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:29:16.594 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:29:16.597 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:29:16.604 INFO 2981023 --- [host-check-host-status4,41bb4ad26838c03a,1187003aba857364] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:29:16.606 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:29:16.609 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:16.610 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:29:16.612 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:29:16.616 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:29:16.617 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.618 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.619 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:16.619 INFO 2981023 --- [host-check-host-status2,41bb4ad26838c03a,7e79f0fca4071ffc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:29:16.619 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:29:16.625 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:29:16.627 INFO 2981023 --- [host-check-host-status3,41bb4ad26838c03a,0afe6a49f3ccb5aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:29:16.633 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:29:16.634 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:29:16.636 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:29:16.645 INFO 2981023 --- [host-check-host-status1,41bb4ad26838c03a,b9a4dcb508e4c23e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:29:16.659 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:29:16.670 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:29:16.673 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:29:16.677 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.678 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.678 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:29:16.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.681 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.682 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:29:16.685 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:29:16.685 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:29:16.688 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:29:16.688 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:29:16.689 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:29:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:29:16.702 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:29:16.702 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:29:16.702 INFO 2981023 --- [ocp-schedules-4,476e8978a57b904c,bf7508a302e8b758] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:29:16.710 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:29:16.368020+08:00 2024-09-11 17:29:17.086 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:17.087 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:17.163 INFO 2981023 --- [ocp-async-9,53fd51c04f5a1254,4e9be401f626e239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,f5444ba743c773ba,e79659e8ab28f64f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,f5444ba743c773ba,e79659e8ab28f64f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:17.748 INFO 2981023 --- [ocp-async-7,63b313905442ffc0,c330e33ddfd7cdf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:29:18.562 INFO 2981023 --- [host-check-clockdiff1,39af3d294ba46426,2d8835cc0b79879a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046958 0 0 , error= 2024-09-11 17:29:19.163 INFO 2981023 --- [ocp-async-2,0fa5209bbb0b0a9b,305a87ca4480c0b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:20.087 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:20.089 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,c5f748adaeb5d579,f63a67cef61e19c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,c5f748adaeb5d579,f63a67cef61e19c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:21.163 INFO 2981023 --- [ocp-async-3,d097aa2d7eae684d,b5c16b9ef620fb26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:21.833 INFO 2981023 --- [ocp-schedules-16,f0dfad136a37ea3e,ab09e59fc1fb5d8a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:29:21.993 INFO 2981023 --- [ocp-schedules-18,be2aad4adffb7d88,3b76cd570e4d2abe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:29:22.043 WARN 2981023 --- [scanner-13492156-1,be2aad4adffb7d88,2d0725340802efb9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:29:22.056 INFO 2981023 --- [host-check-clockdiff3,39af3d294ba46426,70bffe3a5fe201b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046962 1 1 , error= 2024-09-11 17:29:22.467 INFO 2981023 --- [ocp-schedules-26,4db5397c1654e3fb,bb9dabee112e8b7f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:29:22.467 INFO 2981023 --- [ocp-schedules-26,4db5397c1654e3fb,bb9dabee112e8b7f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:29:22.467 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:29:22.467 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:29:22.467 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:29:22.470 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:29:22.471 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:29:22.474 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:29:22.474 INFO 2981023 --- [ocp-schedules-26,4db5397c1654e3fb,bb9dabee112e8b7f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:29:22.474 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:29:22.474 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:29:22.475 INFO 2981023 --- [ocp-schedules-26,4db5397c1654e3fb,bb9dabee112e8b7f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:29:22.475 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:29:22.501 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:29:22.502 INFO 2981023 --- [ocp-schedules-26,4db5397c1654e3fb,bb9dabee112e8b7f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:29:22.502 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:29:22.503 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:29:22.505 INFO 2981023 --- [ocp-schedules-26,4db5397c1654e3fb,bb9dabee112e8b7f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:29:23.089 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:23.091 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:23.163 INFO 2981023 --- [ocp-async-8,552f56ea17923629,f97d8e3d88bb3b4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,a08482cc7dff4ea8,b41354b822aeec3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,a08482cc7dff4ea8,b41354b822aeec3f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:24.188 INFO 2981023 --- [ocp-schedules-4,a51b5ccd1e1162b3,bd9618727aabc3f2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:29:25.164 INFO 2981023 --- [ocp-async-4,7a723c4949b71926,eefe3698437e11ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:26.091 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:26.092 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,a688a991d725d13d,efeea74a2be7af4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,a688a991d725d13d,efeea74a2be7af4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:26.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:29:26.350532+08:00, lastProcessTime=2024-09-11T17:29:16.368020+08:00 2024-09-11 17:29:26.475 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:29:26.350532+08:00 2024-09-11 17:29:27.164 INFO 2981023 --- [ocp-async-6,e432f7bedd8f61ec,3bdd34e2f6b03373] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:27.337 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2551f7c7ade185fe,841fbcc145e7d05e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=2551f7c7ade185fe, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:29:27.337 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,3e32fb30d0507adf,b19f39b1934e3902] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=3e32fb30d0507adf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:29:27.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2551f7c7ade185fe,841fbcc145e7d05e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=2551f7c7ade185fe, duration=5 ms] 2024-09-11 17:29:27.342 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,3e32fb30d0507adf,b19f39b1934e3902] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=3e32fb30d0507adf, duration=9 ms] 2024-09-11 17:29:27.354 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,5b444da192948c44,1974570439aa0560] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=5b444da192948c44, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:29:27.356 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,5b444da192948c44,1974570439aa0560] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=5b444da192948c44, duration=12 ms] 2024-09-11 17:29:27.748 INFO 2981023 --- [ocp-async-2,50439a5c002d25a5,8946f3ca87c68887] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:29:29.092 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:29.093 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:29.164 INFO 2981023 --- [ocp-async-9,0955f7cd9f5740d7,b2c9a812b5f371a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,69736b55caff5c1f,9ab84846833d776d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,69736b55caff5c1f,9ab84846833d776d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:31.164 INFO 2981023 --- [ocp-async-5,82634d9e46563f63,32bb7a1497ee7e99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:31.836 INFO 2981023 --- [ocp-schedules-11,ae43bf32b8eb2fcd,864705d8060064fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:29:32.093 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:32.093 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:32.185 INFO 2981023 --- [ocp-schedules-21,7d37021c98380fe0,0db8a6c988b69d81] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:29:32.210 WARN 2981023 --- [scanner-13492364-1,7d37021c98380fe0,278ba18e6c520670] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:29:32.258 INFO 2981023 --- [alarm-detect-6,7d37021c98380fe0,81ed484e8d716470] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643934, data is empty. 2024-09-11 17:29:32.263 INFO 2981023 --- [alarm-detect-6,7d37021c98380fe0,81ed484e8d716470] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643934, data is empty. 2024-09-11 17:29:32.272 INFO 2981023 --- [alarm-detect-6,7d37021c98380fe0,81ed484e8d716470] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643934, data is empty. 2024-09-11 17:29:32.280 INFO 2981023 --- [alarm-detect-12,7d37021c98380fe0,6493bad742c1ce09] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643934, data is empty. 2024-09-11 17:29:32.281 INFO 2981023 --- [alarm-detect-1,7d37021c98380fe0,820f492008bbd910] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643934, data is empty. 2024-09-11 17:29:32.285 INFO 2981023 --- [alarm-detect-12,7d37021c98380fe0,6493bad742c1ce09] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643934, data is empty. 2024-09-11 17:29:32.288 INFO 2981023 --- [alarm-detect-12,7d37021c98380fe0,6493bad742c1ce09] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643934, data is empty. 2024-09-11 17:29:32.288 INFO 2981023 --- [alarm-detect-1,7d37021c98380fe0,820f492008bbd910] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643934, data is empty. 2024-09-11 17:29:32.294 INFO 2981023 --- [alarm-detect-1,7d37021c98380fe0,820f492008bbd910] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643934, data is empty. 2024-09-11 17:29:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,95195462d4d1adaf,bb0b4d6950e883ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:32.324 INFO 2981023 --- [prometheus-minute-schedule-1,95195462d4d1adaf,bb0b4d6950e883ef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:32.413 INFO 2981023 --- [alarm-detect-5,7d37021c98380fe0,5bb763fcdadfbafd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643934, data is empty. 2024-09-11 17:29:32.421 INFO 2981023 --- [alarm-detect-5,7d37021c98380fe0,5bb763fcdadfbafd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643934, data is empty. 2024-09-11 17:29:32.428 INFO 2981023 --- [alarm-detect-5,7d37021c98380fe0,5bb763fcdadfbafd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643934, data is empty. 2024-09-11 17:29:33.164 INFO 2981023 --- [ocp-async-6,dfe3907c936f9589,bbadd64f3b96e9d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:33.659 INFO 2981023 --- [ocp-schedules-25,683cf6d1c30fb3cb,e909df34ffe99007] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:29:33.671 INFO 2981023 --- [ocp-schedules-25,683cf6d1c30fb3cb,e909df34ffe99007] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:29:33.671 INFO 2981023 --- [ocp-schedules-25,683cf6d1c30fb3cb,e909df34ffe99007] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:29:35.093 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:35.094 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:35.164 INFO 2981023 --- [ocp-async-8,1659cc42a258573e,e41f05ecf0223c08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,3d7dd24138b44980,41d371cbfca48dda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,3d7dd24138b44980,41d371cbfca48dda] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:29:36.336525+08:00, lastProcessTime=2024-09-11T17:29:26.350532+08:00 2024-09-11 17:29:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:29:36.336525+08:00 2024-09-11 17:29:37.164 INFO 2981023 --- [ocp-async-4,acfd1ec21c3f29a1,4195f760a21552c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:37.314 INFO 2981023 --- [prometheus-cache-monitor1,21a109305d1c85fe,e9ca306d3eae4e28] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:29:37.315 INFO 2981023 --- [prometheus-cache-monitor1,21a109305d1c85fe,e9ca306d3eae4e28] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:29:37.748 INFO 2981023 --- [ocp-async-8,8b36faa8daaf7e5a,4d65dd81a2c390e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:29:38.094 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:38.095 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,f56814157a55e112,dfe5f40384f4ee74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,f56814157a55e112,dfe5f40384f4ee74] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:38.424 INFO 2981023 --- [ocp-schedules-8,1b522a00367b3671,1aa1e76dbb9e819a] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:29:38.441 WARN 2981023 --- [ocp-schedules-8,1b522a00367b3671,1aa1e76dbb9e819a] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:29:39.164 INFO 2981023 --- [ocp-async-6,e9ca506afec22b71,0ce109d9821ad74e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:40.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:29:40.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:29:41.095 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:41.096 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:41.164 INFO 2981023 --- [ocp-async-7,c8887f3208ed6992,8b45352e75f425ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,c57d2f2d592c39c6,2bd27fbbe0cf9bc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,c57d2f2d592c39c6,2bd27fbbe0cf9bc6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:41.839 INFO 2981023 --- [ocp-schedules-16,e15b3502739ee4c6,70c87a8181fc0d36] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:29:42.499 INFO 2981023 --- [ocp-schedules-20,a6911412dde49d1f,a28f0288f8be514b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:29:42.549 WARN 2981023 --- [scanner-13492704-1,a6911412dde49d1f,20b2ce4650e89b8b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:29:43.164 INFO 2981023 --- [ocp-async-10,013aa9627a6ef474,27783255cdee5b15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:43.506 INFO 2981023 --- [ocp-schedules-13,77039030b6a17cbe,147e86f04f867a84] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:29:44.096 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:44.097 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,de5abf23a8f7ad51,455e62ac3617d40a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,de5abf23a8f7ad51,455e62ac3617d40a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:45.101 INFO 2981023 --- [prometheus-second-schedule-1,3a6a21a814c7326d,dfec97743477f863] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:29:45.165 INFO 2981023 --- [ocp-async-5,ffebe3afd1944d6e,2c1552c2d536105f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:46.016 INFO 2981023 --- [ocp-schedules-3,f26199334a2a0fef,95e86e80d9c0abe2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:29:46.017 INFO 2981023 --- [host-check-clockdiff1,f26199334a2a0fef,c106c03c6506b28b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:29:46.017 INFO 2981023 --- [host-check-clockdiff2,f26199334a2a0fef,c1c3b88fcb879a20] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:29:46.018 INFO 2981023 --- [host-check-clockdiff3,f26199334a2a0fef,5331eb9885846ef0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:29:46.018 INFO 2981023 --- [host-check-clockdiff4,f26199334a2a0fef,95b3acb0c889afb0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:29:46.027 INFO 2981023 --- [ocp-schedules-19,e2b865135ea625a3,8f661702d11e463a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:29:46.027 INFO 2981023 --- [ocp-schedules-21,197732f3d8c71953,dc75ce7e9cc381db] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:29:46.030 INFO 2981023 --- [host-check-clockdiff1,f26199334a2a0fef,c106c03c6506b28b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046986 0 0 , error= 2024-09-11 17:29:46.032 INFO 2981023 --- [host-check-agent-status1,197732f3d8c71953,48a263c079987bd9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:29:46.032 INFO 2981023 --- [host-check-agent-status2,197732f3d8c71953,4203d73ba1754f96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:29:46.033 INFO 2981023 --- [host-check-agent-status3,197732f3d8c71953,910ae2322773ae6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:29:46.033 INFO 2981023 --- [host-check-agent-status4,197732f3d8c71953,82d4ead8ac63375d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:29:46.034 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:29:46.034 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:29:46.038 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:29:46.038 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:29:46.039 INFO 2981023 --- [host-check-agent-status1,197732f3d8c71953,48a263c079987bd9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:29:46.039 INFO 2981023 --- [host-check-clockdiff4,f26199334a2a0fef,95b3acb0c889afb0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046986 0 0 , error= 2024-09-11 17:29:46.039 INFO 2981023 --- [host-check-clockdiff2,f26199334a2a0fef,c1c3b88fcb879a20] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046986 0 0 , error= 2024-09-11 17:29:46.040 INFO 2981023 --- [host-check-agent-status2,197732f3d8c71953,4203d73ba1754f96] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:29:46.041 INFO 2981023 --- [host-check-agent-status3,197732f3d8c71953,910ae2322773ae6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:29:46.042 INFO 2981023 --- [host-check-agent-status4,197732f3d8c71953,82d4ead8ac63375d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:29:46.043 INFO 2981023 --- [host-check-agent-status1,197732f3d8c71953,48a263c079987bd9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:29:46.043 INFO 2981023 --- [host-check-agent-status4,197732f3d8c71953,82d4ead8ac63375d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:29:46.043 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:29:46.045 INFO 2981023 --- [host-check-agent-status3,197732f3d8c71953,910ae2322773ae6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:29:46.047 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:29:46.047 INFO 2981023 --- [host-check-agent-status2,197732f3d8c71953,4203d73ba1754f96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:29:46.055 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:29:46.057 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:29:46.057 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:29:46.058 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:29:46.059 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:29:46.059 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:29:46.078 INFO 2981023 --- [host-check-agent-status3,197732f3d8c71953,910ae2322773ae6d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:29:46.078 INFO 2981023 --- [host-check-agent-status4,197732f3d8c71953,82d4ead8ac63375d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:29:46.079 INFO 2981023 --- [host-check-agent-status1,197732f3d8c71953,48a263c079987bd9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:29:46.082 INFO 2981023 --- [host-check-agent-status4,197732f3d8c71953,82d4ead8ac63375d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:29:46.082 INFO 2981023 --- [host-check-agent-status4,197732f3d8c71953,82d4ead8ac63375d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:29:46.082 INFO 2981023 --- [host-check-agent-status4,197732f3d8c71953,82d4ead8ac63375d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:29:46.083 INFO 2981023 --- [host-check-agent-status3,197732f3d8c71953,910ae2322773ae6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:29:46.083 INFO 2981023 --- [host-check-agent-status3,197732f3d8c71953,910ae2322773ae6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:29:46.083 INFO 2981023 --- [host-check-agent-status3,197732f3d8c71953,910ae2322773ae6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:29:46.085 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:29:46.087 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:29:46.086796885+08:00 2024-09-11 17:29:46.088 INFO 2981023 --- [host-check-agent-status1,197732f3d8c71953,48a263c079987bd9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:29:46.088 INFO 2981023 --- [host-check-agent-status1,197732f3d8c71953,48a263c079987bd9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:29:46.088 INFO 2981023 --- [host-check-agent-status1,197732f3d8c71953,48a263c079987bd9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:29:46.089 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:29:46.087+08:00 2024-09-11 17:29:46.091 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:29:46.087+08:00 and status AVAILABLE 2024-09-11 17:29:46.099 INFO 2981023 --- [host-check-agent-status2,197732f3d8c71953,4203d73ba1754f96] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:29:46.102 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:29:46.103 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:29:46.103 INFO 2981023 --- [host-check-agent-status2,197732f3d8c71953,4203d73ba1754f96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:29:46.103 INFO 2981023 --- [host-check-agent-status2,197732f3d8c71953,4203d73ba1754f96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:29:46.103 INFO 2981023 --- [host-check-agent-status2,197732f3d8c71953,4203d73ba1754f96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:29:46.103 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:29:46.103573487+08:00 2024-09-11 17:29:46.104 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:29:46.103+08:00 2024-09-11 17:29:46.106 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:29:46.106 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:29:46.103+08:00 and status AVAILABLE 2024-09-11 17:29:46.106 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:29:46.106108477+08:00 2024-09-11 17:29:46.107 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:29:46.106+08:00 2024-09-11 17:29:46.109 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:29:46.109 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:29:46.106+08:00 and status AVAILABLE 2024-09-11 17:29:46.119 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:29:46.103574420+08:00 2024-09-11 17:29:46.123 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:29:46.126 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:29:46.119+08:00 2024-09-11 17:29:46.141 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:29:46.144 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:29:46.145 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:29:46.145 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:29:46.146 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:29:46.146 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:29:46.119+08:00 and status AVAILABLE 2024-09-11 17:29:46.162 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:29:46.163 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:29:46.163 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:29:46.183 INFO 2981023 --- [ocp-schedules-7,76adc4ca0ed0997c,65378629e761e461] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:29:46.187 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:29:46.188 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:29:46.189 INFO 2981023 --- [ocp-schedules-29,a4c2ca1f184cc8d6,18e327513b107428] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:29:46.189 INFO 2981023 --- [ocp-schedules-10,4cccbbe5a2742579,415dce024a285bfa] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:29:46.189 INFO 2981023 --- [ocp-schedules-12,226bd9c7eb8e88ab,c47f015a354e8c80] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:29:46.189 INFO 2981023 --- [ocp-schedules-5,697da63843e04814,01087d7eb0757e75] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:29:46.191 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:29:46.193 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:29:46.193 INFO 2981023 --- [ocp-schedules-10,4cccbbe5a2742579,415dce024a285bfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:29:46.198 INFO 2981023 --- [ocp-schedules-29,a4c2ca1f184cc8d6,18e327513b107428] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:29:46.201 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:29:46.209 INFO 2981023 --- [ocp-schedules-12,226bd9c7eb8e88ab,c47f015a354e8c80] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:29:46.213 INFO 2981023 --- [ocp-schedules-7,76adc4ca0ed0997c,65378629e761e461] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:29:46.214 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:29:46.219 INFO 2981023 --- [ocp-schedules-10,4cccbbe5a2742579,415dce024a285bfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:29:46.227 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:29:46.227 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:29:46.229 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:29:46.236 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:29:46.244 INFO 2981023 --- [ocp-schedules-20,0444952f54bc7416,7161d673a1bc575c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:29:46.254 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:29:46.285 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:29:46.308 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:29:46.321 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:29:46.321 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:29:46.327 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:29:46.334 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:29:46.337 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.337 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.337 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:46.338 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:46.338 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:29:46.339 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:29:46.342 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.342 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:29:46.343 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:46.343 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:29:46.344 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:29:46.344 INFO 2981023 --- [ocp-schedules-29,a4c2ca1f184cc8d6,18e327513b107428] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:29:46.345 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.345 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.345 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:46.345 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:29:46.346 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:29:46.346 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.350 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:29:46.349597+08:00, lastProcessTime=2024-09-11T17:29:36.336525+08:00 2024-09-11 17:29:46.360 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:29:46.362 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:29:46.363 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:29:46.370 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:29:46.375 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.377 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:46.381 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:29:46.391 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:29:46.412 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.413 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:46.416 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:29:46.417 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.419 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.419 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:29:46.420 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:29:46.420 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:29:46.421 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.422 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.425 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.428 INFO 2981023 --- [host-check-host-status1,e2b865135ea625a3,9c8d5415cdf1bc2d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:29:46.430 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:29:46.432 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:29:46.433 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:29:46.438 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:29:46.444 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:29:46.444 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:29:46.453 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:29:46.455 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:29:46.461 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:29:46.485 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:29:46.497 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.498 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.498 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.498 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:46.499 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.502 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:29:46.503 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:29:46.505 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.505 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.505 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:29:46.506 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:29:46.506 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.506 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:29:46.508 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:29:46.508 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.511 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.513 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:29:46.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.514 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:29:46.522 INFO 2981023 --- [host-check-host-status2,e2b865135ea625a3,2e4594e98fed5a79] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:29:46.526 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:46.526 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:29:46.528 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:29:46.533 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:29:46.537 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:29:46.539 INFO 2981023 --- [host-check-host-status4,e2b865135ea625a3,919eaf9c1e90c2df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:29:46.551 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.551 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:46.551 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.552 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:46.552 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:29:46.553 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.555 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:29:46.556 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:29:46.556 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:29:46.557 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.558 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.558 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:29:46.559 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:29:46.559 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.565 INFO 2981023 --- [host-check-host-status3,e2b865135ea625a3,a27c7ccb2cdddb37] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:29:46.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.566 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.566 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:29:46.569 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.570 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.570 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:46.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:46.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:29:46.575 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:29:46.575 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:29:46.579 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:29:46.584 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:29:46.586 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:29:46.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:46.593 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:29:46.595 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.596 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.596 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.597 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:29:46.597 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:29:46.597 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:29:46.598 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.600 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:29:46.601 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:29:46.601 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:29:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:29:46.606 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:29:46.608 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:29:46.609 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:46.609 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:29:46.611 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.612 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.612 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:29:46.613 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:29:46.613 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:29:46.623 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:29:46.628 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.629 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.629 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:29:46.630 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:29:46.635 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:29:46.639 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:29:46.644 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:29:46.665 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:29:46.666 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:29:46.672 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:29:46.679 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.679 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.680 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:29:46.686 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:29:46.690 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:29:46.690 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:29:46.695 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:29:46.698 INFO 2981023 --- [ocp-schedules-17,6a090c212f9b60aa,03c9f37cd5a43162] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:29:46.699 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:29:46.699 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:29:46.706 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.706 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:46.706 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.706 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:46.707 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:29:46.707 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:29:46.708 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:29:46.349597+08:00 2024-09-11 17:29:46.712 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:29:46.712 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046389295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:29:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:29:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046412672000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:29:46.713 INFO 2981023 --- [ocp-schedules-12,226bd9c7eb8e88ab,c47f015a354e8c80] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:29:46.836 INFO 2981023 --- [ocp-schedules-2,d2c1574b2155e518,1085c6042bc90c29] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:29:46.836 INFO 2981023 --- [ocp-schedules-2,d2c1574b2155e518,1085c6042bc90c29] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:29:47.097 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:47.098 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:47.165 INFO 2981023 --- [ocp-async-7,b4041ee7099250c2,92859af2065f5781] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,c8e7f7c1a150cd6e,73eba40fc6e8cf38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,c8e7f7c1a150cd6e,73eba40fc6e8cf38] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:47.748 INFO 2981023 --- [ocp-async-9,fd82db8de9701758,a5683fa3d6f8db9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:29:49.165 INFO 2981023 --- [ocp-async-10,c062e4026c19c87c,32c09afbb7a04438] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:49.295 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a33aed121082a45d,fb97e7f375028c02] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&version=2&ObClusterId=1719994437, client=10.31.35.10, traceId=a33aed121082a45d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkprdocp,1719994437,2,{"ObRegion":"linkprdocp","ObCluster":"linkprdocp","ObRegionId":1719994437,"ObClusterId":1719994437,"Type":"PRIMARY","timestamp":1726046989294018,"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:29:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a33aed121082a45d,fb97e7f375028c02] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkprdocp, obClusterId=1719994437, rootServerJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, groupName=linkprdocp, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:29:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a33aed121082a45d,fb97e7f375028c02] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:29:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a33aed121082a45d,fb97e7f375028c02] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkprdocp, clusterId=1719994437, affectRows=1 2024-09-11 17:29:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a33aed121082a45d,fb97e7f375028c02] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.10, traceId=a33aed121082a45d, duration=9 ms] 2024-09-11 17:29:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:29:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:29:50.098 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:50.100 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:50.201 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:29:50.205 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:29:50.205 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.206 INFO 2981023 --- [ocp-schedules-17,4315f4b91c6055cd,abd303772cf9175d] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:29:50.210 INFO 2981023 --- [ocp-schedules-30,f1b1933af8a25b5e,6e7e6771c38dca3c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:29:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,a7a0e2067bd8169b,15f540601b2cb31c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:50.318 INFO 2981023 --- [prometheus-minute-schedule-1,a7a0e2067bd8169b,15f540601b2cb31c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:51.165 INFO 2981023 --- [ocp-async-5,ea2c043f486aa8e4,db57a48b86392950] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:51.473 INFO 2981023 --- [ocp-schedules-25,39f3616f8ab0d61c,be4c1bbbc9f595ac] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:29:51.473+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:29:51.478 INFO 2981023 --- [ocp-schedules-25,39f3616f8ab0d61c,be4c1bbbc9f595ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:29:51.515 INFO 2981023 --- [ocp-schedules-26,3620f0e1a475c10d,19284664d092c27d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:29:51.517 INFO 2981023 --- [ocp-schedules-26,3620f0e1a475c10d,19284664d092c27d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:29:51.517 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:29:51.518 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:29:51.518 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:29:51.529 INFO 2981023 --- [ocp-schedules-26,3620f0e1a475c10d,19284664d092c27d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:29:51.624 INFO 2981023 --- [ocp-schedules-25,39f3616f8ab0d61c,be4c1bbbc9f595ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 36 to SqlAggregateData 2024-09-11 17:29:51.626 INFO 2981023 --- [ocp-schedules-25,39f3616f8ab0d61c,be4c1bbbc9f595ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:29:51.841 INFO 2981023 --- [ocp-schedules-6,311737ffa82a8a1d,99b5b36bf96d3b63] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:29:52.076 INFO 2981023 --- [host-check-clockdiff3,f26199334a2a0fef,5331eb9885846ef0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726046992 1 1 , error= 2024-09-11 17:29:52.123 INFO 2981023 --- [ocp-schedules-21,eebd6b0bea2e38f2,abf451f17c1244a3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:29:52.124 INFO 2981023 --- [ocp-schedules-21,eebd6b0bea2e38f2,abf451f17c1244a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:29:52.158 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:29:52.158 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:29:52.158 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.158 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:29:52.158 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.158 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.161 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:29:52.162 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:29:52.162 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:29:52.165 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.165 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.166 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.166 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.167 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.167 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.171 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.171 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.177 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.177 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.178 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.178 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:29:52.181 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.181 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.187 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.187 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:29:52.189 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.190 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.190 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:29:52.190 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE21D1D6DA-0-0] 2024-09-11 17:29:52.191 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:29:52.191 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.191 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:29:52.193 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.193 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.194 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.194 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:29:52.203 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.203 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:29:52.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:29:52.211 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.212 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:29:52.212 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.215 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.216 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:29:52.216 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.216 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.217 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:29:52.217 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:29:52.217 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:29:52.217 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:29:52.217 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.217 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.217 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE21D1D6E4-0-0] 2024-09-11 17:29:52.218 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:29:52.218 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:29:52.218 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:29:52.218 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:29:52.218 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.219 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:29:52.219 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.220 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:29:52.220 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.221 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:29:52.222 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.222 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:29:52.223 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:29:52.223 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.224 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:29:52.224 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.225 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:29:52.225 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:29:52.226 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.226 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:29:52.227 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:29:52.227 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.228 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:29:52.229 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.229 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:29:52.230 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.230 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:29:52.231 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.232 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:29:52.233 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.233 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:29:52.233 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:29:52.235 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:52.235 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:29:52.237 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.237 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.240 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:52.241 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.242 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:29:52.248 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:29:52.248 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:29:52.252 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:29:52.252 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.252 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:29:52.253 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:29:52.255 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.256 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:29:52.256 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:29:52.256 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:29:52.258 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:29:52.258 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:29:52.259 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:29:52.260 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.260 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.266 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:29:52.267 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:29:52.267 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:29:52.267 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:29:52.269 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:29:52.271 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:29:52.271 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:52.271 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:29:52.282 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:29:52.283 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:29:52.283 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:29:52.283 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:29:52.283 INFO 2981023 --- [ocp-schedules-21,eebd6b0bea2e38f2,abf451f17c1244a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:29:52.374 INFO 2981023 --- [ocp-schedules-8,4aade87270a5e026,a32e6f9be5577c40] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:29:52.394 WARN 2981023 --- [ocp-schedules-8,4aade87270a5e026,a32e6f9be5577c40] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:29:52.692 INFO 2981023 --- [ocp-schedules-27,ec479326d43deb2f,a63febbbc93d1bbc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:29:52.738 WARN 2981023 --- [scanner-13492929-1,ec479326d43deb2f,63e30666510d1e18] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:29:52.768 INFO 2981023 --- [alarm-detect-16,ec479326d43deb2f,ff63cde66c92a60b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643934, data is empty. 2024-09-11 17:29:52.776 INFO 2981023 --- [alarm-detect-16,ec479326d43deb2f,ff63cde66c92a60b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643970, data is empty. 2024-09-11 17:29:52.782 INFO 2981023 --- [alarm-detect-16,ec479326d43deb2f,ff63cde66c92a60b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643970, data is empty. 2024-09-11 17:29:52.789 INFO 2981023 --- [alarm-detect-20,ec479326d43deb2f,a9ea17fdf0a760dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643934, data is empty. 2024-09-11 17:29:52.792 INFO 2981023 --- [alarm-detect-7,ec479326d43deb2f,69602b866766caa2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643934, data is empty. 2024-09-11 17:29:52.794 INFO 2981023 --- [alarm-detect-20,ec479326d43deb2f,a9ea17fdf0a760dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643970, data is empty. 2024-09-11 17:29:52.797 INFO 2981023 --- [alarm-detect-20,ec479326d43deb2f,a9ea17fdf0a760dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643970, data is empty. 2024-09-11 17:29:52.800 INFO 2981023 --- [alarm-detect-7,ec479326d43deb2f,69602b866766caa2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643970, data is empty. 2024-09-11 17:29:52.808 INFO 2981023 --- [alarm-detect-7,ec479326d43deb2f,69602b866766caa2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643970, data is empty. 2024-09-11 17:29:52.923 INFO 2981023 --- [alarm-detect-11,ec479326d43deb2f,f30432d7f1d62b6b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643934, data is empty. 2024-09-11 17:29:52.940 INFO 2981023 --- [alarm-detect-11,ec479326d43deb2f,f30432d7f1d62b6b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643970, data is empty. 2024-09-11 17:29:52.948 INFO 2981023 --- [alarm-detect-11,ec479326d43deb2f,f30432d7f1d62b6b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643970, data is empty. 2024-09-11 17:29:53.049 INFO 2981023 --- [ocp-schedules-17,ec16b48a01c992ca,d6113d76aeaef8ca] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:29:53.049 INFO 2981023 --- [ocp-schedules-17,ec16b48a01c992ca,d6113d76aeaef8ca] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:29:53.100 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:53.101 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:53.165 INFO 2981023 --- [ocp-async-6,12f85bbba8bcc628,712ccf0d7e7e569c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,a3e3c3699d23d9a2,2df36a30f129aa7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,a3e3c3699d23d9a2,2df36a30f129aa7c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:53.674 INFO 2981023 --- [ocp-schedules-12,59277fc68cd72a4e,ea86ceeabf50b02d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:29:53.683 INFO 2981023 --- [ocp-schedules-12,59277fc68cd72a4e,ea86ceeabf50b02d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:29:53.683 INFO 2981023 --- [ocp-schedules-12,59277fc68cd72a4e,ea86ceeabf50b02d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:29:54.191 INFO 2981023 --- [ocp-schedules-9,1676b2448279260f,70fd48c0862a43de] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:29:55.165 INFO 2981023 --- [ocp-async-2,011ff53cd7ff40d0,829d25c9231b0dd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:55.496 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,b3c51ab95a12ca95,afd32eaa4b6bd615] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=b3c51ab95a12ca95, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:29:55.496 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,37fe5f574a1f93f3,019871e2d4d81a79] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=37fe5f574a1f93f3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:29:55.499 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,37fe5f574a1f93f3,019871e2d4d81a79] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=37fe5f574a1f93f3, duration=7 ms] 2024-09-11 17:29:55.500 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,b3c51ab95a12ca95,afd32eaa4b6bd615] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=b3c51ab95a12ca95, duration=8 ms] 2024-09-11 17:29:55.507 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8c7b3de298fe8f61,66e9ee7601f6476c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=8c7b3de298fe8f61, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:29:55.508 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8c7b3de298fe8f61,66e9ee7601f6476c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=8c7b3de298fe8f61, duration=3 ms] 2024-09-11 17:29:55.641 INFO 2981023 --- [ocp-schedules-13,0c57e73fa9265288,525863b60f2de66d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:29:55.641 INFO 2981023 --- [ocp-schedules-13,0c57e73fa9265288,525863b60f2de66d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:29:55.643 INFO 2981023 --- [ocp-schedules-13,0c57e73fa9265288,525863b60f2de66d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:29:55.643 INFO 2981023 --- [ocp-schedules-13,0c57e73fa9265288,525863b60f2de66d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:29:55.646 INFO 2981023 --- [ocp-schedules-13,0c57e73fa9265288,525863b60f2de66d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:29:55.646 INFO 2981023 --- [ocp-schedules-13,0c57e73fa9265288,525863b60f2de66d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:29:55.713 INFO 2981023 --- [ocp-schedules-9,127d89b944a34ffe,2ea1c1adfe711b1a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:29:55.719 INFO 2981023 --- [ocp-schedules-9,127d89b944a34ffe,2ea1c1adfe711b1a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:29:55.719 INFO 2981023 --- [ocp-schedules-9,127d89b944a34ffe,2ea1c1adfe711b1a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:29:55.916 INFO 2981023 --- [ocp-schedules-7,99904b49085808b3,6d8f47e558e3b755] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:29:55.916 INFO 2981023 --- [ocp-schedules-7,99904b49085808b3,6d8f47e558e3b755] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:29:55.937 INFO 2981023 --- [Collect all tenants major compactions3,99904b49085808b3,e2a4e1e23ad739a0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:29:55.937 INFO 2981023 --- [Collect all tenants major compactions5,99904b49085808b3,41db21f11ddc8656] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:29:55.938 INFO 2981023 --- [Collect all tenants major compactions4,99904b49085808b3,b9930d3cf734dfdc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:29:55.939 INFO 2981023 --- [Collect all tenants major compactions1,99904b49085808b3,251df3c3a05c5646] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:29:55.940 INFO 2981023 --- [Collect all tenants major compactions2,99904b49085808b3,8192a11c77c76bf9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:29:55.940 INFO 2981023 --- [Collect all tenants major compactions6,99904b49085808b3,63e37f74c271a1f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:29:55.952 INFO 2981023 --- [Collect all tenants major compactions3,99904b49085808b3,e2a4e1e23ad739a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:55.953 INFO 2981023 --- [Collect all tenants major compactions3,99904b49085808b3,e2a4e1e23ad739a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:29:55.954 INFO 2981023 --- [Collect all tenants major compactions2,99904b49085808b3,8192a11c77c76bf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:55.955 INFO 2981023 --- [Collect all tenants major compactions4,99904b49085808b3,b9930d3cf734dfdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:55.955 INFO 2981023 --- [Collect all tenants major compactions5,99904b49085808b3,41db21f11ddc8656] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:55.955 INFO 2981023 --- [Collect all tenants major compactions2,99904b49085808b3,8192a11c77c76bf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:29:55.955 INFO 2981023 --- [Collect all tenants major compactions4,99904b49085808b3,b9930d3cf734dfdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:29:55.956 INFO 2981023 --- [Collect all tenants major compactions3,99904b49085808b3,e2a4e1e23ad739a0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:29:55.956 INFO 2981023 --- [Collect all tenants major compactions3,99904b49085808b3,e2a4e1e23ad739a0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:29:55.957 INFO 2981023 --- [Collect all tenants major compactions5,99904b49085808b3,41db21f11ddc8656] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:29:55.957 INFO 2981023 --- [Collect all tenants major compactions1,99904b49085808b3,251df3c3a05c5646] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:55.958 INFO 2981023 --- [Collect all tenants major compactions4,99904b49085808b3,b9930d3cf734dfdc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:29:55.958 INFO 2981023 --- [Collect all tenants major compactions4,99904b49085808b3,b9930d3cf734dfdc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:29:55.958 INFO 2981023 --- [Collect all tenants major compactions1,99904b49085808b3,251df3c3a05c5646] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:29:55.958 INFO 2981023 --- [Collect all tenants major compactions2,99904b49085808b3,8192a11c77c76bf9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:29:55.959 INFO 2981023 --- [Collect all tenants major compactions2,99904b49085808b3,8192a11c77c76bf9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:29:55.959 INFO 2981023 --- [Collect all tenants major compactions5,99904b49085808b3,41db21f11ddc8656] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:29:55.959 INFO 2981023 --- [Collect all tenants major compactions5,99904b49085808b3,41db21f11ddc8656] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:29:55.960 INFO 2981023 --- [Collect all tenants major compactions1,99904b49085808b3,251df3c3a05c5646] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:29:55.960 INFO 2981023 --- [Collect all tenants major compactions1,99904b49085808b3,251df3c3a05c5646] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:29:55.960 INFO 2981023 --- [Collect all tenants major compactions6,99904b49085808b3,63e37f74c271a1f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:29:55.961 INFO 2981023 --- [Collect all tenants major compactions6,99904b49085808b3,63e37f74c271a1f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:29:55.962 INFO 2981023 --- [Collect all tenants major compactions6,99904b49085808b3,63e37f74c271a1f9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:29:55.963 INFO 2981023 --- [Collect all tenants major compactions6,99904b49085808b3,63e37f74c271a1f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:29:55.963 INFO 2981023 --- [ocp-schedules-7,99904b49085808b3,6d8f47e558e3b755] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:29:56.101 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:56.102 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,229fe07ef678b20f,f6ebdc434b2a8625] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,229fe07ef678b20f,f6ebdc434b2a8625] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:29:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:29:56.351756+08:00, lastProcessTime=2024-09-11T17:29:46.349597+08:00 2024-09-11 17:29:56.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:29:56.351756+08:00 2024-09-11 17:29:57.165 INFO 2981023 --- [ocp-async-3,b2f221d437a19922,f861802dffdf82c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:57.748 INFO 2981023 --- [ocp-async-1,567a1662536de3aa,bfe761eab6ea54c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:29:59.102 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:29:59.104 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:29:59.165 INFO 2981023 --- [ocp-async-6,397ae17f548a621f,d98939b08dd9980a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:29:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,ea310741483518ac,eb8ef6c9305cad04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047000, interval=60 2024-09-11 17:29:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,ea310741483518ac,eb8ef6c9305cad04] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:00.089 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:25:01.0, nextRun=2024-09-11 17:30:00.0, currentDate=Wed Sep 11 17:30:00 HKT 2024 2024-09-11 17:30:00.089 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:25:01.0, nextRunTime=2024-09-11 17:30:00.0} 2024-09-11 17:30:00.096 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:30:00.096 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:30:00.096 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:30:00.096+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:30:00.108 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:25:01.0, nextRun=2024-09-11 17:30:00.0, currentDate=Wed Sep 11 17:30:00 HKT 2024 2024-09-11 17:30:00.108 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:25:01.0, nextRunTime=2024-09-11 17:30:00.0} 2024-09-11 17:30:00.138 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2, 1000006] 2024-09-11 17:30:00.147 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:30:00.148 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:30:00.148 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:30:00.148 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,8482da7f98711b18] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2,1000006]]}}, createTime=null, startTime=2024-09-11T17:30:00.148+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:30:00.637 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,09f263edc00b3143] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048162, operation:EXECUTE, state: RUNNING 2024-09-11 17:30:00.651 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,09f263edc00b3143] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048163, operation:EXECUTE, state: RUNNING 2024-09-11 17:30:00.662 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048162, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048182, task_operation=execute, latest_execution_start_time=2024-09-11T17:30:00.630+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048162}, listMap={}}, executor=10.31.35.10 2024-09-11 17:30:00.671 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048163, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048183, task_operation=execute, latest_execution_start_time=2024-09-11T17:30:00.649+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048163}, listMap={cluster_ids=[[1,2,1000006]]}}, executor=10.31.35.10 2024-09-11 17:30:00.673 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:30:00.680 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:30:00.701 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:30:00.705 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:30:00.718 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:30:00.719 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:30:00.719 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:30:00.725 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:30:00.738 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:30:00.749 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:30:00.749 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1000006 2024-09-11 17:30:00.762 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,ebf0b32533cc5587] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048163, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:30:00.794 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:30:00.828 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:30:00.849 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:30:00.860 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:30:00.860 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:30:00.864 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:30:01.008 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:30:01.103 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:30:01.151 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:30:01.165 INFO 2981023 --- [ocp-async-10,323c77a5cbb90570,408a6a395b6b1e02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:01.315 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:30:01.475 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:30:01.490 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:30:01.514 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:30:01.520 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:30:01.520 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:30:01.525 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:30:01.587 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:30:01.620 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:30:01.632 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:30:01.642 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:30:01.657 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:30:01.663 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:30:01.672 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:30:01.674 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:30:01.675 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:30:01.690 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:30:01.702 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:30:01.717 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,00eb05c93e927579] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048162, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:30:01.722 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048183, status:SUCCESS 2024-09-11 17:30:01.725 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048183, status:SUCCESS 2024-09-11 17:30:01.844 INFO 2981023 --- [ocp-schedules-18,e84e20d151cf79ea,7c2c19867be8d622] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:30:01.997 INFO 2981023 --- [ocp-schedules-20,ca9d47a986b1daa2,42e2e200755f3b3d] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:30:01.998 INFO 2981023 --- [ocp-schedules-20,ca9d47a986b1daa2,42e2e200755f3b3d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:30:02.104 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:02.105 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,0bd231540dbdffa7,6264fa99c4f01e14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,0bd231540dbdffa7,6264fa99c4f01e14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:02.415 INFO 2981023 --- [ocp-schedules-17,c29d86ae62a95260,05204b6c28cdb57a] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:30:02.420 INFO 2981023 --- [ocp-schedules-17,c29d86ae62a95260,05204b6c28cdb57a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:30:02.429 INFO 2981023 --- [ocp-schedules-17,c29d86ae62a95260,05204b6c28cdb57a] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:30:02.735 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048182, status:SUCCESS 2024-09-11 17:30:02.735 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048182, status:SUCCESS 2024-09-11 17:30:02.979 INFO 2981023 --- [ocp-schedules-19,95f2eb6eccc26bee,a917bdec94b509e6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:30:03.053 WARN 2981023 --- [scanner-13493446-1,95f2eb6eccc26bee,3734a690ef407965] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:30:03.168 INFO 2981023 --- [ocp-async-3,c90e7e3d7cd82fd0,d15d266a63085106] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:04.484 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,e49b9e108327c5cc] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:20:05.0, nextRun=2024-09-11 17:30:04.0, currentDate=Wed Sep 11 17:30:04 HKT 2024 2024-09-11 17:30:04.484 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,e49b9e108327c5cc] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-09-11 17:20:05.0, nextRunTime=2024-09-11 17:30:04.0} 2024-09-11 17:30:04.491 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,e49b9e108327c5cc] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-09-11 17:30:04.491 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,e49b9e108327c5cc] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:30:04.491 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,e49b9e108327c5cc] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:30:04.491+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:30:04.683 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,86aed0a058018c7d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048164, operation:EXECUTE, state: RUNNING 2024-09-11 17:30:04.707 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048164, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048184, task_operation=execute, latest_execution_start_time=2024-09-11T17:30:04.682+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=4048164}, listMap={}}, executor=10.31.35.10 2024-09-11 17:30:04.714 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 17:30:04.714 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 17:30:04.715 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.715 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:30:04.717 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.718 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.718 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.719 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 17:30:04.730 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.730 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.731 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.731 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 17:30:04.740 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.741 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.741 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.741 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 17:30:04.751 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.751 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.752 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.752 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 17:30:04.762 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.762 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.762 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.763 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 17:30:04.766 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.766 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.767 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.767 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 17:30:04.770 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.770 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.771 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.771 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 17:30:04.774 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.774 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.775 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.775 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 17:30:04.780 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.780 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.781 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.781 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 17:30:04.785 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.785 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.785 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.786 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 17:30:04.789 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.789 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.790 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.790 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 17:30:04.796 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.796 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.797 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.797 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 17:30:04.807 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.807 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.807 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.808 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 17:30:04.811 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.811 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.812 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.812 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 17:30:04.816 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.816 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.817 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.817 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 17:30:04.820 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.821 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.821 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.821 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 17:30:04.824 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.825 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.825 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.825 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 17:30:04.828 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.828 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.829 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.829 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 17:30:04.832 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.832 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.833 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.833 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 17:30:04.836 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.837 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.837 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.837 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 17:30:04.840 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.840 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.841 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.841 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 17:30:04.844 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.844 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.845 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.845 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 17:30:04.848 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.848 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.848 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.849 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 17:30:04.852 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.852 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.853 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.853 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 17:30:04.856 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.856 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.856 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.857 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 17:30:04.859 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.860 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.860 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.860 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 17:30:04.863 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.864 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.864 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.864 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 17:30:04.867 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.867 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.868 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.868 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 17:30:04.871 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.872 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.872 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.872 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 17:30:04.875 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.876 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.876 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.876 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 17:30:04.879 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.879 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.880 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.880 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 17:30:04.883 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.883 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.884 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.884 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 17:30:04.887 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.887 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.887 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.888 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 17:30:04.890 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.891 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.891 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.891 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 17:30:04.896 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.897 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.897 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.897 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 17:30:04.901 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.901 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.902 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.902 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 17:30:04.905 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.906 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.906 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.906 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 17:30:04.909 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.910 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.910 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.910 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 17:30:04.913 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.913 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.913 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.914 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 17:30:04.916 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.916 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.917 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.917 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 17:30:04.919 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.920 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.920 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.921 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 17:30:04.923 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.923 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.924 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.924 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 17:30:04.930 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.931 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.931 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.932 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 17:30:04.947 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.947 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.947 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.948 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 17:30:04.952 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.952 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.953 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.953 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 17:30:04.957 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.957 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.957 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.958 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 17:30:04.962 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.962 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.962 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.963 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 17:30:04.965 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.965 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.966 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.966 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 17:30:04.968 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.969 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.969 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.969 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 17:30:04.972 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 17:30:04.972 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 17:30:04.979 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.980 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:30:04.982 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.982 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.983 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.983 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 17:30:04.987 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.987 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:04.988 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:04.988 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 17:30:04.990 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 17:30:05.003 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,1e7660338820f0b3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048164, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:30:05.105 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:05.107 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:05.168 INFO 2981023 --- [ocp-async-8,d6a7bf010f645ab9,34a8d5e275699f57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,0891aab550315c37,142ebb5e84358d9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,0891aab550315c37,142ebb5e84358d9f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:05.758 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048184, status:SUCCESS 2024-09-11 17:30:05.759 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048184, status:SUCCESS 2024-09-11 17:30:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:30:05.989 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:30:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:30:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:30:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:30:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:30:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:30:06.008 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T17:27:40, target=2024-09-11T17:25:40, granule=120000ms, delay=120000ms) 2024-09-11 17:30:06.008 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:25:40, target=2024-09-11T17:19:41, granule=600000ms, delay=359000ms) 2024-09-11 17:30:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:30:06.010 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:06.010 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:30:06.061 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:27:41, start=2024-09-11T17:25:11, end=2024-09-11T17:27:41, updates=300, granule=120000ms, delay=150000ms 2024-09-11 17:30:06.070 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:27:41, target=2024-09-11T17:27:41, granule=120000ms, delay=0ms) 2024-09-11 17:30:06.070 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:27:41, target=2024-09-11T17:20:51, granule=600000ms, delay=410000ms) 2024-09-11 17:30:06.070 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=103ms, period=120000ms 2024-09-11 17:30:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:30:06.351899+08:00, lastProcessTime=2024-09-11T17:29:56.351756+08:00 2024-09-11 17:30:06.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:30:06.351899+08:00 2024-09-11 17:30:07.168 INFO 2981023 --- [ocp-async-2,54518fbb97a2ff00,4163db176ad87766] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:07.244 INFO 2981023 --- [ocp-schedules-21,56a25015f1b19bcf,035cdbb53df99739] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:30:07.250 INFO 2981023 --- [ocp-schedules-23,c3ee602fd70097bd,445d43ae1c8ed19d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:30:07.250+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:30:07.250 INFO 2981023 --- [ocp-schedules-23,c3ee602fd70097bd,445d43ae1c8ed19d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:30:07.252 INFO 2981023 --- [ocp-schedules-23,c3ee602fd70097bd,445d43ae1c8ed19d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:30:07.252 INFO 2981023 --- [ocp-schedules-23,c3ee602fd70097bd,445d43ae1c8ed19d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:30:07.749 INFO 2981023 --- [ocp-async-6,2a26571ee8e3af98,a26737562fa750ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:30:08.107 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:08.107 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,dddb53116773e8d1,c883ff162842e3a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,dddb53116773e8d1,c883ff162842e3a6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:09.169 INFO 2981023 --- [ocp-async-3,edcd307ff12e8254,90fd6812e2130126] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:10.004 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:30:10.004 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:30:10.913 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:30:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:30:11.107 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:11.108 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:11.169 INFO 2981023 --- [ocp-async-8,f4a6b92852b74599,d5a08735e2d37f9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,88b4127ef8e586e1,af7fda327e208bbd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,88b4127ef8e586e1,af7fda327e208bbd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:11.846 INFO 2981023 --- [ocp-schedules-14,28ab56c39c80cfde,a5beaff9351d4c73] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:30:12.660 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9898dff68bc08cc9,d3b18ed2828a1dd2] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&version=2&ObClusterId=1718689631, client=10.31.35.5, traceId=9898dff68bc08cc9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=rtdob,1718689631,2,{"ObRegion":"rtdob","ObCluster":"rtdob","ObRegionId":1718689631,"ObClusterId":1718689631,"Type":"PRIMARY","timestamp":1726047012658483,"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:30:12.667 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9898dff68bc08cc9,d3b18ed2828a1dd2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=rtdob, obClusterId=1718689631, rootServerJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, groupName=rtdob, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:30:12.667 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9898dff68bc08cc9,d3b18ed2828a1dd2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:30:12.667 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9898dff68bc08cc9,d3b18ed2828a1dd2] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=rtdob, clusterId=1718689631, affectRows=1 2024-09-11 17:30:12.667 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9898dff68bc08cc9,d3b18ed2828a1dd2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.5, traceId=9898dff68bc08cc9, duration=8 ms] 2024-09-11 17:30:13.169 INFO 2981023 --- [ocp-async-2,799c13d6c70366c3,83d402a194203c8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:13.199 INFO 2981023 --- [ocp-schedules-20,0c19d934d2ab7e72,69ed435a72824c9e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:30:13.256 WARN 2981023 --- [scanner-13495126-1,0c19d934d2ab7e72,34f7c6f8310a2106] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:30:13.279 INFO 2981023 --- [alarm-detect-19,0c19d934d2ab7e72,0e78bfe5cc68988e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643970, data is empty. 2024-09-11 17:30:13.284 INFO 2981023 --- [alarm-detect-19,0c19d934d2ab7e72,0e78bfe5cc68988e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643970, data is empty. 2024-09-11 17:30:13.291 INFO 2981023 --- [alarm-detect-19,0c19d934d2ab7e72,0e78bfe5cc68988e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643970, data is empty. 2024-09-11 17:30:13.307 INFO 2981023 --- [alarm-detect-7,0c19d934d2ab7e72,5353734859706d39] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643970, data is empty. 2024-09-11 17:30:13.308 INFO 2981023 --- [alarm-detect-19,0c19d934d2ab7e72,f9f70fc1afac7999] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643970, data is empty. 2024-09-11 17:30:13.312 INFO 2981023 --- [alarm-detect-7,0c19d934d2ab7e72,5353734859706d39] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643970, data is empty. 2024-09-11 17:30:13.316 INFO 2981023 --- [alarm-detect-19,0c19d934d2ab7e72,f9f70fc1afac7999] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643970, data is empty. 2024-09-11 17:30:13.316 INFO 2981023 --- [alarm-detect-7,0c19d934d2ab7e72,5353734859706d39] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643970, data is empty. 2024-09-11 17:30:13.320 INFO 2981023 --- [alarm-detect-19,0c19d934d2ab7e72,f9f70fc1afac7999] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643970, data is empty. 2024-09-11 17:30:13.424 INFO 2981023 --- [alarm-detect-9,0c19d934d2ab7e72,63b8afc68fc23f2e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643970, data is empty. 2024-09-11 17:30:13.428 INFO 2981023 --- [alarm-detect-9,0c19d934d2ab7e72,63b8afc68fc23f2e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643970, data is empty. 2024-09-11 17:30:13.432 INFO 2981023 --- [alarm-detect-9,0c19d934d2ab7e72,63b8afc68fc23f2e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643970, data is empty. 2024-09-11 17:30:13.686 INFO 2981023 --- [ocp-schedules-25,45c00d4e42a14797,8cc3eae685d72f60] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:30:13.694 INFO 2981023 --- [ocp-schedules-25,45c00d4e42a14797,8cc3eae685d72f60] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:30:13.694 INFO 2981023 --- [ocp-schedules-25,45c00d4e42a14797,8cc3eae685d72f60] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:30:14.005 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726046940, nodeCount=480, affectRows=480 2024-09-11 17:30:14.108 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:14.108 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:14.109 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046940, nodeCount=1000, affectRows=1000 2024-09-11 17:30:14.112 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046940, nodeCount=999, validCount=999 2024-09-11 17:30:14.154 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046940, nodeCount=49, affectRows=49 2024-09-11 17:30:14.154 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046940, nodeCount=49, validCount=49 2024-09-11 17:30:14.208 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726046940, nodeCount=1000, affectRows=1000 2024-09-11 17:30:14.210 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726046940, nodeCount=1000, validCount=1000 2024-09-11 17:30:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,8d81b638998e12ed,2bb4e955d008a895] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,c3ad57d21e8fee1a,bb89f323cb3938fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:30:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,8d81b638998e12ed,2bb4e955d008a895] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:15.169 INFO 2981023 --- [ocp-async-5,db44bce966c7ebc6,b00381120c49e1c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:15.837 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726046940, nodeCount=480, validCount=480 2024-09-11 17:30:16.020 INFO 2981023 --- [ocp-schedules-17,be9a780cb8898db3,c3a161785063f7c6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:30:16.020 INFO 2981023 --- [host-check-clockdiff1,be9a780cb8898db3,8c6405504de93020] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:30:16.020 INFO 2981023 --- [host-check-clockdiff2,be9a780cb8898db3,c51b0a4d46f27377] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:30:16.020 INFO 2981023 --- [host-check-clockdiff3,be9a780cb8898db3,7df24dbbba7c9228] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:30:16.022 INFO 2981023 --- [host-check-clockdiff4,be9a780cb8898db3,2e33ce71284bc0ee] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:30:16.027 INFO 2981023 --- [ocp-schedules-19,89b80a2a0d972d98,a79cfd15d4a2ff4c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:30:16.028 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:30:16.028 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:30:16.029 INFO 2981023 --- [ocp-schedules-15,dae6ccd5b71d1124,7ac647d210151c68] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:30:16.030 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:30:16.035 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:30:16.044 INFO 2981023 --- [host-check-agent-status1,dae6ccd5b71d1124,6e1231d126164a69] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:30:16.044 INFO 2981023 --- [host-check-agent-status2,dae6ccd5b71d1124,57b81dd11878ef6a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:30:16.045 INFO 2981023 --- [host-check-agent-status3,dae6ccd5b71d1124,6cf43fb84f05eaf3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:30:16.045 INFO 2981023 --- [host-check-agent-status4,dae6ccd5b71d1124,fc47e6227ff29d44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:30:16.049 INFO 2981023 --- [host-check-clockdiff2,be9a780cb8898db3,c51b0a4d46f27377] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047016 0 0 , error= 2024-09-11 17:30:16.049 INFO 2981023 --- [host-check-clockdiff4,be9a780cb8898db3,2e33ce71284bc0ee] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047016 0 0 , error= 2024-09-11 17:30:16.066 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:30:16.069 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:30:16.070 INFO 2981023 --- [host-check-agent-status3,dae6ccd5b71d1124,6cf43fb84f05eaf3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:30:16.071 INFO 2981023 --- [host-check-agent-status2,dae6ccd5b71d1124,57b81dd11878ef6a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:30:16.071 INFO 2981023 --- [host-check-agent-status4,dae6ccd5b71d1124,fc47e6227ff29d44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:30:16.071 INFO 2981023 --- [host-check-agent-status3,dae6ccd5b71d1124,6cf43fb84f05eaf3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:30:16.071 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:30:16.072 INFO 2981023 --- [host-check-agent-status1,dae6ccd5b71d1124,6e1231d126164a69] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:30:16.072 INFO 2981023 --- [host-check-agent-status4,dae6ccd5b71d1124,fc47e6227ff29d44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:30:16.072 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:30:16.075 INFO 2981023 --- [host-check-agent-status2,dae6ccd5b71d1124,57b81dd11878ef6a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:30:16.076 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:30:16.078 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:30:16.094 INFO 2981023 --- [host-check-agent-status1,dae6ccd5b71d1124,6e1231d126164a69] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:30:16.188 INFO 2981023 --- [ocp-schedules-28,e64a66d854aa9738,07ea278126bd88f8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:30:16.194 INFO 2981023 --- [ocp-schedules-5,b1bbcde8458e4bf1,98adb4f2a305f76f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:30:16.194 INFO 2981023 --- [ocp-schedules-12,e4417336792575a0,b94023551951efe3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:30:16.194 INFO 2981023 --- [ocp-schedules-24,d67936c8a794116e,c5f06131076f918f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:30:16.272 INFO 2981023 --- [ocp-schedules-28,e64a66d854aa9738,07ea278126bd88f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:30:16.287 INFO 2981023 --- [ocp-schedules-12,e4417336792575a0,b94023551951efe3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:30:16.290 INFO 2981023 --- [ocp-schedules-5,b1bbcde8458e4bf1,98adb4f2a305f76f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:30:16.291 INFO 2981023 --- [ocp-schedules-5,b1bbcde8458e4bf1,98adb4f2a305f76f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:30:16.362 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:30:16.363 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:30:16.397 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:30:16.395643+08:00, lastProcessTime=2024-09-11T17:30:06.351899+08:00 2024-09-11 17:30:16.401 INFO 2981023 --- [ocp-schedules-9,52e1a5d3855251f0,2a8cba3de335a794] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:30:16.459 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:30:16.460 INFO 2981023 --- [host-check-agent-status3,dae6ccd5b71d1124,6cf43fb84f05eaf3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:30:16.462 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:30:16.461614501+08:00 2024-09-11 17:30:16.462 INFO 2981023 --- [host-check-agent-status4,dae6ccd5b71d1124,fc47e6227ff29d44] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:30:16.462 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:30:16.462+08:00 2024-09-11 17:30:16.464 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:30:16.465 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:30:16.462+08:00 and status AVAILABLE 2024-09-11 17:30:16.466 INFO 2981023 --- [host-check-agent-status3,dae6ccd5b71d1124,6cf43fb84f05eaf3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:30:16.466 INFO 2981023 --- [host-check-agent-status3,dae6ccd5b71d1124,6cf43fb84f05eaf3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:30:16.466 INFO 2981023 --- [host-check-agent-status3,dae6ccd5b71d1124,6cf43fb84f05eaf3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:30:16.466 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:30:16.466319472+08:00 2024-09-11 17:30:16.467 INFO 2981023 --- [host-check-agent-status4,dae6ccd5b71d1124,fc47e6227ff29d44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:30:16.467 INFO 2981023 --- [host-check-agent-status4,dae6ccd5b71d1124,fc47e6227ff29d44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:30:16.467 INFO 2981023 --- [host-check-agent-status4,dae6ccd5b71d1124,fc47e6227ff29d44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:30:16.467 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:30:16.466+08:00 2024-09-11 17:30:16.470 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:30:16.466+08:00 and status AVAILABLE 2024-09-11 17:30:16.470 INFO 2981023 --- [host-check-agent-status1,dae6ccd5b71d1124,6e1231d126164a69] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:30:16.471 INFO 2981023 --- [host-check-agent-status2,dae6ccd5b71d1124,57b81dd11878ef6a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:30:16.473 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.473 INFO 2981023 --- [host-check-agent-status1,dae6ccd5b71d1124,6e1231d126164a69] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:30:16.473 INFO 2981023 --- [host-check-agent-status1,dae6ccd5b71d1124,6e1231d126164a69] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:30:16.473 INFO 2981023 --- [host-check-agent-status1,dae6ccd5b71d1124,6e1231d126164a69] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:30:16.473 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:16.473 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:30:16.474 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:30:16.474070674+08:00 2024-09-11 17:30:16.474 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:30:16.475 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:30:16.474+08:00 2024-09-11 17:30:16.476 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.476 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:30:16.474+08:00 and status AVAILABLE 2024-09-11 17:30:16.476 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:16.478 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:30:16.479 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:30:16.479 INFO 2981023 --- [host-check-agent-status2,dae6ccd5b71d1124,57b81dd11878ef6a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:30:16.479 INFO 2981023 --- [host-check-agent-status2,dae6ccd5b71d1124,57b81dd11878ef6a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:30:16.479 INFO 2981023 --- [host-check-agent-status2,dae6ccd5b71d1124,57b81dd11878ef6a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:30:16.483 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:30:16.484 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:30:16.485 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:30:16.485341417+08:00 2024-09-11 17:30:16.485 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.486 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:16.488 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.488 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:16.488 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.489 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:16.489 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:30:16.490 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:30:16.492 INFO 2981023 --- [ocp-schedules-28,e64a66d854aa9738,07ea278126bd88f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:30:16.493 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:30:16.485+08:00 2024-09-11 17:30:16.502 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:30:16.485+08:00 and status AVAILABLE 2024-09-11 17:30:16.511 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.512 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:30:16.513 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:16.513 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:30:16.513 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:30:16.514 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:30:16.514 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:30:16.517 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:30:16.519 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:30:16.558 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:30:16.559 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:30:16.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.574 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:30:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:30:16.585 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:30:16.587 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:30:16.596 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.597 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.597 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:30:16.599 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:30:16.610 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:30:16.618 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:30:16.623 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:30:16.626 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.628 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.628 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:16.629 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.629 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.629 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:30:16.630 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.631 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:16.631 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:30:16.631 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:30:16.632 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.632 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:30:16.633 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:30:16.634 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:30:16.635 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:30:16.644 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:30:16.657 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:16.657 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:30:16.662 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:30:16.666 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:30:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:30:16.675 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:30:16.679 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:16.680 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.681 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.681 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:30:16.683 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:30:16.685 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.686 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.686 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:16.687 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:30:16.688 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:16.691 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:30:16.694 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:30:16.695 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:30:16.696 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:30:16.697 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:30:16.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.698 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.699 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:16.699 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:30:16.702 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:30:16.703 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:30:16.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:30:16.710 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:30:16.712 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:30:16.716 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:30:16.717 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.718 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:30:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:16.719 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:30:16.720 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.723 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:30:16.724 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:30:16.730 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:30:16.731 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:30:16.732 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:30:16.734 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:30:16.744 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:30:16.744 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:30:16.744 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:30:16.758 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:16.758 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:30:16.765 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:30:16.768 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:30:16.769 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:30:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:16.786 INFO 2981023 --- [host-check-host-status1,89b80a2a0d972d98,445d2376078fd99f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:30:16.787 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:30:16.787 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.787 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:16.788 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:30:16.789 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:30:16.796 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:30:16.806 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:30:16.808 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:30:16.810 INFO 2981023 --- [host-check-host-status3,89b80a2a0d972d98,2315eddb260bd591] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:30:16.813 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:30:16.813 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:30:16.820 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:30:16.828 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.829 INFO 2981023 --- [host-check-host-status2,89b80a2a0d972d98,ad3bccb253e6a507] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:30:16.829 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.830 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:30:16.840 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:30:16.840 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:30:16.843 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:30:16.849 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:30:16.849 INFO 2981023 --- [host-check-host-status4,89b80a2a0d972d98,710f33da15d6d95d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:30:16.860 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.860 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.860 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:30:16.865 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:30:16.395643+08:00 2024-09-11 17:30:16.867 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:30:16.867 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:30:16.872 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:30:16.877 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:16.878 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:16.878 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:30:16.883 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:30:16.884 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:30:16.884 INFO 2981023 --- [ocp-schedules-12,e4417336792575a0,b94023551951efe3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:30:16.970 INFO 2981023 --- [host-check-clockdiff1,be9a780cb8898db3,8c6405504de93020] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047016 0 0 , error= 2024-09-11 17:30:17.108 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:17.109 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:17.169 INFO 2981023 --- [ocp-async-7,b9086cd5d3795fc5,49f4b3cd03cb4474] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,83d4be5cf604f97d,a4e95511e4b50085] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,83d4be5cf604f97d,a4e95511e4b50085] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:17.749 INFO 2981023 --- [ocp-async-5,dc9f3a992fdb5f59,86bd65b8f17f1f3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:30:19.169 INFO 2981023 --- [ocp-async-4,a331983b33624e57,d8214ad3d8372736] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:20.109 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:20.111 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,12a3fb07d21c40da,395d136d3b24dfcc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,12a3fb07d21c40da,395d136d3b24dfcc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:21.169 INFO 2981023 --- [ocp-async-10,90e73c86c0fabd40,6e4430a115911a47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:21.850 INFO 2981023 --- [ocp-schedules-9,312efb96e969ae75,a7ae44b06b3857af] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:30:22.008 INFO 2981023 --- [host-check-clockdiff3,be9a780cb8898db3,7df24dbbba7c9228] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047022 1 1 , error= 2024-09-11 17:30:22.510 INFO 2981023 --- [ocp-schedules-6,ed0615324e104433,27376d8efe02050e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:30:22.510 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:30:22.510 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:30:22.511 INFO 2981023 --- [ocp-schedules-6,ed0615324e104433,27376d8efe02050e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:30:22.511 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:30:22.516 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:30:22.518 INFO 2981023 --- [ocp-schedules-6,ed0615324e104433,27376d8efe02050e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:30:22.521 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:30:22.521 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:30:22.522 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:30:22.523 INFO 2981023 --- [ocp-schedules-6,ed0615324e104433,27376d8efe02050e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:30:22.524 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:30:22.525 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:30:22.548 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:30:22.548 INFO 2981023 --- [ocp-schedules-6,ed0615324e104433,27376d8efe02050e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:30:22.548 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:30:22.549 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:30:22.550 INFO 2981023 --- [ocp-schedules-6,ed0615324e104433,27376d8efe02050e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:30:23.111 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:23.113 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:23.169 INFO 2981023 --- [ocp-async-5,a8c5898bcc08512f,8b2d48f6daa4acc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,7d95665ee4ce54e4,a78068ff4edd071f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,7d95665ee4ce54e4,a78068ff4edd071f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:23.477 INFO 2981023 --- [ocp-schedules-22,22cc05942c21f66b,938170f8785573fa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:30:23.527 WARN 2981023 --- [scanner-13495490-1,22cc05942c21f66b,72defc70be11f442] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:30:24.194 INFO 2981023 --- [ocp-schedules-6,83efc944ab50b060,cd681ad4c0e789c2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:30:25.170 INFO 2981023 --- [ocp-async-8,ce53a7da3d610475,0a52e84dbb2fae25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:26.113 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:26.113 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,9324f8e58ac5c944,4f3ed8bd07dcf64e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,9324f8e58ac5c944,4f3ed8bd07dcf64e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:30:26.351423+08:00, lastProcessTime=2024-09-11T17:30:16.395643+08:00 2024-09-11 17:30:26.497 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:30:26.351423+08:00 2024-09-11 17:30:26.501 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:30:26.501 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:30:26.504 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:30:26.504 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:30:26.504 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:30:26.504 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:30:26.508 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:30:26.508 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:30:26.508 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:30:26.508 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:30:26.542 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:30:26.547 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:30:26.547 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:30:26.547 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:30:26.547 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:30:26.547 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:30:26.553 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:30:26.852 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:30:27.156 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.648510317S 2024-09-11 17:30:27.157 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:30:27.170 INFO 2981023 --- [ocp-async-4,f725cc0c502f5d22,51abcbeb8f7b020b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:27.212 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:30:27.212 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:30:27.232 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:30:27.303 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:30:27.303 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:30:27.308 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:30:27.308 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:30:27.308 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:30:27.308 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:30:27.316 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:30:27.417 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] o.s.boot.SpringApplication : Started application in 0.912 seconds (JVM running for 82827.289) 2024-09-11 17:30:27.417 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.912873651S 2024-09-11 17:30:27.636 INFO 2981023 --- [ocp-schedules-10,572b94e0c4c1d594,7bec8ecaa6eec491] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:30:27.749 INFO 2981023 --- [ocp-async-6,1bc53141c75f5092,ac2a2cbf425a95a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:30:29.114 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:29.115 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:29.170 INFO 2981023 --- [ocp-async-7,f8ebea958f0f0e0f,74cea5d5f4d5447b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,d8b5e3b52aecae99,85c9399df0b583e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:29.318 INFO 2981023 --- [prometheus-minute-schedule-1,d8b5e3b52aecae99,85c9399df0b583e5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:31.170 INFO 2981023 --- [ocp-async-8,f3b7a2978f1b7d1c,a621784ed8cc48b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:31.854 INFO 2981023 --- [ocp-schedules-18,a454040f5f694bf2,b90f23e1640a25f1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:30:32.115 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:32.116 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,a7ca0ae508be464b,633250f37f8a3fb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,a7ca0ae508be464b,633250f37f8a3fb8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:33.170 INFO 2981023 --- [ocp-async-4,cdc593fc54175c05,c3eb4eefa1144528] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:33.653 INFO 2981023 --- [ocp-schedules-9,748e751e20cd5b5b,0f3bba7ab61ae6b1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:30:33.698 INFO 2981023 --- [ocp-schedules-24,5d887843a7b9efba,f8d9b29d855574b0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:30:33.709 WARN 2981023 --- [scanner-13495712-1,748e751e20cd5b5b,6817af9fdacc6609] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:30:33.712 INFO 2981023 --- [ocp-schedules-24,5d887843a7b9efba,f8d9b29d855574b0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:30:33.712 INFO 2981023 --- [ocp-schedules-24,5d887843a7b9efba,f8d9b29d855574b0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:30:33.726 INFO 2981023 --- [alarm-detect-16,748e751e20cd5b5b,04abcbc6052eabf3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643970, data is empty. 2024-09-11 17:30:33.733 INFO 2981023 --- [alarm-detect-16,748e751e20cd5b5b,04abcbc6052eabf3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643970, data is empty. 2024-09-11 17:30:33.736 INFO 2981023 --- [alarm-detect-16,748e751e20cd5b5b,04abcbc6052eabf3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643970, data is empty. 2024-09-11 17:30:33.754 INFO 2981023 --- [alarm-detect-2,748e751e20cd5b5b,1178c914dcc8f6c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643970, data is empty. 2024-09-11 17:30:33.758 INFO 2981023 --- [alarm-detect-13,748e751e20cd5b5b,25ac2a1a113cb122] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643970, data is empty. 2024-09-11 17:30:33.758 INFO 2981023 --- [alarm-detect-2,748e751e20cd5b5b,1178c914dcc8f6c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643970, data is empty. 2024-09-11 17:30:33.760 INFO 2981023 --- [alarm-detect-13,748e751e20cd5b5b,25ac2a1a113cb122] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643970, data is empty. 2024-09-11 17:30:33.762 INFO 2981023 --- [alarm-detect-2,748e751e20cd5b5b,1178c914dcc8f6c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643970, data is empty. 2024-09-11 17:30:33.765 INFO 2981023 --- [alarm-detect-13,748e751e20cd5b5b,25ac2a1a113cb122] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643970, data is empty. 2024-09-11 17:30:33.858 INFO 2981023 --- [alarm-detect-9,748e751e20cd5b5b,961d0fff0d0a29de] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643970, data is empty. 2024-09-11 17:30:33.865 INFO 2981023 --- [alarm-detect-9,748e751e20cd5b5b,961d0fff0d0a29de] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643970, data is empty. 2024-09-11 17:30:33.869 INFO 2981023 --- [alarm-detect-9,748e751e20cd5b5b,961d0fff0d0a29de] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643970, data is empty. 2024-09-11 17:30:35.116 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:35.117 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:35.170 INFO 2981023 --- [ocp-async-9,ef3902ce2ec3b393,f5bbb00f3909b701] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,7ee6692bfff483a9,4f0eb80c0d877813] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,7ee6692bfff483a9,4f0eb80c0d877813] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:36.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:30:36.351655+08:00, lastProcessTime=2024-09-11T17:30:26.351423+08:00 2024-09-11 17:30:36.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:30:36.351655+08:00 2024-09-11 17:30:37.170 INFO 2981023 --- [ocp-async-2,af46b85566a56002,00872f36e98eb770] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:37.314 INFO 2981023 --- [prometheus-cache-monitor1,5ba2970cdc2e4518,6f5e7a78f6e67694] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:30:37.315 INFO 2981023 --- [prometheus-cache-monitor1,5ba2970cdc2e4518,6f5e7a78f6e67694] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:30:37.749 INFO 2981023 --- [ocp-async-7,ac339d776f3497e8,1fbc42dce2ef1fa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:30:38.117 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:38.117 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,d7a9db62b08051ab,5717008a09c946a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,d7a9db62b08051ab,5717008a09c946a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:38.571 INFO 2981023 --- [ocp-schedules-21,a54e933f1306ef63,3c534701704543d9] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:30:38.589 WARN 2981023 --- [ocp-schedules-21,a54e933f1306ef63,3c534701704543d9] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:30:39.170 INFO 2981023 --- [ocp-async-3,643a4858da9fe925,4e0866e8b12fb00c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:40.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:30:40.003 INFO 2981023 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:30:41.118 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:41.118 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:41.171 INFO 2981023 --- [ocp-async-10,1a9db9ecf53259d8,28f356df57d555cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,5cef7666c0436f69,849dca407e59bea3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,5cef7666c0436f69,849dca407e59bea3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:41.857 INFO 2981023 --- [ocp-schedules-19,79aa22d2c5da96b0,6469b6bbb6276d68] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:30:43.171 INFO 2981023 --- [ocp-async-2,540b5fc65447ced6,6e8cbb1e6b841a8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:43.909 INFO 2981023 --- [ocp-schedules-11,fbbe4e914843a734,e7ced0380d270d04] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:30:43.958 WARN 2981023 --- [scanner-13495990-1,fbbe4e914843a734,8b70010eafb87c72] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:30:43.964 INFO 2981023 --- [ocp-schedules-5,364a91fe36379eae,1818174ed486dd27] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:30:44.119 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:44.122 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,ecc86e3f12c9fe89,81bd1ba5eeb54ff7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,ecc86e3f12c9fe89,81bd1ba5eeb54ff7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:45.101 INFO 2981023 --- [prometheus-second-schedule-1,c64e727be79bc155,d25d3c80ff745ac1] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:30:45.171 INFO 2981023 --- [ocp-async-8,164fa3bc8171edd8,28a05ee16dcc1f47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:46.017 INFO 2981023 --- [ocp-schedules-6,7e84102387027df5,f1aa7d2b05b4d6c4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:30:46.017 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:30:46.017 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:30:46.018 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:30:46.018 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:30:46.026 INFO 2981023 --- [ocp-schedules-21,6aa12b373fa9fdad,592e639a57345081] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:30:46.027 INFO 2981023 --- [host-check-agent-status1,6aa12b373fa9fdad,11930b66ef705136] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:30:46.027 INFO 2981023 --- [host-check-agent-status2,6aa12b373fa9fdad,a2049cd76231c460] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:30:46.028 INFO 2981023 --- [host-check-agent-status3,6aa12b373fa9fdad,6c29d5f32dbdb57a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:30:46.028 INFO 2981023 --- [host-check-agent-status4,6aa12b373fa9fdad,242ba9335ff9d7fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:30:46.029 INFO 2981023 --- [ocp-schedules-16,b77af6616281bff5,d4b047944051dcae] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:30:46.029 INFO 2981023 --- [host-check-clockdiff1,b77af6616281bff5,4ab4d3ab35e911f8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:30:46.036 INFO 2981023 --- [host-check-clockdiff2,b77af6616281bff5,6bccd88117d3a8d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:30:46.037 INFO 2981023 --- [host-check-clockdiff3,b77af6616281bff5,7aa06830e389cb09] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:30:46.039 INFO 2981023 --- [host-check-clockdiff4,b77af6616281bff5,0642eef95486b6e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:30:46.046 INFO 2981023 --- [host-check-clockdiff4,b77af6616281bff5,0642eef95486b6e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047046 0 0 , error= 2024-09-11 17:30:46.061 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:30:46.061 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:30:46.065 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:30:46.065 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:30:46.067 INFO 2981023 --- [host-check-agent-status1,6aa12b373fa9fdad,11930b66ef705136] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:30:46.069 INFO 2981023 --- [host-check-agent-status1,6aa12b373fa9fdad,11930b66ef705136] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:30:46.069 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:30:46.069 INFO 2981023 --- [host-check-agent-status2,6aa12b373fa9fdad,a2049cd76231c460] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:30:46.069 INFO 2981023 --- [host-check-agent-status3,6aa12b373fa9fdad,6c29d5f32dbdb57a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:30:46.071 INFO 2981023 --- [host-check-agent-status4,6aa12b373fa9fdad,242ba9335ff9d7fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:30:46.071 INFO 2981023 --- [host-check-agent-status2,6aa12b373fa9fdad,a2049cd76231c460] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:30:46.072 INFO 2981023 --- [host-check-agent-status4,6aa12b373fa9fdad,242ba9335ff9d7fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:30:46.073 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:30:46.081 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:30:46.082 INFO 2981023 --- [host-check-agent-status3,6aa12b373fa9fdad,6c29d5f32dbdb57a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:30:46.170 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:30:46.175 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:30:46.190 INFO 2981023 --- [ocp-schedules-17,8eecdf8135fd61c4,aaef6d2c747b9f5b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:30:46.190 INFO 2981023 --- [ocp-schedules-3,a89e73a4bcfb660a,a1779cd175504961] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:30:46.196 INFO 2981023 --- [ocp-schedules-30,b1b3aa86b792dcc8,0d50b8e31247b628] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:30:46.196 INFO 2981023 --- [ocp-schedules-27,eb876aa7ca649b33,a3b7c220d6a4e612] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:30:46.269 INFO 2981023 --- [ocp-schedules-12,5888be99cc177ee1,e137a11981d321bd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:30:46.283 INFO 2981023 --- [ocp-schedules-17,8eecdf8135fd61c4,aaef6d2c747b9f5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:30:46.284 INFO 2981023 --- [ocp-schedules-30,b1b3aa86b792dcc8,0d50b8e31247b628] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:30:46.285 INFO 2981023 --- [ocp-schedules-3,a89e73a4bcfb660a,a1779cd175504961] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:30:46.325 INFO 2981023 --- [ocp-schedules-17,8eecdf8135fd61c4,aaef6d2c747b9f5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:30:46.359 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:30:46.363 INFO 2981023 --- [ocp-schedules-12,5888be99cc177ee1,e137a11981d321bd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:30:46.385 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:30:46.381752+08:00, lastProcessTime=2024-09-11T17:30:36.351655+08:00 2024-09-11 17:30:46.389 INFO 2981023 --- [host-check-agent-status1,6aa12b373fa9fdad,11930b66ef705136] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:30:46.392 INFO 2981023 --- [host-check-agent-status1,6aa12b373fa9fdad,11930b66ef705136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:30:46.392 INFO 2981023 --- [host-check-agent-status1,6aa12b373fa9fdad,11930b66ef705136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:30:46.392 INFO 2981023 --- [host-check-agent-status1,6aa12b373fa9fdad,11930b66ef705136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:30:46.396 INFO 2981023 --- [ocp-schedules-13,1ac67e7a0ed19c35,7125fa356edbefc2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:30:46.476 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:30:46.477 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:30:46.477 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:30:46.476937536+08:00 2024-09-11 17:30:46.478 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:30:46.477+08:00 2024-09-11 17:30:46.478 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:30:46.478 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:30:46.478645900+08:00 2024-09-11 17:30:46.480 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:30:46.481 INFO 2981023 --- [host-check-agent-status3,6aa12b373fa9fdad,6c29d5f32dbdb57a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:30:46.484 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:30:46.477+08:00 and status AVAILABLE 2024-09-11 17:30:46.484 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:30:46.481132126+08:00 2024-09-11 17:30:46.484 INFO 2981023 --- [host-check-agent-status2,6aa12b373fa9fdad,a2049cd76231c460] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:30:46.485 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:30:46.484857990+08:00 2024-09-11 17:30:46.486 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:30:46.484+08:00 2024-09-11 17:30:46.487 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:30:46.478+08:00 2024-09-11 17:30:46.487 INFO 2981023 --- [host-check-agent-status3,6aa12b373fa9fdad,6c29d5f32dbdb57a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:30:46.487 INFO 2981023 --- [host-check-agent-status3,6aa12b373fa9fdad,6c29d5f32dbdb57a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:30:46.487 INFO 2981023 --- [host-check-agent-status3,6aa12b373fa9fdad,6c29d5f32dbdb57a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:30:46.488 INFO 2981023 --- [host-check-agent-status2,6aa12b373fa9fdad,a2049cd76231c460] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:30:46.488 INFO 2981023 --- [host-check-agent-status2,6aa12b373fa9fdad,a2049cd76231c460] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:30:46.488 INFO 2981023 --- [host-check-agent-status2,6aa12b373fa9fdad,a2049cd76231c460] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:30:46.488 INFO 2981023 --- [host-check-agent-status4,6aa12b373fa9fdad,242ba9335ff9d7fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:30:46.488 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:30:46.484+08:00 and status AVAILABLE 2024-09-11 17:30:46.490 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:30:46.478+08:00 and status AVAILABLE 2024-09-11 17:30:46.490 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:30:46.485+08:00 2024-09-11 17:30:46.492 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:30:46.485+08:00 and status AVAILABLE 2024-09-11 17:30:46.492 INFO 2981023 --- [host-check-agent-status4,6aa12b373fa9fdad,242ba9335ff9d7fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:30:46.492 INFO 2981023 --- [host-check-agent-status4,6aa12b373fa9fdad,242ba9335ff9d7fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:30:46.492 INFO 2981023 --- [host-check-agent-status4,6aa12b373fa9fdad,242ba9335ff9d7fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:30:46.501 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.502 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:30:46.502 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:30:46.503 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:30:46.503 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.504 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:30:46.504 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:30:46.513 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:30:46.519 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:30:46.520 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.520 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:46.521 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:30:46.523 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.524 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:46.524 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:30:46.525 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.526 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:46.560 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.561 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:46.562 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:30:46.563 INFO 2981023 --- [ocp-schedules-3,a89e73a4bcfb660a,a1779cd175504961] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:30:46.570 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:30:46.571 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:30:46.572 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:30:46.572 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:30:46.572 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:30:46.573 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:30:46.573 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:30:46.574 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:30:46.576 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.578 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:46.588 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.588 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:46.656 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.656 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:30:46.658 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:30:46.661 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.662 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:30:46.662 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.662 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.663 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:30:46.663 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:30:46.663 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.680 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:30:46.680 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:30:46.681 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:30:46.684 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.692 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:30:46.693 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.693 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:46.694 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:30:46.697 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.699 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:30:46.701 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:30:46.702 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:30:46.704 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:30:46.705 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:30:46.711 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:30:46.713 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:30:46.718 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:46.718 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:30:46.725 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:30:46.728 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:30:46.733 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:30:46.741 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.750 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.751 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.751 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:30:46.752 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:46.752 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:30:46.752 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.753 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.753 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:30:46.756 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.757 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:30:46.759 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:30:46.759 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.762 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.765 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:30:46.766 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.766 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:46.767 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:30:46.770 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:30:46.771 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:30:46.775 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:30:46.777 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:30:46.785 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:30:46.792 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.793 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:30:46.794 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.794 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:30:46.794 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:46.796 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.797 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:30:46.799 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:30:46.799 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:30:46.803 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:30:46.804 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:30:46.806 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:30:46.815 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:30:46.822 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:30:46.826 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:30:46.831 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:30:46.838 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:46.838 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:30:46.840 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.840 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.840 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.841 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:30:46.841 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.841 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:30:46.841 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:30:46.841 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:30:46.842 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:30:46.842 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.843 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:30:46.858 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:30:46.874 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:30:46.878 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.879 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.879 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:30:46.880 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:30:46.880 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.883 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:30:46.884 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:30:46.885 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:30:46.381752+08:00 2024-09-11 17:30:46.885 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:46.887 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:30:46.889 INFO 2981023 --- [host-check-host-status3,7e84102387027df5,4745cf7be6ff202b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:30:46.893 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:30:46.898 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:30:46.898 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.899 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:30:46.899 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.900 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:46.901 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:30:46.901 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:46.901 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:30:46.906 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.909 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.910 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:30:46.910 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:30:46.910 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:30:46.915 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:30:46.916 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:30:46.920 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.921 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.921 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:30:46.922 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:30:46.922 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:30:46.922 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:30:46.922 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.925 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:30:46.927 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:30:46.927 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:30:46.934 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:30:46.934 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.934 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:30:46.935 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.936 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:30:46.937 INFO 2981023 --- [host-check-host-status4,7e84102387027df5,4fbe4f20b8a62628] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:30:46.937 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:30:46.937 INFO 2981023 --- [host-check-host-status2,7e84102387027df5,fc1acc03fb24863f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:30:46.941 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:30:46.944 INFO 2981023 --- [host-check-host-status1,7e84102387027df5,0fef317ed6fb9f87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:30:46.946 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:30:46.947 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:30:46.959 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:30:46.972 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:30:46.983 INFO 2981023 --- [host-check-clockdiff2,b77af6616281bff5,6bccd88117d3a8d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047046 0 0 , error= 2024-09-11 17:30:46.988 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:30:46.990 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:30:46.994 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:46.995 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:46.995 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:30:46.998 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:30:46.998 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:30:47.001 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:30:47.001 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:30:47.002 INFO 2981023 --- [ocp-schedules-18,1cb7b1c1a6e6324f,f596713cde137e55] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:30:47.006 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:47.006 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:47.007 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:30:47.011 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:30:47.011 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:30:47.012 INFO 2981023 --- [ocp-schedules-30,b1b3aa86b792dcc8,0d50b8e31247b628] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:30:47.122 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:47.123 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:47.171 INFO 2981023 --- [ocp-async-1,df87cd33a5d3f7dc,796c96b8a875b875] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,7b75da7ead8202ec,2f2570f69dd4e556] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,7b75da7ead8202ec,2f2570f69dd4e556] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:47.749 INFO 2981023 --- [ocp-async-10,fbb602261182502f,82f673883a8d4268] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:30:47.825 INFO 2981023 --- [host-check-clockdiff1,b77af6616281bff5,4ab4d3ab35e911f8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047047 0 0 , error= 2024-09-11 17:30:49.171 INFO 2981023 --- [ocp-async-2,004e4cc746bbe125,5f8371ee2e247700] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:30:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:30:50.123 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:50.125 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:50.213 INFO 2981023 --- [ocp-schedules-20,dcf0716c93f212cd,8f606ee447324035] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:30:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,c3eb4ca517d95ce9,309cf8fadc780534] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,c3eb4ca517d95ce9,309cf8fadc780534] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:51.171 INFO 2981023 --- [ocp-async-8,5be2bc20e29540ef,9a43e81f35dcbbf1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:51.529 INFO 2981023 --- [ocp-schedules-26,0498217076949306,a5b0655852de50a0] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:30:51.530 INFO 2981023 --- [ocp-schedules-26,0498217076949306,a5b0655852de50a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:30:51.530 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:30:51.531 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:30:51.531 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:30:51.531 INFO 2981023 --- [ocp-schedules-26,0498217076949306,a5b0655852de50a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:30:51.645 INFO 2981023 --- [ocp-schedules-13,555771b7eb6c66ba,a55cba41780ff11c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:30:51.645+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:30:51.648 INFO 2981023 --- [ocp-schedules-13,555771b7eb6c66ba,a55cba41780ff11c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:30:51.798 INFO 2981023 --- [ocp-schedules-13,555771b7eb6c66ba,a55cba41780ff11c] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 17 to SqlAggregateData 2024-09-11 17:30:51.800 INFO 2981023 --- [ocp-schedules-13,555771b7eb6c66ba,a55cba41780ff11c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:30:51.860 INFO 2981023 --- [ocp-schedules-8,5f3d11b39ded9415,5e53fd1e2e6009cd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:30:52.013 INFO 2981023 --- [host-check-clockdiff3,b77af6616281bff5,7aa06830e389cb09] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047052 1 1 , error= 2024-09-11 17:30:52.284 INFO 2981023 --- [ocp-schedules-23,d161e67d781a3796,d0786d490e892c32] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:30:52.285 INFO 2981023 --- [ocp-schedules-23,d161e67d781a3796,d0786d490e892c32] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:30:52.322 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:30:52.323 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.323 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:30:52.323 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.324 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:30:52.324 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.326 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:30:52.326 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:30:52.327 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:30:52.332 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.333 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.336 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.336 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.336 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.336 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.343 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.344 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:30:52.349 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.349 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.352 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.352 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:30:52.358 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.358 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.366 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.366 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.366 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.367 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:30:52.367 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:30:52.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2301D6E6-0-0] 2024-09-11 17:30:52.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:30:52.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.368 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:30:52.371 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.371 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.375 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:30:52.381 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.381 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:30:52.382 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.383 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:30:52.384 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.384 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:30:52.385 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.385 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:30:52.386 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:30:52.393 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:30:52.394 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.394 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:30:52.395 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:30:52.395 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2301D6F0-0-0] 2024-09-11 17:30:52.395 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:30:52.397 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.398 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:30:52.398 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.399 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:30:52.400 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.401 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:30:52.402 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.402 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.402 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:30:52.402 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:30:52.403 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.403 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.403 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:30:52.404 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:30:52.404 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.404 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.405 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:30:52.405 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:30:52.405 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.405 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.406 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:30:52.406 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:52.406 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:30:52.406 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.407 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.407 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:30:52.407 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:30:52.408 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.408 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.408 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.408 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:30:52.409 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:30:52.409 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.409 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:30:52.409 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:30:52.411 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:30:52.411 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:30:52.424 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:30:52.427 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:30:52.427 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:30:52.427 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:30:52.427 INFO 2981023 --- [ocp-schedules-24,f9c29a8994c12e8c,2fe793b464b2918e] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:30:52.429 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:52.431 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.431 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:30:52.437 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:30:52.439 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:30:52.440 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.440 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:30:52.440 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:30:52.442 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.442 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:30:52.445 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:30:52.447 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:30:52.447 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.447 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.452 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:30:52.453 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:30:52.453 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:30:52.453 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:30:52.454 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:30:52.455 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:30:52.456 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:52.456 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:30:52.467 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:30:52.468 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:30:52.468 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:30:52.468 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:30:52.470 INFO 2981023 --- [ocp-schedules-23,d161e67d781a3796,d0786d490e892c32] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:30:52.473 WARN 2981023 --- [ocp-schedules-24,f9c29a8994c12e8c,2fe793b464b2918e] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:30:53.056 INFO 2981023 --- [ocp-schedules-25,44929de59504ce9c,3aee2f25c3ccd8f1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:30:53.056 INFO 2981023 --- [ocp-schedules-25,44929de59504ce9c,3aee2f25c3ccd8f1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:30:53.125 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:53.126 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:53.171 INFO 2981023 --- [ocp-async-1,e6e81d8c311aa447,931891c1fea554de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,8cff1cb8ae32f3a0,83d92b97d93635cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,8cff1cb8ae32f3a0,83d92b97d93635cd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:53.715 INFO 2981023 --- [ocp-schedules-21,e1faf2e01d707c67,32417880f94745d7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:30:53.724 INFO 2981023 --- [ocp-schedules-21,e1faf2e01d707c67,32417880f94745d7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:30:53.724 INFO 2981023 --- [ocp-schedules-21,e1faf2e01d707c67,32417880f94745d7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:30:54.196 INFO 2981023 --- [ocp-schedules-7,12ea37cdb65d2cca,2577a6588a4d0a00] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:30:54.202 INFO 2981023 --- [ocp-schedules-12,d523a1b9bb852ee3,b70d173406d4f4cd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:30:54.260 WARN 2981023 --- [scanner-13496240-1,d523a1b9bb852ee3,95d16fe362818fad] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:30:54.265 INFO 2981023 --- [alarm-detect-11,d523a1b9bb852ee3,59855c7e0b8f251d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643970, data is empty. 2024-09-11 17:30:54.269 INFO 2981023 --- [alarm-detect-11,d523a1b9bb852ee3,59855c7e0b8f251d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643987, data is empty. 2024-09-11 17:30:54.275 INFO 2981023 --- [alarm-detect-11,d523a1b9bb852ee3,59855c7e0b8f251d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643987, data is empty. 2024-09-11 17:30:54.297 INFO 2981023 --- [alarm-detect-18,d523a1b9bb852ee3,93a82802df01e664] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643970, data is empty. 2024-09-11 17:30:54.297 INFO 2981023 --- [alarm-detect-8,d523a1b9bb852ee3,a927262be6c2be3d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643970, data is empty. 2024-09-11 17:30:54.301 INFO 2981023 --- [alarm-detect-18,d523a1b9bb852ee3,93a82802df01e664] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643987, data is empty. 2024-09-11 17:30:54.302 INFO 2981023 --- [alarm-detect-8,d523a1b9bb852ee3,a927262be6c2be3d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643987, data is empty. 2024-09-11 17:30:54.305 INFO 2981023 --- [alarm-detect-18,d523a1b9bb852ee3,93a82802df01e664] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643987, data is empty. 2024-09-11 17:30:54.306 INFO 2981023 --- [alarm-detect-8,d523a1b9bb852ee3,a927262be6c2be3d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643987, data is empty. 2024-09-11 17:30:54.407 INFO 2981023 --- [alarm-detect-17,d523a1b9bb852ee3,c22f48cccc4106a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643970, data is empty. 2024-09-11 17:30:54.414 INFO 2981023 --- [alarm-detect-17,d523a1b9bb852ee3,c22f48cccc4106a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643987, data is empty. 2024-09-11 17:30:54.417 INFO 2981023 --- [alarm-detect-17,d523a1b9bb852ee3,c22f48cccc4106a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643987, data is empty. 2024-09-11 17:30:54.718 INFO 2981023 --- [ocp-schedules-27,9e3f3d811b8719b4,7181c960ce8a0fcd] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:30:54.718 INFO 2981023 --- [ocp-schedules-27,9e3f3d811b8719b4,7181c960ce8a0fcd] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:30:54.723 INFO 2981023 --- [ocp-schedules-27,9e3f3d811b8719b4,7181c960ce8a0fcd] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:30:54.723 INFO 2981023 --- [ocp-schedules-27,9e3f3d811b8719b4,7181c960ce8a0fcd] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:30:54.723 INFO 2981023 --- [ocp-schedules-27,9e3f3d811b8719b4,7181c960ce8a0fcd] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:30:54.732 INFO 2981023 --- [ocp-schedules-27,9e3f3d811b8719b4,7181c960ce8a0fcd] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:30:54.732 INFO 2981023 --- [ocp-schedules-27,9e3f3d811b8719b4,7181c960ce8a0fcd] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:30:54.732 INFO 2981023 --- [ocp-schedules-27,9e3f3d811b8719b4,7181c960ce8a0fcd] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:30:54.743 INFO 2981023 --- [ocp-schedules-27,9e3f3d811b8719b4,7181c960ce8a0fcd] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:30:55.171 INFO 2981023 --- [ocp-async-5,7a83b2f9e7212e46,9e6d0570af7e01cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:55.648 INFO 2981023 --- [ocp-schedules-7,9944ed386b4db617,8d912fe25f505ef9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:30:55.648 INFO 2981023 --- [ocp-schedules-7,9944ed386b4db617,8d912fe25f505ef9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:30:55.651 INFO 2981023 --- [ocp-schedules-7,9944ed386b4db617,8d912fe25f505ef9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:30:55.651 INFO 2981023 --- [ocp-schedules-7,9944ed386b4db617,8d912fe25f505ef9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:30:55.653 INFO 2981023 --- [ocp-schedules-7,9944ed386b4db617,8d912fe25f505ef9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:30:55.653 INFO 2981023 --- [ocp-schedules-7,9944ed386b4db617,8d912fe25f505ef9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:30:55.719 INFO 2981023 --- [ocp-schedules-18,eab4045b0956fb74,1f20a5a0809eb13f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:30:55.730 INFO 2981023 --- [ocp-schedules-18,eab4045b0956fb74,1f20a5a0809eb13f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:30:55.730 INFO 2981023 --- [ocp-schedules-18,eab4045b0956fb74,1f20a5a0809eb13f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:30:55.963 INFO 2981023 --- [ocp-schedules-26,d2160f968dcf5a07,7454ec21de9be4de] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:30:55.963 INFO 2981023 --- [ocp-schedules-26,d2160f968dcf5a07,7454ec21de9be4de] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:30:55.977 INFO 2981023 --- [Collect all tenants major compactions3,d2160f968dcf5a07,ca18ded88f9c1609] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:30:55.977 INFO 2981023 --- [Collect all tenants major compactions4,d2160f968dcf5a07,5cbeb3200683ed65] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:30:55.980 INFO 2981023 --- [Collect all tenants major compactions6,d2160f968dcf5a07,b6806325987c3e6c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:30:55.980 INFO 2981023 --- [Collect all tenants major compactions2,d2160f968dcf5a07,efa0e44f76d6c7ba] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:30:55.980 INFO 2981023 --- [Collect all tenants major compactions1,d2160f968dcf5a07,edd22058121df79e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:30:55.980 INFO 2981023 --- [Collect all tenants major compactions5,d2160f968dcf5a07,70e3941087f7c46f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:30:55.993 INFO 2981023 --- [Collect all tenants major compactions3,d2160f968dcf5a07,ca18ded88f9c1609] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:55.994 INFO 2981023 --- [Collect all tenants major compactions2,d2160f968dcf5a07,efa0e44f76d6c7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:55.994 INFO 2981023 --- [Collect all tenants major compactions3,d2160f968dcf5a07,ca18ded88f9c1609] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:30:55.995 INFO 2981023 --- [Collect all tenants major compactions2,d2160f968dcf5a07,efa0e44f76d6c7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:30:55.996 INFO 2981023 --- [Collect all tenants major compactions6,d2160f968dcf5a07,b6806325987c3e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:55.997 INFO 2981023 --- [Collect all tenants major compactions4,d2160f968dcf5a07,5cbeb3200683ed65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:55.997 INFO 2981023 --- [Collect all tenants major compactions6,d2160f968dcf5a07,b6806325987c3e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:30:55.997 INFO 2981023 --- [Collect all tenants major compactions1,d2160f968dcf5a07,edd22058121df79e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:55.998 INFO 2981023 --- [Collect all tenants major compactions4,d2160f968dcf5a07,5cbeb3200683ed65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:30:55.998 INFO 2981023 --- [Collect all tenants major compactions3,d2160f968dcf5a07,ca18ded88f9c1609] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:30:55.999 INFO 2981023 --- [Collect all tenants major compactions1,d2160f968dcf5a07,edd22058121df79e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:30:55.999 INFO 2981023 --- [Collect all tenants major compactions2,d2160f968dcf5a07,efa0e44f76d6c7ba] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:30:56.000 INFO 2981023 --- [Collect all tenants major compactions6,d2160f968dcf5a07,b6806325987c3e6c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:30:56.000 INFO 2981023 --- [Collect all tenants major compactions1,d2160f968dcf5a07,edd22058121df79e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:30:56.001 INFO 2981023 --- [Collect all tenants major compactions4,d2160f968dcf5a07,5cbeb3200683ed65] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:30:56.001 INFO 2981023 --- [Collect all tenants major compactions5,d2160f968dcf5a07,70e3941087f7c46f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:30:56.001 INFO 2981023 --- [Collect all tenants major compactions3,d2160f968dcf5a07,ca18ded88f9c1609] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:30:56.003 INFO 2981023 --- [Collect all tenants major compactions5,d2160f968dcf5a07,70e3941087f7c46f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:30:56.003 INFO 2981023 --- [Collect all tenants major compactions2,d2160f968dcf5a07,efa0e44f76d6c7ba] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:30:56.006 INFO 2981023 --- [Collect all tenants major compactions5,d2160f968dcf5a07,70e3941087f7c46f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:30:56.006 INFO 2981023 --- [Collect all tenants major compactions5,d2160f968dcf5a07,70e3941087f7c46f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:30:56.007 INFO 2981023 --- [Collect all tenants major compactions1,d2160f968dcf5a07,edd22058121df79e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:30:56.008 INFO 2981023 --- [Collect all tenants major compactions6,d2160f968dcf5a07,b6806325987c3e6c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:30:56.009 INFO 2981023 --- [Collect all tenants major compactions4,d2160f968dcf5a07,5cbeb3200683ed65] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:30:56.010 INFO 2981023 --- [ocp-schedules-26,d2160f968dcf5a07,7454ec21de9be4de] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:30:56.126 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:56.127 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,42ed242004ae388e,8031efe30ecc4594] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,42ed242004ae388e,8031efe30ecc4594] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:30:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:30:56.351662+08:00, lastProcessTime=2024-09-11T17:30:46.381752+08:00 2024-09-11 17:30:56.496 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:30:56.351662+08:00 2024-09-11 17:30:57.172 INFO 2981023 --- [ocp-async-8,df61a710c54f3b53,8a68607de079c7bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:57.749 INFO 2981023 --- [ocp-async-4,c5dfac8a374c0315,c94a7a10ac8553b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:30:59.127 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:30:59.128 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:30:59.172 INFO 2981023 --- [ocp-async-3,055677027a5cfd4b,d93027d1f1b963c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:30:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,70b6095bb9b7dac1,59c30765602456be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047060, interval=60 2024-09-11 17:30:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,70b6095bb9b7dac1,59c30765602456be] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:00.932 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:27:00.406, end=2024-09-11T17:29:00.923} 2024-09-11 17:31:00.937 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obC 2024-09-11 17:31:00.937 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:31:00.938 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:00.406, end=2024-09-11T17:29:00.923},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:31:00.938 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:31:00.938 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:31:00.939 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:31:00.942 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:00.406, end=2024-09-11T17:29:00.923},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:31:00.942 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:31:00.942 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:31:01.014 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661014000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.014Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.015 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661015000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.015Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.017 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661017000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.017Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.019 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661019000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.019Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.020 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661020000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.020Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.021 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661021000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.021Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.022 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661022000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.022Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.023 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661023000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.023Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.053 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 59 2024-09-11 17:31:01.054 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.062 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.062 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.081 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:31:01.081 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.088 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.088 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.118 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:31:01.118 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.122 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.122 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.129 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.129 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.132 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.132 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.135 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.135 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.138 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.138 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.142 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.142 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.147 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:31:01.147 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.153 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.153 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.153 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.153 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.153 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.153 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.154 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.155 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.155 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.155 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.158 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.158 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.159 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.159 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.162 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.162 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.178 INFO 2981023 --- [ocp-async-5,979ec36307f62656,90080d758d139a14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:01.186 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.186 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.188 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.188 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.190 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.190 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.192 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.193 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.194 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.194 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.194 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.194 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.195 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.195 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.196 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.196 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.197 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.197 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.198 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.199 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.199 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.201 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.201 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.204 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.206 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.019, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.214 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.214 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.215 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.014, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.215 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.215 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.217 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.015, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.218 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.021, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.219 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.022, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.219 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.017, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.220 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.230 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.230 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.232 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.02, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.251 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.313 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.023, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.336 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661336000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.336Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.346 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661346000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.346Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.403 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661403000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.403Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.417 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:31:01.417 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.421 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.421 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.423 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661423000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.423Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.429 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.429 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.436 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.436 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.439 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661439000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.439Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.453 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661453000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.453Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.459 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.459 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.461 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661461000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.461Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.466 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.466 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.474 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.474 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.482 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.482 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.482 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:31:01.482 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.483 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.483 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.486 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.486 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.487 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.487 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.495 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.495 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.509 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.509 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.517 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661517000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.517Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.518 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.518 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.520 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.520 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.520 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.520 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.528 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:31:01.528 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.529 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.529 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.529 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.529 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.529 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.529 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.529 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.530 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.530 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.530 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.531 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.531 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.533 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.533 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.537 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.336, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.540 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.540 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.542 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.543 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.346, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.545 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.423, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.546 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.546 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.546 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.546 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.547 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.547 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.548 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.548 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.548 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.549 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.553 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.439, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.553 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.453, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.553 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.553 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.554 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.554 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.555 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.555 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.558 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.558 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.558 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.403, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.558 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.461, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.559 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:31:01.561 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.517, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.562 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:31:01.562 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:00.406, end=2024-09-11T17:29:00.923},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:31:01.562 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:31:01.562 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:31:01.563 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:31:01.565 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:00.406, end=2024-09-11T17:29:00.923},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:31:01.567 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:31:01.567 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:31:01.575 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:31:01.575 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:31:01.576 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:00.406, end=2024-09-11T17:29:00.923},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:31:01.576 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:31:01.576 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:31:01.587 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:31:01.601 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960661601000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:31:01.601Z, endTime=2024-09-11T09:29:00.923Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:31:01.603 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:31:01.603 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:31:01.608 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.608 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.609 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:31:01.609 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:31:01.613 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:31:01.613 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726046940923000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:31:01.614 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:31:01.614 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:31:01.601, end=2024-09-11T17:29:00.923}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:31:01.615 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:31:01.615 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:00.406, end=2024-09-11T17:29:00.923},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:31:01.615 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:31:01.615 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:31:01.616 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:31:01.862 INFO 2981023 --- [ocp-schedules-29,bf4533cb4bf1fab7,3be07770f8ff3150] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:31:02.128 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:02.128 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,e41c27344bee9e17,6d45b9e27fb6dcd4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,e41c27344bee9e17,6d45b9e27fb6dcd4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:03.172 INFO 2981023 --- [ocp-async-5,b4e8072db8bafe19,8d5f9824ef767f67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:04.456 INFO 2981023 --- [ocp-schedules-21,81834d3b7b1485be,056b3c28a75598a3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:31:04.510 WARN 2981023 --- [scanner-13496525-1,81834d3b7b1485be,5cae01a9936cd66c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:31:05.128 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:05.130 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:05.172 INFO 2981023 --- [ocp-async-7,aed8fa37602f2730,d87a5bbb71b30e8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f8bea0b48b66dfee,53e71506425825d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,f8bea0b48b66dfee,53e71506425825d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:31:06.352005+08:00, lastProcessTime=2024-09-11T17:30:56.351662+08:00 2024-09-11 17:31:06.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:31:06.352005+08:00 2024-09-11 17:31:06.519 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a3f7eec4104bca59,62dc716828731f36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=a3f7eec4104bca59, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:31:06.530 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a3f7eec4104bca59,62dc716828731f36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=a3f7eec4104bca59, duration=16 ms] 2024-09-11 17:31:06.546 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,66594fe34718b7b0,9f6de97f42d70b04] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=66594fe34718b7b0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:31:06.546 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,66594fe34718b7b0,9f6de97f42d70b04] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=66594fe34718b7b0, duration=2 ms] 2024-09-11 17:31:06.552 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1da6a15291adbdba,46e256abb653c25e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=1da6a15291adbdba, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:31:06.556 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1da6a15291adbdba,46e256abb653c25e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=1da6a15291adbdba, duration=6 ms] 2024-09-11 17:31:07.172 INFO 2981023 --- [ocp-async-6,2f19a97ffc5ef022,367828e17bbd8c7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:07.267 INFO 2981023 --- [ocp-schedules-5,499b8a4570bbfc5d,9b71909b1a9e70b1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:31:07.268 INFO 2981023 --- [ocp-schedules-12,5a572c1e62c17b65,fb44f866852715c9] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:31:07.268+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:31:07.268 INFO 2981023 --- [ocp-schedules-12,5a572c1e62c17b65,fb44f866852715c9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:31:07.269 INFO 2981023 --- [ocp-schedules-12,5a572c1e62c17b65,fb44f866852715c9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:31:07.270 INFO 2981023 --- [ocp-schedules-12,5a572c1e62c17b65,fb44f866852715c9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:31:07.749 INFO 2981023 --- [ocp-async-3,12e7ef09adba2fc3,3f29cbeba4024458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:31:08.130 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:08.132 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,5dfb79b4e62d1fa2,9ce1de89b380f77c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:08.317 INFO 2981023 --- [prometheus-minute-schedule-1,5dfb79b4e62d1fa2,9ce1de89b380f77c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:09.172 INFO 2981023 --- [ocp-async-8,a4e2e87f405b2a48,8e7d4a048a6b1b55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:10.003 INFO 2981023 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:31:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:31:10.918 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:31:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:31:11.132 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:11.132 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:11.172 INFO 2981023 --- [ocp-async-1,d629bcfcc741c696,8ac4713b202f48b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,d688b3afaeed84cb,cefcfd410c786833] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,d688b3afaeed84cb,cefcfd410c786833] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:11.864 INFO 2981023 --- [ocp-schedules-4,8694731240765bb1,b5e3028840a27f2f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:31:13.173 INFO 2981023 --- [ocp-async-6,80f6a4494d77c848,506622c6a2497e1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:13.727 INFO 2981023 --- [ocp-schedules-22,91959c639a7fbcbb,8446f97eb8c11b5c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:31:13.736 INFO 2981023 --- [ocp-schedules-22,91959c639a7fbcbb,8446f97eb8c11b5c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:31:13.737 INFO 2981023 --- [ocp-schedules-22,91959c639a7fbcbb,8446f97eb8c11b5c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:31:14.132 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:14.134 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:14.253 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047000, nodeCount=1000, affectRows=1000 2024-09-11 17:31:14.255 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047000, nodeCount=999, validCount=999 2024-09-11 17:31:14.263 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047000, nodeCount=48, affectRows=48 2024-09-11 17:31:14.263 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047000, nodeCount=48, validCount=48 2024-09-11 17:31:14.283 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047000, nodeCount=1000, affectRows=1000 2024-09-11 17:31:14.285 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047000, nodeCount=1000, validCount=1000 2024-09-11 17:31:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,4fa73085f734444c,296be28b7544a352] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,d3d5a7155bc328a3,5defea31fbbf369a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:31:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,4fa73085f734444c,296be28b7544a352] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:14.666 INFO 2981023 --- [ocp-schedules-12,d3ba9b2261ae4012,eeaab52fa3f54678] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:31:14.725 WARN 2981023 --- [scanner-13498250-1,d3ba9b2261ae4012,891822ea229e7875] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:31:14.749 INFO 2981023 --- [alarm-detect-5,d3ba9b2261ae4012,6503971efa54b9f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643987, data is empty. 2024-09-11 17:31:14.755 INFO 2981023 --- [alarm-detect-5,d3ba9b2261ae4012,6503971efa54b9f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643987, data is empty. 2024-09-11 17:31:14.757 INFO 2981023 --- [alarm-detect-5,d3ba9b2261ae4012,6503971efa54b9f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643987, data is empty. 2024-09-11 17:31:14.782 INFO 2981023 --- [alarm-detect-10,d3ba9b2261ae4012,ac8acd390bbc5e4c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643987, data is empty. 2024-09-11 17:31:14.785 INFO 2981023 --- [alarm-detect-14,d3ba9b2261ae4012,0b0e88537068aa20] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643987, data is empty. 2024-09-11 17:31:14.788 INFO 2981023 --- [alarm-detect-14,d3ba9b2261ae4012,0b0e88537068aa20] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643987, data is empty. 2024-09-11 17:31:14.788 INFO 2981023 --- [alarm-detect-10,d3ba9b2261ae4012,ac8acd390bbc5e4c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643987, data is empty. 2024-09-11 17:31:14.792 INFO 2981023 --- [alarm-detect-10,d3ba9b2261ae4012,ac8acd390bbc5e4c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643987, data is empty. 2024-09-11 17:31:14.792 INFO 2981023 --- [alarm-detect-14,d3ba9b2261ae4012,0b0e88537068aa20] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643987, data is empty. 2024-09-11 17:31:14.897 INFO 2981023 --- [alarm-detect-1,d3ba9b2261ae4012,56ed9552faf23481] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643987, data is empty. 2024-09-11 17:31:14.899 INFO 2981023 --- [alarm-detect-1,d3ba9b2261ae4012,56ed9552faf23481] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643987, data is empty. 2024-09-11 17:31:14.904 INFO 2981023 --- [alarm-detect-1,d3ba9b2261ae4012,56ed9552faf23481] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643987, data is empty. 2024-09-11 17:31:15.173 INFO 2981023 --- [ocp-async-5,6f5f231139d8dd17,e81e64a7fa39ef6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:15.942 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047000, nodeCount=480, affectRows=480 2024-09-11 17:31:16.024 INFO 2981023 --- [ocp-schedules-3,48d116b326f9da03,6b2d2e52e063474e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:31:16.026 INFO 2981023 --- [host-check-clockdiff1,48d116b326f9da03,7056057334eb4d5c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:31:16.026 INFO 2981023 --- [host-check-clockdiff2,48d116b326f9da03,f98961c191cd981d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:31:16.027 INFO 2981023 --- [ocp-schedules-2,9a7b5a0b4fe4195a,643453c96bc548fb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:31:16.027 INFO 2981023 --- [host-check-clockdiff3,48d116b326f9da03,f1cbe4249e11a304] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:31:16.028 INFO 2981023 --- [host-check-clockdiff4,48d116b326f9da03,089a5626f89ad816] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:31:16.033 INFO 2981023 --- [ocp-schedules-16,0136af952aa23cb5,388d83e1003345f3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:31:16.033 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:31:16.035 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:31:16.035 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:31:16.035 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:31:16.035 INFO 2981023 --- [host-check-agent-status1,0136af952aa23cb5,228d8fdf4af4b4fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:31:16.035 INFO 2981023 --- [host-check-agent-status2,0136af952aa23cb5,bed583fb851e3ef8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:31:16.035 INFO 2981023 --- [host-check-agent-status3,0136af952aa23cb5,11c81ffdcd969733] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:31:16.036 INFO 2981023 --- [host-check-agent-status4,0136af952aa23cb5,9da3718359835ed5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:31:16.051 INFO 2981023 --- [host-check-clockdiff4,48d116b326f9da03,089a5626f89ad816] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047076 0 0 , error= 2024-09-11 17:31:16.076 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:31:16.077 INFO 2981023 --- [host-check-agent-status2,0136af952aa23cb5,bed583fb851e3ef8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:31:16.080 INFO 2981023 --- [host-check-agent-status2,0136af952aa23cb5,bed583fb851e3ef8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:31:16.080 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:31:16.084 INFO 2981023 --- [host-check-agent-status4,0136af952aa23cb5,9da3718359835ed5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:31:16.091 INFO 2981023 --- [host-check-agent-status1,0136af952aa23cb5,228d8fdf4af4b4fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:31:16.092 INFO 2981023 --- [host-check-agent-status4,0136af952aa23cb5,9da3718359835ed5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:31:16.096 INFO 2981023 --- [host-check-agent-status3,0136af952aa23cb5,11c81ffdcd969733] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:31:16.096 INFO 2981023 --- [host-check-agent-status1,0136af952aa23cb5,228d8fdf4af4b4fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:31:16.099 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:31:16.101 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:31:16.106 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:31:16.111 INFO 2981023 --- [host-check-agent-status3,0136af952aa23cb5,11c81ffdcd969733] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:31:16.187 INFO 2981023 --- [ocp-schedules-26,7bc64bdc13b6f874,ea75db4b2ddce3e2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:31:16.188 INFO 2981023 --- [ocp-schedules-13,661064af69d011f7,a188c3a721df1b2c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:31:16.188 INFO 2981023 --- [ocp-schedules-12,34deb1a21f56166f,de0a992fe55f61b2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:31:16.188 INFO 2981023 --- [ocp-schedules-4,86e011c155d77a56,2bc054b8c96ff807] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:31:16.194 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:31:16.199 INFO 2981023 --- [ocp-schedules-13,661064af69d011f7,a188c3a721df1b2c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:31:16.206 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:31:16.206 INFO 2981023 --- [ocp-schedules-26,7bc64bdc13b6f874,ea75db4b2ddce3e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:31:16.249 INFO 2981023 --- [ocp-schedules-13,661064af69d011f7,a188c3a721df1b2c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:31:16.269 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:31:16.283 INFO 2981023 --- [ocp-schedules-12,34deb1a21f56166f,de0a992fe55f61b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:31:16.333 INFO 2981023 --- [ocp-schedules-25,5a85925a37b4cf92,bd12bca1083b3b78] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:31:16.371 INFO 2981023 --- [host-check-agent-status3,0136af952aa23cb5,11c81ffdcd969733] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:31:16.378 INFO 2981023 --- [host-check-agent-status3,0136af952aa23cb5,11c81ffdcd969733] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:31:16.378 INFO 2981023 --- [host-check-agent-status3,0136af952aa23cb5,11c81ffdcd969733] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:31:16.378 INFO 2981023 --- [host-check-agent-status3,0136af952aa23cb5,11c81ffdcd969733] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:31:16.387 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:31:16.389 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:31:16.389 INFO 2981023 --- [host-check-agent-status4,0136af952aa23cb5,9da3718359835ed5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:31:16.389 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:31:16.389330592+08:00 2024-09-11 17:31:16.390 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:31:16.390588395+08:00 2024-09-11 17:31:16.392 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:31:16.390+08:00 2024-09-11 17:31:16.393 INFO 2981023 --- [host-check-agent-status4,0136af952aa23cb5,9da3718359835ed5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:31:16.393 INFO 2981023 --- [host-check-agent-status4,0136af952aa23cb5,9da3718359835ed5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:31:16.393 INFO 2981023 --- [host-check-agent-status4,0136af952aa23cb5,9da3718359835ed5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:31:16.393 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:31:16.389+08:00 2024-09-11 17:31:16.395 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:31:16.395 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:31:16.389+08:00 and status AVAILABLE 2024-09-11 17:31:16.395 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:31:16.395720873+08:00 2024-09-11 17:31:16.396 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:31:16.390+08:00 and status AVAILABLE 2024-09-11 17:31:16.399 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:31:16.396+08:00 2024-09-11 17:31:16.406 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:31:16.407 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:31:16.396+08:00 and status AVAILABLE 2024-09-11 17:31:16.408 INFO 2981023 --- [host-check-agent-status1,0136af952aa23cb5,228d8fdf4af4b4fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:31:16.408 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:31:16.407779595+08:00 2024-09-11 17:31:16.408 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:31:16.408 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:31:16.408043+08:00, lastProcessTime=2024-09-11T17:31:06.352005+08:00 2024-09-11 17:31:16.409 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:31:16.408+08:00 2024-09-11 17:31:16.409 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.410 INFO 2981023 --- [host-check-agent-status1,0136af952aa23cb5,228d8fdf4af4b4fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:31:16.410 INFO 2981023 --- [host-check-agent-status1,0136af952aa23cb5,228d8fdf4af4b4fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:31:16.410 INFO 2981023 --- [host-check-agent-status1,0136af952aa23cb5,228d8fdf4af4b4fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:31:16.410 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:16.411 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:31:16.408+08:00 and status AVAILABLE 2024-09-11 17:31:16.412 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:31:16.416 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:31:16.417 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.418 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:16.418 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:31:16.423 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.423 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:31:16.424 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:16.424 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:31:16.424 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:31:16.424 INFO 2981023 --- [ocp-schedules-26,7bc64bdc13b6f874,ea75db4b2ddce3e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:31:16.425 INFO 2981023 --- [host-check-agent-status2,0136af952aa23cb5,bed583fb851e3ef8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:31:16.428 INFO 2981023 --- [host-check-agent-status2,0136af952aa23cb5,bed583fb851e3ef8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:31:16.428 INFO 2981023 --- [host-check-agent-status2,0136af952aa23cb5,bed583fb851e3ef8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:31:16.428 INFO 2981023 --- [host-check-agent-status2,0136af952aa23cb5,bed583fb851e3ef8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:31:16.468 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:31:16.470 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.470 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:16.471 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:31:16.472 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:31:16.472 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:31:16.472 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:31:16.474 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:31:16.474 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:31:16.474 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:31:16.478 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.479 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:16.486 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.486 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:16.506 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:31:16.506 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:31:16.507 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:31:16.516 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.517 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:31:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:31:16.528 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:31:16.537 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:31:16.540 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:31:16.544 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:31:16.568 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:31:16.569 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:31:16.570 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:31:16.574 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:31:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.587 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.588 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.588 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.588 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:16.589 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:31:16.589 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:31:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:31:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:31:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:31:16.603 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:31:16.613 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:31:16.618 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:31:16.637 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:31:16.637 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:31:16.639 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:31:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:16.662 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:31:16.666 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:31:16.669 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:31:16.669 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:31:16.669 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.670 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.670 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:16.672 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.674 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:31:16.675 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:31:16.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.677 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.678 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:16.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:31:16.683 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:31:16.686 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:31:16.688 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:31:16.689 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:16.689 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:31:16.690 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:31:16.690 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:31:16.701 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:31:16.701 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:31:16.710 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.711 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.711 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:31:16.717 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:16.719 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:31:16.725 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:31:16.726 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.727 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.728 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:16.728 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:31:16.729 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:31:16.730 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:31:16.738 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:31:16.739 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:31:16.740 INFO 2981023 --- [host-check-host-status4,9a7b5a0b4fe4195a,8b210df27c65706e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:31:16.740 INFO 2981023 --- [host-check-host-status1,9a7b5a0b4fe4195a,67e0e557248e37ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:31:16.745 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:31:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.746 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:31:16.747 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:31:16.747 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:16.748 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:31:16.749 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.751 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:31:16.752 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:31:16.764 INFO 2981023 --- [host-check-host-status3,9a7b5a0b4fe4195a,92ba45fcbc1f3346] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:31:16.766 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:31:16.766 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:16.766 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:31:16.773 INFO 2981023 --- [host-check-host-status2,9a7b5a0b4fe4195a,46bfd1ce3df4a1a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:31:16.780 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:31:16.796 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:16.806 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.806 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.807 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:16.807 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:31:16.811 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:31:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:31:16.817 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:31:16.824 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:31:16.833 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:31:16.836 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:31:16.408043+08:00 2024-09-11 17:31:16.841 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.842 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.842 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:31:16.845 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:31:16.849 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:31:16.849 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:31:16.852 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.852 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.853 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:31:16.858 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:31:16.858 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046484472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:31:16.864 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:31:16.869 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:16.870 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:16.870 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:31:16.875 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:31:16.875 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:31:16.875 INFO 2981023 --- [ocp-schedules-12,34deb1a21f56166f,de0a992fe55f61b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:31:17.134 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:17.135 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:17.173 INFO 2981023 --- [ocp-async-10,1623d4eba403a293,a94404fb99cc9925] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,71b31e9eb1bc7ba9,3078e555f8ccb529] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,71b31e9eb1bc7ba9,3078e555f8ccb529] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:17.749 INFO 2981023 --- [ocp-async-5,79a10f0ab94ec393,2c304afaea030ac0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:31:18.160 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047000, nodeCount=480, validCount=480 2024-09-11 17:31:19.173 INFO 2981023 --- [ocp-async-3,66b0a2e78b42a58c,2a7190d1aa4c6f17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:19.199 INFO 2981023 --- [host-check-clockdiff2,48d116b326f9da03,f98961c191cd981d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047079 0 0 , error= 2024-09-11 17:31:19.731 INFO 2981023 --- [host-check-clockdiff1,48d116b326f9da03,7056057334eb4d5c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047079 0 0 , error= 2024-09-11 17:31:20.135 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:20.135 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,99f4b0f3c1e9a7b1,ebdc18de38bef901] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,99f4b0f3c1e9a7b1,ebdc18de38bef901] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:21.173 INFO 2981023 --- [ocp-async-9,d2d0e1e453e62b43,c3b27c70f99b5d4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:21.867 INFO 2981023 --- [ocp-schedules-17,b8aa337195c7ef73,de2acf417aa6dfd5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:31:22.030 INFO 2981023 --- [host-check-clockdiff3,48d116b326f9da03,f1cbe4249e11a304] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047082 1 1 , error= 2024-09-11 17:31:22.554 INFO 2981023 --- [ocp-schedules-4,a73d73a1dfabba5f,7758e501b65615c4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:31:22.554 INFO 2981023 --- [ocp-schedules-4,a73d73a1dfabba5f,7758e501b65615c4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:31:22.554 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:31:22.554 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:31:22.554 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:31:22.558 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:31:22.558 INFO 2981023 --- [ocp-schedules-4,a73d73a1dfabba5f,7758e501b65615c4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:31:22.559 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:31:22.559 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:31:22.559 INFO 2981023 --- [ocp-schedules-4,a73d73a1dfabba5f,7758e501b65615c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:31:22.559 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:31:22.560 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:31:22.561 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:31:22.578 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:31:22.579 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:31:22.580 INFO 2981023 --- [ocp-schedules-4,a73d73a1dfabba5f,7758e501b65615c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:31:22.586 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:31:22.588 INFO 2981023 --- [ocp-schedules-4,a73d73a1dfabba5f,7758e501b65615c4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:31:23.135 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:23.136 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:23.173 INFO 2981023 --- [ocp-async-8,c16b1e4ede2be126,ea7a1142818d5335] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,4bc4bef28a79770d,5798cae38c5c47e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,4bc4bef28a79770d,5798cae38c5c47e1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:24.199 INFO 2981023 --- [ocp-schedules-4,317658f91fd472ca,fe202b7da29ab34f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:31:24.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,b1b5e9647e5bd758,76c540454cef567b] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&version=2&ObClusterId=1719994443, client=10.31.35.6, traceId=b1b5e9647e5bd758, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkrtd,1719994443,2,{"ObRegion":"linkrtd","ObCluster":"linkrtd","ObRegionId":1719994443,"ObClusterId":1719994443,"Type":"PRIMARY","timestamp":1726047084469361,"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:31:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,b1b5e9647e5bd758,76c540454cef567b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkrtd, obClusterId=1719994443, rootServerJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, groupName=linkrtd, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:31:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,b1b5e9647e5bd758,76c540454cef567b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:31:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,b1b5e9647e5bd758,76c540454cef567b] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkrtd, clusterId=1719994443, affectRows=1 2024-09-11 17:31:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,b1b5e9647e5bd758,76c540454cef567b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.6, traceId=b1b5e9647e5bd758, duration=9 ms] 2024-09-11 17:31:24.955 INFO 2981023 --- [ocp-schedules-7,8c5a0901f69cb0e1,53d6e17a5e1d1239] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:31:25.008 WARN 2981023 --- [scanner-13498580-1,8c5a0901f69cb0e1,7b3ed60b654785fb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:31:25.066 INFO 2981023 --- [alarm-detect-2,8c5a0901f69cb0e1,845dd182719ab202] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047045 ~ 1726047075 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726047060, value=2.445)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.445, values={1726047060=2.445}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:31:25.066 INFO 2981023 --- [alarm-detect-2,8c5a0901f69cb0e1,845dd182719ab202] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Firing 2024-09-11 17:31:25.067 INFO 2981023 --- [alarm-detect-2,8c5a0901f69cb0e1,845dd182719ab202] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-3] 2024-09-11 17:31:25.101 INFO 2981023 --- [alarm-detect-2,8c5a0901f69cb0e1,845dd182719ab202] c.o.o.alarm.service.OcpAlarmServiceImpl : delete alarm event 6007100 2024-09-11 17:31:25.105 INFO 2981023 --- [alarm-detect-2,8c5a0901f69cb0e1,845dd182719ab202] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007101 2024-09-11 17:31:25.173 INFO 2981023 --- [ocp-async-4,d9e203d0ccbf4cfc,8346f84954ae5ae2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:26.136 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:26.136 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,5f79908d61cafac7,cd9746faa49f2311] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,5f79908d61cafac7,cd9746faa49f2311] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:31:26.337853+08:00, lastProcessTime=2024-09-11T17:31:16.408043+08:00 2024-09-11 17:31:26.457 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:31:26.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:31:26.337853+08:00 2024-09-11 17:31:27.173 INFO 2981023 --- [ocp-async-2,7f2651d13367443a,5f0cd646016a148f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:27.750 INFO 2981023 --- [ocp-async-3,417fe89ac3688740,652a8b362ef98cfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:31:29.137 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:29.137 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:29.173 INFO 2981023 --- [ocp-async-7,e994ac184a4593e5,84d174b06296b5c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,5019f626c0edc718,9b92c4b62b0a0e8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,5019f626c0edc718,9b92c4b62b0a0e8d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:31.174 INFO 2981023 --- [ocp-async-4,f0dfa0dda3bd0003,d907c50b80e9a9b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:31.869 INFO 2981023 --- [ocp-schedules-18,b78e04bedc41af3c,5320481071e4e7bf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:31:32.137 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:32.138 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,905bbfda915e7be9,972caf1d6df11ee7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,905bbfda915e7be9,972caf1d6df11ee7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:33.174 INFO 2981023 --- [ocp-async-2,a94ad6d259862cfc,2ccff00dfb3b5c8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:33.738 INFO 2981023 --- [ocp-schedules-10,0a4f5edf94fddea1,43e9c3b162ebc03b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:31:33.744 INFO 2981023 --- [ocp-schedules-10,0a4f5edf94fddea1,43e9c3b162ebc03b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:31:33.744 INFO 2981023 --- [ocp-schedules-10,0a4f5edf94fddea1,43e9c3b162ebc03b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:31:35.138 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:35.139 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:35.153 INFO 2981023 --- [ocp-schedules-8,e50bbfb5ea2f5506,7000533002f163aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:31:35.174 INFO 2981023 --- [ocp-async-5,4ac0c3ddcf08378b,6128b06b87409433] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:35.194 WARN 2981023 --- [scanner-13498798-1,e50bbfb5ea2f5506,aadd40b3fcd66d66] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:31:35.227 INFO 2981023 --- [alarm-detect-9,e50bbfb5ea2f5506,08b60f13e94471d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643987, data is empty. 2024-09-11 17:31:35.230 INFO 2981023 --- [alarm-detect-9,e50bbfb5ea2f5506,08b60f13e94471d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643987, data is empty. 2024-09-11 17:31:35.232 INFO 2981023 --- [alarm-detect-9,e50bbfb5ea2f5506,08b60f13e94471d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643987, data is empty. 2024-09-11 17:31:35.251 INFO 2981023 --- [alarm-detect-13,e50bbfb5ea2f5506,6999bae3d5ad5e82] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643987, data is empty. 2024-09-11 17:31:35.254 INFO 2981023 --- [alarm-detect-7,e50bbfb5ea2f5506,a9ac53a4d06144ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643987, data is empty. 2024-09-11 17:31:35.255 INFO 2981023 --- [alarm-detect-13,e50bbfb5ea2f5506,6999bae3d5ad5e82] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643987, data is empty. 2024-09-11 17:31:35.258 INFO 2981023 --- [alarm-detect-13,e50bbfb5ea2f5506,6999bae3d5ad5e82] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643987, data is empty. 2024-09-11 17:31:35.261 INFO 2981023 --- [alarm-detect-7,e50bbfb5ea2f5506,a9ac53a4d06144ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643987, data is empty. 2024-09-11 17:31:35.265 INFO 2981023 --- [alarm-detect-7,e50bbfb5ea2f5506,a9ac53a4d06144ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643987, data is empty. 2024-09-11 17:31:35.306 INFO 2981023 --- [alarm-detect-14,e50bbfb5ea2f5506,db45ae1a435b4b71] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047060 ~ 1726047085 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726047075, value=1.922)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.922, values={1726047075=1.922}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:31:35.314 INFO 2981023 --- [prometheus-minute-schedule-1,c60072a9993d76e7,25ae40186b035ad0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:35.317 INFO 2981023 --- [prometheus-minute-schedule-1,c60072a9993d76e7,25ae40186b035ad0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:35.377 INFO 2981023 --- [alarm-detect-18,e50bbfb5ea2f5506,a2ed240b432d0f38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643987, data is empty. 2024-09-11 17:31:35.419 INFO 2981023 --- [alarm-detect-18,e50bbfb5ea2f5506,a2ed240b432d0f38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643987, data is empty. 2024-09-11 17:31:35.424 INFO 2981023 --- [alarm-detect-18,e50bbfb5ea2f5506,a2ed240b432d0f38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643987, data is empty. 2024-09-11 17:31:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:31:36.337189+08:00, lastProcessTime=2024-09-11T17:31:26.337853+08:00 2024-09-11 17:31:36.466 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:31:36.337189+08:00 2024-09-11 17:31:37.174 INFO 2981023 --- [ocp-async-10,7b45acf907db772d,c4abee4403c2a32a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:37.314 INFO 2981023 --- [prometheus-cache-monitor1,8dd78702d05cdc1b,7ddda7a3f632294e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:31:37.315 INFO 2981023 --- [prometheus-cache-monitor1,8dd78702d05cdc1b,7ddda7a3f632294e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:31:37.750 INFO 2981023 --- [ocp-async-7,acbf17709c59496b,a3a81b137076aa40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:31:38.139 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:38.140 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,14d22fd0e6cf4a6a,4a885e4457b0a440] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,14d22fd0e6cf4a6a,4a885e4457b0a440] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:38.721 INFO 2981023 --- [ocp-schedules-26,0306303345ab37d3,5933705ac54ab562] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:31:38.740 WARN 2981023 --- [ocp-schedules-26,0306303345ab37d3,5933705ac54ab562] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:31:39.174 INFO 2981023 --- [ocp-async-2,a60e4e1bcda8a598,d1ea22754ca7a6ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:40.003 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:31:40.003 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:31:41.140 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:41.140 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:41.174 INFO 2981023 --- [ocp-async-5,bbf76fbeb88b0462,dc4216ef0e46c626] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,15e90ec4fd4bb5a6,a7cf143c5d681801] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,15e90ec4fd4bb5a6,a7cf143c5d681801] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:41.872 INFO 2981023 --- [ocp-schedules-10,db563f8158c484fd,a7a3447119a83c90] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:31:43.174 INFO 2981023 --- [ocp-async-10,bda439e9df7c52ed,3df669bdfb9712e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:44.141 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:44.142 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,64af8decc112540c,483c8cd6e51f50bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:44.317 INFO 2981023 --- [prometheus-minute-schedule-1,64af8decc112540c,483c8cd6e51f50bd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:44.544 INFO 2981023 --- [ocp-schedules-15,b8c032de9360786c,f84b763cf1467329] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:31:45.101 INFO 2981023 --- [prometheus-second-schedule-1,9905bf28f017944a,7793989653d20baf] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:31:45.174 INFO 2981023 --- [ocp-async-6,fc2b1d5b22e6fe6b,bfe2ea0d4c4f0c15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:45.451 INFO 2981023 --- [ocp-schedules-11,89a8557704a77def,e84149983c5efbf3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:31:45.500 WARN 2981023 --- [scanner-13499077-1,89a8557704a77def,ba31be6b12e4af6c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:31:45.543 INFO 2981023 --- [alarm-detect-6,89a8557704a77def,e2959a97abc1c2dc] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047075 ~ 1726047095 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726047090, value=1.558)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.558, values={1726047090=1.558}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:31:46.014 INFO 2981023 --- [ocp-schedules-22,c497105a844d4e29,970e06e8904b0050] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:31:46.015 INFO 2981023 --- [host-check-clockdiff1,c497105a844d4e29,2df0fca6b6f3f25e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:31:46.015 INFO 2981023 --- [host-check-clockdiff2,c497105a844d4e29,be6429a78442c101] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:31:46.015 INFO 2981023 --- [host-check-clockdiff3,c497105a844d4e29,50bbb1b7dfa66acf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:31:46.017 INFO 2981023 --- [host-check-clockdiff4,c497105a844d4e29,b239f876f36930e0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:31:46.030 INFO 2981023 --- [ocp-schedules-23,c760f8b56d1adc0c,731f840923359111] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:31:46.030 INFO 2981023 --- [ocp-schedules-14,250a55fd6aeba8b9,a5de3e632dc4a337] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:31:46.030 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:31:46.030 INFO 2981023 --- [host-check-clockdiff4,c497105a844d4e29,b239f876f36930e0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047106 0 0 , error= 2024-09-11 17:31:46.031 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:31:46.031 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:31:46.031 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:31:46.031 INFO 2981023 --- [host-check-agent-status1,250a55fd6aeba8b9,282bc894d081706d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:31:46.032 INFO 2981023 --- [host-check-agent-status2,250a55fd6aeba8b9,ad84c7ddd4f8b85a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:31:46.032 INFO 2981023 --- [host-check-agent-status3,250a55fd6aeba8b9,9230b1a65fa74a36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:31:46.033 INFO 2981023 --- [host-check-agent-status4,250a55fd6aeba8b9,2d90b0137b944a63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:31:46.046 INFO 2981023 --- [host-check-clockdiff2,c497105a844d4e29,be6429a78442c101] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047106 0 0 , error= 2024-09-11 17:31:46.057 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:31:46.057 INFO 2981023 --- [host-check-agent-status3,250a55fd6aeba8b9,9230b1a65fa74a36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:31:46.061 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:31:46.061 INFO 2981023 --- [host-check-agent-status3,250a55fd6aeba8b9,9230b1a65fa74a36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:31:46.061 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:31:46.062 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:31:46.063 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:31:46.064 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:31:46.065 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:31:46.068 INFO 2981023 --- [host-check-agent-status2,250a55fd6aeba8b9,ad84c7ddd4f8b85a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:31:46.069 INFO 2981023 --- [host-check-agent-status2,250a55fd6aeba8b9,ad84c7ddd4f8b85a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:31:46.070 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:31:46.102 INFO 2981023 --- [host-check-agent-status1,250a55fd6aeba8b9,282bc894d081706d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:31:46.182 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:31:46.188 INFO 2981023 --- [ocp-schedules-6,2675470504e6adb6,6cb5621c1d9591d5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:31:46.193 INFO 2981023 --- [ocp-schedules-7,cfaad41d8b1ba219,72726e715cca4511] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:31:46.196 INFO 2981023 --- [ocp-schedules-18,60d4559c2c7b60d0,eb55f5f24724c113] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:31:46.196 INFO 2981023 --- [ocp-schedules-13,c0303b4ead5f0e60,4efe34a2c821d771] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:31:46.227 INFO 2981023 --- [ocp-schedules-12,17f946b461a39191,e5f374ffc3f7eb6d] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:31:46.227 INFO 2981023 --- [ocp-schedules-12,17f946b461a39191,e5f374ffc3f7eb6d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:31:46.265 INFO 2981023 --- [ocp-schedules-1,16456c0fa8b857ce,a4de031d4ae6978b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:31:46.266 INFO 2981023 --- [host-check-agent-status1,250a55fd6aeba8b9,282bc894d081706d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:31:46.272 INFO 2981023 --- [ocp-schedules-6,2675470504e6adb6,6cb5621c1d9591d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:31:46.274 INFO 2981023 --- [host-check-agent-status4,250a55fd6aeba8b9,2d90b0137b944a63] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:31:46.274 INFO 2981023 --- [ocp-schedules-18,60d4559c2c7b60d0,eb55f5f24724c113] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:31:46.299 INFO 2981023 --- [ocp-schedules-7,cfaad41d8b1ba219,72726e715cca4511] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:31:46.300 INFO 2981023 --- [ocp-schedules-7,cfaad41d8b1ba219,72726e715cca4511] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:31:46.302 INFO 2981023 --- [host-check-agent-status4,250a55fd6aeba8b9,2d90b0137b944a63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:31:46.374 INFO 2981023 --- [ocp-schedules-1,16456c0fa8b857ce,a4de031d4ae6978b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:31:46.381 INFO 2981023 --- [ocp-schedules-26,8faafb373ec87378,f159b912662ceb44] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:31:46.394 INFO 2981023 --- [ocp-schedules-15,12999366b5dd0947,4b77ad62827e1ccf] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:31:46.395 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:31:46.395 INFO 2981023 --- [ocp-schedules-26,8faafb373ec87378,f159b912662ceb44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:31:46.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:31:46.471513+08:00, lastProcessTime=2024-09-11T17:31:36.337189+08:00 2024-09-11 17:31:46.489 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:31:46.491 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:31:46.490981621+08:00 2024-09-11 17:31:46.492 INFO 2981023 --- [host-check-agent-status2,250a55fd6aeba8b9,ad84c7ddd4f8b85a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:31:46.495 INFO 2981023 --- [host-check-agent-status2,250a55fd6aeba8b9,ad84c7ddd4f8b85a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:31:46.495 INFO 2981023 --- [host-check-agent-status2,250a55fd6aeba8b9,ad84c7ddd4f8b85a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:31:46.495 INFO 2981023 --- [host-check-agent-status2,250a55fd6aeba8b9,ad84c7ddd4f8b85a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:31:46.498 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:31:46.491+08:00 2024-09-11 17:31:46.504 INFO 2981023 --- [host-check-agent-status3,250a55fd6aeba8b9,9230b1a65fa74a36] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:31:46.504 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:31:46.491+08:00 and status AVAILABLE 2024-09-11 17:31:46.509 INFO 2981023 --- [host-check-agent-status3,250a55fd6aeba8b9,9230b1a65fa74a36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:31:46.509 INFO 2981023 --- [host-check-agent-status3,250a55fd6aeba8b9,9230b1a65fa74a36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:31:46.509 INFO 2981023 --- [host-check-agent-status3,250a55fd6aeba8b9,9230b1a65fa74a36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:31:46.556 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:31:46.556 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:31:46.556433337+08:00 2024-09-11 17:31:46.561 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:31:46.556+08:00 2024-09-11 17:31:46.562 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:31:46.564 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:31:46.564390511+08:00 2024-09-11 17:31:46.565 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:31:46.566 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:31:46.556+08:00 and status AVAILABLE 2024-09-11 17:31:46.567 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:31:46.567 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:31:46.564+08:00 2024-09-11 17:31:46.567 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:31:46.567482513+08:00 2024-09-11 17:31:46.570 INFO 2981023 --- [host-check-agent-status1,250a55fd6aeba8b9,282bc894d081706d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:31:46.571 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:31:46.567+08:00 2024-09-11 17:31:46.571 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:31:46.564+08:00 and status AVAILABLE 2024-09-11 17:31:46.573 INFO 2981023 --- [host-check-agent-status1,250a55fd6aeba8b9,282bc894d081706d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:31:46.573 INFO 2981023 --- [host-check-agent-status1,250a55fd6aeba8b9,282bc894d081706d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:31:46.573 INFO 2981023 --- [host-check-agent-status1,250a55fd6aeba8b9,282bc894d081706d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:31:46.575 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:31:46.567+08:00 and status AVAILABLE 2024-09-11 17:31:46.577 INFO 2981023 --- [host-check-agent-status4,250a55fd6aeba8b9,2d90b0137b944a63] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:31:46.582 INFO 2981023 --- [host-check-agent-status4,250a55fd6aeba8b9,2d90b0137b944a63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:31:46.582 INFO 2981023 --- [host-check-agent-status4,250a55fd6aeba8b9,2d90b0137b944a63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:31:46.582 INFO 2981023 --- [host-check-agent-status4,250a55fd6aeba8b9,2d90b0137b944a63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:31:46.582 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.582 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:31:46.582 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:31:46.584 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.584 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:46.585 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:31:46.588 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:31:46.588 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.589 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:46.590 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:31:46.597 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,97087e5da6ca1d63,f62a26f071c8f219] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=97087e5da6ca1d63, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:31:46.597 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,97087e5da6ca1d63,f62a26f071c8f219] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=97087e5da6ca1d63, duration=0 ms] 2024-09-11 17:31:46.598 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.599 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:46.600 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.600 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:31:46.600 INFO 2981023 --- [ocp-schedules-6,2675470504e6adb6,6cb5621c1d9591d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:31:46.600 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:31:46.601 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:31:46.605 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.608 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:31:46.610 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.610 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:31:46.611 INFO 2981023 --- [pool-10-thread-4,84ee5c5089eeb924,b14b0cb3ffd2e6bc] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:31:46.611 INFO 2981023 --- [pool-10-thread-4,84ee5c5089eeb924,b14b0cb3ffd2e6bc] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:31:46.621 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,35024da2180cab97,6bc492a56ba9603d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=35024da2180cab97, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:31:46.622 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,35024da2180cab97,6bc492a56ba9603d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=35024da2180cab97, duration=1 ms] 2024-09-11 17:31:46.623 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.623 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:31:46.630 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.633 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:31:46.656 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:46.658 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.658 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.659 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:31:46.663 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:31:46.663 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:46.664 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,790c7a6dd0c7be82,e483a3c085a6bb5a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&ObClusterId=1719994443&version=2, client=10.31.35.10, traceId=790c7a6dd0c7be82, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkrtd,1719994443,false,2,] 2024-09-11 17:31:46.664 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,790c7a6dd0c7be82,e483a3c085a6bb5a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=790c7a6dd0c7be82, duration=0 ms] 2024-09-11 17:31:46.665 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:31:46.665 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.666 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.666 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.668 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:31:46.669 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:31:46.669 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:31:46.670 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.673 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.676 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:31:46.677 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:31:46.677 INFO 2981023 --- [ocp-schedules-26,8faafb373ec87378,f159b912662ceb44] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:31:46.678 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:31:46.679 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:31:46.680 INFO 2981023 --- [pool-10-thread-3,84ee5c5089eeb924,c67d771161059fec] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:31:46.681 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:31:46.681 INFO 2981023 --- [pool-10-thread-1,84ee5c5089eeb924,a62821e844036a52] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:31:46.682 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:31:46.683 INFO 2981023 --- [pool-10-thread-2,84ee5c5089eeb924,95c3492014063c85] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:31:46.684 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:31:46.686 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:31:46.687 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.687 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:46.690 INFO 2981023 --- [ForkJoinPool-6-worker-4,84ee5c5089eeb924,0328e13db5fe87c3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:31:46.690 INFO 2981023 --- [ForkJoinPool-6-worker-4,84ee5c5089eeb924,0328e13db5fe87c3] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:31:46.694 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:31:46.694 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:31:46.695 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:31:46.695 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:31:46.707 INFO 2981023 --- [pool-10-thread-2,84ee5c5089eeb924,95c3492014063c85] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:31:46.707 INFO 2981023 --- [pool-10-thread-2,84ee5c5089eeb924,95c3492014063c85] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:31:46.754 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.755 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.756 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:31:46.756 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.756 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:31:46.763 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:31:46.764 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.765 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:31:46.768 WARN 2981023 --- [ForkJoinPool-6-worker-4,84ee5c5089eeb924,0328e13db5fe87c3] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:31:46.776 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:31:46.783 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:31:46.784 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.784 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:31:46.784 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:31:46.785 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.785 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:31:46.786 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:31:46.792 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:31:46.794 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.795 INFO 2981023 --- [ForkJoinPool-6-worker-8,84ee5c5089eeb924,95da8a0034074eaa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.797 INFO 2981023 --- [ForkJoinPool-6-worker-8,84ee5c5089eeb924,95da8a0034074eaa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:31:46.800 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:31:46.802 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:31:46.805 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:31:46.820 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:31:46.826 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:31:46.826 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.827 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.827 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:31:46.828 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:31:46.828 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.828 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.829 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:31:46.830 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:31:46.831 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.831 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.831 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.832 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:31:46.832 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:31:46.832 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.833 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:46.833 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:31:46.833 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:31:46.834 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.837 INFO 2981023 --- [ocp-schedules-11,e2fdbb910277f09e,7445d46b7b50e02e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:31:46.837 INFO 2981023 --- [ocp-schedules-11,e2fdbb910277f09e,7445d46b7b50e02e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:31:46.837 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:31:46.838 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:31:46.839 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.840 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.843 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:31:46.843 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:31:46.843 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.846 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:31:46.855 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:31:46.859 INFO 2981023 --- [ForkJoinPool-6-worker-8,84ee5c5089eeb924,95da8a0034074eaa] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:31:46.871 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:46.871 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:31:46.872 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:31:46.872 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.875 INFO 2981023 --- [ForkJoinPool-6-worker-8,84ee5c5089eeb924,95da8a0034074eaa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.875 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.875 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:31:46, 2024-09-11 17:31:46] 2024-09-11 17:31:46.875 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:31:46.877 INFO 2981023 --- [ForkJoinPool-6-worker-8,84ee5c5089eeb924,95da8a0034074eaa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:31:46.878 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:31:46, 2024-09-11 17:31:46] 2024-09-11 17:31:46.878 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:31:46.878 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.879 INFO 2981023 --- [ForkJoinPool-6-worker-8,84ee5c5089eeb924,95da8a0034074eaa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.884 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:31:46.884 INFO 2981023 --- [ForkJoinPool-6-worker-13,84ee5c5089eeb924,636d9adbd9f60c50] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:31:46.886 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:31:46.887 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:31:46.891 INFO 2981023 --- [ForkJoinPool-6-worker-8,84ee5c5089eeb924,95da8a0034074eaa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:31:46.892 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.892 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:31:46.893 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:31:46.894 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:31:46.894 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.899 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:31:46, 2024-09-11 17:31:46] 2024-09-11 17:31:46.900 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:31:46.900 INFO 2981023 --- [ForkJoinPool-6-worker-11,84ee5c5089eeb924,2ae83b71ad86ffa4] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:31:46.901 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.902 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.902 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:46.903 INFO 2981023 --- [pool-10-thread-3,84ee5c5089eeb924,c67d771161059fec] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:31:46.903 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.905 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:31:46.905 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:31:46.906 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:31:46.913 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:46.923 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:31:46.925 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:31:46.932 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.933 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.934 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.934 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:46.935 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:31:46.939 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:31:46.940 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:31:46.943 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:31:46.944 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:31:46.945 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.946 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:31:46.947 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:31:46.947 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.948 INFO 2981023 --- [ForkJoinPool-6-worker-6,84ee5c5089eeb924,04bc6d4b5e13377d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:31:46.949 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:46.949 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:31:46.952 INFO 2981023 --- [pool-10-thread-1,84ee5c5089eeb924,a62821e844036a52] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:31:46.953 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:31:46.956 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.958 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.959 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:31:46.959 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:31:46.960 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.961 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:31:46.962 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:31:46.962 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.962 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:31:46.966 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:46.968 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:31:46.968 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:31:46.971 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:46.974 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:31:46.977 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:31:46.982 INFO 2981023 --- [host-check-clockdiff1,c497105a844d4e29,2df0fca6b6f3f25e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047106 0 0 , error= 2024-09-11 17:31:46.985 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:31:46.986 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:31:46.996 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:31:46.999 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:31:47.004 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:47.012 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:31:47.025 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:47.025 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:31:47.025 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.026 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:31:47.031 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:47.032 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.033 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:47.033 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:31:47.038 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:31:47.038 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:31:47.047 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:31:47.051 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:31:47.057 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:31:47.058 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:31:47.059 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:31:47.066 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:31:47.068 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:47.071 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.072 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:31:47.073 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:31:47.074 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:47.074 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.074 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.074 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:31:47.076 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:31:47.076 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.079 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:31:47.080 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:31:47.082 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:31:47.087 INFO 2981023 --- [host-check-host-status3,c760f8b56d1adc0c,390c427ad1d2ec56] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:31:47.089 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:31:47.097 INFO 2981023 --- [host-check-host-status2,c760f8b56d1adc0c,850465ecd02671f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:31:47.099 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:31:47.100 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:31:47.101 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:31:47.101 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:47.101 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:31:47.111 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:31:47.116 INFO 2981023 --- [host-check-host-status4,c760f8b56d1adc0c,9e5f351af4a31cf3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:31:47.120 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:31:47.122 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:47.122 INFO 2981023 --- [host-check-host-status1,c760f8b56d1adc0c,e7b4e408aa564a23] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:31:47.123 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:31:47.125 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.127 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:31:47.132 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:47.132 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.133 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:31:47.134 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:31:47.139 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:31:47.142 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:47.143 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:31:47.143 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:47.144 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:31:46.471513+08:00 2024-09-11 17:31:47.145 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:31:47.145 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:31:47.145 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:31:47.151 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:47.152 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.152 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:31:47.153 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:31:47.153 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.162 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:31:47.164 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:31:47.173 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:47.173 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.174 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:31:47.174 INFO 2981023 --- [ocp-async-8,651b292aee64fdbb,dc35a774468ec1ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:47.179 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:31:47.179 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:31:47.183 INFO 2981023 --- [pool-10-thread-4,84ee5c5089eeb924,b14b0cb3ffd2e6bc] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:31:47.188 INFO 2981023 --- [pool-10-thread-4,84ee5c5089eeb924,b14b0cb3ffd2e6bc] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:31:47.190 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:31:47.196 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:47.197 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:47.197 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:31:47.203 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:31:47.203 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:31:47.203 INFO 2981023 --- [ocp-schedules-18,60d4559c2c7b60d0,eb55f5f24724c113] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:31:47.205 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:31:47.220 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:31:47.226 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:31:47.228 INFO 2981023 --- [ocp-schedules-17,c8236c9ced3043af,e1e822154d669239] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:31:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,2c8193a0e59afc7b,d03fa8b3e620b138] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,2c8193a0e59afc7b,d03fa8b3e620b138] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:47.750 INFO 2981023 --- [ocp-async-7,b8a9dcd6da7558d7,f9fd560c8c66e6d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:31:49.175 INFO 2981023 --- [ocp-async-4,57095399f7758dac,77696f3633baf077] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:31:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:31:50.143 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:50.144 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:50.216 INFO 2981023 --- [ocp-schedules-28,2feda4c7555e306e,4054f191323be098] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:31:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,88e6ef653c63d017,3a7c37782b6c056b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,88e6ef653c63d017,3a7c37782b6c056b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:51.175 INFO 2981023 --- [ocp-async-2,9a0cc9018882dac3,7556bfe8b77bb082] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:51.531 INFO 2981023 --- [ocp-schedules-2,65f9abc7f1b861ae,eedfc1fffb7e44f3] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:31:51.532 INFO 2981023 --- [ocp-schedules-2,65f9abc7f1b861ae,eedfc1fffb7e44f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:31:51.532 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:31:51.533 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:31:51.533 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:31:51.533 INFO 2981023 --- [ocp-schedules-2,65f9abc7f1b861ae,eedfc1fffb7e44f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:31:51.813 INFO 2981023 --- [ocp-schedules-21,5c2028d659e1d7dd,bd7527ecd272173c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:31:51.813+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:31:51.816 INFO 2981023 --- [ocp-schedules-21,5c2028d659e1d7dd,bd7527ecd272173c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:31:51.875 INFO 2981023 --- [ocp-schedules-24,f3bda4c84c7d2c72,30016c52ca024375] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:31:51.954 INFO 2981023 --- [ocp-schedules-21,5c2028d659e1d7dd,bd7527ecd272173c] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 60 to SqlAggregateData 2024-09-11 17:31:51.956 INFO 2981023 --- [ocp-schedules-21,5c2028d659e1d7dd,bd7527ecd272173c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:31:51.978 INFO 2981023 --- [host-check-clockdiff3,c497105a844d4e29,50bbb1b7dfa66acf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047111 1 1 , error= 2024-09-11 17:31:52.470 INFO 2981023 --- [ocp-schedules-8,8eedf0eee0450f80,2da4ad6c1e1bfb57] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:31:52.471 INFO 2981023 --- [ocp-schedules-8,8eedf0eee0450f80,2da4ad6c1e1bfb57] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:31:52.501 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:31:52.501 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.505 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:31:52.509 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.510 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.512 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:31:52.512 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:31:52.512 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.512 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.519 INFO 2981023 --- [ocp-schedules-3,e5511f807d192453,0ce22b40668c5933] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:31:52.523 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:31:52.524 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.524 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.526 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:31:52.528 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.528 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.542 WARN 2981023 --- [ocp-schedules-3,e5511f807d192453,0ce22b40668c5933] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:31:52.546 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.546 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.548 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.548 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.551 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.551 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.554 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.554 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.566 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.566 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.567 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:31:52.567 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2401D6DB-0-0] 2024-09-11 17:31:52.567 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:31:52.567 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.568 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:31:52.571 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.572 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.573 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:31:52.574 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:31:52.574 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.575 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:31:52.577 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.578 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:31:52.578 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:31:52.581 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.581 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:31:52.581 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:31:52.582 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:31:52.588 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.588 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:31:52.594 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.594 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:31:52.596 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.596 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:31:52.597 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:31:52.597 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2401D6E5-0-0] 2024-09-11 17:31:52.597 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:31:52.604 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.605 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:31:52.605 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.606 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:31:52.607 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.607 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:52.607 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.607 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:31:52.608 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:31:52.608 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.608 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.609 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:31:52.609 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:31:52.609 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.609 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.609 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.610 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.610 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:31:52.610 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:31:52.611 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.611 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.611 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:31:52.611 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:31:52.612 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:31:52.612 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.613 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:31:52.613 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.614 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:31:52.614 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:31:52.614 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.615 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:31:52.616 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.616 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:31:52.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:31:52.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:31:52.621 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:31:52.625 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:31:52.628 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:31:52.628 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:31:52.628 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:31:52.628 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:52.630 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.630 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:31:52.635 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:31:52.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:31:52.642 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:31:52.645 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:31:52.646 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:31:52.646 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.647 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.647 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:31:52.647 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:31:52.655 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:31:52.656 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:31:52.657 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:52.657 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:31:52.658 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:31:52.659 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:31:52.659 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:31:52.659 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:31:52.664 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:31:52.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:31:52.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:31:52.665 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:31:52.665 INFO 2981023 --- [ocp-schedules-8,8eedf0eee0450f80,2da4ad6c1e1bfb57] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:31:53.062 INFO 2981023 --- [ocp-schedules-23,0a3271773297f1b9,2495928bd106f795] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:31:53.062 INFO 2981023 --- [ocp-schedules-23,0a3271773297f1b9,2495928bd106f795] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:31:53.144 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:53.145 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:53.175 INFO 2981023 --- [ocp-async-8,72b3f5762ab34784,443d37d836d56e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,d8df3506d34c5bb9,0e5783f6107479b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,d8df3506d34c5bb9,0e5783f6107479b4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:53.747 INFO 2981023 --- [ocp-schedules-9,392758ed26125293,c356b361e6bc58b4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:31:53.756 INFO 2981023 --- [ocp-schedules-9,392758ed26125293,c356b361e6bc58b4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:31:53.756 INFO 2981023 --- [ocp-schedules-9,392758ed26125293,c356b361e6bc58b4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:31:54.201 INFO 2981023 --- [ocp-schedules-26,3b4d79bc142c330d,e10bd436f3516e3e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:31:55.175 INFO 2981023 --- [ocp-async-3,54063374f4f4f1f2,84b18313af3af339] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:55.643 INFO 2981023 --- [ocp-schedules-13,417b3c9775014dfd,173691ceb206c72e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:31:55.656 INFO 2981023 --- [ocp-schedules-29,f085c77fac4f0b6e,3aa378a9e267534d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:31:55.656 INFO 2981023 --- [ocp-schedules-29,f085c77fac4f0b6e,3aa378a9e267534d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:31:55.658 INFO 2981023 --- [ocp-schedules-29,f085c77fac4f0b6e,3aa378a9e267534d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:31:55.658 INFO 2981023 --- [ocp-schedules-29,f085c77fac4f0b6e,3aa378a9e267534d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:31:55.659 INFO 2981023 --- [ocp-schedules-29,f085c77fac4f0b6e,3aa378a9e267534d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:31:55.659 INFO 2981023 --- [ocp-schedules-29,f085c77fac4f0b6e,3aa378a9e267534d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:31:55.688 WARN 2981023 --- [scanner-13499332-1,417b3c9775014dfd,f6fc7ce04ca378f8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:31:55.718 INFO 2981023 --- [alarm-detect-14,417b3c9775014dfd,68a3d5a58f159670] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8643987, data is empty. 2024-09-11 17:31:55.726 INFO 2981023 --- [alarm-detect-14,417b3c9775014dfd,68a3d5a58f159670] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644047, data is empty. 2024-09-11 17:31:55.730 INFO 2981023 --- [ocp-schedules-26,c5fb40c6eb3f7b62,ea3e4c1d1ba96ce4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:31:55.731 INFO 2981023 --- [alarm-detect-14,417b3c9775014dfd,68a3d5a58f159670] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644047, data is empty. 2024-09-11 17:31:55.743 INFO 2981023 --- [ocp-schedules-26,c5fb40c6eb3f7b62,ea3e4c1d1ba96ce4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:31:55.743 INFO 2981023 --- [ocp-schedules-26,c5fb40c6eb3f7b62,ea3e4c1d1ba96ce4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:31:55.746 INFO 2981023 --- [alarm-detect-14,417b3c9775014dfd,170b5498f8993577] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8643987, data is empty. 2024-09-11 17:31:55.747 INFO 2981023 --- [alarm-detect-11,417b3c9775014dfd,9a6fe496bf6b211e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8643987, data is empty. 2024-09-11 17:31:55.752 INFO 2981023 --- [alarm-detect-14,417b3c9775014dfd,170b5498f8993577] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644047, data is empty. 2024-09-11 17:31:55.756 INFO 2981023 --- [alarm-detect-14,417b3c9775014dfd,170b5498f8993577] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644047, data is empty. 2024-09-11 17:31:55.757 INFO 2981023 --- [alarm-detect-11,417b3c9775014dfd,9a6fe496bf6b211e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644047, data is empty. 2024-09-11 17:31:55.764 INFO 2981023 --- [alarm-detect-11,417b3c9775014dfd,9a6fe496bf6b211e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644047, data is empty. 2024-09-11 17:31:55.868 INFO 2981023 --- [alarm-detect-11,417b3c9775014dfd,985bc30b25c8e25b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8643987, data is empty. 2024-09-11 17:31:55.875 INFO 2981023 --- [alarm-detect-11,417b3c9775014dfd,985bc30b25c8e25b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644047, data is empty. 2024-09-11 17:31:55.878 INFO 2981023 --- [alarm-detect-11,417b3c9775014dfd,985bc30b25c8e25b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644047, data is empty. 2024-09-11 17:31:56.010 INFO 2981023 --- [ocp-schedules-4,effd9dbe51e9579c,de456c25c41ce5f5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:31:56.010 INFO 2981023 --- [ocp-schedules-4,effd9dbe51e9579c,de456c25c41ce5f5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:31:56.023 INFO 2981023 --- [Collect all tenants major compactions1,effd9dbe51e9579c,5d3de213a4ead1c2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:31:56.023 INFO 2981023 --- [Collect all tenants major compactions2,effd9dbe51e9579c,7a113a8c9f651ae4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:31:56.024 INFO 2981023 --- [Collect all tenants major compactions3,effd9dbe51e9579c,2f5baba8a4742258] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:31:56.026 INFO 2981023 --- [Collect all tenants major compactions5,effd9dbe51e9579c,84d8d4e100641e4c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:31:56.026 INFO 2981023 --- [Collect all tenants major compactions4,effd9dbe51e9579c,c25fcde4328743a8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:31:56.027 INFO 2981023 --- [Collect all tenants major compactions6,effd9dbe51e9579c,58a992d748ed5a17] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:31:56.038 INFO 2981023 --- [Collect all tenants major compactions2,effd9dbe51e9579c,7a113a8c9f651ae4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:56.038 INFO 2981023 --- [Collect all tenants major compactions6,effd9dbe51e9579c,58a992d748ed5a17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:56.038 INFO 2981023 --- [Collect all tenants major compactions3,effd9dbe51e9579c,2f5baba8a4742258] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:56.038 INFO 2981023 --- [Collect all tenants major compactions1,effd9dbe51e9579c,5d3de213a4ead1c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:56.038 INFO 2981023 --- [Collect all tenants major compactions5,effd9dbe51e9579c,84d8d4e100641e4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:56.039 INFO 2981023 --- [Collect all tenants major compactions2,effd9dbe51e9579c,7a113a8c9f651ae4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:31:56.039 INFO 2981023 --- [Collect all tenants major compactions4,effd9dbe51e9579c,c25fcde4328743a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:31:56.039 INFO 2981023 --- [Collect all tenants major compactions3,effd9dbe51e9579c,2f5baba8a4742258] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:31:56.039 INFO 2981023 --- [Collect all tenants major compactions5,effd9dbe51e9579c,84d8d4e100641e4c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:31:56.040 INFO 2981023 --- [Collect all tenants major compactions6,effd9dbe51e9579c,58a992d748ed5a17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:31:56.040 INFO 2981023 --- [Collect all tenants major compactions1,effd9dbe51e9579c,5d3de213a4ead1c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:31:56.040 INFO 2981023 --- [Collect all tenants major compactions4,effd9dbe51e9579c,c25fcde4328743a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:31:56.043 INFO 2981023 --- [Collect all tenants major compactions2,effd9dbe51e9579c,7a113a8c9f651ae4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:31:56.043 INFO 2981023 --- [Collect all tenants major compactions5,effd9dbe51e9579c,84d8d4e100641e4c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:31:56.043 INFO 2981023 --- [Collect all tenants major compactions2,effd9dbe51e9579c,7a113a8c9f651ae4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:31:56.043 INFO 2981023 --- [Collect all tenants major compactions5,effd9dbe51e9579c,84d8d4e100641e4c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:31:56.043 INFO 2981023 --- [Collect all tenants major compactions3,effd9dbe51e9579c,2f5baba8a4742258] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:31:56.043 INFO 2981023 --- [Collect all tenants major compactions3,effd9dbe51e9579c,2f5baba8a4742258] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:31:56.043 INFO 2981023 --- [Collect all tenants major compactions6,effd9dbe51e9579c,58a992d748ed5a17] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:31:56.043 INFO 2981023 --- [Collect all tenants major compactions6,effd9dbe51e9579c,58a992d748ed5a17] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:31:56.044 INFO 2981023 --- [Collect all tenants major compactions1,effd9dbe51e9579c,5d3de213a4ead1c2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:31:56.044 INFO 2981023 --- [Collect all tenants major compactions1,effd9dbe51e9579c,5d3de213a4ead1c2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:31:56.044 INFO 2981023 --- [Collect all tenants major compactions4,effd9dbe51e9579c,c25fcde4328743a8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:31:56.044 INFO 2981023 --- [Collect all tenants major compactions4,effd9dbe51e9579c,c25fcde4328743a8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:31:56.045 INFO 2981023 --- [ocp-schedules-4,effd9dbe51e9579c,de456c25c41ce5f5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:31:56.145 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:56.146 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,071a988638130f33,b07bea9e6dae0b13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,071a988638130f33,b07bea9e6dae0b13] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:31:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:31:56.350733+08:00, lastProcessTime=2024-09-11T17:31:46.471513+08:00 2024-09-11 17:31:56.480 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:31:56.350733+08:00 2024-09-11 17:31:57.175 INFO 2981023 --- [ocp-async-2,372d8154ebf6a2ce,645d01af0c431a37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:57.750 INFO 2981023 --- [ocp-async-4,9d14f97e645ccbf6,9a8b34038e7b759d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:31:59.146 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:31:59.146 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:31:59.175 INFO 2981023 --- [ocp-async-8,e51dedeec970648c,29675bd20738b4b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:31:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,961cd32ed49e6f4d,ba2e967b22843ef2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047120, interval=60 2024-09-11 17:31:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,961cd32ed49e6f4d,ba2e967b22843ef2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:01.177 INFO 2981023 --- [ocp-async-10,66abb3f228734712,26ab6be4bcfdf6c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:01.878 INFO 2981023 --- [ocp-schedules-16,76834773db126b96,17d74150fe645ad3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:32:02.146 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:02.147 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,195ab2728b7b7600,eac79201dc766c9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,195ab2728b7b7600,eac79201dc766c9b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:02.429 INFO 2981023 --- [ocp-schedules-8,8c820dec0e5f8580,ba9e84af8b60ba77] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:32:02.430 INFO 2981023 --- [ocp-schedules-8,8c820dec0e5f8580,ba9e84af8b60ba77] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:02.438 INFO 2981023 --- [ocp-schedules-8,8c820dec0e5f8580,ba9e84af8b60ba77] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:32:03.175 INFO 2981023 --- [ocp-async-5,8340d806860f6d84,d49f2f963e41455a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:05.147 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:05.148 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:05.175 INFO 2981023 --- [ocp-async-1,d1f874f14f3bf2f0,6e96223f6d00073c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,fd5e9da72a64e38a,6844af2fb1bbd44a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,fd5e9da72a64e38a,6844af2fb1bbd44a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:05.335 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,62aa3039479019f6,44014091c8b5b212] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=62aa3039479019f6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:32:05.336 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,62aa3039479019f6,44014091c8b5b212] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=62aa3039479019f6, duration=5 ms] 2024-09-11 17:32:05.344 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,15b1d7237942953e,967627b384b283b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=15b1d7237942953e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:32:05.346 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,15b1d7237942953e,967627b384b283b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=15b1d7237942953e, duration=4 ms] 2024-09-11 17:32:05.348 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,8138446996748a8a,e0e78b885a2284d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=8138446996748a8a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:32:05.350 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,8138446996748a8a,e0e78b885a2284d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=8138446996748a8a, duration=4 ms] 2024-09-11 17:32:05.912 INFO 2981023 --- [ocp-schedules-20,007f8e9416ba7d78,f3be2852f8dd01c6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:32:05.956 WARN 2981023 --- [scanner-13499650-1,007f8e9416ba7d78,52fae85ca12c1446] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:32:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:32:05.972 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:05.972 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:32:05.973 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:32:06.048 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:32:06.063 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:32:06.069 INFO 2981023 --- [alarm-detect-4,007f8e9416ba7d78,19c1329bbf4f0795] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047090 ~ 1726047116 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726047105, value=2.16)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.16, values={1726047105=2.16}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:32:06.069 INFO 2981023 --- [alarm-detect-4,007f8e9416ba7d78,19c1329bbf4f0795] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.10-3, currentState=Pending 2024-09-11 17:32:06.073 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:32:06.091 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:32:06.091 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:32:06.091 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:32:06.097 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T17:29:41, target=2024-09-11T17:27:41, granule=120000ms, delay=120000ms) 2024-09-11 17:32:06.097 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:27:41, target=2024-09-11T17:20:51, granule=600000ms, delay=410000ms) 2024-09-11 17:32:06.100 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:06.101 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:32:06.103 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:06.104 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:32:06.124 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:29:40, start=2024-09-11T17:25:40, end=2024-09-11T17:28:10, updates=130, granule=120000ms, delay=240000ms 2024-09-11 17:32:06.130 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:29:40, target=2024-09-11T17:27:41, granule=120000ms, delay=119000ms) 2024-09-11 17:32:06.130 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:27:41, target=2024-09-11T17:19:41, granule=600000ms, delay=480000ms) 2024-09-11 17:32:06.130 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=157ms, period=120000ms 2024-09-11 17:32:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:32:06.336532+08:00, lastProcessTime=2024-09-11T17:31:56.350733+08:00 2024-09-11 17:32:06.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:32:06.336532+08:00 2024-09-11 17:32:07.176 INFO 2981023 --- [ocp-async-4,c41d56d78a7d7898,e9b251d99e7997a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:07.284 INFO 2981023 --- [ocp-schedules-8,d4f3fd53f8eef5b3,2f3f4be21c859e78] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:32:07.284 INFO 2981023 --- [ocp-schedules-17,cf5153b03ee7d0b3,8ae03735dd3a66e6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:32:07.284+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:32:07.285 INFO 2981023 --- [ocp-schedules-17,cf5153b03ee7d0b3,8ae03735dd3a66e6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:32:07.286 INFO 2981023 --- [ocp-schedules-17,cf5153b03ee7d0b3,8ae03735dd3a66e6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:32:07.287 INFO 2981023 --- [ocp-schedules-17,cf5153b03ee7d0b3,8ae03735dd3a66e6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:32:07.750 INFO 2981023 --- [ocp-async-1,22fcceb42197cc80,ced687463063f541] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:32:08.148 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:08.148 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,6c657857e4c7f255,26df573cee032d41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,6c657857e4c7f255,26df573cee032d41] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:09.176 INFO 2981023 --- [ocp-async-2,2e6dfd47bc60d0b1,1078614d37de87e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:09.319 INFO 2981023 --- [ocp-schedules-4,d777cfb2ca5ccba4,b15370f8e7c6510b] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-09-11 17:32:09.319 INFO 2981023 --- [ocp-schedules-4,d777cfb2ca5ccba4,b15370f8e7c6510b] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-09-11 17:32:09.319 INFO 2981023 --- [ocp-schedules-4,d777cfb2ca5ccba4,b15370f8e7c6510b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=11634 2024-09-11 17:32:09.537 INFO 2981023 --- [ocp-schedules-4,d777cfb2ca5ccba4,b15370f8e7c6510b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=11634 2024-09-11 17:32:10.003 INFO 2981023 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:32:10.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:32:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:32:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:32:11.148 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:11.149 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:11.176 INFO 2981023 --- [ocp-async-9,2791c9cee30b0b56,604f85ed072a1490] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,2bb4a430702943f3,2c512557034ad40f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,2bb4a430702943f3,2c512557034ad40f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:11.881 INFO 2981023 --- [ocp-schedules-20,8bec28aeb801377a,7bf7caa05a848ab5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:32:12.561 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:32:12.571 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:12.572 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:12.583 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:12.584 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:12.596 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:32:12.596 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:32:12.608 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:12.608 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:12.620 INFO 2981023 --- [ocp-schedules-14,4cb3352b33f7f208,b9b53539b1f31df3] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=5 2024-09-11 17:32:13.176 INFO 2981023 --- [ocp-async-8,5d3defe2827278db,0a36309be2591c11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:13.245 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047060, nodeCount=480, affectRows=480 2024-09-11 17:32:13.757 INFO 2981023 --- [ocp-schedules-4,560752c07d4af7bd,77a9f9acee306ff8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:32:13.763 INFO 2981023 --- [ocp-schedules-4,560752c07d4af7bd,77a9f9acee306ff8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:32:13.763 INFO 2981023 --- [ocp-schedules-4,560752c07d4af7bd,77a9f9acee306ff8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:32:13.864 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,93749b445a51e2d4,eee697c332dcd42c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=93749b445a51e2d4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:32:13.866 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,93749b445a51e2d4,eee697c332dcd42c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=93749b445a51e2d4, duration=4 ms] 2024-09-11 17:32:13.875 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f9a72e2ecd9a2497,853d6827c8deb9df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=f9a72e2ecd9a2497, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:32:13.876 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f9a72e2ecd9a2497,853d6827c8deb9df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=f9a72e2ecd9a2497, duration=3 ms] 2024-09-11 17:32:13.878 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,d046bb3c4a191dff,525045dee4e40c7e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=d046bb3c4a191dff, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:32:13.881 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,d046bb3c4a191dff,525045dee4e40c7e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=d046bb3c4a191dff, duration=6 ms] 2024-09-11 17:32:14.149 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:14.150 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:14.310 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047060, nodeCount=48, affectRows=48 2024-09-11 17:32:14.311 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047060, nodeCount=48, validCount=48 2024-09-11 17:32:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,90e0ad8dd354ba01,fe9c12bede24900c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:32:14.320 INFO 2981023 --- [prometheus-minute-schedule-1,768e84bde7b6ab27,a53cb4ce8c18eefc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:14.323 INFO 2981023 --- [prometheus-minute-schedule-1,768e84bde7b6ab27,a53cb4ce8c18eefc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:14.422 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047060, nodeCount=1000, affectRows=1000 2024-09-11 17:32:14.423 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047060, nodeCount=1000, affectRows=1000 2024-09-11 17:32:14.425 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047060, nodeCount=999, validCount=999 2024-09-11 17:32:14.425 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047060, nodeCount=1000, validCount=1000 2024-09-11 17:32:15.115 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047060, nodeCount=480, validCount=480 2024-09-11 17:32:15.176 INFO 2981023 --- [ocp-async-6,3ba4ff750a93be9f,1d1206e96051bfcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:16.018 INFO 2981023 --- [ocp-schedules-16,921bfa9c5361a741,1f062618ad631099] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:32:16.018 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:32:16.019 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:32:16.019 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:32:16.019 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:32:16.023 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:16.023 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:16.024 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:16.024 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:16.026 INFO 2981023 --- [ocp-schedules-20,c949c451fd98f0ae,8004dc5ea08f2c92] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:32:16.032 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:16.032 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:32:16.032 INFO 2981023 --- [host-check-clockdiff1,c949c451fd98f0ae,fecf5d511c7c0082] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:32:16.033 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:32:16.034 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:16.034 INFO 2981023 --- [host-check-clockdiff2,c949c451fd98f0ae,55582bebcd958c2f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:32:16.036 INFO 2981023 --- [ocp-schedules-3,a371aec4d45bfe9c,e18e73e5c8172675] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:32:16.036 INFO 2981023 --- [host-check-agent-status1,a371aec4d45bfe9c,37984d0d7d9aa0ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:32:16.038 INFO 2981023 --- [host-check-agent-status2,a371aec4d45bfe9c,4e2ff7c73d5fc04a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:32:16.039 INFO 2981023 --- [host-check-clockdiff4,c949c451fd98f0ae,fef4acf978809da4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:32:16.040 INFO 2981023 --- [host-check-clockdiff3,c949c451fd98f0ae,3615f481bfee3e04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:32:16.040 INFO 2981023 --- [host-check-agent-status4,a371aec4d45bfe9c,9a8c106aa20497b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:32:16.041 INFO 2981023 --- [host-check-agent-status3,a371aec4d45bfe9c,0aaa245f3e602885] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:32:16.049 INFO 2981023 --- [host-check-agent-status2,a371aec4d45bfe9c,4e2ff7c73d5fc04a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:16.051 INFO 2981023 --- [host-check-agent-status2,a371aec4d45bfe9c,4e2ff7c73d5fc04a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:16.053 INFO 2981023 --- [host-check-agent-status1,a371aec4d45bfe9c,37984d0d7d9aa0ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:16.055 INFO 2981023 --- [host-check-agent-status1,a371aec4d45bfe9c,37984d0d7d9aa0ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:16.056 INFO 2981023 --- [host-check-agent-status4,a371aec4d45bfe9c,9a8c106aa20497b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:16.058 INFO 2981023 --- [host-check-clockdiff4,c949c451fd98f0ae,fef4acf978809da4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047136 0 0 , error= 2024-09-11 17:32:16.059 INFO 2981023 --- [host-check-agent-status3,a371aec4d45bfe9c,0aaa245f3e602885] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:32:16.060 INFO 2981023 --- [host-check-clockdiff2,c949c451fd98f0ae,55582bebcd958c2f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047136 0 0 , error= 2024-09-11 17:32:16.060 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:32:16.060 INFO 2981023 --- [host-check-agent-status4,a371aec4d45bfe9c,9a8c106aa20497b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:16.061 INFO 2981023 --- [host-check-agent-status3,a371aec4d45bfe9c,0aaa245f3e602885] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:32:16.062 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:32:16.063 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:32:16.062370986+08:00 2024-09-11 17:32:16.064 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:32:16.064527523+08:00 2024-09-11 17:32:16.064 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:32:16.063+08:00 2024-09-11 17:32:16.066 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:32:16.064+08:00 2024-09-11 17:32:16.068 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:32:16.063+08:00 and status AVAILABLE 2024-09-11 17:32:16.068 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:32:16.064+08:00 and status AVAILABLE 2024-09-11 17:32:16.071 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:32:16.074 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:32:16.073120918+08:00 2024-09-11 17:32:16.075 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:32:16.074+08:00 2024-09-11 17:32:16.076 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:32:16.077 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:32:16.074+08:00 and status AVAILABLE 2024-09-11 17:32:16.079 INFO 2981023 --- [host-check-agent-status2,a371aec4d45bfe9c,4e2ff7c73d5fc04a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:32:16.081 INFO 2981023 --- [host-check-agent-status2,a371aec4d45bfe9c,4e2ff7c73d5fc04a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:32:16.081 INFO 2981023 --- [host-check-agent-status2,a371aec4d45bfe9c,4e2ff7c73d5fc04a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:32:16.081 INFO 2981023 --- [host-check-agent-status2,a371aec4d45bfe9c,4e2ff7c73d5fc04a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:32:16.088 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:16.089 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:16.090 INFO 2981023 --- [host-check-agent-status1,a371aec4d45bfe9c,37984d0d7d9aa0ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:32:16.093 INFO 2981023 --- [host-check-agent-status1,a371aec4d45bfe9c,37984d0d7d9aa0ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:32:16.093 INFO 2981023 --- [host-check-agent-status1,a371aec4d45bfe9c,37984d0d7d9aa0ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:32:16.093 INFO 2981023 --- [host-check-agent-status1,a371aec4d45bfe9c,37984d0d7d9aa0ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:32:16.094 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:32:16.094 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:32:16.098 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:32:16.098344808+08:00 2024-09-11 17:32:16.099 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:32:16.099 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:32:16.098+08:00 2024-09-11 17:32:16.102 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:32:16.098+08:00 and status AVAILABLE 2024-09-11 17:32:16.107 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:16.108 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:32:16.109 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:16.117 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:32:16.118 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:32:16.119 INFO 2981023 --- [host-check-agent-status4,a371aec4d45bfe9c,9a8c106aa20497b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:32:16.122 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:32:16.122 INFO 2981023 --- [host-check-agent-status4,a371aec4d45bfe9c,9a8c106aa20497b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:32:16.122 INFO 2981023 --- [host-check-agent-status4,a371aec4d45bfe9c,9a8c106aa20497b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:32:16.122 INFO 2981023 --- [host-check-agent-status4,a371aec4d45bfe9c,9a8c106aa20497b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:32:16.138 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:16.139 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:16.143 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:32:16.144 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:32:16.155 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:32:16.158 INFO 2981023 --- [ocp-schedules-11,02dadbbe855c36fb,907c503c731564fb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:32:16.160 INFO 2981023 --- [host-check-agent-status3,a371aec4d45bfe9c,0aaa245f3e602885] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:32:16.165 INFO 2981023 --- [host-check-agent-status3,a371aec4d45bfe9c,0aaa245f3e602885] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:32:16.165 INFO 2981023 --- [host-check-agent-status3,a371aec4d45bfe9c,0aaa245f3e602885] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:32:16.165 INFO 2981023 --- [host-check-agent-status3,a371aec4d45bfe9c,0aaa245f3e602885] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:32:16.165 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:32:16.168 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:32:16.183 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:32:16.184 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:32:16.185 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:32:16.191 INFO 2981023 --- [ocp-schedules-10,916714b9a2a69afe,6b91258ae2ed61e3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:32:16.191 INFO 2981023 --- [ocp-schedules-5,e467b3664b3f49b7,921a654192767f31] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:32:16.191 INFO 2981023 --- [ocp-schedules-1,d0804c3e4d0290ce,7c888dd9ed52643b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:32:16.192 INFO 2981023 --- [ocp-schedules-6,c2db964f83ec7076,95b0c2005f56bd04] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:32:16.194 INFO 2981023 --- [ocp-schedules-1,d0804c3e4d0290ce,7c888dd9ed52643b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:16.195 INFO 2981023 --- [ocp-schedules-10,916714b9a2a69afe,6b91258ae2ed61e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:16.198 INFO 2981023 --- [ocp-schedules-5,e467b3664b3f49b7,921a654192767f31] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:16.204 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:32:16.229 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:32:16.233 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:32:16.235 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:32:16.238 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:32:16.239 INFO 2981023 --- [ocp-schedules-8,f1a982c1bd3fcc31,165bac1b324da5bc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:32:16.246 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:32:16.255 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:32:16.255 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:32:16.264 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:32:16.265 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:32:16.265 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:32:16.271 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:32:16.280 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:32:16.294 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:32:16.301 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:32:16.305 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:32:16.313 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:32:16.314 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:32:16.319 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:32:16.326 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:32:16.330 INFO 2981023 --- [host-check-host-status1,921bfa9c5361a741,36a64383d0f51109] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:32:16.334 INFO 2981023 --- [host-check-host-status3,921bfa9c5361a741,5c248066fcfe5c69] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:32:16.335 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:32:16.346 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:32:16.347 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:32:16.347250+08:00, lastProcessTime=2024-09-11T17:32:06.336532+08:00 2024-09-11 17:32:16.366 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:32:16.382 INFO 2981023 --- [host-check-host-status4,921bfa9c5361a741,a9771ff964b75d08] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:32:16.384 INFO 2981023 --- [host-check-host-status2,921bfa9c5361a741,4724e4432d4b3c2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:32:16.413 INFO 2981023 --- [ocp-schedules-5,e467b3664b3f49b7,921a654192767f31] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:32:16.419 WARN 2981023 --- [scanner-13501374-1,02dadbbe855c36fb,d0b19bb80f2fa33c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:32:16.441 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.441 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:16.450 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:32:16.455 INFO 2981023 --- [alarm-detect-2,02dadbbe855c36fb,50fcc4a9ad13445e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644047, data is empty. 2024-09-11 17:32:16.464 INFO 2981023 --- [alarm-detect-2,02dadbbe855c36fb,50fcc4a9ad13445e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644047, data is empty. 2024-09-11 17:32:16.471 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.471 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:16.472 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:32:16.473 INFO 2981023 --- [alarm-detect-2,02dadbbe855c36fb,50fcc4a9ad13445e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644047, data is empty. 2024-09-11 17:32:16.479 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.480 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:16.481 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:32:16.482 INFO 2981023 --- [ocp-schedules-10,916714b9a2a69afe,6b91258ae2ed61e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:32:16.483 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.483 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.483 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:16.484 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:16.490 INFO 2981023 --- [alarm-detect-2,02dadbbe855c36fb,04d310055cbf2473] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644047, data is empty. 2024-09-11 17:32:16.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:16.500 INFO 2981023 --- [alarm-detect-19,02dadbbe855c36fb,0edd2f7ebaddb282] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644047, data is empty. 2024-09-11 17:32:16.504 INFO 2981023 --- [alarm-detect-19,02dadbbe855c36fb,0edd2f7ebaddb282] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644047, data is empty. 2024-09-11 17:32:16.506 INFO 2981023 --- [alarm-detect-19,02dadbbe855c36fb,0edd2f7ebaddb282] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644047, data is empty. 2024-09-11 17:32:16.508 INFO 2981023 --- [alarm-detect-2,02dadbbe855c36fb,04d310055cbf2473] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644047, data is empty. 2024-09-11 17:32:16.512 INFO 2981023 --- [alarm-detect-2,02dadbbe855c36fb,04d310055cbf2473] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644047, data is empty. 2024-09-11 17:32:16.563 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.566 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:32:16.571 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.573 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.573 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:32:16.589 INFO 2981023 --- [alarm-detect-10,02dadbbe855c36fb,3773165510afe85f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047105 ~ 1726047126 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726047120, value=1.68)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.68, values={1726047120=1.68}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:32:16.614 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.617 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.619 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:16.620 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:32:16.621 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.625 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:32:16.626 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.626 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:32:16.627 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.627 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:16.629 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.631 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:32:16.632 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:32:16.653 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:16.653 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:32:16.658 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:16.658 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:32:16.682 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.697 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:16.699 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.699 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.699 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:32:16.700 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.700 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:16.701 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:32:16.703 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:32:16.347250+08:00 2024-09-11 17:32:16.712 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:32:16.713 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.715 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.715 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:16.716 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:32:16.722 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:16.725 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:32:16.726 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:16.728 INFO 2981023 --- [alarm-detect-7,02dadbbe855c36fb,74efd84ed376b941] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644047, data is empty. 2024-09-11 17:32:16.736 INFO 2981023 --- [alarm-detect-7,02dadbbe855c36fb,74efd84ed376b941] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644047, data is empty. 2024-09-11 17:32:16.739 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:16.741 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:16.743 INFO 2981023 --- [alarm-detect-7,02dadbbe855c36fb,74efd84ed376b941] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644047, data is empty. 2024-09-11 17:32:16.754 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.756 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.760 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:32:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:16.761 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:32:16.761 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.766 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:32:16.767 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:32:16.770 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:32:16.782 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:16.782 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:32:16.799 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:32:16.815 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:32:16.819 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:32:16.822 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:16.823 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:16.830 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.830 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.831 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:32:16.838 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:32:16.840 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:32:16.840 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:32:16.842 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:32:16.844 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.844 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.845 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:32:16.850 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:32:16.850 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:32:16.872 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:32:16.878 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:16.878 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:16.879 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:32:16.884 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:32:16.884 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:32:16.884 INFO 2981023 --- [ocp-schedules-1,d0804c3e4d0290ce,7c888dd9ed52643b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:32:17.150 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:17.151 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:17.176 INFO 2981023 --- [ocp-async-7,128aac2632ac862c,5ff3b1fb1df399aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,b0c20a1747c8ba7a,c128046c69164a5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,b0c20a1747c8ba7a,c128046c69164a5c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:17.750 INFO 2981023 --- [ocp-async-6,aa558cb17c60245d,0c75230a3bd16c49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:32:18.561 INFO 2981023 --- [host-check-clockdiff1,c949c451fd98f0ae,fecf5d511c7c0082] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047138 0 0 , error= 2024-09-11 17:32:19.176 INFO 2981023 --- [ocp-async-8,5725798c750472b3,8c314bca772fe5a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:20.151 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:20.153 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,7f50ee11090b94df,26038d8b92b679ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,7f50ee11090b94df,26038d8b92b679ca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:21.176 INFO 2981023 --- [ocp-async-6,27f9fc126365689d,fc71f62dcd2d0b89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:21.887 INFO 2981023 --- [ocp-schedules-14,0ed0e7d728e916e3,56789608d9dbe3be] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:32:22.025 INFO 2981023 --- [host-check-clockdiff3,c949c451fd98f0ae,3615f481bfee3e04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047142 1 1 , error= 2024-09-11 17:32:22.591 INFO 2981023 --- [ocp-schedules-15,26b868bd909c69bc,3778aaf225236351] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:32:22.591 INFO 2981023 --- [ocp-schedules-15,26b868bd909c69bc,3778aaf225236351] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:32:22.592 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:32:22.592 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:32:22.592 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:32:22.596 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:22.596 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:22.597 INFO 2981023 --- [ocp-schedules-15,26b868bd909c69bc,3778aaf225236351] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:32:22.597 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:22.600 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:22.600 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:22.601 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:22.601 INFO 2981023 --- [ocp-schedules-15,26b868bd909c69bc,3778aaf225236351] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:32:22.628 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:32:22.628 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:32:22.628 INFO 2981023 --- [ocp-schedules-15,26b868bd909c69bc,3778aaf225236351] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:32:22.628 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:32:22.631 INFO 2981023 --- [ocp-schedules-15,26b868bd909c69bc,3778aaf225236351] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:32:23.153 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:23.153 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:23.176 INFO 2981023 --- [ocp-async-2,e124cd547797464f,a730691b25840418] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,f1eb1c1de8620c5c,ac067d811146c695] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,f1eb1c1de8620c5c,ac067d811146c695] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:24.204 INFO 2981023 --- [ocp-schedules-15,0d9068a6e7106077,3af9997917375ed9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:32:25.177 INFO 2981023 --- [ocp-async-9,93c78f9bdf9ee2d2,42e0e1cb5f6afcb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:26.153 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:26.154 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,fbb70cb0ee70cbeb,973c41d46ee17636] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,fbb70cb0ee70cbeb,973c41d46ee17636] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:32:26.351246+08:00, lastProcessTime=2024-09-11T17:32:16.347250+08:00 2024-09-11 17:32:26.474 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:32:26.351246+08:00 2024-09-11 17:32:26.800 INFO 2981023 --- [ocp-schedules-14,2e9533965944bb95,fb8bb5d7f0c4bc95] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:32:26.844 WARN 2981023 --- [scanner-13501684-1,2e9533965944bb95,db11fac0a402299a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:32:27.177 INFO 2981023 --- [ocp-async-8,98d0154d149d2344,0ef18e61147c319c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:27.750 INFO 2981023 --- [ocp-async-9,37cc1bd487d7152f,fdc5036d49323746] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:32:29.154 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:29.155 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:29.177 INFO 2981023 --- [ocp-async-5,d9cc1867e8c2e7fe,3ace196cc6bd0de0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,ea7b2ce9c6df1330,dfb124ee6271987a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,ea7b2ce9c6df1330,dfb124ee6271987a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:30.804 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,e2394706dd7b1a7a,7816551dbecd9331] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=e2394706dd7b1a7a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:32:30.805 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,e2394706dd7b1a7a,7816551dbecd9331] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=e2394706dd7b1a7a, duration=5 ms] 2024-09-11 17:32:30.813 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a496d96234564df3,8aabad0013913e9e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=a496d96234564df3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:32:30.815 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,69d156e7e58f0fa0,76fac18825b7b980] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=69d156e7e58f0fa0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:32:30.819 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a496d96234564df3,8aabad0013913e9e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=a496d96234564df3, duration=8 ms] 2024-09-11 17:32:30.822 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,69d156e7e58f0fa0,76fac18825b7b980] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=69d156e7e58f0fa0, duration=9 ms] 2024-09-11 17:32:31.177 INFO 2981023 --- [ocp-async-1,7d89293ffdccb71e,6eec139bdc58699b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:31.890 INFO 2981023 --- [ocp-schedules-14,7a92cb11c3ca9372,6efa5250029526a7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:32:32.155 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:32.156 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,81a48651be2fa3b7,825834cc90c56a74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,81a48651be2fa3b7,825834cc90c56a74] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:33.177 INFO 2981023 --- [ocp-async-3,93426ba96cd1991d,31170dc96308f7a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:33.766 INFO 2981023 --- [ocp-schedules-9,99a23a74583731a4,9d74504b3e31d634] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:32:33.774 INFO 2981023 --- [ocp-schedules-9,99a23a74583731a4,9d74504b3e31d634] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:32:33.774 INFO 2981023 --- [ocp-schedules-9,99a23a74583731a4,9d74504b3e31d634] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:32:35.156 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:35.157 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:35.177 INFO 2981023 --- [ocp-async-6,66c5149d86086955,65270a4a130d1c67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,45f38ae1c8ace7cd,e4d3da84ef16767d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,45f38ae1c8ace7cd,e4d3da84ef16767d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:32:36.337438+08:00, lastProcessTime=2024-09-11T17:32:26.351246+08:00 2024-09-11 17:32:36.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:32:36.337438+08:00 2024-09-11 17:32:36.462 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9bae84a6905bde71,79a83a1660f57e58] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2024-09-11T16%3A32%3A35%2B08%3A00&endTime=2024-09-11T17%3A32%3A35%2B08%3A00&maxPoints=360, client=10.31.33.41, traceId=9bae84a6905bde71, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:32:35+08:00,2024-09-11T17:32:35+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2024-09-11 17:32:36.473 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7dc360844226cf89,30dbd358b1700dc3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.31.33.41, traceId=7dc360844226cf89, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2024-09-11 17:32:36.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7dc360844226cf89,30dbd358b1700dc3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.31.33.41, traceId=7dc360844226cf89, duration=8 ms] 2024-09-11 17:32:36.487 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9bae84a6905bde71,79a83a1660f57e58] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=9bae84a6905bde71, duration=28 ms] 2024-09-11 17:32:36.492 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,5c22ba183d90ffda,6c2660502b02491c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP&maxPoints=360, client=10.31.33.41, traceId=5c22ba183d90ffda, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,] 2024-09-11 17:32:36.501 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,5c22ba183d90ffda,6c2660502b02491c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=5c22ba183d90ffda, duration=31 ms] 2024-09-11 17:32:36.570 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,feb9c5267477acfa,f40e51083ae7d0cd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.31.33.41, traceId=feb9c5267477acfa, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2024-09-11 17:32:36.581 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,feb9c5267477acfa,f40e51083ae7d0cd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=feb9c5267477acfa, duration=14 ms] 2024-09-11 17:32:36.597 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,19a56b049362a685,30e2396ff46cc15d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=19a56b049362a685, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 17:32:36.614 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,42d3f201cc2d3c78,62407060d75e9096] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.31.33.41, traceId=42d3f201cc2d3c78, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2024-09-11 17:32:36.638 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,42d3f201cc2d3c78,62407060d75e9096] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.31.33.41, traceId=42d3f201cc2d3c78, duration=28 ms] 2024-09-11 17:32:36.653 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:36.655 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:32:36.659 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:36.660 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:32:36.665 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,19a56b049362a685,30e2396ff46cc15d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=19a56b049362a685, duration=70 ms] 2024-09-11 17:32:36.694 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:36.695 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:32:36.989 INFO 2981023 --- [ocp-schedules-21,3518e5f38d0a517b,08013245f79f7fb4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:32:37.019 WARN 2981023 --- [scanner-13501882-1,3518e5f38d0a517b,0505953479c07694] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:32:37.051 INFO 2981023 --- [alarm-detect-13,3518e5f38d0a517b,a39bbf3e4a21d923] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644047, data is empty. 2024-09-11 17:32:37.057 INFO 2981023 --- [alarm-detect-13,3518e5f38d0a517b,a39bbf3e4a21d923] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644047, data is empty. 2024-09-11 17:32:37.062 INFO 2981023 --- [alarm-detect-13,3518e5f38d0a517b,a39bbf3e4a21d923] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644047, data is empty. 2024-09-11 17:32:37.082 INFO 2981023 --- [alarm-detect-8,3518e5f38d0a517b,9feab9e6487ee61f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644047, data is empty. 2024-09-11 17:32:37.086 INFO 2981023 --- [alarm-detect-7,3518e5f38d0a517b,58a41904e85596ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644047, data is empty. 2024-09-11 17:32:37.092 INFO 2981023 --- [alarm-detect-8,3518e5f38d0a517b,9feab9e6487ee61f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644047, data is empty. 2024-09-11 17:32:37.093 INFO 2981023 --- [alarm-detect-7,3518e5f38d0a517b,58a41904e85596ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644047, data is empty. 2024-09-11 17:32:37.097 INFO 2981023 --- [alarm-detect-8,3518e5f38d0a517b,9feab9e6487ee61f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644047, data is empty. 2024-09-11 17:32:37.099 INFO 2981023 --- [alarm-detect-7,3518e5f38d0a517b,58a41904e85596ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644047, data is empty. 2024-09-11 17:32:37.177 INFO 2981023 --- [ocp-async-5,f8a65011ba849b72,21f41eec854146eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:37.207 INFO 2981023 --- [alarm-detect-1,3518e5f38d0a517b,d7cd160b01ffdb5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644047, data is empty. 2024-09-11 17:32:37.214 INFO 2981023 --- [alarm-detect-1,3518e5f38d0a517b,d7cd160b01ffdb5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644047, data is empty. 2024-09-11 17:32:37.221 INFO 2981023 --- [alarm-detect-1,3518e5f38d0a517b,d7cd160b01ffdb5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644047, data is empty. 2024-09-11 17:32:37.314 INFO 2981023 --- [prometheus-cache-monitor1,db67f79b94724b90,a398c473c11861f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2050,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:32:37.315 INFO 2981023 --- [prometheus-cache-monitor1,db67f79b94724b90,a398c473c11861f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:32:37.750 INFO 2981023 --- [ocp-async-2,e24b48558b19ca92,9bc81d46022fee0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:32:38.157 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:38.157 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,20b0848c0f0a2252,a44194ec39803ab2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,20b0848c0f0a2252,a44194ec39803ab2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:38.874 INFO 2981023 --- [ocp-schedules-22,98a4d95df026f75c,04268d3617ab7bbf] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:32:38.893 WARN 2981023 --- [ocp-schedules-22,98a4d95df026f75c,04268d3617ab7bbf] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:32:39.177 INFO 2981023 --- [ocp-async-10,0f48fe125f98012a,96fd6721571f9dac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:40.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:32:40.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:32:41.028 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,df00ad1fd5b1bbb0,bbdb70da0f3ea47f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.31.33.41, traceId=df00ad1fd5b1bbb0, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-09-11 17:32:41.028 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,df00ad1fd5b1bbb0,bbdb70da0f3ea47f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.31.33.41, traceId=df00ad1fd5b1bbb0, duration=3 ms] 2024-09-11 17:32:41.029 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,68055c1ac760f6fc,3898c57ec7287f25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.31.33.41, traceId=68055c1ac760f6fc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 17:32:41.029 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fbd6399c68c18a1c,c045f1d12f3d7921] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.31.33.41, traceId=fbd6399c68c18a1c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-09-11 17:32:41.033 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fbd6399c68c18a1c,c045f1d12f3d7921] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.31.33.41, traceId=fbd6399c68c18a1c, duration=7 ms] 2024-09-11 17:32:41.038 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,68055c1ac760f6fc,3898c57ec7287f25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=68055c1ac760f6fc, duration=16 ms] 2024-09-11 17:32:41.038 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,9155fa1098cef740,645e0b3b5ec2b8af] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.31.33.41, traceId=9155fa1098cef740, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-09-11 17:32:41.038 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,9155fa1098cef740,645e0b3b5ec2b8af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.31.33.41, traceId=9155fa1098cef740, duration=2 ms] 2024-09-11 17:32:41.041 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ccde2f61c11a5bac,3d7047ee57992d43] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ccde2f61c11a5bac, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:32:41.051 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,ccde2f61c11a5bac,3d7047ee57992d43] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ccde2f61c11a5bac, duration=14 ms] 2024-09-11 17:32:41.131 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,8048016eedc3eb67,0e861e59106c9e5c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.31.33.41, traceId=8048016eedc3eb67, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-09-11 17:32:41.131 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,8048016eedc3eb67,0e861e59106c9e5c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.31.33.41, traceId=8048016eedc3eb67, duration=2 ms] 2024-09-11 17:32:41.157 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:41.158 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:41.169 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,534234103e4e46f2,3c8cdc39ed947ea2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.31.33.41, traceId=534234103e4e46f2, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-09-11 17:32:41.171 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,534234103e4e46f2,3c8cdc39ed947ea2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.31.33.41, traceId=534234103e4e46f2, duration=6 ms] 2024-09-11 17:32:41.178 INFO 2981023 --- [ocp-async-7,e2a1f6c7045dcbf8,b9559bbc48f2ce02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:41.183 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,76f4a50f757fb314,04cb67faf8afb21d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.31.33.41, traceId=76f4a50f757fb314, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-09-11 17:32:41.183 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,76f4a50f757fb314,04cb67faf8afb21d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.31.33.41, traceId=76f4a50f757fb314, duration=7 ms] 2024-09-11 17:32:41.236 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,e973cffe015eafc1,6b8a2bd0520810e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.31.33.41, traceId=e973cffe015eafc1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:32:41.280 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 17:32:41.280 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-8,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, client=10.31.33.41, traceId=] 2024-09-11 17:32:41.280 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 17:32:41.285 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 17:32:41.285 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, client=10.31.33.41, traceId=] 2024-09-11 17:32:41.285 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 17:32:41.305 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,dd8b302360795ed0,538131622cf81a0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/summary?id=1000006, client=10.31.33.41, traceId=dd8b302360795ed0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1000006,] 2024-09-11 17:32:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,0a0568e281ff4bda,b2d78d34c4b9c8e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:41.318 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,52bd7f98c7da8e37,99dcb386cdcc82e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/topSql/switch?id=1000006, client=10.31.33.41, traceId=52bd7f98c7da8e37, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1000006,] 2024-09-11 17:32:41.319 INFO 2981023 --- [prometheus-minute-schedule-1,0a0568e281ff4bda,b2d78d34c4b9c8e9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:41.355 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:41.355 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,dd8b302360795ed0,538131622cf81a0e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:41.356 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:32:41.357 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,dd8b302360795ed0,538131622cf81a0e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:32:41.360 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:41.362 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT ZONE, SVR_IP, SVR_PORT, NAME, VALUE, DATA_TYPE, SECTION, EDIT_LEVEL, INFO FROM GV$OB_PARAMETERS WHERE SCOPE = 'CLUSTER' AND name = ?, args: [_lcl_op_interval] 2024-09-11 17:32:41.370 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,dd8b302360795ed0,538131622cf81a0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/summary client=10.31.33.41, traceId=dd8b302360795ed0, duration=67 ms] 2024-09-11 17:32:41.377 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:41.377 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,52bd7f98c7da8e37,99dcb386cdcc82e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/topSql/switch client=10.31.33.41, traceId=52bd7f98c7da8e37, duration=64 ms] 2024-09-11 17:32:41.381 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1] 2024-09-11 17:32:41.383 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:41.384 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,92ddb1ccd110e5b6,3ef089eafa0eb283] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/stats?id=1000006, client=10.31.33.41, traceId=92ddb1ccd110e5b6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1000006,] 2024-09-11 17:32:41.385 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2024-09-11 17:32:41.387 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:41.388 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [resource_hard_limit] 2024-09-11 17:32:41.399 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,92ddb1ccd110e5b6,3ef089eafa0eb283] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:41.400 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,92ddb1ccd110e5b6,3ef089eafa0eb283] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:32:41.402 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,92ddb1ccd110e5b6,3ef089eafa0eb283] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/stats client=10.31.33.41, traceId=92ddb1ccd110e5b6, duration=22 ms] 2024-09-11 17:32:41.405 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:41.406 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:41.410 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1] 2024-09-11 17:32:41.451 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,e973cffe015eafc1,6b8a2bd0520810e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.31.33.41, traceId=e973cffe015eafc1, duration=218 ms] 2024-09-11 17:32:41.892 INFO 2981023 --- [ocp-schedules-22,245bd7dd9f587c86,ed022e56eabd29d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:32:42.016 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,73cbb46c242b0893,b1f89f5a2e36083d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.31.33.41, traceId=73cbb46c242b0893, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-09-11 17:32:42.022 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,73cbb46c242b0893,b1f89f5a2e36083d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.31.33.41, traceId=73cbb46c242b0893, duration=8 ms] 2024-09-11 17:32:43.178 INFO 2981023 --- [ocp-async-4,1f8a63e7fbddd17f,67d45df41804646e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:44.158 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:44.159 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,d99f3b608b469790,744944c881ba2921] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,d99f3b608b469790,744944c881ba2921] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:44.352 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8067db951249678f,c4fdbb4f3fbcf0d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.31.33.41, traceId=8067db951249678f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 17:32:44.352 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,46923f4592322567,c9d3b9bbc000e986] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/alarmSummaries?id=1000006, client=10.31.33.41, traceId=46923f4592322567, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=1000006,] 2024-09-11 17:32:44.359 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9e06375c1e55b53d,761572f860abea98] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/serverStats?id=1000006, client=10.31.33.41, traceId=9e06375c1e55b53d, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=1000006,] 2024-09-11 17:32:44.361 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8067db951249678f,c4fdbb4f3fbcf0d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=8067db951249678f, duration=12 ms] 2024-09-11 17:32:44.363 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,46923f4592322567,c9d3b9bbc000e986] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/alarmSummaries client=10.31.33.41, traceId=46923f4592322567, duration=14 ms] 2024-09-11 17:32:44.366 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6d65c3126623cbea,0a072597138e8032] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/eventMeta?id=1000006, client=10.31.33.41, traceId=6d65c3126623cbea, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1000006,] 2024-09-11 17:32:44.366 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6d65c3126623cbea,0a072597138e8032] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/eventMeta client=10.31.33.41, traceId=6d65c3126623cbea, duration=3 ms] 2024-09-11 17:32:44.367 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,7f1b88065c15c975,14f60777d17e6b5d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?status=AVAILABLE&status=ONLINE&excludedServiceType=OB_CLUSTER&excludedServiceType=BACKUP_AGENT&excludedServiceType=RESTORE_AGENT&excludedServiceType=BACKUP_RESTORE_AGENT&excludedServiceType=ARBITRATION&excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=7f1b88065c15c975, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[AVAILABLE, ONLINE],[OB_CLUSTER, BACKUP_AGENT, RESTORE_AGENT, BACKUP_RESTORE_AGENT, ARBITRATION, BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:32:44.373 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-7,7f1b88065c15c975,14f60777d17e6b5d] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:32:44.379 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9e06375c1e55b53d,761572f860abea98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:44.381 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9e06375c1e55b53d,761572f860abea98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:32:44.382 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9e06375c1e55b53d,761572f860abea98] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/serverStats client=10.31.33.41, traceId=9e06375c1e55b53d, duration=29 ms] 2024-09-11 17:32:44.400 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,7f1b88065c15c975,14f60777d17e6b5d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=7f1b88065c15c975, duration=39 ms] 2024-09-11 17:32:44.470 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e86f0c97728ce4ff,092985ecf396d83e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.31.33.41, traceId=e86f0c97728ce4ff, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:32:44.471 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e86f0c97728ce4ff,092985ecf396d83e] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:32:44.472 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,6b90e51be12076d5,6dc6b8650f490c29] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=6b90e51be12076d5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:32:44.476 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,6b90e51be12076d5,6dc6b8650f490c29] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=6b90e51be12076d5, duration=6 ms] 2024-09-11 17:32:44.501 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a275a19394c7c309,05729f22e3533677] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top/realtime?metrics=active_session%2Cqps%2Ctps&labels=app%3AOB%2Cob_cluster_name%3Alinkrtd%2Cob_cluster_id%3A1719994443&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&queryAsync=true&endTime=2024-09-11T17%3A32%3A43%2B08%3A00&limit=1&maxPoints=360, client=10.31.33.41, traceId=a275a19394c7c309, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryRealtimeTopMetrics(OffsetDateTime,Long,String,String,Long,Long,Integer,Integer,String,Boolean,QueryRouteType), args=2024-09-11T17:32:43+08:00,60,active_session,qps,tps,app,ob_cluster_name,ob_cluster_id,0,0,360,1,app:OB,ob_cluster_name:linkrtd,ob_cluster_id:1719994443,true,NOT_ROUTED,] 2024-09-11 17:32:44.501 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,40a19b46f073697f,e49a356aa99591d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/topSql/status/events?null, client=10.31.33.41, traceId=40a19b46f073697f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitchEvents(Long,OffsetDateTime,OffsetDateTime,Pageable), args=1000006,Page request [number: 0, size 5000, sort: timestamp: ASC],] 2024-09-11 17:32:44.506 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a275a19394c7c309,05729f22e3533677] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = qps 2024-09-11 17:32:44.506 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a275a19394c7c309,05729f22e3533677] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = qps, minTimestamp = 0, minTimeBuffer = 0, startTime = 1726047105 2024-09-11 17:32:44.510 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,40a19b46f073697f,e49a356aa99591d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/topSql/status/events client=10.31.33.41, traceId=40a19b46f073697f, duration=12 ms] 2024-09-11 17:32:44.511 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a275a19394c7c309,05729f22e3533677] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top/realtime client=10.31.33.41, traceId=a275a19394c7c309, duration=12 ms] 2024-09-11 17:32:44.513 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e86f0c97728ce4ff,092985ecf396d83e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=e86f0c97728ce4ff, duration=46 ms] 2024-09-11 17:32:44.531 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,53868f83229323a6,425b99611fab28f3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=53868f83229323a6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:32:44.534 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-7,53868f83229323a6,425b99611fab28f3] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:32:44.545 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,53868f83229323a6,425b99611fab28f3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=53868f83229323a6, duration=16 ms] 2024-09-11 17:32:44.580 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,9378c790937a4862,66d9e753edb61a91] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top/realtime?metrics=active_session%2Cqps%2Ctps&labels=app%3AOB%2Cob_cluster_name%3Alinkrtd%2Cob_cluster_id%3A1719994443&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&queryAsync=true&minStep=1&endTime=2024-09-11T17%3A32%3A43%2B08%3A00&limit=1&maxPoints=360, client=10.31.33.41, traceId=9378c790937a4862, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryRealtimeTopMetrics(OffsetDateTime,Long,String,String,Long,Long,Integer,Integer,String,Boolean,QueryRouteType), args=2024-09-11T17:32:43+08:00,60,active_session,qps,tps,app,ob_cluster_name,ob_cluster_id,0,1,360,1,app:OB,ob_cluster_name:linkrtd,ob_cluster_id:1719994443,true,NOT_ROUTED,] 2024-09-11 17:32:44.581 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-8,9378c790937a4862,66d9e753edb61a91] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = qps 2024-09-11 17:32:44.581 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-8,9378c790937a4862,66d9e753edb61a91] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = qps, minTimestamp = 0, minTimeBuffer = 0, startTime = 1726047105 2024-09-11 17:32:44.583 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,9378c790937a4862,66d9e753edb61a91] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top/realtime client=10.31.33.41, traceId=9378c790937a4862, duration=6 ms] 2024-09-11 17:32:44.620 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0c6fe49a0262a552,3f715df159f4c214] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.31.33.41, traceId=0c6fe49a0262a552, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:32:44.622 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0c6fe49a0262a552,3f715df159f4c214] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.31.33.41, traceId=0c6fe49a0262a552, duration=4 ms] 2024-09-11 17:32:44.645 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,5713fba2c6c870bc,7f67c17b723357bd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=5713fba2c6c870bc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:32:44.646 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,5713fba2c6c870bc,7f67c17b723357bd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=5713fba2c6c870bc, duration=3 ms] 2024-09-11 17:32:44.761 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c6ce09a54b99d03f,1be2b36904dfa918] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=c6ce09a54b99d03f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:32:44.762 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c6ce09a54b99d03f,1be2b36904dfa918] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=c6ce09a54b99d03f, duration=4 ms] 2024-09-11 17:32:44.944 INFO 2981023 --- [ocp-schedules-2,6e164a857dad189d,dcc5109676f8f5c6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-09-11 17:32:44.945 INFO 2981023 --- [ocp-schedules-2,6e164a857dad189d,dcc5109676f8f5c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:44.989 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:44.989 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:44.992 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:44.993 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 02:03:06.281634, 2024-09-11 17:32:44.965, 1000] 2024-09-11 17:32:44.993 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:21:42.083265, 2024-09-11 17:32:44.968, 1000] 2024-09-11 17:32:44.994 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000006, startTime 2024-09-11 17:21:42.083265, endTime 2024-09-11 17:32:44.968 2024-09-11 17:32:44.994 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-09-11 02:03:06.281634, endTime 2024-09-11 17:32:44.965 2024-09-11 17:32:44.996 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 10:40:18.658505, 2024-09-11 17:32:44.972, 1000] 2024-09-11 17:32:44.998 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2024-09-11 10:40:18.658505, endTime 2024-09-11 17:32:44.972 2024-09-11 17:32:45.001 INFO 2981023 --- [ocp-schedules-17,fe5ea18bbc345e76,6a58e6b182b3a0a3] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:32:45.008 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:45.012 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:45.023 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,c618c7864b0ef8bc,3ce7f8c26e832a43] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/events?startTime=2024-09-04T17%3A32%3A42%2B08%3A00&endTime=2024-09-11T17%3A32%3A42%2B08%3A00&page=1&size=7, client=10.31.33.41, traceId=c618c7864b0ef8bc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1000006,2024-09-04T17:32:42+08:00,2024-09-11T17:32:42+08:00,Page request [number: 0, size 7, sort: timestamp: DESC],] 2024-09-11 17:32:45.033 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:21:42.083371, 2024-09-11 17:32:45.0, 1000] 2024-09-11 17:32:45.034 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000006, startTime 2024-09-11 17:21:42.083371, endTime 2024-09-11 17:32:45.0 2024-09-11 17:32:45.044 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,c618c7864b0ef8bc,3ce7f8c26e832a43] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/events client=10.31.33.41, traceId=c618c7864b0ef8bc, duration=25 ms] 2024-09-11 17:32:45.057 INFO 2981023 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:45.057 INFO 2981023 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:45.058 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:45.058 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:45.088 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 08:00:10.321716, 2024-09-11 17:32:45.05, 1000] 2024-09-11 17:32:45.088 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 16:48:30.801707, 2024-09-11 17:32:45.05, 1000] 2024-09-11 17:32:45.088 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2024-09-11 16:48:30.801707, endTime 2024-09-11 17:32:45.05 2024-09-11 17:32:45.089 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-09-11 08:00:10.321716, endTime 2024-09-11 17:32:45.05 2024-09-11 17:32:45.089 INFO 2981023 --- [ocp-schedules-2,6e164a857dad189d,dcc5109676f8f5c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, tryCount=3, successCount=3 2024-09-11 17:32:45.101 INFO 2981023 --- [prometheus-second-schedule-1,1e2ecb51e77a0a7a,ac0dbc5e2574007e] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:32:45.178 INFO 2981023 --- [ocp-async-3,68ad0d1e86bcbde1,79ccb0751176fbbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:46.016 INFO 2981023 --- [ocp-schedules-7,f04829c11c4c1366,1350c23389622fad] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:32:46.023 INFO 2981023 --- [host-check-clockdiff1,f04829c11c4c1366,bbdfb0111fb2c537] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:32:46.025 INFO 2981023 --- [ocp-schedules-26,448ef332d77dc7dd,7736631135bb8402] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:32:46.026 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:32:46.026 INFO 2981023 --- [host-check-clockdiff2,f04829c11c4c1366,fa05450cffab7557] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:32:46.026 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:32:46.026 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:32:46.027 INFO 2981023 --- [ocp-schedules-27,b1baaec11ed22ae5,ea79c73677848202] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:32:46.027 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:32:46.028 INFO 2981023 --- [host-check-clockdiff3,f04829c11c4c1366,f0c8036bbde227d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:32:46.032 INFO 2981023 --- [host-check-agent-status1,b1baaec11ed22ae5,e6059950dab46721] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:32:46.032 INFO 2981023 --- [host-check-agent-status2,b1baaec11ed22ae5,92ef78f7fbd60d89] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:32:46.033 INFO 2981023 --- [host-check-agent-status3,b1baaec11ed22ae5,548d528e8d07e406] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:32:46.033 INFO 2981023 --- [host-check-clockdiff4,f04829c11c4c1366,7fa5df6dd8b5dfab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:32:46.034 INFO 2981023 --- [host-check-agent-status4,b1baaec11ed22ae5,c3ee064a1ce0ea60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:32:46.036 INFO 2981023 --- [host-check-agent-status2,b1baaec11ed22ae5,92ef78f7fbd60d89] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:46.037 INFO 2981023 --- [host-check-agent-status2,b1baaec11ed22ae5,92ef78f7fbd60d89] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:46.039 INFO 2981023 --- [host-check-agent-status1,b1baaec11ed22ae5,e6059950dab46721] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:46.040 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:46.041 INFO 2981023 --- [host-check-clockdiff4,f04829c11c4c1366,7fa5df6dd8b5dfab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047166 0 0 , error= 2024-09-11 17:32:46.041 INFO 2981023 --- [host-check-agent-status1,b1baaec11ed22ae5,e6059950dab46721] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:46.043 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:46.046 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:32:46.046 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:46.048 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:32:46.048 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:46.048 INFO 2981023 --- [host-check-agent-status3,b1baaec11ed22ae5,548d528e8d07e406] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:32:46.049 INFO 2981023 --- [host-check-agent-status4,b1baaec11ed22ae5,c3ee064a1ce0ea60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:46.049 INFO 2981023 --- [host-check-agent-status3,b1baaec11ed22ae5,548d528e8d07e406] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:32:46.050 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:46.052 INFO 2981023 --- [host-check-agent-status4,b1baaec11ed22ae5,c3ee064a1ce0ea60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:46.052 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:46.065 INFO 2981023 --- [host-check-agent-status2,b1baaec11ed22ae5,92ef78f7fbd60d89] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:32:46.069 INFO 2981023 --- [host-check-agent-status2,b1baaec11ed22ae5,92ef78f7fbd60d89] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:32:46.069 INFO 2981023 --- [host-check-agent-status2,b1baaec11ed22ae5,92ef78f7fbd60d89] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:32:46.069 INFO 2981023 --- [host-check-agent-status2,b1baaec11ed22ae5,92ef78f7fbd60d89] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:32:46.075 INFO 2981023 --- [host-check-agent-status1,b1baaec11ed22ae5,e6059950dab46721] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:32:46.076 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:32:46.076 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:32:46.076298530+08:00 2024-09-11 17:32:46.077 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:32:46.077 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:32:46.076+08:00 2024-09-11 17:32:46.078 INFO 2981023 --- [host-check-agent-status3,b1baaec11ed22ae5,548d528e8d07e406] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:32:46.078 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:32:46.079 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:32:46.078794724+08:00 2024-09-11 17:32:46.079 INFO 2981023 --- [host-check-agent-status1,b1baaec11ed22ae5,e6059950dab46721] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:32:46.079 INFO 2981023 --- [host-check-agent-status1,b1baaec11ed22ae5,e6059950dab46721] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:32:46.079 INFO 2981023 --- [host-check-agent-status1,b1baaec11ed22ae5,e6059950dab46721] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:32:46.079 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:32:46.076+08:00 and status AVAILABLE 2024-09-11 17:32:46.080 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:32:46.079+08:00 2024-09-11 17:32:46.081 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:32:46.081 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:32:46.081207216+08:00 2024-09-11 17:32:46.082 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:32:46.079+08:00 and status AVAILABLE 2024-09-11 17:32:46.082 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:32:46.082616859+08:00 2024-09-11 17:32:46.083 INFO 2981023 --- [host-check-agent-status3,b1baaec11ed22ae5,548d528e8d07e406] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:32:46.083 INFO 2981023 --- [host-check-agent-status3,b1baaec11ed22ae5,548d528e8d07e406] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:32:46.083 INFO 2981023 --- [host-check-agent-status3,b1baaec11ed22ae5,548d528e8d07e406] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:32:46.090 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:32:46.082+08:00 2024-09-11 17:32:46.092 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:32:46.082+08:00 and status AVAILABLE 2024-09-11 17:32:46.093 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:32:46.081+08:00 2024-09-11 17:32:46.098 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:32:46.081+08:00 and status AVAILABLE 2024-09-11 17:32:46.099 INFO 2981023 --- [host-check-agent-status4,b1baaec11ed22ae5,c3ee064a1ce0ea60] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:32:46.102 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:32:46.103 INFO 2981023 --- [host-check-agent-status4,b1baaec11ed22ae5,c3ee064a1ce0ea60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:32:46.103 INFO 2981023 --- [host-check-agent-status4,b1baaec11ed22ae5,c3ee064a1ce0ea60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:32:46.103 INFO 2981023 --- [host-check-agent-status4,b1baaec11ed22ae5,c3ee064a1ce0ea60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:32:46.108 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:46.108 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:46.109 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:32:46.123 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:32:46.130 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:32:46.132 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:32:46.138 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:46.139 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:46.140 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:32:46.141 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:32:46.148 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:46.159 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:46.162 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:32:46.166 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:32:46.175 INFO 2981023 --- [ocp-schedules-20,546f13d5d70896c8,e008f9734043f586] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:32:46.180 INFO 2981023 --- [ocp-schedules-20,546f13d5d70896c8,e008f9734043f586] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:32:46.187 INFO 2981023 --- [ocp-schedules-2,49ee8cbdc9941968,82aef71e025ae033] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:32:46.187 INFO 2981023 --- [ocp-schedules-22,9ef702356ae8d138,e91d3362c1bfaf20] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:32:46.187 INFO 2981023 --- [ocp-schedules-16,2e66697469134716,e8c533481544216f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:32:46.187 INFO 2981023 --- [ocp-schedules-10,190adc16e3f9cf91,764feb5b53e0733d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:32:46.189 INFO 2981023 --- [ocp-schedules-22,9ef702356ae8d138,e91d3362c1bfaf20] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:46.189 INFO 2981023 --- [ocp-schedules-16,2e66697469134716,e8c533481544216f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:46.191 INFO 2981023 --- [ocp-schedules-2,49ee8cbdc9941968,82aef71e025ae033] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:46.221 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:32:46.228 INFO 2981023 --- [ocp-schedules-6,7f9eaf1b2daee18c,3e5aee043b1f7865] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:32:46.230 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:32:46.232 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:32:46.234 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:32:46.235 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.236 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:46.236 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:32:46.245 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.246 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:46.246 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:32:46.249 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:32:46.250 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:32:46.256 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:32:46.260 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:32:46.261 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:32:46.269 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:32:46.273 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:32:46.281 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:32:46.288 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.289 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:32:46.290 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:32:46.290 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.294 INFO 2981023 --- [ocp-schedules-2,49ee8cbdc9941968,82aef71e025ae033] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:32:46.294 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:32:46.303 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:32:46.306 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:32:46.309 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:32:46.312 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.312 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:46.315 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:32:46.328 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:32:46.330 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:32:46.335 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.336 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:46.336 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:32:46.337 INFO 2981023 --- [ocp-schedules-22,9ef702356ae8d138,e91d3362c1bfaf20] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:32:46.342 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:32:46.346 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:32:46.347 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:32:46.347051+08:00, lastProcessTime=2024-09-11T17:32:36.337438+08:00 2024-09-11 17:32:46.348 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.348 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:46.353 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:32:46.374 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:32:46.383 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:32:46.385 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.386 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.386 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:32:46.387 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:32:46.387 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.387 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.388 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:46.390 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.391 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.392 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:32:46.396 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:32:46.408 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:32:46.412 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:32:46.417 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.417 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:32:46.418 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.418 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:46.419 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:32:46.419 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.423 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:32:46.424 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:32:46.424 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:32:46.424 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:32:46.425 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:32:46.433 INFO 2981023 --- [host-check-host-status3,448ef332d77dc7dd,967b59e3dc986866] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:32:46.436 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:32:46.443 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.444 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.445 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:32:46.448 INFO 2981023 --- [host-check-host-status1,448ef332d77dc7dd,2ad40d2c86ab98d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:32:46.449 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:46.449 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:32:46.453 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:32:46.454 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:32:46.461 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.464 INFO 2981023 --- [host-check-host-status2,448ef332d77dc7dd,9c1887a78ee104f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:32:46.465 INFO 2981023 --- [host-check-host-status4,448ef332d77dc7dd,e662ddb10241ef4c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:32:46.476 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.477 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.477 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:32:46.478 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:32:46.478 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.478 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:32:46.479 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:32:46.479 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.482 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.483 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.483 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:46.485 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.485 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:32:46.487 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:46.487 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:32:46.488 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:32:46.497 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.499 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:46.499 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:32:46.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.502 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:46.503 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:32:46.509 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:32:46.510 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:32:46.515 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:46.515 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:32:46.517 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:32:46.525 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.526 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.526 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:46.527 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:32:46.531 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:32:46.532 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.532 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.532 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.532 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.533 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:32:46.533 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:32:46.534 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:32:46.534 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.534 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:32:46.535 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:32:46.541 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:32:46.548 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:32:46.561 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:32:46.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.567 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.568 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:32:46.568 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:32:46.569 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.571 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:32:46.572 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:32:46.572 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:32:46.578 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:46.578 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:32:46.590 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:32:46.592 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.592 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.593 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:32:46.594 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:32:46.594 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.599 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.600 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.600 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:32:46.601 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:32:46.605 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:32:46.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:32:46.608 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:32:46.623 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:32:46.628 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:32:46.632 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.634 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.634 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:32:46.635 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:32:46.643 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:32:46.643 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:32:46.646 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.646 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:32:46.347051+08:00 2024-09-11 17:32:46.647 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.648 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:32:46.649 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:32:46.653 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:32:46.653 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:32:46.662 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:32:46.666 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:32:46.670 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:46.670 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:46.671 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:32:46.673 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:32:46.677 INFO 2981023 --- [ocp-schedules-19,e8fc70ebb4c32c0f,5fa70c8e5a720417] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:32:46.677 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:32:46.677 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:32:46.678 INFO 2981023 --- [ocp-schedules-16,2e66697469134716,e8c533481544216f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:32:46.971 INFO 2981023 --- [host-check-clockdiff2,f04829c11c4c1366,fa05450cffab7557] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047166 0 0 , error= 2024-09-11 17:32:47.159 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:47.160 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:47.178 INFO 2981023 --- [ocp-async-7,681735204c793004,486f3afdee21391b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:47.245 INFO 2981023 --- [ocp-schedules-10,2e37929b1131e9e1,9f8c46dedfc2818e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:32:47.294 WARN 2981023 --- [scanner-13502222-1,2e37929b1131e9e1,06646405a4990d2a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:32:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,6cf99fe6abd6a991,bfc2042bad61225d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:47.326 INFO 2981023 --- [prometheus-minute-schedule-1,6cf99fe6abd6a991,bfc2042bad61225d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:47.397 INFO 2981023 --- [alarm-detect-7,2e37929b1131e9e1,c029218f76119340] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047135 ~ 1726047157 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726047150, value=1.562)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.562, values={1726047150=1.562}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:32:47.442 INFO 2981023 --- [ocp-schedules-18,665e2930db5b7b40,c0c657c69dbd156d] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 1 2024-09-11 17:32:47.445 INFO 2981023 --- [ocp-schedules-18,665e2930db5b7b40,c0c657c69dbd156d] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-09-11 17:32:47.445 INFO 2981023 --- [ocp-schedules-18,665e2930db5b7b40,c0c657c69dbd156d] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 1 2024-09-11 17:32:47.750 INFO 2981023 --- [ocp-async-8,f9ae0082246021e4,c82f3b867b241773] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:32:47.812 INFO 2981023 --- [host-check-clockdiff1,f04829c11c4c1366,bbdfb0111fb2c537] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047167 0 0 , error= 2024-09-11 17:32:49.178 INFO 2981023 --- [ocp-async-1,ead0fc0e5a8dd375,8baf6580d6acf665] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:32:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:32:50.160 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:50.162 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:50.209 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:32:50.213 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:32:50.213 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.213 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.213 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.213 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.213 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.213 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.213 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.213 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.214 INFO 2981023 --- [ocp-schedules-5,0b3d4cbe751e29be,5e421a826a403838] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:32:50.218 INFO 2981023 --- [ocp-schedules-14,b7a70bcef3f45ce2,22fad5818fcd8195] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:32:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,b8e090eb8fcacf53,58533b58fd2c7cae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,b8e090eb8fcacf53,58533b58fd2c7cae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:51.178 INFO 2981023 --- [ocp-async-6,dd6e06c4f51e156a,52a861dbea981693] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:51.533 INFO 2981023 --- [ocp-schedules-22,d1d0ff155883590d,cd97435ae2816b10] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:32:51.534 INFO 2981023 --- [ocp-schedules-22,d1d0ff155883590d,cd97435ae2816b10] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:51.534 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:32:51.534 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:32:51.534 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:32:51.535 INFO 2981023 --- [ocp-schedules-22,d1d0ff155883590d,cd97435ae2816b10] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:32:51.895 INFO 2981023 --- [ocp-schedules-2,0ae45b3e11ef8d4e,c51eeb2c3a460567] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:32:51.972 INFO 2981023 --- [host-check-clockdiff3,f04829c11c4c1366,f0c8036bbde227d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047171 1 1 , error= 2024-09-11 17:32:51.979 INFO 2981023 --- [ocp-schedules-16,b6d922ce58a6e021,ae7f5f242261e714] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:32:51.979+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:32:51.982 INFO 2981023 --- [ocp-schedules-16,b6d922ce58a6e021,ae7f5f242261e714] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:32:52.104 INFO 2981023 --- [ocp-schedules-16,b6d922ce58a6e021,ae7f5f242261e714] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 38 to SqlAggregateData 2024-09-11 17:32:52.106 INFO 2981023 --- [ocp-schedules-16,b6d922ce58a6e021,ae7f5f242261e714] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:32:52.581 INFO 2981023 --- [ocp-schedules-27,150e6f45ae5ddda6,b63469b00a569903] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:32:52.601 WARN 2981023 --- [ocp-schedules-27,150e6f45ae5ddda6,b63469b00a569903] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:32:52.665 INFO 2981023 --- [ocp-schedules-30,e01b856dd862bb99,d532a01d37a09b82] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:32:52.667 INFO 2981023 --- [ocp-schedules-30,e01b856dd862bb99,d532a01d37a09b82] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:32:52.699 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:32:52.699 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.699 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:32:52.699 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.699 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:32:52.700 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.702 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:32:52.703 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:32:52.703 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:32:52.707 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.707 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.707 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.707 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.707 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.707 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.712 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.712 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.717 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.718 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:32:52.719 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.720 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.723 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.724 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.726 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.726 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:32:52.731 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.732 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.732 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:32:52.732 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.733 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:32:52.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2501D74E-0-0] 2024-09-11 17:32:52.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:32:52.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.734 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:32:52.739 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.739 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.739 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:32:52.740 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.745 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.745 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:32:52.751 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.751 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.751 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:32:52.751 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:32:52.752 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.752 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.753 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:32:52.753 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:32:52.753 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.753 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.754 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:32:52.754 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:32:52.754 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:32:52.756 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.756 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:32:52.757 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.757 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:32:52.758 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.758 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:32:52.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.759 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:32:52.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:32:52.759 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.759 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:32:52.760 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2501D758-0-0] 2024-09-11 17:32:52.760 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:32:52.760 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:32:52.760 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.761 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:32:52.762 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.763 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:32:52.763 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:32:52.766 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:32:52.766 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.767 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:32:52.767 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.768 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:32:52.769 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.769 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:32:52.770 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.770 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:32:52.771 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:32:52.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.773 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:32:52.773 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:32:52.774 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:52.775 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:32:52.776 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.776 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.779 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:52.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:32:52.788 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.788 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:32:52.789 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.789 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:32:52.791 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:32:52.793 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:32:52.793 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:32:52.793 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:32:52.794 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:32:52.796 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:32:52.796 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.796 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:32:52.799 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:32:52.801 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:32:52.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:32:52.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.802 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:32:52.803 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:52.803 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:32:52.807 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:32:52.809 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:32:52.809 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:32:52.809 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:32:52.812 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:32:52.814 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:32:52.814 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:32:52.814 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:32:52.814 INFO 2981023 --- [ocp-schedules-30,e01b856dd862bb99,d532a01d37a09b82] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:32:53.065 INFO 2981023 --- [ocp-schedules-12,e4a02e4e1a254bd4,b9437bc31bf8896c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:32:53.065 INFO 2981023 --- [ocp-schedules-12,e4a02e4e1a254bd4,b9437bc31bf8896c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:32:53.162 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:53.163 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:53.179 INFO 2981023 --- [ocp-async-7,612c5574b20308b6,833bfa03d1d10cc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,95816e312bcfdd5c,0d08dc2e850eb752] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,95816e312bcfdd5c,0d08dc2e850eb752] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:53.777 INFO 2981023 --- [ocp-schedules-4,7e48bfdbac81c720,9750fd0cce430509] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:32:53.785 INFO 2981023 --- [ocp-schedules-4,7e48bfdbac81c720,9750fd0cce430509] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:32:53.785 INFO 2981023 --- [ocp-schedules-4,7e48bfdbac81c720,9750fd0cce430509] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:32:54.206 INFO 2981023 --- [ocp-schedules-26,acd263136cd7e88c,e1d6da5a3afe59eb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:32:55.179 INFO 2981023 --- [ocp-async-4,1653b6bef80a9ec9,809f79d9817ec2b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:55.659 INFO 2981023 --- [ocp-schedules-27,7677703c14e1abff,a71bed954c5b8d85] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:32:55.659 INFO 2981023 --- [ocp-schedules-27,7677703c14e1abff,a71bed954c5b8d85] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:32:55.660 INFO 2981023 --- [ocp-schedules-27,7677703c14e1abff,a71bed954c5b8d85] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:32:55.660 INFO 2981023 --- [ocp-schedules-27,7677703c14e1abff,a71bed954c5b8d85] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:32:55.660 INFO 2981023 --- [ocp-schedules-27,7677703c14e1abff,a71bed954c5b8d85] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:32:55.660 INFO 2981023 --- [ocp-schedules-27,7677703c14e1abff,a71bed954c5b8d85] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:32:55.743 INFO 2981023 --- [ocp-schedules-26,03eaab85a5b88390,2d79d542786aeeb0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:32:55.749 INFO 2981023 --- [ocp-schedules-26,03eaab85a5b88390,2d79d542786aeeb0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:32:55.749 INFO 2981023 --- [ocp-schedules-26,03eaab85a5b88390,2d79d542786aeeb0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:32:56.045 INFO 2981023 --- [ocp-schedules-28,4aadb5727f9f8d5d,20211cc99c6197af] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:32:56.045 INFO 2981023 --- [ocp-schedules-28,4aadb5727f9f8d5d,20211cc99c6197af] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:32:56.058 INFO 2981023 --- [Collect all tenants major compactions3,4aadb5727f9f8d5d,2f912daa5272f7f8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:32:56.058 INFO 2981023 --- [Collect all tenants major compactions1,4aadb5727f9f8d5d,db239746309c32cd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:32:56.059 INFO 2981023 --- [Collect all tenants major compactions2,4aadb5727f9f8d5d,7674de7549f4388f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:32:56.059 INFO 2981023 --- [Collect all tenants major compactions6,4aadb5727f9f8d5d,88d0e8306925ed9b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:32:56.059 INFO 2981023 --- [Collect all tenants major compactions4,4aadb5727f9f8d5d,68c1c33d9abca51d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:32:56.061 INFO 2981023 --- [Collect all tenants major compactions5,4aadb5727f9f8d5d,fbc8b103aba4f5d9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:32:56.070 INFO 2981023 --- [Collect all tenants major compactions5,4aadb5727f9f8d5d,fbc8b103aba4f5d9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:56.071 INFO 2981023 --- [Collect all tenants major compactions3,4aadb5727f9f8d5d,2f912daa5272f7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:56.072 INFO 2981023 --- [Collect all tenants major compactions5,4aadb5727f9f8d5d,fbc8b103aba4f5d9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:32:56.072 INFO 2981023 --- [Collect all tenants major compactions1,4aadb5727f9f8d5d,db239746309c32cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:56.073 INFO 2981023 --- [Collect all tenants major compactions3,4aadb5727f9f8d5d,2f912daa5272f7f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:32:56.073 INFO 2981023 --- [Collect all tenants major compactions4,4aadb5727f9f8d5d,68c1c33d9abca51d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:56.073 INFO 2981023 --- [Collect all tenants major compactions1,4aadb5727f9f8d5d,db239746309c32cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:32:56.074 INFO 2981023 --- [Collect all tenants major compactions4,4aadb5727f9f8d5d,68c1c33d9abca51d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:32:56.074 INFO 2981023 --- [Collect all tenants major compactions6,4aadb5727f9f8d5d,88d0e8306925ed9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:56.075 INFO 2981023 --- [Collect all tenants major compactions2,4aadb5727f9f8d5d,7674de7549f4388f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:32:56.075 INFO 2981023 --- [Collect all tenants major compactions6,4aadb5727f9f8d5d,88d0e8306925ed9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:32:56.076 INFO 2981023 --- [Collect all tenants major compactions5,4aadb5727f9f8d5d,fbc8b103aba4f5d9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:32:56.076 INFO 2981023 --- [Collect all tenants major compactions5,4aadb5727f9f8d5d,fbc8b103aba4f5d9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:32:56.076 INFO 2981023 --- [Collect all tenants major compactions3,4aadb5727f9f8d5d,2f912daa5272f7f8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:32:56.076 INFO 2981023 --- [Collect all tenants major compactions3,4aadb5727f9f8d5d,2f912daa5272f7f8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:32:56.077 INFO 2981023 --- [Collect all tenants major compactions2,4aadb5727f9f8d5d,7674de7549f4388f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:32:56.077 INFO 2981023 --- [Collect all tenants major compactions4,4aadb5727f9f8d5d,68c1c33d9abca51d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:32:56.077 INFO 2981023 --- [Collect all tenants major compactions4,4aadb5727f9f8d5d,68c1c33d9abca51d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:32:56.078 INFO 2981023 --- [Collect all tenants major compactions6,4aadb5727f9f8d5d,88d0e8306925ed9b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:32:56.078 INFO 2981023 --- [Collect all tenants major compactions6,4aadb5727f9f8d5d,88d0e8306925ed9b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:32:56.078 INFO 2981023 --- [Collect all tenants major compactions2,4aadb5727f9f8d5d,7674de7549f4388f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:32:56.078 INFO 2981023 --- [Collect all tenants major compactions2,4aadb5727f9f8d5d,7674de7549f4388f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:32:56.083 INFO 2981023 --- [Collect all tenants major compactions1,4aadb5727f9f8d5d,db239746309c32cd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:32:56.083 INFO 2981023 --- [Collect all tenants major compactions1,4aadb5727f9f8d5d,db239746309c32cd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:32:56.083 INFO 2981023 --- [ocp-schedules-28,4aadb5727f9f8d5d,20211cc99c6197af] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:32:56.163 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:56.163 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,08a1643132b5b9d3,395cfae11f9a4e92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,08a1643132b5b9d3,395cfae11f9a4e92] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:32:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:32:56.351161+08:00, lastProcessTime=2024-09-11T17:32:46.347051+08:00 2024-09-11 17:32:56.480 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:32:56.351161+08:00 2024-09-11 17:32:57.179 INFO 2981023 --- [ocp-async-8,d9f1fe994009caf7,d32606e1459f00b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:57.474 INFO 2981023 --- [ocp-schedules-25,d0edf37f8e5e2d3b,b2f4480389153995] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:32:57.499 WARN 2981023 --- [scanner-13502404-1,d0edf37f8e5e2d3b,33682e4816b24337] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:32:57.544 INFO 2981023 --- [alarm-detect-15,d0edf37f8e5e2d3b,f03d61f6c0fefee4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644047, data is empty. 2024-09-11 17:32:57.551 INFO 2981023 --- [alarm-detect-15,d0edf37f8e5e2d3b,f03d61f6c0fefee4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644085, data is empty. 2024-09-11 17:32:57.555 INFO 2981023 --- [alarm-detect-15,d0edf37f8e5e2d3b,f03d61f6c0fefee4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644085, data is empty. 2024-09-11 17:32:57.612 INFO 2981023 --- [alarm-detect-19,d0edf37f8e5e2d3b,f9daea89073cead7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644047, data is empty. 2024-09-11 17:32:57.614 INFO 2981023 --- [alarm-detect-4,d0edf37f8e5e2d3b,f356d0cbfb545d48] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644047, data is empty. 2024-09-11 17:32:57.620 INFO 2981023 --- [alarm-detect-4,d0edf37f8e5e2d3b,f356d0cbfb545d48] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644085, data is empty. 2024-09-11 17:32:57.622 INFO 2981023 --- [alarm-detect-4,d0edf37f8e5e2d3b,f356d0cbfb545d48] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644085, data is empty. 2024-09-11 17:32:57.629 INFO 2981023 --- [alarm-detect-19,d0edf37f8e5e2d3b,f9daea89073cead7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644085, data is empty. 2024-09-11 17:32:57.640 INFO 2981023 --- [alarm-detect-19,d0edf37f8e5e2d3b,f9daea89073cead7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644085, data is empty. 2024-09-11 17:32:57.732 INFO 2981023 --- [alarm-detect-11,d0edf37f8e5e2d3b,1aadbdc6cb93ee53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644047, data is empty. 2024-09-11 17:32:57.737 INFO 2981023 --- [alarm-detect-11,d0edf37f8e5e2d3b,1aadbdc6cb93ee53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644085, data is empty. 2024-09-11 17:32:57.742 INFO 2981023 --- [alarm-detect-11,d0edf37f8e5e2d3b,1aadbdc6cb93ee53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644085, data is empty. 2024-09-11 17:32:57.751 INFO 2981023 --- [ocp-async-3,f3313928d33aa8b9,3f1f982f2151815f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:32:58.277 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.279 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obCluster 2024-09-11 17:32:58.289 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.297 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.300 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.308 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.309 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.310 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.310 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.311 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.314 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.317 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.320 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.323 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.324 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.324 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.325 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.326 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.327 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.327 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.328 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.329 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.332 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.335 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.337 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.339 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:26:58.202, end=2024-09-11T17:31:58.273} 2024-09-11 17:32:58.339 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:32:59.163 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:32:59.164 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:32:59.179 INFO 2981023 --- [ocp-async-5,a41b477cb13492cb,5892b1d50b5970db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:32:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,4ddfc07e2d2685e8,1dbdb5b160555dc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047180, interval=60 2024-09-11 17:32:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,4ddfc07e2d2685e8,1dbdb5b160555dc7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:01.179 INFO 2981023 --- [ocp-async-1,bcc04267f87df110,586330328387fac4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:01.630 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:29:00.923, end=2024-09-11T17:31:01.626} 2024-09-11 17:33:01.633 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719 2024-09-11 17:33:01.633 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:33:01.636 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:29:00.923, end=2024-09-11T17:31:01.626},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:33:01.636 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:33:01.636 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:33:01.654 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:33:01.671 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781671000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.671Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.674 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:33:01.674 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.679 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.679 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.683 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.683 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.687 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.687 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.688 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:33:01.688 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.671, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.689 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:33:01.689 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:29:00.923, end=2024-09-11T17:31:01.626},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:33:01.689 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:33:01.689 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:33:01.690 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:33:01.692 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:29:00.923, end=2024-09-11T17:31:01.626},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:33:01.692 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:33:01.692 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:33:01.735 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781735000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.735Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.737 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781737000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.737Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.741 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781741000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.741Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.742 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781742000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.742Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.746 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781746000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.746Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.746 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781746000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.746Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.748 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781748000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.748Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.749 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781749000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.749Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.757 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.757 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.760 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.760 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.762 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.762 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.768 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 60 2024-09-11 17:33:01.768 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.772 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.772 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.774 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.774 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.775 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.775 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.776 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.776 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.801 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:33:01.801 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.801 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:33:01.801 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.802 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.802 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.803 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.803 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.808 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.808 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.809 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.809 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.809 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.809 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.811 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.811 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.811 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:33:01.811 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.819 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.819 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.820 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.820 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.820 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.821 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.821 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.821 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.821 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.821 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.823 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.823 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.823 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.823 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.823 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.823 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.824 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.824 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.827 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.827 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.831 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.833 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.833 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.834 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.741, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.834 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.836 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.836 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.838 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.737, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.839 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.839 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.840 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.842 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.735, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.844 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.746, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.846 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.742, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.848 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.848 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.848 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.848 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.851 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.852 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.857 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.857 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.748, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.857 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.858 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.746, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.860 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.868 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.749, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.875 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781875000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.875Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.876 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781876000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.876Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.877 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781877000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.877Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.887 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.887 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.888 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781888000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.888Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.891 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:33:01.891 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.891 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781891000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.891Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.894 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.894 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.898 INFO 2981023 --- [ocp-schedules-13,f6cb1bf4328b5270,3624b34d9d577663] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:33:01.899 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.899 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.901 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:33:01.901 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.902 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781902000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.902Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.903 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781903000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3043450C260575FA1FB0F432BD07FEB0, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.903Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.904 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.904 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.908 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.908 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.911 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.911 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.911 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.911 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.913 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781913000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.913Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.914 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.914 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.917 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.917 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.918 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.918 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.918 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.919 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.921 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.921 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.921 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.921 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.923 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.923 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.923 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.923 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.924 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.924 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.925 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.925 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.925 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.932 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.875, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.934 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.934 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.936 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.936 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.936 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.937 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.937 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.938 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.938 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.938 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.938 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.938 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3043450C260575FA1FB0F432BD07FEB0, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.939 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.941 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.877, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.941 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.941 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.942 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.942 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.943 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.943 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.943 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.876, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.943 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.943 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.943 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.943 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.944 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.891, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.946 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.950 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.888, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.952 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.952 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.953 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.953 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3043450C260575FA1FB0F432BD07FEB0, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.953 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.953 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.954 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.955 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.955 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.958 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.913, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.959 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960781958000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:33:01.958Z, endTime=2024-09-11T09:31:01.626Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:33:01.959 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.902, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.959 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.903, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=3043450C260575FA1FB0F432BD07FEB0, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.964 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:33:01.964 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:33:01.969 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.969 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.970 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:33:01.970 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:33:01.975 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:33:01.975 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047061626000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:33:01.976 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:33:01.978 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:33:01.958, end=2024-09-11T17:31:01.626}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:33:01.979 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:33:01.980 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:29:00.923, end=2024-09-11T17:31:01.626},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:33:01.980 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:33:01.980 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:33:01.988 INFO 2981023 --- [plan_change_detector18,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:33:01.988 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:33:01.989 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:29:00.923, end=2024-09-11T17:31:01.626},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:33:01.989 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:33:01.989 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:33:01.990 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:33:01.990 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:29:00.923, end=2024-09-11T17:31:01.626},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:33:01.990 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:33:01.990 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:33:01.991 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:33:02.164 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:02.165 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,f3a3d80fcd385f56,5e6bcac260465764] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,f3a3d80fcd385f56,5e6bcac260465764] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:03.179 INFO 2981023 --- [ocp-async-8,a91395d43f185663,e5b2bc880a84cf4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:05.165 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:05.166 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:05.179 INFO 2981023 --- [ocp-async-2,cc63e59c9a99a29f,372ba4e6d0c92ed1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,9cc69f1c58eece86,3704559162e878b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,9cc69f1c58eece86,3704559162e878b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:33:06.337821+08:00, lastProcessTime=2024-09-11T17:32:56.351161+08:00 2024-09-11 17:33:06.465 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:33:06.337821+08:00 2024-09-11 17:33:07.179 INFO 2981023 --- [ocp-async-4,3d468246f5a9b592,e7cfd10ede132b5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:07.298 INFO 2981023 --- [ocp-schedules-4,ac4838660b4e6003,6ff6185507bca493] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:33:07.300 INFO 2981023 --- [ocp-schedules-30,38facba11b903436,b7f5111d2f6c0e10] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:33:07.300+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:33:07.303 INFO 2981023 --- [ocp-schedules-30,38facba11b903436,b7f5111d2f6c0e10] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:33:07.306 INFO 2981023 --- [ocp-schedules-30,38facba11b903436,b7f5111d2f6c0e10] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:33:07.306 INFO 2981023 --- [ocp-schedules-30,38facba11b903436,b7f5111d2f6c0e10] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:33:07.751 INFO 2981023 --- [ocp-async-2,186fbeea8e7ff718,b79a903064f25986] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:33:07.779 INFO 2981023 --- [ocp-schedules-11,6af1e8863fd23aa9,0cbaa9059fd12f2d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:33:07.823 WARN 2981023 --- [scanner-13502728-1,6af1e8863fd23aa9,af3b99fd2c7353d1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:33:08.166 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:08.166 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,a7b72561007dd777,f639d86aec5ea330] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,a7b72561007dd777,f639d86aec5ea330] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:09.179 INFO 2981023 --- [ocp-async-8,c97f3c9b83994f09,3c14fff6a4acaf91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:10.003 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:33:10.004 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:33:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:33:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:33:11.166 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:11.168 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:11.180 INFO 2981023 --- [ocp-async-5,b36275b97285395f,0e628121b0015f86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,5a9c7bc404126545,6189ba89692f0a95] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,5a9c7bc404126545,6189ba89692f0a95] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:11.434 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047120, nodeCount=480, affectRows=480 2024-09-11 17:33:11.911 INFO 2981023 --- [ocp-schedules-25,c7921a31a660fd4c,05fa234b1a3ac1ab] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:33:13.180 INFO 2981023 --- [ocp-async-9,2bca3bc0b299df09,6c4f3200f0c1508b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:13.549 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047120, nodeCount=480, validCount=480 2024-09-11 17:33:13.787 INFO 2981023 --- [ocp-schedules-15,cb9971c5bd3c4f5d,a479b32c95165512] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:33:13.792 INFO 2981023 --- [ocp-schedules-15,cb9971c5bd3c4f5d,a479b32c95165512] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:33:13.792 INFO 2981023 --- [ocp-schedules-15,cb9971c5bd3c4f5d,a479b32c95165512] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:33:14.168 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:14.169 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,ce88eba7f652248b,ab83e38b172f7958] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,3fc5242a7daa64f2,cae4bb8d78946bbd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:33:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,ce88eba7f652248b,ab83e38b172f7958] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:14.435 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047120, nodeCount=26, affectRows=26 2024-09-11 17:33:14.435 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047120, nodeCount=26, validCount=26 2024-09-11 17:33:14.437 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047120, nodeCount=25, affectRows=25 2024-09-11 17:33:14.437 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047120, nodeCount=25, validCount=25 2024-09-11 17:33:14.464 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047120, nodeCount=1000, affectRows=1000 2024-09-11 17:33:14.467 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047120, nodeCount=1000, validCount=1000 2024-09-11 17:33:14.473 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047120, nodeCount=1000, affectRows=1000 2024-09-11 17:33:14.475 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047120, nodeCount=999, validCount=999 2024-09-11 17:33:15.180 INFO 2981023 --- [ocp-async-10,d0a48ab61324bd30,43d36d4c78fc3254] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:16.023 INFO 2981023 --- [ocp-schedules-19,a3bcd0148747d10d,cc5d911ece9e83e7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:33:16.023 INFO 2981023 --- [ocp-schedules-3,fcd12fa6d8be6195,ab999efa13d18193] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:33:16.024 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:33:16.026 INFO 2981023 --- [host-check-clockdiff1,fcd12fa6d8be6195,42bbbf8c3264b03f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:33:16.026 INFO 2981023 --- [host-check-clockdiff2,fcd12fa6d8be6195,b5a80c27a092fb73] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:33:16.027 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:33:16.027 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:33:16.028 INFO 2981023 --- [host-check-clockdiff3,fcd12fa6d8be6195,5c1adf6dbd6c4b82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:33:16.028 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:33:16.029 INFO 2981023 --- [host-check-clockdiff4,fcd12fa6d8be6195,5aa03e54c335efda] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:33:16.031 INFO 2981023 --- [ocp-schedules-5,c9817367a4669287,29178a3ff5bb55ad] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:33:16.033 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:33:16.034 INFO 2981023 --- [host-check-agent-status1,c9817367a4669287,332e17d59f20273b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:33:16.034 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:33:16.034 INFO 2981023 --- [host-check-agent-status2,c9817367a4669287,eebaf10eef6dbdd1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:33:16.035 INFO 2981023 --- [host-check-agent-status3,c9817367a4669287,fb25c76fa22d5914] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:33:16.035 INFO 2981023 --- [host-check-agent-status4,c9817367a4669287,6d04b31d34344742] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:33:16.041 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:33:16.041 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:33:16.042 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:33:16.044 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:33:16.044 INFO 2981023 --- [host-check-clockdiff4,fcd12fa6d8be6195,5aa03e54c335efda] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047196 0 0 , error= 2024-09-11 17:33:16.044 INFO 2981023 --- [host-check-clockdiff2,fcd12fa6d8be6195,b5a80c27a092fb73] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047196 0 0 , error= 2024-09-11 17:33:16.046 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:33:16.047 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:33:16.048 INFO 2981023 --- [host-check-agent-status2,c9817367a4669287,eebaf10eef6dbdd1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:33:16.049 INFO 2981023 --- [host-check-agent-status4,c9817367a4669287,6d04b31d34344742] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:33:16.049 INFO 2981023 --- [host-check-agent-status3,c9817367a4669287,fb25c76fa22d5914] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:33:16.050 INFO 2981023 --- [host-check-agent-status2,c9817367a4669287,eebaf10eef6dbdd1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:33:16.051 INFO 2981023 --- [host-check-agent-status3,c9817367a4669287,fb25c76fa22d5914] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:33:16.051 INFO 2981023 --- [host-check-agent-status1,c9817367a4669287,332e17d59f20273b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:33:16.052 INFO 2981023 --- [host-check-agent-status4,c9817367a4669287,6d04b31d34344742] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:33:16.056 INFO 2981023 --- [host-check-agent-status1,c9817367a4669287,332e17d59f20273b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:33:16.077 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:33:16.081 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:33:16.081 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:33:16.078939631+08:00 2024-09-11 17:33:16.083 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:33:16.084 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:33:16.084 INFO 2981023 --- [host-check-agent-status1,c9817367a4669287,332e17d59f20273b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:33:16.085 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:33:16.085343101+08:00 2024-09-11 17:33:16.086 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:33:16.085+08:00 2024-09-11 17:33:16.086 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:33:16.086148287+08:00 2024-09-11 17:33:16.087 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:33:16.086+08:00 2024-09-11 17:33:16.088 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:33:16.085+08:00 and status AVAILABLE 2024-09-11 17:33:16.088 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:33:16.086+08:00 and status AVAILABLE 2024-09-11 17:33:16.091 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:33:16.081+08:00 2024-09-11 17:33:16.092 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:33:16.088600405+08:00 2024-09-11 17:33:16.093 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:33:16.081+08:00 and status AVAILABLE 2024-09-11 17:33:16.094 INFO 2981023 --- [host-check-agent-status1,c9817367a4669287,332e17d59f20273b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:33:16.095 INFO 2981023 --- [host-check-agent-status2,c9817367a4669287,eebaf10eef6dbdd1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:33:16.095 INFO 2981023 --- [host-check-agent-status4,c9817367a4669287,6d04b31d34344742] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:33:16.095 INFO 2981023 --- [host-check-agent-status1,c9817367a4669287,332e17d59f20273b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:33:16.095 INFO 2981023 --- [host-check-agent-status1,c9817367a4669287,332e17d59f20273b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:33:16.097 INFO 2981023 --- [host-check-agent-status2,c9817367a4669287,eebaf10eef6dbdd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:33:16.097 INFO 2981023 --- [host-check-agent-status2,c9817367a4669287,eebaf10eef6dbdd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:33:16.098 INFO 2981023 --- [host-check-agent-status2,c9817367a4669287,eebaf10eef6dbdd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:33:16.098 INFO 2981023 --- [host-check-agent-status4,c9817367a4669287,6d04b31d34344742] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:33:16.098 INFO 2981023 --- [host-check-agent-status4,c9817367a4669287,6d04b31d34344742] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:33:16.098 INFO 2981023 --- [host-check-agent-status4,c9817367a4669287,6d04b31d34344742] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:33:16.098 INFO 2981023 --- [host-check-agent-status3,c9817367a4669287,fb25c76fa22d5914] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:33:16.099 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:33:16.092+08:00 2024-09-11 17:33:16.100 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:33:16.092+08:00 and status AVAILABLE 2024-09-11 17:33:16.102 INFO 2981023 --- [host-check-agent-status3,c9817367a4669287,fb25c76fa22d5914] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:33:16.102 INFO 2981023 --- [host-check-agent-status3,c9817367a4669287,fb25c76fa22d5914] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:33:16.102 INFO 2981023 --- [host-check-agent-status3,c9817367a4669287,fb25c76fa22d5914] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:33:16.108 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:33:16.117 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:33:16.129 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:33:16.131 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:33:16.141 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:33:16.142 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:33:16.143 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:33:16.143 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:33:16.143 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:33:16.145 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:33:16.149 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:33:16.151 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:33:16.182 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:33:16.187 INFO 2981023 --- [ocp-schedules-8,62bd5cd2b76d6016,0cddda6a1c90376e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:33:16.187 INFO 2981023 --- [ocp-schedules-29,7d85e42c926c78f0,70b7033d7070ea49] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:33:16.187 INFO 2981023 --- [ocp-schedules-12,f26e6650e3332a5c,325103e0167062dc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:33:16.187 INFO 2981023 --- [ocp-schedules-10,02ce94df65bee760,9a2bcc8d74a107d7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:33:16.194 INFO 2981023 --- [ocp-schedules-8,62bd5cd2b76d6016,0cddda6a1c90376e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:33:16.194 INFO 2981023 --- [ocp-schedules-29,7d85e42c926c78f0,70b7033d7070ea49] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:33:16.195 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:33:16.196 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:33:16.197 INFO 2981023 --- [ocp-schedules-21,f361063bc05c8df2,4e7ef0b34c7e88b7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:33:16.197 INFO 2981023 --- [ocp-schedules-12,f26e6650e3332a5c,325103e0167062dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:33:16.204 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:33:16.224 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:33:16.267 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:33:16.279 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:33:16.280 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:33:16.287 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.287 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:16.291 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:33:16.293 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:33:16.293 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:33:16.297 INFO 2981023 --- [ocp-schedules-8,62bd5cd2b76d6016,0cddda6a1c90376e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:33:16.298 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:33:16.301 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.301 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:16.330 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.331 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.331 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:33:16.339 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.340 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:16.342 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:33:16.343 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:33:16.342430+08:00, lastProcessTime=2024-09-11T17:33:06.337821+08:00 2024-09-11 17:33:16.346 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:33:16.348 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.349 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:16.350 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:33:16.358 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:33:16.362 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.363 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:16.363 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:33:16.364 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:33:16.365 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.366 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:16.366 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:33:16.370 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:33:16.375 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.375 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:33:16.375 INFO 2981023 --- [ocp-schedules-29,7d85e42c926c78f0,70b7033d7070ea49] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:33:16.376 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.376 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:16.377 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:33:16.377 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.379 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:33:16.381 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:33:16.383 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:33:16.394 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:16.394 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:33:16.395 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:33:16.399 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:33:16.405 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:33:16.410 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:33:16.418 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:33:16.421 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:33:16.426 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:16.426 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:33:16.429 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.430 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.430 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:33:16.434 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:33:16.434 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.435 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.435 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:16.436 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:33:16.440 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:33:16.441 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:33:16.444 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:33:16.446 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:33:16.451 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:33:16.451 INFO 2981023 --- [host-check-host-status3,a3bcd0148747d10d,1338d824bf200842] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:33:16.460 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:33:16.463 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.464 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.464 INFO 2981023 --- [host-check-host-status1,a3bcd0148747d10d,47508083e542208a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:33:16.464 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:16.465 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.467 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:33:16.468 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:33:16.469 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:33:16.472 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:33:16.477 INFO 2981023 --- [host-check-host-status2,a3bcd0148747d10d,78ebcfbaa1ae97d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:33:16.488 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:16.488 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:33:16.503 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:16.503 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.504 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.505 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:33:16.508 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:33:16.513 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.516 INFO 2981023 --- [host-check-host-status4,a3bcd0148747d10d,6a895f2f523bd07f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:33:16.517 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:16.519 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:33:16.524 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:33:16.528 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.528 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.529 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:16.529 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:33:16.530 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.532 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:33:16.533 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:33:16.535 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:33:16.537 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:33:16.545 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:16.545 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:33:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:16.561 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:33:16.561 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:33:16.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.566 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.567 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:16.567 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:33:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:33:16.571 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:33:16.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:33:16.575 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:33:16.581 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:33:16.581 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:33:16.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:33:16.607 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:33:16.342430+08:00 2024-09-11 17:33:16.617 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:33:16.622 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:33:16.623 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.623 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.624 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:33:16.628 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:16.629 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:16.630 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:33:16.631 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:33:16.631 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:33:16.638 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:33:16.638 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:33:16.640 INFO 2981023 --- [ocp-schedules-12,f26e6650e3332a5c,325103e0167062dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:33:17.169 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:17.170 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:17.180 INFO 2981023 --- [ocp-async-7,d27269f5cdf71fbf,38e76237bcd3d3c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,352c6c02a330cc01,18f6fe097c118856] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:17.323 INFO 2981023 --- [prometheus-minute-schedule-1,352c6c02a330cc01,18f6fe097c118856] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:17.751 INFO 2981023 --- [ocp-async-10,3435f7b35acec91b,33dc669551f222e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:33:17.967 INFO 2981023 --- [ocp-schedules-9,400ea491b3fe78af,2d47fd04c204d027] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:33:18.027 WARN 2981023 --- [scanner-13504455-1,400ea491b3fe78af,5159e6e1242a0146] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:33:18.135 INFO 2981023 --- [alarm-detect-8,400ea491b3fe78af,39746224a0c929bc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644085, data is empty. 2024-09-11 17:33:18.136 INFO 2981023 --- [alarm-detect-5,400ea491b3fe78af,970b14d4da6bf332] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644085, data is empty. 2024-09-11 17:33:18.138 INFO 2981023 --- [alarm-detect-8,400ea491b3fe78af,39746224a0c929bc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644085, data is empty. 2024-09-11 17:33:18.140 INFO 2981023 --- [alarm-detect-4,400ea491b3fe78af,17c5d453c8e0058f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644085, data is empty. 2024-09-11 17:33:18.141 INFO 2981023 --- [alarm-detect-5,400ea491b3fe78af,970b14d4da6bf332] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644085, data is empty. 2024-09-11 17:33:18.144 INFO 2981023 --- [alarm-detect-8,400ea491b3fe78af,39746224a0c929bc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644085, data is empty. 2024-09-11 17:33:18.147 INFO 2981023 --- [alarm-detect-4,400ea491b3fe78af,17c5d453c8e0058f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644085, data is empty. 2024-09-11 17:33:18.152 INFO 2981023 --- [alarm-detect-5,400ea491b3fe78af,970b14d4da6bf332] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644085, data is empty. 2024-09-11 17:33:18.153 INFO 2981023 --- [alarm-detect-4,400ea491b3fe78af,17c5d453c8e0058f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644085, data is empty. 2024-09-11 17:33:18.245 INFO 2981023 --- [alarm-detect-3,400ea491b3fe78af,c968c2827be4feda] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644085, data is empty. 2024-09-11 17:33:18.250 INFO 2981023 --- [alarm-detect-3,400ea491b3fe78af,c968c2827be4feda] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644085, data is empty. 2024-09-11 17:33:18.254 INFO 2981023 --- [alarm-detect-3,400ea491b3fe78af,c968c2827be4feda] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644085, data is empty. 2024-09-11 17:33:19.180 INFO 2981023 --- [ocp-async-4,6d917e24f3a05141,1012fc0fc65d136b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:20.170 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:20.170 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,16aeda17f50eb310,e37025fde205c292] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,16aeda17f50eb310,e37025fde205c292] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:21.180 INFO 2981023 --- [ocp-async-10,d230b169eed644ca,080d18c117018808] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:21.281 INFO 2981023 --- [host-check-clockdiff1,fcd12fa6d8be6195,42bbbf8c3264b03f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047201 0 0 , error= 2024-09-11 17:33:21.871 INFO 2981023 --- [ocp-schedules-17,081aa51fa4564752,6a52b5e57b85f780] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 17:33:21.871 INFO 2981023 --- [ocp-schedules-17,081aa51fa4564752,6a52b5e57b85f780] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 17:33:21.887 INFO 2981023 --- [pool-546-thread-2,081aa51fa4564752,ba1ada85f80d1f36] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:33:21.888 INFO 2981023 --- [pool-546-thread-3,081aa51fa4564752,b7eef6ffd0a7023c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:33:21.888 INFO 2981023 --- [pool-546-thread-1,081aa51fa4564752,a9124b29fbb02d12] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:33:21.908 INFO 2981023 --- [pool-546-thread-1,081aa51fa4564752,a9124b29fbb02d12] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:21.910 INFO 2981023 --- [pool-546-thread-3,081aa51fa4564752,b7eef6ffd0a7023c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:21.910 INFO 2981023 --- [pool-546-thread-2,081aa51fa4564752,ba1ada85f80d1f36] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:21.912 INFO 2981023 --- [pool-546-thread-1,081aa51fa4564752,a9124b29fbb02d12] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046601000000, 1726047201000000] 2024-09-11 17:33:21.912 INFO 2981023 --- [pool-546-thread-3,081aa51fa4564752,b7eef6ffd0a7023c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046601000000, 1726047201000000] 2024-09-11 17:33:21.914 INFO 2981023 --- [ocp-schedules-7,b39e4a919bf33b0f,64a34e8ff8fa2bb4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:33:21.914 INFO 2981023 --- [pool-546-thread-2,081aa51fa4564752,ba1ada85f80d1f36] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046601000000, 1726047201000000] 2024-09-11 17:33:21.916 INFO 2981023 --- [pool-546-thread-3,081aa51fa4564752,b7eef6ffd0a7023c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:33:21.987 INFO 2981023 --- [pool-546-thread-1,081aa51fa4564752,a9124b29fbb02d12] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:33:22.001 INFO 2981023 --- [host-check-clockdiff3,fcd12fa6d8be6195,5c1adf6dbd6c4b82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047201 1 1 , error= 2024-09-11 17:33:22.058 INFO 2981023 --- [pool-546-thread-2,081aa51fa4564752,ba1ada85f80d1f36] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:33:22.059 INFO 2981023 --- [ocp-schedules-17,081aa51fa4564752,6a52b5e57b85f780] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 17:33:22.634 INFO 2981023 --- [ocp-schedules-27,785665554a8f04aa,e6d5a598b54c94da] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:33:22.634 INFO 2981023 --- [ocp-schedules-27,785665554a8f04aa,e6d5a598b54c94da] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:33:22.635 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:33:22.635 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:33:22.635 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:33:22.639 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:33:22.639 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:33:22.640 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:33:22.643 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:33:22.644 INFO 2981023 --- [ocp-schedules-27,785665554a8f04aa,e6d5a598b54c94da] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:33:22.645 INFO 2981023 --- [ocp-schedules-27,785665554a8f04aa,e6d5a598b54c94da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:33:22.645 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:33:22.646 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:33:22.659 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:33:22.660 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:33:22.663 INFO 2981023 --- [ocp-schedules-27,785665554a8f04aa,e6d5a598b54c94da] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:33:22.664 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:33:22.666 INFO 2981023 --- [ocp-schedules-27,785665554a8f04aa,e6d5a598b54c94da] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:33:23.170 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:23.171 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:23.180 INFO 2981023 --- [ocp-async-6,805ff2513850d3fe,a3e59dcac0a754d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,9b483ad9f6f70e95,71cafeba65bb2f27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,9b483ad9f6f70e95,71cafeba65bb2f27] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:24.208 INFO 2981023 --- [ocp-schedules-14,8bd2b80493795b5e,ac777b7d46002a75] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:33:25.180 INFO 2981023 --- [ocp-async-5,20030032b9870455,a609c1146f1113f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:26.171 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:26.171 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,f7719e2b805557ac,6cfc96bf27d28f38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,f7719e2b805557ac,6cfc96bf27d28f38] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:33:26.337291+08:00, lastProcessTime=2024-09-11T17:33:16.342430+08:00 2024-09-11 17:33:26.475 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:33:26.337291+08:00 2024-09-11 17:33:26.703 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.705 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obC 2024-09-11 17:33:26.717 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.717 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:33:26.724 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.733 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.747 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.748 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.748 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:33:26.749 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.750 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.751 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.751 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.757 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.758 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.759 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.760 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.764 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.764 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:33:26.766 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.770 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.774 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.778 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.782 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 37 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.844 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 8 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.866 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 169 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.895 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.897 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.898 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.899 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.899 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.900 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:27:26.552, end=2024-09-11T17:32:26.7} 2024-09-11 17:33:26.900 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 4 dubious sqls 2024-09-11 17:33:27.180 INFO 2981023 --- [ocp-async-1,972393e67b4dc10c,9b51c43af0f7ba7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:27.636 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:33:27.636 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:33:27.639 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:33:27.639 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:33:27.639 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:33:27.639 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:33:27.641 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:33:27.641 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:33:27.641 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:33:27.641 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:33:27.663 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:33:27.668 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:33:27.668 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:33:27.668 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:33:27.668 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:33:27.668 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:33:27.672 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:33:27.751 INFO 2981023 --- [ocp-async-5,c32f3a924869da34,4279d3b9f3cdcf09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:33:27.912 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:33:28.163 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.522386433S 2024-09-11 17:33:28.163 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:33:28.211 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:33:28.211 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:33:28.225 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:33:28.262 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:33:28.262 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:33:28.262 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:33:28.262 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:33:28.262 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:33:28.262 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:33:28.268 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:33:28.294 INFO 2981023 --- [ocp-schedules-20,c3aee0a216ffc6c1,41359c65768fdfcf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:33:28.365 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] o.s.boot.SpringApplication : Started application in 0.726 seconds (JVM running for 83008.238) 2024-09-11 17:33:28.365 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.726717442S 2024-09-11 17:33:28.431 WARN 2981023 --- [scanner-13504760-1,c3aee0a216ffc6c1,14e56bbbebcb1b1d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:33:28.708 INFO 2981023 --- [ocp-schedules-5,94b15f5ba934f1f0,0f8f7a15acf2ad3e] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:33:29.172 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:29.172 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:29.180 INFO 2981023 --- [ocp-async-7,476e2c9de4f7ca2b,16c9a49528bff10f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,a963a909be278bbe,fa42186aa52d9bab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,a963a909be278bbe,fa42186aa52d9bab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:31.190 INFO 2981023 --- [ocp-async-2,8e13ee6c6778c8ed,dbe3c0d6c60cea14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:31.916 INFO 2981023 --- [ocp-schedules-7,04f2469196b2868d,2bb0210942a2abad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:33:32.172 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:32.173 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,7a7d123eae2dad2a,94045d2663f42464] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,7a7d123eae2dad2a,94045d2663f42464] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:33.181 INFO 2981023 --- [ocp-async-3,456a05721979093f,a8cd9d70abdbdb86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:33.795 INFO 2981023 --- [ocp-schedules-8,9c184907693d28d5,516e710bb2b89445] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:33:33.803 INFO 2981023 --- [ocp-schedules-8,9c184907693d28d5,516e710bb2b89445] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:33:33.803 INFO 2981023 --- [ocp-schedules-8,9c184907693d28d5,516e710bb2b89445] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:33:35.173 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:35.175 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:35.181 INFO 2981023 --- [ocp-async-10,61aaa36e683fde86,b7572c735a69471e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,d10ef62d88b997f0,1d380e2443254d27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:35.317 INFO 2981023 --- [prometheus-minute-schedule-1,d10ef62d88b997f0,1d380e2443254d27] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:36.321 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,bda24238fd9e9d8a,7d833e3a02f28796] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.31.33.41, traceId=bda24238fd9e9d8a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:33:36.356 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:33:36.355163+08:00, lastProcessTime=2024-09-11T17:33:26.337291+08:00 2024-09-11 17:33:36.508 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,bda24238fd9e9d8a,7d833e3a02f28796] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.31.33.41, traceId=bda24238fd9e9d8a, duration=193 ms] 2024-09-11 17:33:36.516 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:33:36.355163+08:00 2024-09-11 17:33:37.181 INFO 2981023 --- [ocp-async-5,38d10a185352a019,b13369cfe6a5b527] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:37.314 INFO 2981023 --- [prometheus-cache-monitor1,67c5f0f48c8a0879,5781cbb5e41fc268] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:33:37.314 INFO 2981023 --- [prometheus-cache-monitor1,67c5f0f48c8a0879,5781cbb5e41fc268] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:33:37.751 INFO 2981023 --- [ocp-async-6,894fdd3afd495ed3,ff5cda640bebb8f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:33:38.175 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:38.177 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,f0dde6f2d7afe2c0,a0038f4a573d8ada] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,f0dde6f2d7afe2c0,a0038f4a573d8ada] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:38.773 INFO 2981023 --- [ocp-schedules-14,d6e12051bc3bc724,0bf677efc962b3f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:33:38.819 WARN 2981023 --- [scanner-13505026-1,d6e12051bc3bc724,cad989c00e79cef0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:33:38.842 INFO 2981023 --- [alarm-detect-5,d6e12051bc3bc724,61ca87cdf9b9f828] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644085, data is empty. 2024-09-11 17:33:38.851 INFO 2981023 --- [alarm-detect-5,d6e12051bc3bc724,61ca87cdf9b9f828] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644085, data is empty. 2024-09-11 17:33:38.859 INFO 2981023 --- [alarm-detect-5,d6e12051bc3bc724,61ca87cdf9b9f828] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644085, data is empty. 2024-09-11 17:33:38.872 INFO 2981023 --- [alarm-detect-5,d6e12051bc3bc724,603144ecc5e0bbcf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644085, data is empty. 2024-09-11 17:33:38.873 INFO 2981023 --- [alarm-detect-14,d6e12051bc3bc724,368d3f87f26c59db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644085, data is empty. 2024-09-11 17:33:38.881 INFO 2981023 --- [alarm-detect-5,d6e12051bc3bc724,603144ecc5e0bbcf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644085, data is empty. 2024-09-11 17:33:38.881 INFO 2981023 --- [alarm-detect-14,d6e12051bc3bc724,368d3f87f26c59db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644085, data is empty. 2024-09-11 17:33:38.886 INFO 2981023 --- [alarm-detect-5,d6e12051bc3bc724,603144ecc5e0bbcf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644085, data is empty. 2024-09-11 17:33:38.887 INFO 2981023 --- [alarm-detect-14,d6e12051bc3bc724,368d3f87f26c59db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644085, data is empty. 2024-09-11 17:33:39.004 INFO 2981023 --- [alarm-detect-12,d6e12051bc3bc724,5691bc25dd1b1247] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644085, data is empty. 2024-09-11 17:33:39.007 INFO 2981023 --- [alarm-detect-12,d6e12051bc3bc724,5691bc25dd1b1247] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644085, data is empty. 2024-09-11 17:33:39.012 INFO 2981023 --- [alarm-detect-12,d6e12051bc3bc724,5691bc25dd1b1247] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644085, data is empty. 2024-09-11 17:33:39.112 INFO 2981023 --- [ocp-schedules-1,8130d705d1cf69fb,64810228daadf35a] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:33:39.132 WARN 2981023 --- [ocp-schedules-1,8130d705d1cf69fb,64810228daadf35a] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:33:39.181 INFO 2981023 --- [ocp-async-4,683faa4b4ffeb522,6009d5b46cfde49c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:40.003 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:33:40.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:33:41.177 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:41.178 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:41.181 INFO 2981023 --- [ocp-async-8,94f1e76f1a751e38,9325043d9a788f08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,5749e29c89554804,f601d85736d71f43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,5749e29c89554804,f601d85736d71f43] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:41.919 INFO 2981023 --- [ocp-schedules-8,6cb6b4209663dd67,d2dd13f1c6182168] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:33:43.181 INFO 2981023 --- [ocp-async-9,c49d1adbc9053008,5a257b9e6bb9c528] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:44.178 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:44.179 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,3049a13ea13efa11,589d2451987319fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,3049a13ea13efa11,589d2451987319fb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:45.101 INFO 2981023 --- [prometheus-second-schedule-1,3c38d09346a42e46,648f90221a64cba9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:33:45.181 INFO 2981023 --- [ocp-async-2,56c90cc90db6895d,f2fdb62c91af9a28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:45.463 INFO 2981023 --- [ocp-schedules-15,fc3b97414256f2ee,969e5b420980e20e] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:33:46.019 INFO 2981023 --- [ocp-schedules-30,bc9326ab76cce52b,3c78d59756b0a952] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:33:46.019 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:33:46.019 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:33:46.020 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:33:46.020 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:33:46.061 INFO 2981023 --- [ocp-schedules-20,33d87b72cc68fc07,6e784154abb0bc97] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:33:46.062 INFO 2981023 --- [ocp-schedules-6,7fd8600332c23631,81546aa3b552b8f1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:33:46.062 INFO 2981023 --- [host-check-agent-status1,33d87b72cc68fc07,ea869ff2890e4439] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:33:46.062 INFO 2981023 --- [host-check-agent-status2,33d87b72cc68fc07,077545223fe02bd6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:33:46.062 INFO 2981023 --- [host-check-agent-status3,33d87b72cc68fc07,46026476bd515754] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:33:46.066 INFO 2981023 --- [host-check-agent-status4,33d87b72cc68fc07,2eb89d2f4945a19b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:33:46.066 INFO 2981023 --- [host-check-clockdiff1,7fd8600332c23631,6d7cc2fcbbbc5441] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:33:46.067 INFO 2981023 --- [host-check-clockdiff2,7fd8600332c23631,dafe679afff56750] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:33:46.068 INFO 2981023 --- [host-check-clockdiff3,7fd8600332c23631,aa2497bc5e0ac9a4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:33:46.070 INFO 2981023 --- [host-check-clockdiff4,7fd8600332c23631,607f86558b78f454] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:33:46.070 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:33:46.075 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:33:46.078 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:33:46.117 INFO 2981023 --- [host-check-agent-status1,33d87b72cc68fc07,ea869ff2890e4439] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:33:46.118 INFO 2981023 --- [host-check-agent-status1,33d87b72cc68fc07,ea869ff2890e4439] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:33:46.120 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:33:46.120 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:33:46.121 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:33:46.122 INFO 2981023 --- [host-check-agent-status4,33d87b72cc68fc07,2eb89d2f4945a19b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:33:46.124 INFO 2981023 --- [host-check-agent-status2,33d87b72cc68fc07,077545223fe02bd6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:33:46.125 INFO 2981023 --- [host-check-agent-status2,33d87b72cc68fc07,077545223fe02bd6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:33:46.127 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:33:46.127 INFO 2981023 --- [host-check-agent-status4,33d87b72cc68fc07,2eb89d2f4945a19b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:33:46.142 INFO 2981023 --- [host-check-agent-status3,33d87b72cc68fc07,46026476bd515754] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:33:46.144 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:33:46.150 INFO 2981023 --- [host-check-clockdiff4,7fd8600332c23631,607f86558b78f454] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047226 0 0 , error= 2024-09-11 17:33:46.181 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:33:46.188 INFO 2981023 --- [ocp-schedules-22,a02d2cd47c8aa48b,09bc94b84042ad21] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:33:46.189 INFO 2981023 --- [ocp-schedules-14,2486fdde0898c4c8,d678582949758749] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:33:46.191 INFO 2981023 --- [ocp-schedules-27,5dfed053edf84022,207a3c9a35a7b3da] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:33:46.191 INFO 2981023 --- [ocp-schedules-12,161b32afee7a68c3,740d3c0f8ec92864] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:33:46.272 INFO 2981023 --- [ocp-schedules-27,5dfed053edf84022,207a3c9a35a7b3da] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:33:46.272 INFO 2981023 --- [host-check-agent-status3,33d87b72cc68fc07,46026476bd515754] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:33:46.287 INFO 2981023 --- [ocp-schedules-22,a02d2cd47c8aa48b,09bc94b84042ad21] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:33:46.295 INFO 2981023 --- [ocp-schedules-10,f017677a7afd0883,04ba05b270178135] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:33:46.296 INFO 2981023 --- [ocp-schedules-14,2486fdde0898c4c8,d678582949758749] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:33:46.327 INFO 2981023 --- [ocp-schedules-14,2486fdde0898c4c8,d678582949758749] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:33:46.357 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:33:46.363 INFO 2981023 --- [host-check-agent-status2,33d87b72cc68fc07,077545223fe02bd6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:33:46.368 INFO 2981023 --- [host-check-agent-status2,33d87b72cc68fc07,077545223fe02bd6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:33:46.368 INFO 2981023 --- [host-check-agent-status2,33d87b72cc68fc07,077545223fe02bd6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:33:46.368 INFO 2981023 --- [host-check-agent-status2,33d87b72cc68fc07,077545223fe02bd6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:33:46.370 INFO 2981023 --- [ocp-schedules-4,e844b0843b013e1a,b514658b2a167a8f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:33:46.387 INFO 2981023 --- [ocp-schedules-10,f017677a7afd0883,04ba05b270178135] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:33:46.406 INFO 2981023 --- [host-check-agent-status1,33d87b72cc68fc07,ea869ff2890e4439] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:33:46.410 INFO 2981023 --- [host-check-agent-status1,33d87b72cc68fc07,ea869ff2890e4439] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:33:46.410 INFO 2981023 --- [host-check-agent-status1,33d87b72cc68fc07,ea869ff2890e4439] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:33:46.410 INFO 2981023 --- [host-check-agent-status1,33d87b72cc68fc07,ea869ff2890e4439] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:33:46.457 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:33:46.461 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:33:46.461421945+08:00 2024-09-11 17:33:46.464 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:33:46.462+08:00 2024-09-11 17:33:46.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:33:46.463605+08:00, lastProcessTime=2024-09-11T17:33:36.355163+08:00 2024-09-11 17:33:46.468 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:33:46.471 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:33:46.471088813+08:00 2024-09-11 17:33:46.471 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:33:46.462+08:00 and status AVAILABLE 2024-09-11 17:33:46.472 INFO 2981023 --- [host-check-agent-status4,33d87b72cc68fc07,2eb89d2f4945a19b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:33:46.473 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:33:46.471+08:00 2024-09-11 17:33:46.477 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:33:46.471+08:00 and status AVAILABLE 2024-09-11 17:33:46.477 INFO 2981023 --- [host-check-agent-status3,33d87b72cc68fc07,46026476bd515754] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:33:46.477 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:33:46.478 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:33:46.477974588+08:00 2024-09-11 17:33:46.478 INFO 2981023 --- [host-check-agent-status4,33d87b72cc68fc07,2eb89d2f4945a19b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:33:46.478 INFO 2981023 --- [host-check-agent-status4,33d87b72cc68fc07,2eb89d2f4945a19b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:33:46.478 INFO 2981023 --- [host-check-agent-status4,33d87b72cc68fc07,2eb89d2f4945a19b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:33:46.479 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:33:46.478+08:00 2024-09-11 17:33:46.480 INFO 2981023 --- [host-check-agent-status3,33d87b72cc68fc07,46026476bd515754] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:33:46.480 INFO 2981023 --- [host-check-agent-status3,33d87b72cc68fc07,46026476bd515754] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:33:46.480 INFO 2981023 --- [host-check-agent-status3,33d87b72cc68fc07,46026476bd515754] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:33:46.481 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:33:46.478+08:00 and status AVAILABLE 2024-09-11 17:33:46.484 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:33:46.485 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:33:46.486 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.486 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:33:46.486062224+08:00 2024-09-11 17:33:46.487 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:46.487 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:33:46.488 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:33:46.486+08:00 2024-09-11 17:33:46.491 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:33:46.486+08:00 and status AVAILABLE 2024-09-11 17:33:46.495 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.495 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.496 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:46.497 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:33:46.498 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.498 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:46.499 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:33:46.499 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:33:46.499 INFO 2981023 --- [ocp-schedules-22,a02d2cd47c8aa48b,09bc94b84042ad21] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:33:46.499 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:33:46.500 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.505 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:33:46.508 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:33:46.515 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:33:46.516 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:46.517 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:33:46.521 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:33:46.522 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.522 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:33:46.523 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:46.554 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:33:46.557 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:33:46.559 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:33:46.562 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.562 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:46.562 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:33:46.563 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:33:46.563 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:33:46.584 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:33:46.586 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:33:46.595 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.600 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:33:46.605 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.606 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.606 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:33:46.607 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:33:46.608 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:33:46.608 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.610 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:33:46.619 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:33:46.623 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:33:46.635 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:33:46.641 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:33:46.644 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:33:46.646 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:33:46.655 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:33:46.661 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:33:46.667 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.670 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.670 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:33:46.670 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:33:46.672 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:33:46.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:46.677 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:33:46.681 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.684 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:33:46.685 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:33:46.686 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.700 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.700 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.701 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:33:46.701 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:33:46.702 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.702 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.703 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:46.704 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.705 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.705 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:33:46.706 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:33:46.706 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:33:46.706 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.706 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:33:46.712 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:33:46.719 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:46.719 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:33:46.733 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:46.733 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:33:46.734 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:33:46.739 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:33:46.743 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:33:46.753 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:46.754 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:33:46.759 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:33:46.760 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:33:46.761 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:33:46.766 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.768 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.768 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:46.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:46.769 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:33:46.771 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:33:46.774 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:33:46.777 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:33:46.779 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:33:46.779 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:33:46.780 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:33:46.781 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:33:46.784 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.785 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.785 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.786 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:46.787 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:33:46.787 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.788 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.788 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.788 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:33:46.789 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:33:46.789 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:33:46.789 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.790 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:33:46.791 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:33:46.791 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:33:46.806 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:33:46.808 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:33:46.820 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:33:46.822 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:33:46.824 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:33:46.825 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:33:46.830 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:33:46.836 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:33:46.837 INFO 2981023 --- [ocp-schedules-26,fc03e5bc98a10616,d115285a9cd371ae] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:33:46.837 INFO 2981023 --- [ocp-schedules-26,fc03e5bc98a10616,d115285a9cd371ae] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:33:46.844 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:33:46.847 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:33:46.849 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:33:46.851 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:33:46.851 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.855 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.858 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.858 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:33:46.859 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:33:46.860 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:33:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.860 INFO 2981023 --- [host-check-host-status1,bc9326ab76cce52b,d07b0050124d81f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:33:46.861 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:33:46.861 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.862 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:33:46.863 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:33:46.864 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:33:46.869 INFO 2981023 --- [host-check-host-status2,bc9326ab76cce52b,b66e111a2d2ac42a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:33:46.876 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:33:46.881 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:46.881 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:33:46.888 INFO 2981023 --- [host-check-host-status3,bc9326ab76cce52b,5246aeb3cd327e83] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:33:46.890 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:33:46.891 INFO 2981023 --- [host-check-host-status4,bc9326ab76cce52b,2ed9581c6f78f1ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:33:46.896 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:33:46.463605+08:00 2024-09-11 17:33:46.900 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:33:46.902 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.906 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.907 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:33:46.909 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.911 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.911 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:33:46.912 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:33:46.915 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:33:46.915 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:33:46.921 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:33:46.924 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:33:46.926 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:33:46.929 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:33:46.948 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:33:46.949 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:33:46.955 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:33:46.958 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:33:46.961 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.961 INFO 2981023 --- [ocp-schedules-25,bc1b70d95fd4c420,b4be8ca3d45d258c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:33:46.961 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.962 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:33:46.967 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:33:46.967 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:33:46.981 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:33:46.986 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:46.986 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:46.987 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:33:46.992 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:33:46.992 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:33:46.992 INFO 2981023 --- [ocp-schedules-27,5dfed053edf84022,207a3c9a35a7b3da] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:33:47.179 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:47.179 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:47.181 INFO 2981023 --- [ocp-async-3,d9946eafe5543ec1,c09d4c2f25831a1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,3bf9a0f38aea5a7d,563477385303bc08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,3bf9a0f38aea5a7d,563477385303bc08] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:47.751 INFO 2981023 --- [ocp-async-6,0aecf6fe1eba5335,5a8a261bbeb0accf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:33:47.874 INFO 2981023 --- [host-check-clockdiff1,7fd8600332c23631,6d7cc2fcbbbc5441] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047227 0 0 , error= 2024-09-11 17:33:49.051 INFO 2981023 --- [ocp-schedules-23,66341afa29e9f1c5,a7f4b099e0b6463e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:33:49.084 WARN 2981023 --- [scanner-13505314-1,66341afa29e9f1c5,d4395a7c1b2c954d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:33:49.183 INFO 2981023 --- [ocp-async-5,dc43e6bcb91e02ad,229f483861cbfe68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:49.238 INFO 2981023 --- [host-check-clockdiff2,7fd8600332c23631,dafe679afff56750] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047229 0 0 , error= 2024-09-11 17:33:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:33:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:33:50.179 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:50.181 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:50.221 INFO 2981023 --- [ocp-schedules-27,0594ef28fdb015fc,dd0b78e1f02ee30d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:33:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,c1026aacc09cc419,4f9dbd78a1b05afe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,c1026aacc09cc419,4f9dbd78a1b05afe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:51.183 INFO 2981023 --- [ocp-async-4,8e47dc2e187b9515,af9706a7ef37bebf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:51.535 INFO 2981023 --- [ocp-schedules-10,d95d383374ea1848,6d41a88e5f2f43d8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:33:51.536 INFO 2981023 --- [ocp-schedules-10,d95d383374ea1848,6d41a88e5f2f43d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:33:51.536 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:33:51.536 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:33:51.536 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:33:51.537 INFO 2981023 --- [ocp-schedules-10,d95d383374ea1848,6d41a88e5f2f43d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:33:51.922 INFO 2981023 --- [ocp-schedules-24,72073931b187dd50,e3789eb75b6ebc83] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:33:52.079 INFO 2981023 --- [host-check-clockdiff3,7fd8600332c23631,aa2497bc5e0ac9a4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047232 1 1 , error= 2024-09-11 17:33:52.126 INFO 2981023 --- [ocp-schedules-23,413d01d0a80a4e9c,9ac112f0d317e787] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:33:52.126+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:33:52.131 INFO 2981023 --- [ocp-schedules-23,413d01d0a80a4e9c,9ac112f0d317e787] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:33:52.266 INFO 2981023 --- [ocp-schedules-23,413d01d0a80a4e9c,9ac112f0d317e787] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 50 to SqlAggregateData 2024-09-11 17:33:52.267 INFO 2981023 --- [ocp-schedules-23,413d01d0a80a4e9c,9ac112f0d317e787] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:33:52.631 INFO 2981023 --- [ocp-schedules-11,130fa229403c1d63,a1082d9880d97997] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:33:52.649 WARN 2981023 --- [ocp-schedules-11,130fa229403c1d63,a1082d9880d97997] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:33:52.814 INFO 2981023 --- [ocp-schedules-30,f0fc08c66a72c8e4,7907af6f4b51d21d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:33:52.815 INFO 2981023 --- [ocp-schedules-30,f0fc08c66a72c8e4,7907af6f4b51d21d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:33:52.851 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:33:52.851 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:33:52.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.854 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:33:52.854 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.854 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:33:52.856 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:33:52.858 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:33:52.863 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.864 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.864 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.864 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.865 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.865 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.872 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.872 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.875 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.875 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.876 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.877 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.878 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.878 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:33:52.886 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.887 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:33:52.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.890 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.890 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:33:52.890 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2631D6DB-0-0] 2024-09-11 17:33:52.891 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:33:52.891 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.891 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.891 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:33:52.891 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.893 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.893 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:33:52.904 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.904 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:33:52.905 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.905 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:33:52.906 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.913 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:33:52.914 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.915 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:33:52.915 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.916 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:33:52.916 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:33:52.917 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.917 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:33:52.918 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:33:52.918 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2631D6E5-0-0] 2024-09-11 17:33:52.918 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:33:52.920 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:33:52.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:33:52.922 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.922 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:33:52.923 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.923 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:33:52.924 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:33:52.924 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.925 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:33:52.925 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.926 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:33:52.927 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.927 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:33:52.928 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.928 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:33:52.929 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.929 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:33:52.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:33:52.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.930 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:33:52.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:33:52.931 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.931 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:33:52.932 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:33:52.932 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:33:52.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.934 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:33:52.935 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.935 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:33:52.935 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:33:52.937 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:33:52.944 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:52.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.946 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:33:52.950 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:52.952 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.952 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:33:52.954 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:33:52.955 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:33:52.956 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.957 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.960 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:33:52.964 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:33:52.964 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.964 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:33:52.969 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:33:52.992 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:33:52.993 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:33:52.993 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.993 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:33:52.993 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:52.993 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.994 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:33:52.995 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:33:52.995 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:33:52.995 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:33:52.999 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:33:53.001 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:33:53.001 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:33:53.001 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:33:53.005 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:33:53.007 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:33:53.007 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:33:53.007 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:33:53.007 INFO 2981023 --- [ocp-schedules-30,f0fc08c66a72c8e4,7907af6f4b51d21d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:33:53.072 INFO 2981023 --- [ocp-schedules-5,3314a77136344c43,d0320313894a7da0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:33:53.072 INFO 2981023 --- [ocp-schedules-5,3314a77136344c43,d0320313894a7da0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:33:53.181 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:53.182 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:53.183 INFO 2981023 --- [ocp-async-3,e2cf67a6e4fc4b04,95df5a8714cf2c4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,19f30a3e9e192dac,d0dba1f5a2e292a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,19f30a3e9e192dac,d0dba1f5a2e292a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:53.807 INFO 2981023 --- [ocp-schedules-28,54f55a34ceb4071b,f4d0fcfd2f9fd793] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:33:53.815 INFO 2981023 --- [ocp-schedules-28,54f55a34ceb4071b,f4d0fcfd2f9fd793] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:33:53.815 INFO 2981023 --- [ocp-schedules-28,54f55a34ceb4071b,f4d0fcfd2f9fd793] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:33:54.209 INFO 2981023 --- [ocp-schedules-26,4ce5797367bcd7a1,b14686149a59ffe1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:33:54.559 INFO 2981023 --- [ocp-schedules-15,56904fcce830ac0c,d16ab6e1dcf67c30] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T17:33:54.559+08:00, tenantSize=6 2024-09-11 17:33:54.562 INFO 2981023 --- [ocp-schedules-15,56904fcce830ac0c,d16ab6e1dcf67c30] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726047234559000 2024-09-11 17:33:54.562 INFO 2981023 --- [ocp-schedules-15,56904fcce830ac0c,d16ab6e1dcf67c30] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726046934535000 2024-09-11 17:33:54.577 INFO 2981023 --- [ocp-schedules-15,56904fcce830ac0c,d16ab6e1dcf67c30] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 17:33:54.743 INFO 2981023 --- [ocp-schedules-5,eb2cfa88b207c979,dcb2853ca0906bae] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:33:54.743 INFO 2981023 --- [ocp-schedules-5,eb2cfa88b207c979,dcb2853ca0906bae] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:33:54.748 INFO 2981023 --- [ocp-schedules-5,eb2cfa88b207c979,dcb2853ca0906bae] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:33:54.748 INFO 2981023 --- [ocp-schedules-5,eb2cfa88b207c979,dcb2853ca0906bae] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:33:54.748 INFO 2981023 --- [ocp-schedules-5,eb2cfa88b207c979,dcb2853ca0906bae] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:33:54.751 INFO 2981023 --- [ocp-schedules-5,eb2cfa88b207c979,dcb2853ca0906bae] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:33:54.751 INFO 2981023 --- [ocp-schedules-5,eb2cfa88b207c979,dcb2853ca0906bae] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:33:54.751 INFO 2981023 --- [ocp-schedules-5,eb2cfa88b207c979,dcb2853ca0906bae] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:33:54.760 INFO 2981023 --- [ocp-schedules-5,eb2cfa88b207c979,dcb2853ca0906bae] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:33:55.183 INFO 2981023 --- [ocp-async-2,b65d4a9a6e590eda,f9870a031959d38c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:55.661 INFO 2981023 --- [ocp-schedules-9,a0e91c4fa125282a,726fd18033aa3a21] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:33:55.661 INFO 2981023 --- [ocp-schedules-9,a0e91c4fa125282a,726fd18033aa3a21] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:33:55.666 INFO 2981023 --- [ocp-schedules-9,a0e91c4fa125282a,726fd18033aa3a21] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:33:55.666 INFO 2981023 --- [ocp-schedules-9,a0e91c4fa125282a,726fd18033aa3a21] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:33:55.671 INFO 2981023 --- [ocp-schedules-9,a0e91c4fa125282a,726fd18033aa3a21] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:33:55.671 INFO 2981023 --- [ocp-schedules-9,a0e91c4fa125282a,726fd18033aa3a21] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:33:55.750 INFO 2981023 --- [ocp-schedules-26,0ac9721d041309fd,1589a13d61a4aa55] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:33:55.756 INFO 2981023 --- [ocp-schedules-26,0ac9721d041309fd,1589a13d61a4aa55] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:33:55.756 INFO 2981023 --- [ocp-schedules-26,0ac9721d041309fd,1589a13d61a4aa55] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:33:56.083 INFO 2981023 --- [ocp-schedules-17,f15e9710b4ec1365,90ffc2097cc3a2c7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:33:56.083 INFO 2981023 --- [ocp-schedules-17,f15e9710b4ec1365,90ffc2097cc3a2c7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:33:56.096 INFO 2981023 --- [Collect all tenants major compactions1,f15e9710b4ec1365,6fa4e801390b1034] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:33:56.097 INFO 2981023 --- [Collect all tenants major compactions2,f15e9710b4ec1365,18c1339e2e9279e5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:33:56.099 INFO 2981023 --- [Collect all tenants major compactions5,f15e9710b4ec1365,e6dfd69ad1aed3c9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:33:56.100 INFO 2981023 --- [Collect all tenants major compactions6,f15e9710b4ec1365,a1a2396d81514885] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:33:56.100 INFO 2981023 --- [Collect all tenants major compactions4,f15e9710b4ec1365,41f539286da03bd8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:33:56.100 INFO 2981023 --- [Collect all tenants major compactions3,f15e9710b4ec1365,03df340fedcd5fdd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:33:56.115 INFO 2981023 --- [Collect all tenants major compactions1,f15e9710b4ec1365,6fa4e801390b1034] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:56.115 INFO 2981023 --- [Collect all tenants major compactions1,f15e9710b4ec1365,6fa4e801390b1034] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:33:56.116 INFO 2981023 --- [Collect all tenants major compactions6,f15e9710b4ec1365,a1a2396d81514885] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:56.117 INFO 2981023 --- [Collect all tenants major compactions2,f15e9710b4ec1365,18c1339e2e9279e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:56.117 INFO 2981023 --- [Collect all tenants major compactions5,f15e9710b4ec1365,e6dfd69ad1aed3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:56.118 INFO 2981023 --- [Collect all tenants major compactions6,f15e9710b4ec1365,a1a2396d81514885] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:33:56.118 INFO 2981023 --- [Collect all tenants major compactions5,f15e9710b4ec1365,e6dfd69ad1aed3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:33:56.119 INFO 2981023 --- [Collect all tenants major compactions2,f15e9710b4ec1365,18c1339e2e9279e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:33:56.120 INFO 2981023 --- [Collect all tenants major compactions4,f15e9710b4ec1365,41f539286da03bd8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:56.120 INFO 2981023 --- [Collect all tenants major compactions1,f15e9710b4ec1365,6fa4e801390b1034] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:33:56.121 INFO 2981023 --- [Collect all tenants major compactions6,f15e9710b4ec1365,a1a2396d81514885] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:33:56.121 INFO 2981023 --- [Collect all tenants major compactions2,f15e9710b4ec1365,18c1339e2e9279e5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:33:56.121 INFO 2981023 --- [Collect all tenants major compactions3,f15e9710b4ec1365,03df340fedcd5fdd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:33:56.122 INFO 2981023 --- [Collect all tenants major compactions5,f15e9710b4ec1365,e6dfd69ad1aed3c9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:33:56.122 INFO 2981023 --- [Collect all tenants major compactions4,f15e9710b4ec1365,41f539286da03bd8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:33:56.123 INFO 2981023 --- [Collect all tenants major compactions3,f15e9710b4ec1365,03df340fedcd5fdd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:33:56.124 INFO 2981023 --- [Collect all tenants major compactions4,f15e9710b4ec1365,41f539286da03bd8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:33:56.125 INFO 2981023 --- [Collect all tenants major compactions6,f15e9710b4ec1365,a1a2396d81514885] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:33:56.126 INFO 2981023 --- [Collect all tenants major compactions3,f15e9710b4ec1365,03df340fedcd5fdd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:33:56.126 INFO 2981023 --- [Collect all tenants major compactions3,f15e9710b4ec1365,03df340fedcd5fdd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:33:56.126 INFO 2981023 --- [Collect all tenants major compactions1,f15e9710b4ec1365,6fa4e801390b1034] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:33:56.128 INFO 2981023 --- [Collect all tenants major compactions2,f15e9710b4ec1365,18c1339e2e9279e5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:33:56.129 INFO 2981023 --- [Collect all tenants major compactions5,f15e9710b4ec1365,e6dfd69ad1aed3c9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:33:56.130 INFO 2981023 --- [Collect all tenants major compactions4,f15e9710b4ec1365,41f539286da03bd8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:33:56.130 INFO 2981023 --- [ocp-schedules-17,f15e9710b4ec1365,90ffc2097cc3a2c7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:33:56.182 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:56.182 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,999e7a5d9fc0b6aa,d4a9ff438f20794f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,999e7a5d9fc0b6aa,d4a9ff438f20794f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:33:56.350822+08:00, lastProcessTime=2024-09-11T17:33:46.463605+08:00 2024-09-11 17:33:56.483 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:33:56.350822+08:00 2024-09-11 17:33:57.183 INFO 2981023 --- [ocp-async-6,570b2ed7e72e5c95,ae8478fa930afd0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:57.751 INFO 2981023 --- [ocp-async-5,f1871d5e8edd3e8c,37528978f7f4f542] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:33:59.183 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:33:59.183 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:33:59.183 INFO 2981023 --- [ocp-async-3,94e200d2ec390a16,6fcd02a2cb0eee89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:33:59.258 INFO 2981023 --- [ocp-schedules-18,7d59ea3a3a290371,ff289e5f2c77ede0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:33:59.295 WARN 2981023 --- [scanner-13505546-1,7d59ea3a3a290371,31d6f2d676797213] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:33:59.314 INFO 2981023 --- [prometheus-minute-schedule-1,e0fd86cd3cbbf95f,255661074b755cb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047240, interval=60 2024-09-11 17:33:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,e0fd86cd3cbbf95f,255661074b755cb9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:33:59.319 INFO 2981023 --- [alarm-detect-15,7d59ea3a3a290371,4604883b185219e8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644085, data is empty. 2024-09-11 17:33:59.331 INFO 2981023 --- [alarm-detect-15,7d59ea3a3a290371,4604883b185219e8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644135, data is empty. 2024-09-11 17:33:59.338 INFO 2981023 --- [alarm-detect-15,7d59ea3a3a290371,4604883b185219e8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644135, data is empty. 2024-09-11 17:33:59.357 INFO 2981023 --- [alarm-detect-6,7d59ea3a3a290371,a1ee1a2e54e1e7d9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644085, data is empty. 2024-09-11 17:33:59.357 INFO 2981023 --- [alarm-detect-18,7d59ea3a3a290371,e29ec00122801a16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644085, data is empty. 2024-09-11 17:33:59.363 INFO 2981023 --- [alarm-detect-18,7d59ea3a3a290371,e29ec00122801a16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644135, data is empty. 2024-09-11 17:33:59.363 INFO 2981023 --- [alarm-detect-6,7d59ea3a3a290371,a1ee1a2e54e1e7d9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644135, data is empty. 2024-09-11 17:33:59.366 INFO 2981023 --- [alarm-detect-6,7d59ea3a3a290371,a1ee1a2e54e1e7d9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644135, data is empty. 2024-09-11 17:33:59.367 INFO 2981023 --- [alarm-detect-18,7d59ea3a3a290371,e29ec00122801a16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644135, data is empty. 2024-09-11 17:33:59.476 INFO 2981023 --- [alarm-detect-4,7d59ea3a3a290371,c1c79530f84f8ea4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644085, data is empty. 2024-09-11 17:33:59.481 INFO 2981023 --- [alarm-detect-4,7d59ea3a3a290371,c1c79530f84f8ea4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644135, data is empty. 2024-09-11 17:33:59.486 INFO 2981023 --- [alarm-detect-4,7d59ea3a3a290371,c1c79530f84f8ea4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644135, data is empty. 2024-09-11 17:34:01.184 INFO 2981023 --- [ocp-async-2,20efe82dc5abdca5,f045781777fc5c7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:01.925 INFO 2981023 --- [ocp-schedules-8,cca43de9cdf5f915,1552bd1c99ff212c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:34:02.020 INFO 2981023 --- [ocp-schedules-13,615271446d6129b2,9bf2e4515c4e045f] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:34:02.020 INFO 2981023 --- [ocp-schedules-13,615271446d6129b2,9bf2e4515c4e045f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:34:02.183 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:02.184 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,3c399bd93013a618,f47aa7fceaecc10e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,3c399bd93013a618,f47aa7fceaecc10e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:02.438 INFO 2981023 --- [ocp-schedules-25,b5c7340f3a0ec517,3e748508fe4c1419] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:34:02.439 INFO 2981023 --- [ocp-schedules-25,b5c7340f3a0ec517,3e748508fe4c1419] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:34:02.448 INFO 2981023 --- [ocp-schedules-25,b5c7340f3a0ec517,3e748508fe4c1419] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:34:03.183 INFO 2981023 --- [ocp-async-2,08802d985311a1cd,587d59f45cd2d8c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:05.184 INFO 2981023 --- [ocp-async-10,ac8edf4cfb7e75d0,544497f53f6cfde6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:05.184 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:05.185 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,62b6e3b5c3d0ae33,d72d296b802e54af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,62b6e3b5c3d0ae33,d72d296b802e54af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:34:05.971 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:05.971 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:34:05.972 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:34:05.984 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:34:05.989 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:34:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:34:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:34:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:34:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:34:06.003 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:06.004 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:06.004 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:34:06.005 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:34:06.005 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:06.006 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:06.006 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:06.006 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:06.063 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:31:40, start=2024-09-11T17:27:41, end=2024-09-11T17:30:11, updates=112, granule=120000ms, delay=239000ms 2024-09-11 17:34:06.066 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:31:41, start=2024-09-11T17:27:41, end=2024-09-11T17:30:11, updates=307, granule=120000ms, delay=240000ms 2024-09-11 17:34:06.072 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:31:40, target=2024-09-11T17:30:10, granule=120000ms, delay=90000ms) 2024-09-11 17:34:06.072 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:30:10, target=2024-09-11T17:19:41, granule=600000ms, delay=629000ms) 2024-09-11 17:34:06.075 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:31:41, target=2024-09-11T17:30:10, granule=120000ms, delay=91000ms) 2024-09-11 17:34:06.075 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:30:10, target=2024-09-11T17:20:51, granule=600000ms, delay=559000ms) 2024-09-11 17:34:06.075 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=103ms, period=120000ms 2024-09-11 17:34:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:34:06.337350+08:00, lastProcessTime=2024-09-11T17:33:56.350822+08:00 2024-09-11 17:34:06.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:34:06.337350+08:00 2024-09-11 17:34:07.184 INFO 2981023 --- [ocp-async-9,5f3f7fe4e42b538a,f2091bd2f750d6ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:07.312 INFO 2981023 --- [ocp-schedules-17,37150b9520bf853b,20a913db632078ce] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:34:07.320 INFO 2981023 --- [ocp-schedules-5,00c394b46fbee399,5912da31dd0d9c92] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:34:07.320+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:34:07.320 INFO 2981023 --- [ocp-schedules-5,00c394b46fbee399,5912da31dd0d9c92] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:34:07.322 INFO 2981023 --- [ocp-schedules-5,00c394b46fbee399,5912da31dd0d9c92] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:34:07.322 INFO 2981023 --- [ocp-schedules-5,00c394b46fbee399,5912da31dd0d9c92] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:34:07.751 INFO 2981023 --- [ocp-async-3,4d4fb7ed01c72b49,7858504b1d1bbfaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:34:08.185 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:08.186 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,386f54b62a50629a,636c9ac30c766b45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,386f54b62a50629a,636c9ac30c766b45] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:09.184 INFO 2981023 --- [ocp-async-6,c2616c56943885ab,428ea49d28369a27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:09.520 INFO 2981023 --- [ocp-schedules-13,73f5188b1eb98534,c94dca2f2716aea5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:34:09.553 WARN 2981023 --- [scanner-13505823-1,73f5188b1eb98534,fdda25af5dfa1a17] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:34:10.001 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:34:10.002 INFO 2981023 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:34:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:34:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:34:11.184 INFO 2981023 --- [ocp-async-10,1f29e45d848bad41,3a0f6a1aac283e99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:11.186 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:11.187 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,8e901e781b2085cf,da130cf7c05e1999] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,8e901e781b2085cf,da130cf7c05e1999] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:11.699 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047180, nodeCount=1000, affectRows=1000 2024-09-11 17:34:11.703 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047180, nodeCount=1000, validCount=1000 2024-09-11 17:34:11.707 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047180, nodeCount=1000, affectRows=1000 2024-09-11 17:34:11.710 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047180, nodeCount=999, validCount=999 2024-09-11 17:34:11.725 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047180, nodeCount=49, affectRows=49 2024-09-11 17:34:11.726 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047180, nodeCount=49, validCount=49 2024-09-11 17:34:11.928 INFO 2981023 --- [ocp-schedules-9,dd645389e75fc172,0b3601931a5d4303] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:34:13.184 INFO 2981023 --- [ocp-async-1,95b1440f61771532,2655840743804803] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:13.797 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047180, nodeCount=480, affectRows=480 2024-09-11 17:34:13.817 INFO 2981023 --- [ocp-schedules-25,b7fe56d1c3780262,09af5301315fc9da] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:34:13.828 INFO 2981023 --- [ocp-schedules-25,b7fe56d1c3780262,09af5301315fc9da] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:34:13.828 INFO 2981023 --- [ocp-schedules-25,b7fe56d1c3780262,09af5301315fc9da] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:34:14.187 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:14.188 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,756a240866ed9795,19499356ccec0358] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,6729dfb6224f5322,cc7051c29327ed5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:34:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,756a240866ed9795,19499356ccec0358] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:15.184 INFO 2981023 --- [ocp-async-8,742a02390ecd7d1b,93ed51a4f5b1939c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:15.536 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047180, nodeCount=480, validCount=480 2024-09-11 17:34:15.997 INFO 2981023 --- [ocp-schedules-24,5e1b797759212617,9322a6f99abe0261] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:34:15.998 INFO 2981023 --- [host-check-clockdiff1,5e1b797759212617,4c6a2c0d5c7acd31] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:34:15.998 INFO 2981023 --- [host-check-clockdiff2,5e1b797759212617,348ac489c90025a2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:34:15.999 INFO 2981023 --- [host-check-clockdiff4,5e1b797759212617,923ef4cd5946daec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:34:15.999 INFO 2981023 --- [host-check-clockdiff3,5e1b797759212617,afcfd5d971261972] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:34:16.006 INFO 2981023 --- [ocp-schedules-22,27d66fff2aa00fb8,9a2c35d280f2d0e8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:34:16.007 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:34:16.007 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:34:16.007 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:34:16.012 INFO 2981023 --- [host-check-clockdiff4,5e1b797759212617,923ef4cd5946daec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047256 0 0 , error= 2024-09-11 17:34:16.012 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:34:16.013 INFO 2981023 --- [host-check-clockdiff2,5e1b797759212617,348ac489c90025a2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047256 0 0 , error= 2024-09-11 17:34:16.017 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:16.017 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:34:16.017 INFO 2981023 --- [ocp-schedules-26,ec0a89e74ebbf98d,1a765cf324fd675d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:34:16.018 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:16.018 INFO 2981023 --- [host-check-agent-status1,ec0a89e74ebbf98d,6f1b0e0ad96be2b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:34:16.021 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:34:16.022 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:16.022 INFO 2981023 --- [host-check-agent-status2,ec0a89e74ebbf98d,0d5f7934cb9e5033] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:34:16.022 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:16.022 INFO 2981023 --- [host-check-agent-status3,ec0a89e74ebbf98d,3bfa50420577a415] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:34:16.026 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:16.028 INFO 2981023 --- [host-check-agent-status4,ec0a89e74ebbf98d,1c1c7af863cb5e42] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:34:16.030 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:16.031 INFO 2981023 --- [host-check-agent-status1,ec0a89e74ebbf98d,6f1b0e0ad96be2b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:16.035 INFO 2981023 --- [host-check-agent-status3,ec0a89e74ebbf98d,3bfa50420577a415] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:34:16.035 INFO 2981023 --- [host-check-agent-status1,ec0a89e74ebbf98d,6f1b0e0ad96be2b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:16.036 INFO 2981023 --- [host-check-agent-status3,ec0a89e74ebbf98d,3bfa50420577a415] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:34:16.038 INFO 2981023 --- [host-check-agent-status4,ec0a89e74ebbf98d,1c1c7af863cb5e42] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:16.039 INFO 2981023 --- [host-check-agent-status2,ec0a89e74ebbf98d,0d5f7934cb9e5033] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:16.043 INFO 2981023 --- [host-check-agent-status2,ec0a89e74ebbf98d,0d5f7934cb9e5033] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:16.043 INFO 2981023 --- [host-check-agent-status4,ec0a89e74ebbf98d,1c1c7af863cb5e42] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:16.061 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:34:16.062 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:34:16.063 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:34:16.065 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:34:16.064859584+08:00 2024-09-11 17:34:16.066 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:34:16.066113985+08:00 2024-09-11 17:34:16.066 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:34:16.066+08:00 2024-09-11 17:34:16.067 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:34:16.066+08:00 2024-09-11 17:34:16.068 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:34:16.067485020+08:00 2024-09-11 17:34:16.068 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:34:16.068 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:34:16.066+08:00 and status AVAILABLE 2024-09-11 17:34:16.069 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:34:16.068+08:00 2024-09-11 17:34:16.069 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:34:16.066+08:00 and status AVAILABLE 2024-09-11 17:34:16.069 INFO 2981023 --- [host-check-agent-status1,ec0a89e74ebbf98d,6f1b0e0ad96be2b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:34:16.070 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:34:16.070416047+08:00 2024-09-11 17:34:16.071 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:34:16.068+08:00 and status AVAILABLE 2024-09-11 17:34:16.072 INFO 2981023 --- [host-check-agent-status1,ec0a89e74ebbf98d,6f1b0e0ad96be2b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:34:16.072 INFO 2981023 --- [host-check-agent-status1,ec0a89e74ebbf98d,6f1b0e0ad96be2b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:34:16.072 INFO 2981023 --- [host-check-agent-status1,ec0a89e74ebbf98d,6f1b0e0ad96be2b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:34:16.074 INFO 2981023 --- [host-check-agent-status3,ec0a89e74ebbf98d,3bfa50420577a415] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:34:16.074 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:34:16.070+08:00 2024-09-11 17:34:16.076 INFO 2981023 --- [host-check-agent-status2,ec0a89e74ebbf98d,0d5f7934cb9e5033] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:34:16.077 INFO 2981023 --- [host-check-agent-status4,ec0a89e74ebbf98d,1c1c7af863cb5e42] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:34:16.078 INFO 2981023 --- [host-check-agent-status3,ec0a89e74ebbf98d,3bfa50420577a415] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:34:16.078 INFO 2981023 --- [host-check-agent-status3,ec0a89e74ebbf98d,3bfa50420577a415] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:34:16.078 INFO 2981023 --- [host-check-agent-status3,ec0a89e74ebbf98d,3bfa50420577a415] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:34:16.078 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:34:16.070+08:00 and status AVAILABLE 2024-09-11 17:34:16.078 INFO 2981023 --- [host-check-agent-status2,ec0a89e74ebbf98d,0d5f7934cb9e5033] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:34:16.078 INFO 2981023 --- [host-check-agent-status2,ec0a89e74ebbf98d,0d5f7934cb9e5033] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:34:16.078 INFO 2981023 --- [host-check-agent-status2,ec0a89e74ebbf98d,0d5f7934cb9e5033] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:34:16.085 INFO 2981023 --- [host-check-agent-status4,ec0a89e74ebbf98d,1c1c7af863cb5e42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:34:16.085 INFO 2981023 --- [host-check-agent-status4,ec0a89e74ebbf98d,1c1c7af863cb5e42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:34:16.085 INFO 2981023 --- [host-check-agent-status4,ec0a89e74ebbf98d,1c1c7af863cb5e42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:34:16.088 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:34:16.093 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:34:16.093 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:34:16.103 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:34:16.112 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:16.113 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:16.120 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:16.122 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:16.127 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:34:16.129 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:34:16.136 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:34:16.157 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:16.160 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:34:16.161 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:16.181 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:34:16.187 INFO 2981023 --- [ocp-schedules-12,9cd786ce402fdac1,b06f72aa47fddb68] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:34:16.187 INFO 2981023 --- [ocp-schedules-11,e9c018ce35872317,4b164713273c30a0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:34:16.190 INFO 2981023 --- [ocp-schedules-11,e9c018ce35872317,4b164713273c30a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:34:16.191 INFO 2981023 --- [ocp-schedules-12,9cd786ce402fdac1,b06f72aa47fddb68] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:34:16.192 INFO 2981023 --- [ocp-schedules-28,cec7997b2a789eb2,da60944a1198f334] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:34:16.192 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:34:16.193 INFO 2981023 --- [ocp-schedules-28,cec7997b2a789eb2,da60944a1198f334] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:34:16.195 INFO 2981023 --- [ocp-schedules-20,ee084a0150dca3ee,58ede5c25ba39fcc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:34:16.203 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:34:16.208 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:34:16.208 INFO 2981023 --- [ocp-schedules-1,e781b01fd14f746f,666f8e238697f2d0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:34:16.219 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:34:16.228 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:34:16.240 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:34:16.243 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:34:16.247 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:34:16.247 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:34:16.250 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.251 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:16.252 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:34:16.261 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:34:16.263 INFO 2981023 --- [ocp-schedules-11,e9c018ce35872317,4b164713273c30a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:34:16.269 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:34:16.276 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:34:16.290 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:34:16.293 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:34:16.296 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.297 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:16.297 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.297 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:34:16.298 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:16.300 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:34:16.300 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.301 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:16.302 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:34:16.302 INFO 2981023 --- [ocp-schedules-12,9cd786ce402fdac1,b06f72aa47fddb68] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:34:16.311 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.312 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:16.313 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.313 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:34:16.314 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:16.315 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:34:16.325 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:34:16.338 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:34:16.339 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:34:16.348 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:34:16.345096+08:00, lastProcessTime=2024-09-11T17:34:06.337350+08:00 2024-09-11 17:34:16.353 INFO 2981023 --- [host-check-host-status1,27d66fff2aa00fb8,9b20bb2f2ca37306] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:34:16.360 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:34:16.361 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.362 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.363 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:34:16.364 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:34:16.368 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:34:16.369 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:34:16.371 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.372 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.373 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:34:16.385 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:34:16.394 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.395 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.395 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:16.395 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:34:16.396 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:34:16.396 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.398 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:34:16.399 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:34:16.403 INFO 2981023 --- [host-check-host-status3,27d66fff2aa00fb8,90701c90d8ddb850] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:34:16.405 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.405 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:34:16.405 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.406 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:16.407 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.410 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:34:16.410 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:34:16.415 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:16.415 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:34:16.418 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:16.418 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:34:16.431 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:16.433 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:34:16.435 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:16.437 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:34:16.440 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.441 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.441 INFO 2981023 --- [host-check-host-status4,27d66fff2aa00fb8,5b86c1ae6d971c72] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:34:16.442 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:16.443 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:34:16.446 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.447 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.447 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:16.447 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:34:16.448 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:34:16.451 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:34:16.454 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:16.455 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:16.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:16.457 INFO 2981023 --- [host-check-host-status2,27d66fff2aa00fb8,357764e6b85f74af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:34:16.457 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:16.466 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.466 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.467 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:34:16.473 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:34:16.475 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:34:16.490 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.490 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.491 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:16.491 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:34:16.491 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.494 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:34:16.495 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:34:16.504 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:16.504 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:34:16.517 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:16.526 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.527 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.527 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:16.528 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:34:16.531 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:34:16.535 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:16.536 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:16.537 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:34:16.539 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:34:16.548 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.549 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.550 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:34:16.553 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.553 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.554 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:34:16.557 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:34:16.558 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:34:16.558 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:34:16.561 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:34:16.561 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:34:16.574 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:34:16.345096+08:00 2024-09-11 17:34:16.586 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:34:16.591 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:16.591 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:16.592 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:34:16.597 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:34:16.597 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:34:16.597 INFO 2981023 --- [ocp-schedules-28,cec7997b2a789eb2,da60944a1198f334] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:34:17.184 INFO 2981023 --- [ocp-async-7,26fa2a03da6181c5,cfa7b73b3184f3b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:17.188 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:17.188 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,f1b0bdae20d9d26b,819fc57d59ed0bdc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,f1b0bdae20d9d26b,819fc57d59ed0bdc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:17.751 INFO 2981023 --- [ocp-async-8,c1fb705078a2e2df,52cef01b036d960d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:34:17.788 INFO 2981023 --- [host-check-clockdiff1,5e1b797759212617,4c6a2c0d5c7acd31] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047257 0 0 , error= 2024-09-11 17:34:19.184 INFO 2981023 --- [ocp-async-9,367d84571f343d9d,036904e9fe5d3ed3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:19.311 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:34:19.311 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000008 2024-09-11 17:34:19.311 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000009 2024-09-11 17:34:19.311 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000007 2024-09-11 17:34:19.311 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2024-09-11 17:34:19.331 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:34:19.355 WARN 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:34:19.360 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:34:19.365 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:34:19.374 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:34:19.411 WARN 2981023 --- [ForkJoinPool.commonPool-worker-2,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:34:19.414 WARN 2981023 --- [ForkJoinPool.commonPool-worker-3,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:34:19.420 WARN 2981023 --- [ForkJoinPool.commonPool-worker-0,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:34:19.430 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:34:19.430 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:34:19.430 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:34:19.431 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000009 agent sql collect property 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_all_error_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000008 agent sql collect property 2024-09-11 17:34:19.432 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:34:19.432 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:34:19.432 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:34:19.432 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:34:19.432 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000007 agent sql collect property 2024-09-11 17:34:19.438 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:34:19.442 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2000009 to set sql collect property inactive 2024-09-11 17:34:19.444 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:19.444 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:19.446 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:19.446 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:19.453 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:34:19.453 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:34:19.454 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:34:19.455 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2024-09-11 17:34:19.464 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.ob.trace.status=inactive), (monagent.pipeline.ob.log.status=inactive)]), params:null 2024-09-11 17:34:19.468 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:19.470 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:19.471 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994437), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.10), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkprdocp), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"include 2024-09-11 17:34:19.472 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.6), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4 2024-09-11 17:34:19.484 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000008, do nothing 2024-09-11 17:34:19.486 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000009, do nothing 2024-09-11 17:34:19.487 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000007, do nothing 2024-09-11 17:34:19.489 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1718689631), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.5), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=rtdob), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"in 2024-09-11 17:34:19.508 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2024-09-11 17:34:19.509 INFO 2981023 --- [ocp-schedules-25,631c6496f3c4b5bd,b3fd429263badb77] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-09-11 17:34:19.715 INFO 2981023 --- [ocp-schedules-20,4920e60a4cecf3c6,b58e5d3f40a09a9c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:34:19.772 WARN 2981023 --- [scanner-13507581-1,4920e60a4cecf3c6,91bcad064dd37e1f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:34:19.793 INFO 2981023 --- [alarm-detect-5,4920e60a4cecf3c6,d11af04547d79eb3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644135, data is empty. 2024-09-11 17:34:19.797 INFO 2981023 --- [alarm-detect-5,4920e60a4cecf3c6,d11af04547d79eb3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644135, data is empty. 2024-09-11 17:34:19.802 INFO 2981023 --- [alarm-detect-5,4920e60a4cecf3c6,d11af04547d79eb3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644135, data is empty. 2024-09-11 17:34:19.816 INFO 2981023 --- [alarm-detect-10,4920e60a4cecf3c6,21ae291a799f0910] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644135, data is empty. 2024-09-11 17:34:19.821 INFO 2981023 --- [alarm-detect-12,4920e60a4cecf3c6,0dbed4535202471b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644135, data is empty. 2024-09-11 17:34:19.821 INFO 2981023 --- [alarm-detect-10,4920e60a4cecf3c6,21ae291a799f0910] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644135, data is empty. 2024-09-11 17:34:19.824 INFO 2981023 --- [alarm-detect-10,4920e60a4cecf3c6,21ae291a799f0910] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644135, data is empty. 2024-09-11 17:34:19.825 INFO 2981023 --- [alarm-detect-12,4920e60a4cecf3c6,0dbed4535202471b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644135, data is empty. 2024-09-11 17:34:19.831 INFO 2981023 --- [alarm-detect-12,4920e60a4cecf3c6,0dbed4535202471b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644135, data is empty. 2024-09-11 17:34:19.942 INFO 2981023 --- [alarm-detect-10,4920e60a4cecf3c6,ee3d9ccb81df4586] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644135, data is empty. 2024-09-11 17:34:19.949 INFO 2981023 --- [alarm-detect-10,4920e60a4cecf3c6,ee3d9ccb81df4586] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644135, data is empty. 2024-09-11 17:34:19.955 INFO 2981023 --- [alarm-detect-10,4920e60a4cecf3c6,ee3d9ccb81df4586] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644135, data is empty. 2024-09-11 17:34:20.188 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:20.190 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,2673b5bd77efc163,5d13bb05a070d26e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:20.318 INFO 2981023 --- [prometheus-minute-schedule-1,2673b5bd77efc163,5d13bb05a070d26e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:21.185 INFO 2981023 --- [ocp-async-8,1a65e2e3d09ee983,49ad76551f0700cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:21.931 INFO 2981023 --- [ocp-schedules-17,4d33f28de984ea33,d7c66f2afa718873] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:34:21.989 INFO 2981023 --- [host-check-clockdiff3,5e1b797759212617,afcfd5d971261972] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047261 1 1 , error= 2024-09-11 17:34:22.669 INFO 2981023 --- [ocp-schedules-9,e2363a16bbeb9ef8,f95365f32306ee8c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:34:22.669 INFO 2981023 --- [ocp-schedules-9,e2363a16bbeb9ef8,f95365f32306ee8c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:34:22.669 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:34:22.669 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:34:22.670 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:34:22.675 INFO 2981023 --- [ocp-schedules-9,e2363a16bbeb9ef8,f95365f32306ee8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:34:22.676 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:22.677 INFO 2981023 --- [ocp-schedules-9,e2363a16bbeb9ef8,f95365f32306ee8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:34:22.677 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:22.677 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:22.678 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:22.678 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:22.679 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:22.701 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:34:22.701 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:34:22.702 INFO 2981023 --- [ocp-schedules-9,e2363a16bbeb9ef8,f95365f32306ee8c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:34:22.703 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:34:22.706 INFO 2981023 --- [ocp-schedules-9,e2363a16bbeb9ef8,f95365f32306ee8c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:34:23.185 INFO 2981023 --- [ocp-async-7,74a826ecc8550b2a,4c667751f46615ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:23.190 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:23.191 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,a557eecc57c146e2,cb7ad3cf51a041b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,a557eecc57c146e2,cb7ad3cf51a041b3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:24.212 INFO 2981023 --- [ocp-schedules-13,fa04beb73165e0b0,416f5f9c00a33d29] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:34:25.185 INFO 2981023 --- [ocp-async-3,06104e8837a93432,2446cdd045bca5b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:26.191 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:26.192 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,44ee62d94bc0dcc4,040586b1d87822fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,44ee62d94bc0dcc4,040586b1d87822fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:34:26.351302+08:00, lastProcessTime=2024-09-11T17:34:16.345096+08:00 2024-09-11 17:34:26.473 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:34:26.351302+08:00 2024-09-11 17:34:26.850 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 17:34:26.850 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 17:34:26.851 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.851 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:34:26.852 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.852 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.852 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.853 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 17:34:26.867 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.867 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.868 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.868 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 17:34:26.877 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.877 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.877 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.878 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 17:34:26.886 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.886 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.886 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.887 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 17:34:26.896 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.896 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.897 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.897 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 17:34:26.900 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.901 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.901 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.902 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 17:34:26.904 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.904 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.905 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.905 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 17:34:26.908 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.908 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.909 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.909 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 17:34:26.913 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.913 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.914 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.914 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 17:34:26.917 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.918 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.918 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.918 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 17:34:26.921 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.922 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.922 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.922 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 17:34:26.928 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.928 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.929 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.929 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 17:34:26.938 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.938 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.939 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.939 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 17:34:26.942 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.942 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.943 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.943 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 17:34:26.946 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.947 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.947 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.947 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 17:34:26.950 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.951 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.951 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.951 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 17:34:26.954 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.954 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.955 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.955 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 17:34:26.958 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.958 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.959 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.959 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 17:34:26.962 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.962 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.962 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.963 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 17:34:26.966 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.966 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.966 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.967 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 17:34:26.969 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.970 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.970 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.970 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 17:34:26.973 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.973 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.974 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.974 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 17:34:26.977 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.977 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.977 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.978 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 17:34:26.980 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.981 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.981 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.981 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 17:34:26.984 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.984 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.985 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.985 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 17:34:26.988 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.988 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.988 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.989 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 17:34:26.991 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.992 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.992 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.992 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 17:34:26.995 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.995 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:26.996 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.996 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 17:34:26.999 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:26.999 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.000 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.000 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 17:34:27.003 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.003 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.003 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.004 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 17:34:27.006 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.007 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.007 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.007 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 17:34:27.010 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.010 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.011 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.011 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 17:34:27.014 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.014 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.014 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.015 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 17:34:27.017 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.017 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.018 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.018 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 17:34:27.023 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.023 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.024 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.024 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 17:34:27.029 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.029 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.029 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.030 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 17:34:27.033 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.033 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.033 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.034 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 17:34:27.037 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.037 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.037 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.038 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 17:34:27.040 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.040 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.040 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.041 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 17:34:27.043 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.043 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.044 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.044 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 17:34:27.046 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.046 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.047 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.047 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 17:34:27.049 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.049 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.050 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.050 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 17:34:27.057 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.057 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.057 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.058 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 17:34:27.073 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.073 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.074 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.074 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 17:34:27.078 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.078 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.079 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.079 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 17:34:27.083 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.083 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.084 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.084 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 17:34:27.088 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.088 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.089 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.089 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 17:34:27.091 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.091 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.092 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.092 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 17:34:27.095 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.095 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.095 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.096 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 17:34:27.098 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 17:34:27.098 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 17:34:27.099 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.099 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:34:27.101 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.101 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.102 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.102 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 17:34:27.106 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.106 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:34:27.106 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:27.107 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 17:34:27.109 INFO 2981023 --- [ocp-schedules-17,937dc07040d7d2e3,cf51242699bf0fcc] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 17:34:27.185 INFO 2981023 --- [ocp-async-9,d99f57a37e825928,398d007196763c22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:27.752 INFO 2981023 --- [ocp-async-3,d5f2f889c1bf69c9,500cf14a27787e50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:34:29.185 INFO 2981023 --- [ocp-async-6,f7da55ab5a00a310,ae26e3e781528c8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:29.192 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:29.192 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,d455794e060f98d0,a5397bd4fa04ff0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,d455794e060f98d0,a5397bd4fa04ff0a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:29.994 INFO 2981023 --- [ocp-schedules-14,e5f5de5e5db81d4c,050055a3b0931434] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:34:30.040 WARN 2981023 --- [scanner-13507913-1,e5f5de5e5db81d4c,f4b8edca82d358e8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:34:31.185 INFO 2981023 --- [ocp-async-1,efaf2acf86e54bed,f01fdd5c6045e6d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:31.933 INFO 2981023 --- [ocp-schedules-14,4002c454f0e3e28c,a7a6a4c3e076282a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:34:32.192 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:32.193 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,cf61b2220a6b029b,98f6c70c21dd9912] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,cf61b2220a6b029b,98f6c70c21dd9912] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:33.192 INFO 2981023 --- [ocp-async-5,6b9086482c152c41,45e550106348b1a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:33.831 INFO 2981023 --- [ocp-schedules-27,317742a21558f882,43e0ebf4230c209e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:34:33.852 INFO 2981023 --- [ocp-schedules-27,317742a21558f882,43e0ebf4230c209e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:34:33.852 INFO 2981023 --- [ocp-schedules-27,317742a21558f882,43e0ebf4230c209e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:34:35.186 INFO 2981023 --- [ocp-async-7,19a8dbc4e9e04879,a67323af86269870] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:35.193 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:35.194 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,70dc6f9606ddfd6e,8b4ebbc5f911f7d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,70dc6f9606ddfd6e,8b4ebbc5f911f7d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:34:36.337201+08:00, lastProcessTime=2024-09-11T17:34:26.351302+08:00 2024-09-11 17:34:36.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:34:36.337201+08:00 2024-09-11 17:34:37.186 INFO 2981023 --- [ocp-async-6,fa3a447fe700be79,c5b58ab15a773a03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:37.314 INFO 2981023 --- [prometheus-cache-monitor1,c66c8143f70ec341,a5bd8feebaeda5d3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:34:37.314 INFO 2981023 --- [prometheus-cache-monitor1,c66c8143f70ec341,a5bd8feebaeda5d3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:34:37.752 INFO 2981023 --- [ocp-async-4,b2c0eed1fe915234,9e65a9c998d16899] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:34:38.194 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:38.195 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,5ff208057a344723,809b882e398c4afe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,5ff208057a344723,809b882e398c4afe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:39.186 INFO 2981023 --- [ocp-async-9,ba1e559bd34460a1,9929a3488d908a35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:39.264 INFO 2981023 --- [ocp-schedules-30,4f5640c4feabfff6,2b6e8cf206f32010] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:34:39.282 WARN 2981023 --- [ocp-schedules-30,4f5640c4feabfff6,2b6e8cf206f32010] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:34:40.003 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:34:40.003 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:34:40.189 INFO 2981023 --- [ocp-schedules-6,da7a65337eee9506,9cc8f5d99eac2494] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:34:40.233 WARN 2981023 --- [scanner-13508131-1,da7a65337eee9506,14f913bc457c4a95] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:34:40.256 INFO 2981023 --- [alarm-detect-5,da7a65337eee9506,d4c08ad2ce0d1623] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644135, data is empty. 2024-09-11 17:34:40.260 INFO 2981023 --- [alarm-detect-5,da7a65337eee9506,d4c08ad2ce0d1623] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644135, data is empty. 2024-09-11 17:34:40.267 INFO 2981023 --- [alarm-detect-5,da7a65337eee9506,d4c08ad2ce0d1623] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644135, data is empty. 2024-09-11 17:34:40.285 INFO 2981023 --- [alarm-detect-11,da7a65337eee9506,67c892bcd3e02d03] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644135, data is empty. 2024-09-11 17:34:40.286 INFO 2981023 --- [alarm-detect-6,da7a65337eee9506,c023a666a13efa7c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644135, data is empty. 2024-09-11 17:34:40.289 INFO 2981023 --- [alarm-detect-6,da7a65337eee9506,c023a666a13efa7c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644135, data is empty. 2024-09-11 17:34:40.289 INFO 2981023 --- [alarm-detect-11,da7a65337eee9506,67c892bcd3e02d03] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644135, data is empty. 2024-09-11 17:34:40.293 INFO 2981023 --- [alarm-detect-11,da7a65337eee9506,67c892bcd3e02d03] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644135, data is empty. 2024-09-11 17:34:40.296 INFO 2981023 --- [alarm-detect-6,da7a65337eee9506,c023a666a13efa7c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644135, data is empty. 2024-09-11 17:34:40.317 INFO 2981023 --- [alarm-detect-18,da7a65337eee9506,116463e41c7d2cfe] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-2, alarm_template_id=0:host=10.31.35.10-3, alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.6-2, alarm_template_id=0:host=10.31.35.5-2, alarm_template_id=0:host=10.31.35.7-3, alarm_template_id=0:host=10.31.35.7-2, alarm_template_id=0:host=10.31.35.6-3] 2024-09-11 17:34:40.408 INFO 2981023 --- [alarm-detect-1,da7a65337eee9506,67836a5362821406] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644135, data is empty. 2024-09-11 17:34:40.411 INFO 2981023 --- [alarm-detect-1,da7a65337eee9506,67836a5362821406] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644135, data is empty. 2024-09-11 17:34:40.415 INFO 2981023 --- [alarm-detect-1,da7a65337eee9506,67836a5362821406] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644135, data is empty. 2024-09-11 17:34:41.186 INFO 2981023 --- [ocp-async-7,13e377ecddaedf29,901f985b340ba7a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:41.195 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:41.196 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,f824d67da2f62391,d0aed1b710e0fa4d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,f824d67da2f62391,d0aed1b710e0fa4d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:41.936 INFO 2981023 --- [ocp-schedules-20,b3725d85e25c3318,b28bac588f3d6edb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:34:43.186 INFO 2981023 --- [ocp-async-10,6e018c6b02b1ed5f,eb1e1b122eeee73d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:44.196 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:44.196 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,863eea33373054ad,d033475e3157ec22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,863eea33373054ad,d033475e3157ec22] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:45.101 INFO 2981023 --- [prometheus-second-schedule-1,b2c873bbb38a2138,3501836335658734] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:34:45.186 INFO 2981023 --- [ocp-async-2,231500d775dc5754,4979a8ed1f21f0a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:45.909 INFO 2981023 --- [ocp-schedules-20,f85210a6abda7a41,4f0ddafe23f7b835] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:34:46.021 INFO 2981023 --- [ocp-schedules-16,84b59a7e49e14cf2,5ce442bb149c2ece] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:34:46.026 INFO 2981023 --- [host-check-clockdiff1,84b59a7e49e14cf2,48b50806001ad510] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:34:46.027 INFO 2981023 --- [host-check-clockdiff2,84b59a7e49e14cf2,6944c3fb6db3ff8e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:34:46.028 INFO 2981023 --- [host-check-clockdiff3,84b59a7e49e14cf2,366d3d89907344e2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:34:46.028 INFO 2981023 --- [ocp-schedules-9,31dde8eef75aec9b,7ffc85a7b598d091] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:34:46.029 INFO 2981023 --- [host-check-clockdiff4,84b59a7e49e14cf2,d7c309ee3c613314] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:34:46.031 INFO 2981023 --- [ocp-schedules-30,535647265d94fb59,d97b0601eef54b34] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:34:46.031 INFO 2981023 --- [host-check-agent-status1,31dde8eef75aec9b,f1803701ff45ef10] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:34:46.032 INFO 2981023 --- [host-check-agent-status2,31dde8eef75aec9b,3a90eb60117aa7e5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:34:46.032 INFO 2981023 --- [host-check-agent-status3,31dde8eef75aec9b,18f0d81458ad8cd7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:34:46.043 INFO 2981023 --- [host-check-agent-status4,31dde8eef75aec9b,52036fc2d274bed9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:34:46.048 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:34:46.049 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:34:46.049 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:34:46.049 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:34:46.050 INFO 2981023 --- [host-check-clockdiff4,84b59a7e49e14cf2,d7c309ee3c613314] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047286 0 0 , error= 2024-09-11 17:34:46.051 INFO 2981023 --- [host-check-agent-status1,31dde8eef75aec9b,f1803701ff45ef10] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:46.052 INFO 2981023 --- [host-check-agent-status2,31dde8eef75aec9b,3a90eb60117aa7e5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:46.052 INFO 2981023 --- [host-check-agent-status4,31dde8eef75aec9b,52036fc2d274bed9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:46.056 INFO 2981023 --- [host-check-agent-status1,31dde8eef75aec9b,f1803701ff45ef10] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:46.056 INFO 2981023 --- [host-check-agent-status4,31dde8eef75aec9b,52036fc2d274bed9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:46.056 INFO 2981023 --- [host-check-agent-status2,31dde8eef75aec9b,3a90eb60117aa7e5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:46.059 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:46.059 INFO 2981023 --- [host-check-agent-status3,31dde8eef75aec9b,18f0d81458ad8cd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:34:46.060 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:46.060 INFO 2981023 --- [host-check-agent-status3,31dde8eef75aec9b,18f0d81458ad8cd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:34:46.060 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:46.062 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:46.062 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:46.062 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:34:46.064 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:46.064 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:34:46.088 INFO 2981023 --- [host-check-agent-status2,31dde8eef75aec9b,3a90eb60117aa7e5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:34:46.090 INFO 2981023 --- [host-check-agent-status1,31dde8eef75aec9b,f1803701ff45ef10] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:34:46.093 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:34:46.095 INFO 2981023 --- [host-check-agent-status2,31dde8eef75aec9b,3a90eb60117aa7e5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:34:46.095 INFO 2981023 --- [host-check-agent-status2,31dde8eef75aec9b,3a90eb60117aa7e5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:34:46.095 INFO 2981023 --- [host-check-agent-status2,31dde8eef75aec9b,3a90eb60117aa7e5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:34:46.097 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:34:46.098 INFO 2981023 --- [host-check-agent-status1,31dde8eef75aec9b,f1803701ff45ef10] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:34:46.098 INFO 2981023 --- [host-check-agent-status1,31dde8eef75aec9b,f1803701ff45ef10] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:34:46.099 INFO 2981023 --- [host-check-agent-status1,31dde8eef75aec9b,f1803701ff45ef10] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:34:46.100 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:34:46.094827985+08:00 2024-09-11 17:34:46.102 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:34:46.100+08:00 2024-09-11 17:34:46.103 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:34:46.099754658+08:00 2024-09-11 17:34:46.105 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:34:46.100+08:00 and status AVAILABLE 2024-09-11 17:34:46.116 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:34:46.103+08:00 2024-09-11 17:34:46.128 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:34:46.103+08:00 and status AVAILABLE 2024-09-11 17:34:46.129 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:34:46.130 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:34:46.133 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:34:46.133472513+08:00 2024-09-11 17:34:46.134 INFO 2981023 --- [host-check-agent-status4,31dde8eef75aec9b,52036fc2d274bed9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:34:46.134 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:34:46.133+08:00 2024-09-11 17:34:46.137 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:34:46.133+08:00 and status AVAILABLE 2024-09-11 17:34:46.152 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:34:46.155 INFO 2981023 --- [host-check-agent-status4,31dde8eef75aec9b,52036fc2d274bed9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:34:46.155 INFO 2981023 --- [host-check-agent-status4,31dde8eef75aec9b,52036fc2d274bed9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:34:46.155 INFO 2981023 --- [host-check-agent-status4,31dde8eef75aec9b,52036fc2d274bed9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:34:46.155 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:34:46.154348987+08:00 2024-09-11 17:34:46.156 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:46.157 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:46.165 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:34:46.169 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:34:46.155+08:00 2024-09-11 17:34:46.172 INFO 2981023 --- [ocp-schedules-13,544e080dea81eb4a,e53a855c97d585cd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:34:46.172 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:34:46.155+08:00 and status AVAILABLE 2024-09-11 17:34:46.175 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:34:46.179 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:34:46.180 INFO 2981023 --- [ocp-schedules-13,544e080dea81eb4a,e53a855c97d585cd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:34:46.184 INFO 2981023 --- [host-check-agent-status3,31dde8eef75aec9b,18f0d81458ad8cd7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:34:46.186 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:34:46.187 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:34:46.187 INFO 2981023 --- [ocp-schedules-2,4558195f4e0c3e73,9d80018430ac3055] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:34:46.187 INFO 2981023 --- [ocp-schedules-1,0a323a99fb634d29,2d6da60bbdc22a90] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:34:46.187 INFO 2981023 --- [ocp-schedules-26,50e9b8bb11c19021,9671b15316fed7ab] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:34:46.188 INFO 2981023 --- [ocp-schedules-6,f80e0ca3307c38a2,2f6a596c262dd039] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:34:46.190 INFO 2981023 --- [ocp-schedules-2,4558195f4e0c3e73,9d80018430ac3055] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:34:46.190 INFO 2981023 --- [ocp-schedules-1,0a323a99fb634d29,2d6da60bbdc22a90] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:34:46.191 INFO 2981023 --- [ocp-schedules-26,50e9b8bb11c19021,9671b15316fed7ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:34:46.196 INFO 2981023 --- [ocp-schedules-1,0a323a99fb634d29,2d6da60bbdc22a90] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:34:46.197 INFO 2981023 --- [host-check-agent-status3,31dde8eef75aec9b,18f0d81458ad8cd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:34:46.197 INFO 2981023 --- [host-check-agent-status3,31dde8eef75aec9b,18f0d81458ad8cd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:34:46.197 INFO 2981023 --- [host-check-agent-status3,31dde8eef75aec9b,18f0d81458ad8cd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:34:46.200 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:34:46.201 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:34:46.204 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:34:46.216 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:34:46.267 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:46.273 INFO 2981023 --- [ocp-schedules-10,0ff0eff7656a124a,76bd90e8a00fef60] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:34:46.273 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:46.277 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:34:46.277 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:34:46.279 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:46.284 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:46.301 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:34:46.312 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.316 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.318 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:34:46.319 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:34:46.320 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.320 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:34:46.334 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:34:46.338 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:34:46.347 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.347 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:46.348 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.349 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:46.350 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:34:46.356 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.358 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.358 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:46.358 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.358 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:46.360 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:34:46.360 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:46.361 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:34:46.362 INFO 2981023 --- [ocp-schedules-2,4558195f4e0c3e73,9d80018430ac3055] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:34:46.363 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:34:46.364 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:34:46.364112+08:00, lastProcessTime=2024-09-11T17:34:36.337201+08:00 2024-09-11 17:34:46.366 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:34:46.376 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.376 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:46.384 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:34:46.391 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:34:46.393 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:34:46.400 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:34:46.403 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:34:46.410 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.411 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:34:46.411 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:34:46.411 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.415 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:34:46.423 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:34:46.424 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:34:46.424 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:34:46.430 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.430 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.432 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:34:46.432 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:34:46.438 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:34:46.441 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:34:46.445 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:34:46.446 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:34:46.450 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:34:46.463 INFO 2981023 --- [host-check-host-status3,535647265d94fb59,354f2fcd809ef73b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:34:46.467 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:34:46.472 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:34:46.483 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:34:46.484 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.486 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.494 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.495 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.496 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.496 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:46.496 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:46.496 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:34:46.497 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.498 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.498 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:34:46.498 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:34:46.499 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:34:46.499 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.499 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:34:46.499 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:34:46.500 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:34:46.500 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:34:46.500 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.500 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.502 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:34:46.503 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:34:46.504 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:34:46.506 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:34:46.513 INFO 2981023 --- [host-check-host-status2,535647265d94fb59,101eb9c0d2c769dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:34:46.514 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:34:46.523 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:46.523 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:34:46.524 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:46.524 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:34:46.535 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:34:46.535 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:34:46.547 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:46.548 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:34:46.549 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:34:46.553 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:46.553 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.554 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.554 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:34:46.561 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.562 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.562 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:46.563 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:34:46.563 INFO 2981023 --- [host-check-host-status4,535647265d94fb59,2f6df9c95bdc6a24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:34:46.564 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.565 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.565 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:34:46.566 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:34:46.566 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.567 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.567 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:34:46.569 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.569 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:46.570 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:34:46.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:34:46.575 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:34:46.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:34:46.580 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:34:46.581 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:34:46.582 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:34:46.591 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:34:46.593 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.594 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.594 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:34:46.595 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:34:46.595 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.597 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:34:46.598 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:34:46.600 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:34:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:34:46.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:34:46.606 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:46.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:34:46.616 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:34:46.621 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.622 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:34:46.622 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.623 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:34:46.624 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:34:46.625 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.630 INFO 2981023 --- [host-check-host-status1,535647265d94fb59,35ae1681c3cea0b4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:34:46.646 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.646 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.647 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:34:46.647 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:34:46.650 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:34:46.654 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:34:46.656 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:34:46.664 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:34:46.677 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.678 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:34:46.686 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:34:46.686 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.686 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:34:46.706 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:34:46.706 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:34:46.706 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:34:46.708 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:34:46.716 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:34:46.720 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.720 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:34:46.721 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.721 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:34:46.721 INFO 2981023 --- [ocp-schedules-17,09f66485833ce90a,d5b79f1d67566c1f] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:34:46.726 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:34:46.729 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:46.730 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:46.730 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:34:46.733 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:34:46.364112+08:00 2024-09-11 17:34:46.733 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:34:46.733 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:34:46.736 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:34:46.736 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:34:46.737 INFO 2981023 --- [ocp-schedules-26,50e9b8bb11c19021,9671b15316fed7ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:34:46.971 INFO 2981023 --- [host-check-clockdiff1,84b59a7e49e14cf2,48b50806001ad510] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047286 0 0 , error= 2024-09-11 17:34:46.973 INFO 2981023 --- [host-check-clockdiff2,84b59a7e49e14cf2,6944c3fb6db3ff8e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047286 0 0 , error= 2024-09-11 17:34:47.186 INFO 2981023 --- [ocp-async-4,1b2239ceda5ab0ca,5392925454553efd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:47.196 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:47.197 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,d924a46913d51699,f8ad5ac2804677ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:47.315 INFO 2981023 --- [prometheus-minute-schedule-1,d924a46913d51699,f8ad5ac2804677ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:47.752 INFO 2981023 --- [ocp-async-7,17c2e5b3da9a9024,bc51d7611e01f644] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:34:49.186 INFO 2981023 --- [ocp-async-1,ecd6011ee40f8a4d,60417d9fe6791686] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:34:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:34:50.197 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:50.197 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:50.223 INFO 2981023 --- [ocp-schedules-21,03200e94d161dbc5,06c27d624db87a35] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:34:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,513d090444cbb0d0,b95d10db6f1ea449] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,513d090444cbb0d0,b95d10db6f1ea449] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:50.454 INFO 2981023 --- [ocp-schedules-2,0ee8e506cf923010,2846de0123f8c994] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:34:50.495 WARN 2981023 --- [scanner-13508429-1,0ee8e506cf923010,7952199365e1845f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:34:51.186 INFO 2981023 --- [ocp-async-8,7edc5592e6211ceb,faf9351ae717a282] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:51.537 INFO 2981023 --- [ocp-schedules-15,30e6887991b89f1e,87a9394606905d71] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:34:51.538 INFO 2981023 --- [ocp-schedules-15,30e6887991b89f1e,87a9394606905d71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:34:51.538 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:34:51.538 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:34:51.538 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:34:51.539 INFO 2981023 --- [ocp-schedules-15,30e6887991b89f1e,87a9394606905d71] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:34:51.938 INFO 2981023 --- [ocp-schedules-21,64772bc5ca41d291,cd0f4e719a16a591] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:34:52.009 INFO 2981023 --- [host-check-clockdiff3,84b59a7e49e14cf2,366d3d89907344e2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047292 1 1 , error= 2024-09-11 17:34:52.281 INFO 2981023 --- [ocp-schedules-22,9312cc4cb4c368da,65d3a2d6ace8763f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:34:52.281+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:34:52.283 INFO 2981023 --- [ocp-schedules-22,9312cc4cb4c368da,65d3a2d6ace8763f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:34:52.336 INFO 2981023 --- [ocp-schedules-22,9312cc4cb4c368da,65d3a2d6ace8763f] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 44 to SqlAggregateData 2024-09-11 17:34:52.337 INFO 2981023 --- [ocp-schedules-22,9312cc4cb4c368da,65d3a2d6ace8763f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:34:52.677 INFO 2981023 --- [ocp-schedules-24,9b4d271c8cb5b6b3,6b231da44a140973] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:34:52.694 WARN 2981023 --- [ocp-schedules-24,9b4d271c8cb5b6b3,6b231da44a140973] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:34:53.007 INFO 2981023 --- [ocp-schedules-6,45c63bde156ee6ad,63c4e303f9d3b391] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:34:53.008 INFO 2981023 --- [ocp-schedules-6,45c63bde156ee6ad,63c4e303f9d3b391] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:34:53.048 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:34:53.048 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.049 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:34:53.049 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:34:53.049 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.049 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.052 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:34:53.052 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:34:53.053 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:34:53.056 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.056 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.057 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.057 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.057 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.058 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.062 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.063 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.068 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.069 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.070 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.070 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:34:53.070 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.071 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.078 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.078 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:34:53.080 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.080 INFO 2981023 --- [ocp-schedules-18,42983653eec9676c,4cb253540dc7a04a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:34:53.080 INFO 2981023 --- [ocp-schedules-18,42983653eec9676c,4cb253540dc7a04a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:34:53.080 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.080 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:34:53.081 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2751D6DC-0-0] 2024-09-11 17:34:53.081 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:34:53.081 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.081 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:34:53.083 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.084 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:34:53.092 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.092 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:34:53.093 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.094 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:34:53.096 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.097 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:34:53.105 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.106 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:34:53.106 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.106 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.107 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.107 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:34:53.107 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:34:53.107 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:34:53.107 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.107 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:34:53.107 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2751D6E6-0-0] 2024-09-11 17:34:53.108 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:34:53.108 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:34:53.108 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.109 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:34:53.109 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.109 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:34:53.110 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:34:53.110 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.111 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:34:53.113 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.113 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:34:53.113 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:34:53.114 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.114 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.114 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:34:53.115 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:34:53.115 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.115 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.115 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:34:53.116 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.116 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:34:53.116 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.116 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:34:53.117 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.117 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:34:53.117 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:34:53.118 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:34:53.118 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.118 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:34:53.119 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:34:53.124 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.124 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:34:53.125 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:34:53.126 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:34:53.131 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:53.133 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.133 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:53.133 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.137 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.137 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:34:53.139 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:34:53.140 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.140 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:34:53.143 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:34:53.145 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:34:53.146 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.146 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:34:53.146 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:34:53.148 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:34:53.148 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:34:53.148 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:34:53.152 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:34:53.152 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:34:53.154 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:34:53.154 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:34:53.154 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.154 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:53.154 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.154 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:34:53.160 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:34:53.161 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:34:53.161 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:34:53.161 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:34:53.165 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:34:53.166 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:34:53.166 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:34:53.166 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:34:53.167 INFO 2981023 --- [ocp-schedules-6,45c63bde156ee6ad,63c4e303f9d3b391] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:34:53.186 INFO 2981023 --- [ocp-async-9,b684a34a89a8b549,5dd3c16eea7fefbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:53.197 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:53.199 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,b29a084d3f616f97,babfa51bb4184809] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,b29a084d3f616f97,babfa51bb4184809] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:53.855 INFO 2981023 --- [ocp-schedules-23,222b55d054363567,a4558a84428810a6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:34:53.864 INFO 2981023 --- [ocp-schedules-23,222b55d054363567,a4558a84428810a6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:34:53.864 INFO 2981023 --- [ocp-schedules-23,222b55d054363567,a4558a84428810a6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:34:54.215 INFO 2981023 --- [ocp-schedules-10,1c8b9362fe46512d,1ee9610847971a33] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:34:55.187 INFO 2981023 --- [ocp-async-3,d6d4436f4a3880ed,3bb67e2bf682ce23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:55.675 INFO 2981023 --- [ocp-schedules-10,eb099d3f37fcbd50,e3647a62d24b1b14] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:34:55.675 INFO 2981023 --- [ocp-schedules-10,eb099d3f37fcbd50,e3647a62d24b1b14] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:34:55.677 INFO 2981023 --- [ocp-schedules-10,eb099d3f37fcbd50,e3647a62d24b1b14] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:34:55.677 INFO 2981023 --- [ocp-schedules-10,eb099d3f37fcbd50,e3647a62d24b1b14] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:34:55.677 INFO 2981023 --- [ocp-schedules-10,eb099d3f37fcbd50,e3647a62d24b1b14] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:34:55.677 INFO 2981023 --- [ocp-schedules-10,eb099d3f37fcbd50,e3647a62d24b1b14] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:34:55.756 INFO 2981023 --- [ocp-schedules-7,4c0408d6ee0badfa,02bbdf00a7484054] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:34:55.763 INFO 2981023 --- [ocp-schedules-7,4c0408d6ee0badfa,02bbdf00a7484054] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:34:55.763 INFO 2981023 --- [ocp-schedules-7,4c0408d6ee0badfa,02bbdf00a7484054] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:34:56.130 INFO 2981023 --- [ocp-schedules-26,e8bb6a3d11743214,17bb8b9b2a54864b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:34:56.130 INFO 2981023 --- [ocp-schedules-26,e8bb6a3d11743214,17bb8b9b2a54864b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:34:56.147 INFO 2981023 --- [Collect all tenants major compactions1,e8bb6a3d11743214,f8804394933b713f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:34:56.148 INFO 2981023 --- [Collect all tenants major compactions2,e8bb6a3d11743214,dccc4782404065ff] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:34:56.148 INFO 2981023 --- [Collect all tenants major compactions4,e8bb6a3d11743214,286e020d623de427] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:34:56.148 INFO 2981023 --- [Collect all tenants major compactions6,e8bb6a3d11743214,8a0a8df6406dfe6b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:34:56.148 INFO 2981023 --- [Collect all tenants major compactions5,e8bb6a3d11743214,3517607390d97a0c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:34:56.150 INFO 2981023 --- [Collect all tenants major compactions3,e8bb6a3d11743214,f7ace0db8af761d2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:34:56.160 INFO 2981023 --- [Collect all tenants major compactions4,e8bb6a3d11743214,286e020d623de427] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:56.161 INFO 2981023 --- [Collect all tenants major compactions5,e8bb6a3d11743214,3517607390d97a0c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:56.161 INFO 2981023 --- [Collect all tenants major compactions1,e8bb6a3d11743214,f8804394933b713f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:56.161 INFO 2981023 --- [Collect all tenants major compactions2,e8bb6a3d11743214,dccc4782404065ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:56.161 INFO 2981023 --- [Collect all tenants major compactions6,e8bb6a3d11743214,8a0a8df6406dfe6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:56.162 INFO 2981023 --- [Collect all tenants major compactions4,e8bb6a3d11743214,286e020d623de427] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:34:56.162 INFO 2981023 --- [Collect all tenants major compactions5,e8bb6a3d11743214,3517607390d97a0c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:34:56.162 INFO 2981023 --- [Collect all tenants major compactions1,e8bb6a3d11743214,f8804394933b713f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:34:56.163 INFO 2981023 --- [Collect all tenants major compactions6,e8bb6a3d11743214,8a0a8df6406dfe6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:34:56.164 INFO 2981023 --- [Collect all tenants major compactions2,e8bb6a3d11743214,dccc4782404065ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:34:56.165 INFO 2981023 --- [Collect all tenants major compactions3,e8bb6a3d11743214,f7ace0db8af761d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:34:56.165 INFO 2981023 --- [Collect all tenants major compactions5,e8bb6a3d11743214,3517607390d97a0c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:34:56.165 INFO 2981023 --- [Collect all tenants major compactions5,e8bb6a3d11743214,3517607390d97a0c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:34:56.166 INFO 2981023 --- [Collect all tenants major compactions1,e8bb6a3d11743214,f8804394933b713f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:34:56.166 INFO 2981023 --- [Collect all tenants major compactions1,e8bb6a3d11743214,f8804394933b713f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:34:56.166 INFO 2981023 --- [Collect all tenants major compactions2,e8bb6a3d11743214,dccc4782404065ff] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:34:56.166 INFO 2981023 --- [Collect all tenants major compactions2,e8bb6a3d11743214,dccc4782404065ff] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:34:56.166 INFO 2981023 --- [Collect all tenants major compactions6,e8bb6a3d11743214,8a0a8df6406dfe6b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:34:56.166 INFO 2981023 --- [Collect all tenants major compactions6,e8bb6a3d11743214,8a0a8df6406dfe6b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:34:56.166 INFO 2981023 --- [Collect all tenants major compactions4,e8bb6a3d11743214,286e020d623de427] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:34:56.166 INFO 2981023 --- [Collect all tenants major compactions4,e8bb6a3d11743214,286e020d623de427] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:34:56.167 INFO 2981023 --- [Collect all tenants major compactions3,e8bb6a3d11743214,f7ace0db8af761d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:34:56.168 INFO 2981023 --- [Collect all tenants major compactions3,e8bb6a3d11743214,f7ace0db8af761d2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:34:56.168 INFO 2981023 --- [Collect all tenants major compactions3,e8bb6a3d11743214,f7ace0db8af761d2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:34:56.169 INFO 2981023 --- [ocp-schedules-26,e8bb6a3d11743214,17bb8b9b2a54864b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:34:56.199 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:56.199 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,3f06e97d2c403ae8,1d9fedf2242e7ecf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,3f06e97d2c403ae8,1d9fedf2242e7ecf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:34:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:34:56.351504+08:00, lastProcessTime=2024-09-11T17:34:46.364112+08:00 2024-09-11 17:34:56.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:34:56.351504+08:00 2024-09-11 17:34:57.187 INFO 2981023 --- [ocp-async-5,334d1ec0f890e978,baf94fde10c230fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:57.752 INFO 2981023 --- [ocp-async-2,245188df1cf0e434,458ace3d0f187f68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:34:59.187 INFO 2981023 --- [ocp-async-9,dc59ed82688d319e,fa205d5a621e2553] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:34:59.199 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:34:59.201 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:34:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,0e640921ca82841f,1f95a9929ceb4f6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047300, interval=60 2024-09-11 17:34:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,0e640921ca82841f,1f95a9929ceb4f6f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:00.044 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:30:00.0, nextRun=2024-09-11 17:35:00.0, currentDate=Wed Sep 11 17:35:00 HKT 2024 2024-09-11 17:35:00.044 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:30:00.0, nextRunTime=2024-09-11 17:35:00.0} 2024-09-11 17:35:00.051 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:35:00.051 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:35:00.051 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:35:00.051+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:35:00.079 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:30:00.0, nextRun=2024-09-11 17:35:00.0, currentDate=Wed Sep 11 17:35:00 HKT 2024 2024-09-11 17:35:00.079 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:30:00.0, nextRunTime=2024-09-11 17:35:00.0} 2024-09-11 17:35:00.112 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2, 1000006] 2024-09-11 17:35:00.119 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:35:00.120 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:35:00.120 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:35:00.120 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,189cf4d9cf552a93] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2,1000006]]}}, createTime=null, startTime=2024-09-11T17:35:00.120+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:35:00.522 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,c9094e1858eb1bbf] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048165, operation:EXECUTE, state: RUNNING 2024-09-11 17:35:00.532 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,c9094e1858eb1bbf] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048166, operation:EXECUTE, state: RUNNING 2024-09-11 17:35:00.542 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048165, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048185, task_operation=execute, latest_execution_start_time=2024-09-11T17:35:00.515+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048165}, listMap={}}, executor=10.31.35.10 2024-09-11 17:35:00.550 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048166, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048186, task_operation=execute, latest_execution_start_time=2024-09-11T17:35:00.530+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048166}, listMap={cluster_ids=[[1,2,1000006]]}}, executor=10.31.35.10 2024-09-11 17:35:00.551 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:35:00.557 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:35:00.578 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:35:00.581 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:35:00.593 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:35:00.594 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:35:00.594 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:35:00.602 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:35:00.615 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:35:00.625 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:35:00.625 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1000006 2024-09-11 17:35:00.630 INFO 2981023 --- [ocp-schedules-26,82d40e204cbc0e11,198e381d714a330a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:35:00.647 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,b1cf8fcca6e136ee] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048166, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:35:00.685 WARN 2981023 --- [scanner-13508694-1,82d40e204cbc0e11,09a4876bbc065edd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:35:00.702 INFO 2981023 --- [alarm-detect-18,82d40e204cbc0e11,d7c459a5465454d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644135, data is empty. 2024-09-11 17:35:00.711 INFO 2981023 --- [alarm-detect-18,82d40e204cbc0e11,d7c459a5465454d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644179, data is empty. 2024-09-11 17:35:00.715 INFO 2981023 --- [alarm-detect-18,82d40e204cbc0e11,d7c459a5465454d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644179, data is empty. 2024-09-11 17:35:00.732 INFO 2981023 --- [alarm-detect-3,82d40e204cbc0e11,ea1f0cd82959b117] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644135, data is empty. 2024-09-11 17:35:00.735 INFO 2981023 --- [alarm-detect-4,82d40e204cbc0e11,69de8f814ae22fa6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644135, data is empty. 2024-09-11 17:35:00.736 INFO 2981023 --- [alarm-detect-3,82d40e204cbc0e11,ea1f0cd82959b117] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644179, data is empty. 2024-09-11 17:35:00.739 INFO 2981023 --- [alarm-detect-3,82d40e204cbc0e11,ea1f0cd82959b117] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644179, data is empty. 2024-09-11 17:35:00.740 INFO 2981023 --- [alarm-detect-4,82d40e204cbc0e11,69de8f814ae22fa6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644179, data is empty. 2024-09-11 17:35:00.746 INFO 2981023 --- [alarm-detect-4,82d40e204cbc0e11,69de8f814ae22fa6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644179, data is empty. 2024-09-11 17:35:00.811 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:35:00.862 INFO 2981023 --- [alarm-detect-12,82d40e204cbc0e11,b0e43cc32f926b93] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644135, data is empty. 2024-09-11 17:35:00.866 INFO 2981023 --- [alarm-detect-12,82d40e204cbc0e11,b0e43cc32f926b93] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644179, data is empty. 2024-09-11 17:35:00.873 INFO 2981023 --- [alarm-detect-12,82d40e204cbc0e11,b0e43cc32f926b93] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644179, data is empty. 2024-09-11 17:35:00.917 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:35:00.941 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:35:00.956 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:35:00.956 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:35:00.967 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:35:01.100 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:35:01.191 INFO 2981023 --- [ocp-async-1,733a131cb4ce3906,d77e6d0befea48c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:01.318 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048186, status:SUCCESS 2024-09-11 17:35:01.377 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048186, status:SUCCESS 2024-09-11 17:35:01.411 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:35:01.446 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:35:01.471 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:35:01.520 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:35:01.531 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:35:01.544 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:35:01.550 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:35:01.550 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:35:01.555 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:35:01.625 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:35:01.658 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:35:01.670 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:35:01.676 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:35:01.684 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:35:01.689 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:35:01.695 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:35:01.697 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:35:01.697 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:35:01.709 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:35:01.717 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:35:01.737 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,b058e7bca656322c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048165, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:35:01.941 INFO 2981023 --- [ocp-schedules-3,d2ae7f70b788b2d4,256d5c6026982b2f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:35:02.019 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:31:01.626, end=2024-09-11T17:33:02.014} 2024-09-11 17:35:02.023 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=17 2024-09-11 17:35:02.023 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:35:02.025 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:01.626, end=2024-09-11T17:33:02.014},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:35:02.025 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:35:02.025 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:35:02.033 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:35:02.053 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902053000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=AD63995D4A6E8C3787D81BDA88E06774, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.053Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.053 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902053000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.053Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.055 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:35:02.055 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.064 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.064 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.067 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.067 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.068 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 252 2024-09-11 17:35:02.068 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.072 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.072 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.073 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:35:02.073 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.053, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.074 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.074 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.075 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.075 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.080 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.080 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.082 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.086 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.053, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=4, sqlId=AD63995D4A6E8C3787D81BDA88E06774, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.086 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:35:02.089 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:01.626, end=2024-09-11T17:33:02.014},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:35:02.089 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:35:02.089 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:35:02.129 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902129000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.129Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.133 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902133000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.133Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.141 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902141000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.141Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.143 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902143000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.143Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.146 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902146000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.146Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.146 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 60 2024-09-11 17:35:02.146 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.147 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902147000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.147Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.148 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902148000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.148Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.155 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.155 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.155 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902155000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.155Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.164 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.164 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.164 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:35:02.164 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.168 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.168 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.173 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.173 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.173 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.173 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.177 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.177 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.181 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:35:02.181 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.182 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.182 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.183 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.183 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.185 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.185 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.186 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.186 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.189 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:35:02.189 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.190 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.190 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.192 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.192 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.192 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.192 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.195 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.195 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.195 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.195 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.196 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.196 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.197 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.197 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.198 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.198 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.201 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:02.202 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.202 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.202 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.202 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:02.203 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.203 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.204 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.204 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.205 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.207 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.207 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.207 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.209 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.209 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.209 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.209 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.210 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.210 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.211 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.211 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.212 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.141, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.212 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.212 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.213 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.213 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.214 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.214 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.214 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.133, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.215 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.129, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.219 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.143, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.221 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.148, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.221 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.146, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.223 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.147, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.226 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.226 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.229 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.236 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.155, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.250 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902250000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.250Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.252 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902252000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.252Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.254 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902254000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.254Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.258 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902258000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.258Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.262 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:35:02.262 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.266 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902266000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.266Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.266 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902266000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.266Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.268 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.268 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.269 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902269000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.269Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.271 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.271 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.277 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.277 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.278 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.278 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.278 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.278 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.280 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.280 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.283 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:35:02.283 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.285 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.285 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.286 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.286 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.286 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.286 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.288 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.288 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.290 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.290 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.291 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.291 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.292 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.292 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.294 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725960902294000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:35:02.294Z, endTime=2024-09-11T09:33:02.014Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:35:02.299 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.299 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.299 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.299 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.303 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.303 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.304 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.304 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.305 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.305 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.306 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:35:02.306 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:35:02.306 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.306 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.307 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.307 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.307 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.307 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.308 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.308 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.309 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.309 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.309 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.309 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.313 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.313 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.314 INFO 2981023 --- [prometheus-minute-schedule-1,35f4ba7a8e99410b,99c5339292e0dc6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:02.314 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.254, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.315 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.252, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.315 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.258, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.316 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.25, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.319 INFO 2981023 --- [prometheus-minute-schedule-1,35f4ba7a8e99410b,99c5339292e0dc6d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:02.324 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.324 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.327 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.327 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.327 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.327 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.328 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.328 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.329 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.330 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.331 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.332 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:35:02.332 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:35:02.336 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.269, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.337 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.266, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.337 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.266, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.340 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:35:02.340 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047182014000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:35:02.342 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:35:02.344 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:35:02.294, end=2024-09-11T17:33:02.014}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:35:02.345 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:35:02.345 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:01.626, end=2024-09-11T17:33:02.014},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:35:02.345 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:35:02.345 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:35:02.346 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:35:02.347 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:01.626, end=2024-09-11T17:33:02.014},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:35:02.347 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:35:02.347 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:35:02.347 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:35:02.349 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 5 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:01.626, end=2024-09-11T17:33:02.014},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:35:02.358 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:35:02.358 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:35:02.367 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:35:02.367 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:35:02.368 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:01.626, end=2024-09-11T17:33:02.014},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:35:02.368 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:35:02.368 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:35:02.369 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:35:02.403 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048185, status:SUCCESS 2024-09-11 17:35:02.403 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048185, status:SUCCESS 2024-09-11 17:35:03.187 INFO 2981023 --- [ocp-async-5,4eeff943ba477113,83ca73ffbbd059e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:05.187 INFO 2981023 --- [ocp-async-6,5daacbe3b2f52acc,8a40d093b485302e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:05.203 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:05.203 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,867b6835e1744300,8321e638d702ffd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,867b6835e1744300,8321e638d702ffd0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:35:06.352542+08:00, lastProcessTime=2024-09-11T17:34:56.351504+08:00 2024-09-11 17:35:06.478 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:35:06.352542+08:00 2024-09-11 17:35:07.187 INFO 2981023 --- [ocp-async-2,e8794491b101933e,96d9e5e6ece281a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:07.329 INFO 2981023 --- [ocp-schedules-3,d800165ccf06a6e1,4653f19083dd7ec2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:35:07.336 INFO 2981023 --- [ocp-schedules-22,296ea939597fe601,8bc8af570179e011] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:35:07.336+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:35:07.337 INFO 2981023 --- [ocp-schedules-22,296ea939597fe601,8bc8af570179e011] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:35:07.338 INFO 2981023 --- [ocp-schedules-22,296ea939597fe601,8bc8af570179e011] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:35:07.339 INFO 2981023 --- [ocp-schedules-22,296ea939597fe601,8bc8af570179e011] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:35:07.752 INFO 2981023 --- [ocp-async-6,c2c5d5f353379e91,781d33c8672726a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:35:08.203 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:08.204 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,0892c6854a38060c,d673a991a9f380cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,0892c6854a38060c,d673a991a9f380cd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:09.187 INFO 2981023 --- [ocp-async-1,48ce0a6f9031e9ed,402e37acedc74ec0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:10.003 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:35:10.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:35:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:35:10.919 INFO 2981023 --- [ocp-schedules-20,d4f9f2d069a8aa06,9bc64222ea289b71] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:35:10.920 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:35:10.955 WARN 2981023 --- [scanner-13509150-1,d4f9f2d069a8aa06,2f06bfbd90add47e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:35:11.189 INFO 2981023 --- [ocp-async-9,fd19ada9351d196b,99b8748e95acf073] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:11.204 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:11.204 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,f27247ed9bfcd4f4,1386b0bfdd4a277b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,f27247ed9bfcd4f4,1386b0bfdd4a277b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:11.922 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047240, nodeCount=1000, affectRows=1000 2024-09-11 17:35:11.924 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047240, nodeCount=999, validCount=999 2024-09-11 17:35:11.940 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047240, nodeCount=49, affectRows=49 2024-09-11 17:35:11.940 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047240, nodeCount=49, validCount=49 2024-09-11 17:35:11.940 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047240, nodeCount=1000, affectRows=1000 2024-09-11 17:35:11.944 INFO 2981023 --- [ocp-schedules-22,2ff4cf250272322a,7ae8be6ca33bc509] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:35:11.944 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047240, nodeCount=1000, validCount=1000 2024-09-11 17:35:13.188 INFO 2981023 --- [ocp-async-2,cdcca3b1e5eede63,b4c77e70d18a1fe2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:13.867 INFO 2981023 --- [ocp-schedules-9,7d59997104981075,b107b530171b3999] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:35:13.876 INFO 2981023 --- [ocp-schedules-9,7d59997104981075,b107b530171b3999] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:35:13.876 INFO 2981023 --- [ocp-schedules-9,7d59997104981075,b107b530171b3999] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:35:14.204 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:14.205 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,ab360cad652098fb,ad664716ebfef059] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,da6bee89feb149d3,1830d48580a389d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:35:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,ab360cad652098fb,ad664716ebfef059] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:15.188 INFO 2981023 --- [ocp-async-5,13e48114c88c6bf6,37c7fb305ffd5f48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:15.222 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047240, nodeCount=480, affectRows=480 2024-09-11 17:35:16.018 INFO 2981023 --- [ocp-schedules-28,3b6b46fb67228175,dfd2d04fed224620] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:35:16.018 INFO 2981023 --- [host-check-clockdiff1,3b6b46fb67228175,c3b135b727833b28] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:35:16.019 INFO 2981023 --- [host-check-clockdiff2,3b6b46fb67228175,a3d208dcf296cbc4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:35:16.019 INFO 2981023 --- [host-check-clockdiff4,3b6b46fb67228175,c361aab71112d556] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:35:16.019 INFO 2981023 --- [host-check-clockdiff3,3b6b46fb67228175,7ecfbe80bf48bc83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:35:16.024 INFO 2981023 --- [ocp-schedules-12,e140916e859b954f,aba34f605cc6bd49] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:35:16.027 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:35:16.027 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:35:16.027 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:35:16.037 INFO 2981023 --- [host-check-clockdiff2,3b6b46fb67228175,a3d208dcf296cbc4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047316 0 0 , error= 2024-09-11 17:35:16.037 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:35:16.039 INFO 2981023 --- [ocp-schedules-24,5339d6c213202bc5,cdcf50bd78a3c6bb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:35:16.057 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:35:16.058 INFO 2981023 --- [host-check-agent-status1,5339d6c213202bc5,d1fca26a154791fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:35:16.061 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:35:16.063 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:35:16.065 INFO 2981023 --- [host-check-agent-status2,5339d6c213202bc5,2fdbf32d4620bd0d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:35:16.065 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:35:16.065 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:35:16.066 INFO 2981023 --- [host-check-agent-status3,5339d6c213202bc5,ae9558ce8e13ca2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:35:16.072 INFO 2981023 --- [host-check-agent-status4,5339d6c213202bc5,10bda6ba80d0dd81] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:35:16.072 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:35:16.071 INFO 2981023 --- [host-check-clockdiff1,3b6b46fb67228175,c3b135b727833b28] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047316 0 0 , error= 2024-09-11 17:35:16.073 INFO 2981023 --- [host-check-clockdiff4,3b6b46fb67228175,c361aab71112d556] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047316 0 0 , error= 2024-09-11 17:35:16.074 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:35:16.075 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:35:16.078 INFO 2981023 --- [host-check-agent-status3,5339d6c213202bc5,ae9558ce8e13ca2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:35:16.081 INFO 2981023 --- [host-check-agent-status1,5339d6c213202bc5,d1fca26a154791fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:35:16.095 INFO 2981023 --- [host-check-agent-status1,5339d6c213202bc5,d1fca26a154791fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:35:16.102 INFO 2981023 --- [host-check-agent-status3,5339d6c213202bc5,ae9558ce8e13ca2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:35:16.196 INFO 2981023 --- [ocp-schedules-19,f5429ae6f5b20160,623199e967658f4d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:35:16.197 INFO 2981023 --- [ocp-schedules-17,e63de3651c0829c9,1ab4d39fe5386cfa] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:35:16.197 INFO 2981023 --- [ocp-schedules-16,52e49b78788a1317,0536a4c6a51ef9f5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:35:16.197 INFO 2981023 --- [ocp-schedules-7,a150e41da14ebdea,b46fe91404a70963] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:35:16.258 INFO 2981023 --- [host-check-agent-status4,5339d6c213202bc5,10bda6ba80d0dd81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:35:16.262 INFO 2981023 --- [host-check-agent-status4,5339d6c213202bc5,10bda6ba80d0dd81] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:35:16.278 INFO 2981023 --- [host-check-agent-status2,5339d6c213202bc5,2fdbf32d4620bd0d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:35:16.282 INFO 2981023 --- [host-check-agent-status2,5339d6c213202bc5,2fdbf32d4620bd0d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:35:16.282 INFO 2981023 --- [ocp-schedules-17,e63de3651c0829c9,1ab4d39fe5386cfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:35:16.287 INFO 2981023 --- [ocp-schedules-16,52e49b78788a1317,0536a4c6a51ef9f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:35:16.287 INFO 2981023 --- [ocp-schedules-19,f5429ae6f5b20160,623199e967658f4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:35:16.290 INFO 2981023 --- [host-check-agent-status3,5339d6c213202bc5,ae9558ce8e13ca2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:35:16.295 INFO 2981023 --- [host-check-agent-status3,5339d6c213202bc5,ae9558ce8e13ca2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:35:16.295 INFO 2981023 --- [host-check-agent-status3,5339d6c213202bc5,ae9558ce8e13ca2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:35:16.295 INFO 2981023 --- [host-check-agent-status3,5339d6c213202bc5,ae9558ce8e13ca2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:35:16.297 INFO 2981023 --- [ocp-schedules-10,53dfba151ea32039,c7e5f201160bc67f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:35:16.304 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:35:16.305 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:35:16.306 INFO 2981023 --- [host-check-agent-status1,5339d6c213202bc5,d1fca26a154791fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:35:16.306 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:35:16.305762596+08:00 2024-09-11 17:35:16.306 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:35:16.306+08:00 2024-09-11 17:35:16.307 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:35:16.307456279+08:00 2024-09-11 17:35:16.308 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:35:16.307+08:00 2024-09-11 17:35:16.308 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:35:16.306+08:00 and status AVAILABLE 2024-09-11 17:35:16.309 INFO 2981023 --- [host-check-agent-status1,5339d6c213202bc5,d1fca26a154791fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:35:16.309 INFO 2981023 --- [host-check-agent-status1,5339d6c213202bc5,d1fca26a154791fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:35:16.309 INFO 2981023 --- [host-check-agent-status1,5339d6c213202bc5,d1fca26a154791fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:35:16.312 INFO 2981023 --- [host-check-agent-status2,5339d6c213202bc5,2fdbf32d4620bd0d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:35:16.313 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:35:16.307+08:00 and status AVAILABLE 2024-09-11 17:35:16.315 INFO 2981023 --- [host-check-agent-status2,5339d6c213202bc5,2fdbf32d4620bd0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:35:16.315 INFO 2981023 --- [host-check-agent-status2,5339d6c213202bc5,2fdbf32d4620bd0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:35:16.315 INFO 2981023 --- [host-check-agent-status2,5339d6c213202bc5,2fdbf32d4620bd0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:35:16.290 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:35:16.316 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:35:16.316690875+08:00 2024-09-11 17:35:16.319 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:35:16.319 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:35:16.320 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:35:16.316+08:00 2024-09-11 17:35:16.322 INFO 2981023 --- [ocp-schedules-17,e63de3651c0829c9,1ab4d39fe5386cfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:35:16.359 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:35:16.316+08:00 and status AVAILABLE 2024-09-11 17:35:16.361 INFO 2981023 --- [host-check-agent-status4,5339d6c213202bc5,10bda6ba80d0dd81] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:35:16.375 INFO 2981023 --- [host-check-agent-status4,5339d6c213202bc5,10bda6ba80d0dd81] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:35:16.375 INFO 2981023 --- [host-check-agent-status4,5339d6c213202bc5,10bda6ba80d0dd81] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:35:16.375 INFO 2981023 --- [host-check-agent-status4,5339d6c213202bc5,10bda6ba80d0dd81] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:35:16.380 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:35:16.386 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:35:16.390 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:35:16.394 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:35:16.395 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:35:16.393304+08:00, lastProcessTime=2024-09-11T17:35:06.352542+08:00 2024-09-11 17:35:16.397 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:35:16.411 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:35:16.413 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:35:16.422 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:35:16.423 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.423 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:35:16.423274967+08:00 2024-09-11 17:35:16.423 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:16.424 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.424 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:35:16.424 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:35:16.423+08:00 2024-09-11 17:35:16.425 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:16.425 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:35:16.426 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:35:16.427 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:35:16.423+08:00 and status AVAILABLE 2024-09-11 17:35:16.428 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.429 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:16.429 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:35:16.430 INFO 2981023 --- [ocp-schedules-19,f5429ae6f5b20160,623199e967658f4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:35:16.432 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.433 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:16.434 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.434 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:16.453 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:35:16.453 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:35:16.458 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:35:16.465 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:35:16.467 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:35:16.467 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.468 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:35:16.468 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:16.474 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:35:16.480 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:35:16.490 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:35:16.503 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:35:16.514 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:35:16.515 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:35:16.517 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.518 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.519 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:35:16.521 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:35:16.526 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:35:16.538 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.538 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.539 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:35:16.539 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:35:16.541 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:35:16.555 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:35:16.555 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.556 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.558 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:16.561 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.562 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:35:16.564 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:35:16.564 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:35:16.565 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:35:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:35:16.578 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:16.578 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:35:16.580 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:35:16.592 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:35:16.593 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.593 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:35:16.594 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.595 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:35:16.597 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:16.601 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:35:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:35:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:35:16.607 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:35:16.608 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.610 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.613 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:16.614 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:35:16.614 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:35:16.617 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:35:16.617 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:35:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:35:16.621 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:35:16.623 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.624 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:35:16.625 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.625 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:35:16.625 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:35:16.626 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:16.626 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:35:16.627 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.627 INFO 2981023 --- [host-check-host-status3,e140916e859b954f,a9e758be9500022d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:35:16.630 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:35:16.631 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:35:16.632 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:35:16.636 INFO 2981023 --- [host-check-host-status2,e140916e859b954f,1051cfd0c162c973] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:35:16.643 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:16.643 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:35:16.648 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:35:16.653 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:16.660 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:35:16.665 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.666 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:16.666 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:35:16.671 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:35:16.673 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.674 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.674 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:16.674 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:35:16.675 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:35:16.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:35:16.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:35:16.678 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:35:16.683 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:35:16.684 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:35:16.687 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:35:16.695 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:35:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:35:16.713 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:35:16.715 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:35:16.721 INFO 2981023 --- [host-check-host-status4,e140916e859b954f,a01a3172abd0c58b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:35:16.722 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.723 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.723 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:35:16.724 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:35:16.731 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:35:16.731 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:35:16.742 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:35:16.742 INFO 2981023 --- [host-check-host-status1,e140916e859b954f,74834108f40329ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:35:16.749 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.750 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.750 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:35:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:35:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:35:16.761 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:35:16.393304+08:00 2024-09-11 17:35:16.772 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:35:16.777 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:16.778 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:16.778 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:35:16.783 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:35:16.783 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:35:16.783 INFO 2981023 --- [ocp-schedules-16,52e49b78788a1317,0536a4c6a51ef9f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:35:17.188 INFO 2981023 --- [ocp-async-8,11b08057c67cc932,4677e3c8248b0a10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:17.205 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:17.205 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,23e8f6de5e4bfd7f,f586d4d37812ac43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,23e8f6de5e4bfd7f,f586d4d37812ac43] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:17.344 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047240, nodeCount=480, validCount=480 2024-09-11 17:35:17.752 INFO 2981023 --- [ocp-async-5,6df09bf8a9227217,e3e5fa563848992a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:35:19.188 INFO 2981023 --- [ocp-async-2,7c179b7f896443b8,a29bb66106d16088] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:20.205 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:20.207 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,1970e498d615d5a6,be791999c2207c42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,1970e498d615d5a6,be791999c2207c42] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:21.120 INFO 2981023 --- [ocp-schedules-6,0144e2a604f9bb86,cf7f53d4ff200f11] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:35:21.172 WARN 2981023 --- [scanner-13510879-1,0144e2a604f9bb86,940acb3020866c25] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:35:21.185 INFO 2981023 --- [alarm-detect-2,0144e2a604f9bb86,6bb911447bd42961] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644179, data is empty. 2024-09-11 17:35:21.188 INFO 2981023 --- [ocp-async-7,dd6da05ff79246b5,3d2296877e7ffc54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:21.189 INFO 2981023 --- [alarm-detect-2,0144e2a604f9bb86,6bb911447bd42961] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644179, data is empty. 2024-09-11 17:35:21.192 INFO 2981023 --- [alarm-detect-2,0144e2a604f9bb86,6bb911447bd42961] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644179, data is empty. 2024-09-11 17:35:21.230 INFO 2981023 --- [alarm-detect-19,0144e2a604f9bb86,987b2e655923fbf0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644179, data is empty. 2024-09-11 17:35:21.230 INFO 2981023 --- [alarm-detect-17,0144e2a604f9bb86,ce833b0dbf00789c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644179, data is empty. 2024-09-11 17:35:21.233 INFO 2981023 --- [alarm-detect-17,0144e2a604f9bb86,ce833b0dbf00789c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644179, data is empty. 2024-09-11 17:35:21.233 INFO 2981023 --- [alarm-detect-19,0144e2a604f9bb86,987b2e655923fbf0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644179, data is empty. 2024-09-11 17:35:21.236 INFO 2981023 --- [alarm-detect-17,0144e2a604f9bb86,ce833b0dbf00789c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644179, data is empty. 2024-09-11 17:35:21.237 INFO 2981023 --- [alarm-detect-19,0144e2a604f9bb86,987b2e655923fbf0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644179, data is empty. 2024-09-11 17:35:21.357 INFO 2981023 --- [alarm-detect-12,0144e2a604f9bb86,208500129a2b814b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644179, data is empty. 2024-09-11 17:35:21.365 INFO 2981023 --- [alarm-detect-12,0144e2a604f9bb86,208500129a2b814b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644179, data is empty. 2024-09-11 17:35:21.368 INFO 2981023 --- [alarm-detect-12,0144e2a604f9bb86,208500129a2b814b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644179, data is empty. 2024-09-11 17:35:21.949 INFO 2981023 --- [ocp-schedules-12,e37cc37742f20171,db79646da84b5957] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:35:21.981 INFO 2981023 --- [host-check-clockdiff3,3b6b46fb67228175,7ecfbe80bf48bc83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047321 1 1 , error= 2024-09-11 17:35:22.709 INFO 2981023 --- [ocp-schedules-27,24bd92f14c2ee203,301ff16cea6e6bf7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:35:22.709 INFO 2981023 --- [ocp-schedules-27,24bd92f14c2ee203,301ff16cea6e6bf7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:35:22.709 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:35:22.709 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:35:22.709 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:35:22.715 INFO 2981023 --- [ocp-schedules-27,24bd92f14c2ee203,301ff16cea6e6bf7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:35:22.715 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:35:22.719 INFO 2981023 --- [ocp-schedules-27,24bd92f14c2ee203,301ff16cea6e6bf7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:35:22.719 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:35:22.719 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:35:22.719 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:35:22.720 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:35:22.723 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:35:22.745 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:35:22.747 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:35:22.748 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:35:22.748 INFO 2981023 --- [ocp-schedules-27,24bd92f14c2ee203,301ff16cea6e6bf7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:35:22.750 INFO 2981023 --- [ocp-schedules-27,24bd92f14c2ee203,301ff16cea6e6bf7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:35:23.188 INFO 2981023 --- [ocp-async-4,a2e629d4fe8be332,b88533939fe3c510] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:23.207 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:23.209 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,5e57d669fdcbe3fc,bd46c2c154d9aae8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,5e57d669fdcbe3fc,bd46c2c154d9aae8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:24.218 INFO 2981023 --- [ocp-schedules-5,cb696fd111105934,98e5cbd28c6dab32] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:35:25.188 INFO 2981023 --- [ocp-async-6,af2d1052e270670b,a850af75dc11360f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:26.209 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:26.209 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,4e72afcfdcc66a66,474e930428cb59cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,4e72afcfdcc66a66,474e930428cb59cd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:35:26.351686+08:00, lastProcessTime=2024-09-11T17:35:16.393304+08:00 2024-09-11 17:35:26.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:35:26.351686+08:00 2024-09-11 17:35:27.189 INFO 2981023 --- [ocp-async-3,10b9a965bde77479,3b5a18bff80919af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:27.752 INFO 2981023 --- [ocp-async-6,ebc54b43120e2923,8f7d1a888ca466ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:35:29.189 INFO 2981023 --- [ocp-async-1,73c4c0241e8388f4,864e37ba51bc9169] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:29.209 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:29.210 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,9a733622860c9fee,8f115943e2b6607a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,9a733622860c9fee,8f115943e2b6607a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:31.189 INFO 2981023 --- [ocp-async-10,ffd54e8fff8265aa,a32f131424416095] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:31.413 INFO 2981023 --- [ocp-schedules-4,3b6e974c6bc637ad,2fc3ac7c328f974e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:35:31.514 WARN 2981023 --- [scanner-13511195-1,3b6e974c6bc637ad,f020fb3172ab7f99] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:35:31.952 INFO 2981023 --- [ocp-schedules-28,6656030888d76915,801b5757d3989f8b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:35:32.210 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:32.210 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,61a7fbb06817ddfb,da69f89d99148016] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,61a7fbb06817ddfb,da69f89d99148016] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:33.189 INFO 2981023 --- [ocp-async-5,7dc77dae3abd028f,ef6b23babf678bea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:33.878 INFO 2981023 --- [ocp-schedules-20,bef2cf30ad74c5af,74421fe4337607df] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:35:33.886 INFO 2981023 --- [ocp-schedules-20,bef2cf30ad74c5af,74421fe4337607df] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:35:33.886 INFO 2981023 --- [ocp-schedules-20,bef2cf30ad74c5af,74421fe4337607df] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:35:35.189 INFO 2981023 --- [ocp-async-8,91ff44bd3b0afc7c,619413a8a4c05850] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:35.210 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:35.211 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,f0ef07534d790fd1,3303bd2d069534e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,f0ef07534d790fd1,3303bd2d069534e1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:35:36.338030+08:00, lastProcessTime=2024-09-11T17:35:26.351686+08:00 2024-09-11 17:35:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:35:36.338030+08:00 2024-09-11 17:35:37.189 INFO 2981023 --- [ocp-async-1,c102b5f193235fe8,c7543e34f5620233] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:37.314 INFO 2981023 --- [prometheus-cache-monitor1,9a2f5629a3a7569d,a506a91bf4aa09c1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8197,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:35:37.315 INFO 2981023 --- [prometheus-cache-monitor1,9a2f5629a3a7569d,a506a91bf4aa09c1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:35:37.752 INFO 2981023 --- [ocp-async-8,7e92916d40c301af,b8f2cdda3c73c4d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:35:38.211 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:38.211 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,b73115ef080ac47d,43db1499605d3907] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,b73115ef080ac47d,43db1499605d3907] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:39.189 INFO 2981023 --- [ocp-async-2,98eab1d3a8b41f49,b1aab9765412d002] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:39.416 INFO 2981023 --- [ocp-schedules-9,c4041a5f25ed97e6,4d8a9074ffd15646] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:35:39.434 WARN 2981023 --- [ocp-schedules-9,c4041a5f25ed97e6,4d8a9074ffd15646] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:35:40.003 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:35:40.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:35:41.189 INFO 2981023 --- [ocp-async-4,ed58aa9d8fd292c3,ddefdb1b739095f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:41.211 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:41.212 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,9742a1ca6c137ff4,c23d5d4b46904c65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,9742a1ca6c137ff4,c23d5d4b46904c65] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:41.622 INFO 2981023 --- [ocp-schedules-12,286a502717522ef5,19d187e110f23e0c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:35:41.633 WARN 2981023 --- [scanner-13511380-1,286a502717522ef5,c3e7f7afb7f4eca5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:35:41.720 INFO 2981023 --- [alarm-detect-19,286a502717522ef5,0d3290074015c012] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644179, data is empty. 2024-09-11 17:35:41.725 INFO 2981023 --- [alarm-detect-19,286a502717522ef5,0d3290074015c012] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644179, data is empty. 2024-09-11 17:35:41.729 INFO 2981023 --- [alarm-detect-19,286a502717522ef5,0d3290074015c012] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644179, data is empty. 2024-09-11 17:35:41.746 INFO 2981023 --- [alarm-detect-3,286a502717522ef5,9fa3f9cf0dcd1bdc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644179, data is empty. 2024-09-11 17:35:41.747 INFO 2981023 --- [alarm-detect-10,286a502717522ef5,03105b5f52c968d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644179, data is empty. 2024-09-11 17:35:41.750 INFO 2981023 --- [alarm-detect-3,286a502717522ef5,9fa3f9cf0dcd1bdc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644179, data is empty. 2024-09-11 17:35:41.752 INFO 2981023 --- [alarm-detect-10,286a502717522ef5,03105b5f52c968d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644179, data is empty. 2024-09-11 17:35:41.753 INFO 2981023 --- [alarm-detect-3,286a502717522ef5,9fa3f9cf0dcd1bdc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644179, data is empty. 2024-09-11 17:35:41.756 INFO 2981023 --- [alarm-detect-10,286a502717522ef5,03105b5f52c968d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644179, data is empty. 2024-09-11 17:35:41.856 INFO 2981023 --- [alarm-detect-5,286a502717522ef5,90dac01d6f833f3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644179, data is empty. 2024-09-11 17:35:41.861 INFO 2981023 --- [alarm-detect-5,286a502717522ef5,90dac01d6f833f3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644179, data is empty. 2024-09-11 17:35:41.867 INFO 2981023 --- [alarm-detect-5,286a502717522ef5,90dac01d6f833f3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644179, data is empty. 2024-09-11 17:35:41.954 INFO 2981023 --- [ocp-schedules-22,f02d52b6afb2b96b,8cbe3101c8f1a11e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:35:43.189 INFO 2981023 --- [ocp-async-6,64b5b941d438a224,07001bc4f26f1208] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:44.212 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:44.212 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,9fdaa84b9ab4078e,6173278fcaefeb3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,9fdaa84b9ab4078e,6173278fcaefeb3f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:45.101 INFO 2981023 --- [prometheus-second-schedule-1,1c62c71670b2085b,b8045d0bb68dd10d] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:35:45.190 INFO 2981023 --- [ocp-async-3,9d5296cd57aecd7a,f460b87d5f9bb226] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:46.016 INFO 2981023 --- [ocp-schedules-4,6888e4f0b9ed0aab,1bbf75d26db15f63] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:35:46.016 INFO 2981023 --- [host-check-clockdiff1,6888e4f0b9ed0aab,84ab245aea806364] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:35:46.017 INFO 2981023 --- [host-check-clockdiff2,6888e4f0b9ed0aab,b6299a5d60770fed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:35:46.018 INFO 2981023 --- [host-check-clockdiff3,6888e4f0b9ed0aab,efb6d620d19d2638] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:35:46.019 INFO 2981023 --- [host-check-clockdiff4,6888e4f0b9ed0aab,0e6ab8fbff12f0f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:35:46.030 INFO 2981023 --- [host-check-clockdiff2,6888e4f0b9ed0aab,b6299a5d60770fed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047346 0 0 , error= 2024-09-11 17:35:46.036 INFO 2981023 --- [ocp-schedules-7,51baa56b89326838,f05ef5ee4c71f566] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:35:46.036 INFO 2981023 --- [ocp-schedules-28,b9f2760de4bb0f9c,ff99f2c7b2d545a6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:35:46.036 INFO 2981023 --- [host-check-clockdiff4,6888e4f0b9ed0aab,0e6ab8fbff12f0f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047346 0 0 , error= 2024-09-11 17:35:46.036 INFO 2981023 --- [host-check-agent-status1,51baa56b89326838,d1981ed7d8142fcb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:35:46.036 INFO 2981023 --- [host-check-agent-status2,51baa56b89326838,9db9c5fad8beac25] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:35:46.037 INFO 2981023 --- [host-check-agent-status3,51baa56b89326838,b451a536343eaaa0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:35:46.037 INFO 2981023 --- [host-check-agent-status4,51baa56b89326838,67090837818a40f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:35:46.037 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:35:46.037 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:35:46.043 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:35:46.047 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:35:46.049 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:35:46.049 INFO 2981023 --- [host-check-agent-status2,51baa56b89326838,9db9c5fad8beac25] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:35:46.050 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:35:46.050 INFO 2981023 --- [host-check-agent-status2,51baa56b89326838,9db9c5fad8beac25] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:35:46.050 INFO 2981023 --- [host-check-agent-status1,51baa56b89326838,d1981ed7d8142fcb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:35:46.051 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:35:46.051 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:35:46.052 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:35:46.052 INFO 2981023 --- [host-check-agent-status1,51baa56b89326838,d1981ed7d8142fcb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:35:46.052 INFO 2981023 --- [host-check-agent-status4,51baa56b89326838,67090837818a40f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:35:46.053 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:35:46.054 INFO 2981023 --- [host-check-agent-status3,51baa56b89326838,b451a536343eaaa0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:35:46.054 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:35:46.054 INFO 2981023 --- [host-check-agent-status4,51baa56b89326838,67090837818a40f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:35:46.055 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:35:46.056 INFO 2981023 --- [host-check-agent-status3,51baa56b89326838,b451a536343eaaa0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:35:46.080 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:35:46.080 INFO 2981023 --- [host-check-agent-status3,51baa56b89326838,b451a536343eaaa0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:35:46.082 INFO 2981023 --- [host-check-agent-status1,51baa56b89326838,d1981ed7d8142fcb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:35:46.083 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:35:46.083209286+08:00 2024-09-11 17:35:46.083 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:35:46.085 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:35:46.083+08:00 2024-09-11 17:35:46.086 INFO 2981023 --- [host-check-agent-status3,51baa56b89326838,b451a536343eaaa0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:35:46.086 INFO 2981023 --- [host-check-agent-status3,51baa56b89326838,b451a536343eaaa0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:35:46.086 INFO 2981023 --- [host-check-agent-status3,51baa56b89326838,b451a536343eaaa0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:35:46.086 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:35:46.085990267+08:00 2024-09-11 17:35:46.087 INFO 2981023 --- [host-check-agent-status1,51baa56b89326838,d1981ed7d8142fcb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:35:46.087 INFO 2981023 --- [host-check-agent-status1,51baa56b89326838,d1981ed7d8142fcb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:35:46.087 INFO 2981023 --- [host-check-agent-status1,51baa56b89326838,d1981ed7d8142fcb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:35:46.088 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:35:46.083+08:00 and status AVAILABLE 2024-09-11 17:35:46.088 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:35:46.086+08:00 2024-09-11 17:35:46.098 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:35:46.100 INFO 2981023 --- [host-check-agent-status4,51baa56b89326838,67090837818a40f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:35:46.100 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:35:46.099869484+08:00 2024-09-11 17:35:46.101 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:35:46.100+08:00 2024-09-11 17:35:46.102 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:35:46.086+08:00 and status AVAILABLE 2024-09-11 17:35:46.110 INFO 2981023 --- [host-check-agent-status4,51baa56b89326838,67090837818a40f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:35:46.110 INFO 2981023 --- [host-check-agent-status4,51baa56b89326838,67090837818a40f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:35:46.110 INFO 2981023 --- [host-check-agent-status4,51baa56b89326838,67090837818a40f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:35:46.110 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:35:46.100+08:00 and status AVAILABLE 2024-09-11 17:35:46.116 INFO 2981023 --- [host-check-agent-status2,51baa56b89326838,9db9c5fad8beac25] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:35:46.117 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:35:46.118 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:35:46.118764584+08:00 2024-09-11 17:35:46.118 INFO 2981023 --- [host-check-agent-status2,51baa56b89326838,9db9c5fad8beac25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:35:46.118 INFO 2981023 --- [host-check-agent-status2,51baa56b89326838,9db9c5fad8beac25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:35:46.118 INFO 2981023 --- [host-check-agent-status2,51baa56b89326838,9db9c5fad8beac25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:35:46.119 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:35:46.118+08:00 2024-09-11 17:35:46.120 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:35:46.121 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:35:46.118+08:00 and status AVAILABLE 2024-09-11 17:35:46.138 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:35:46.140 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:35:46.141 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:35:46.143 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:35:46.144 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:35:46.151 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:35:46.152 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:35:46.155 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:35:46.157 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:35:46.160 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:35:46.168 INFO 2981023 --- [ocp-schedules-16,f5a7e4454f262668,7d9816f67ab44804] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:35:46.182 INFO 2981023 --- [ocp-schedules-16,f5a7e4454f262668,7d9816f67ab44804] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:35:46.187 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:35:46.188 INFO 2981023 --- [ocp-schedules-21,3180d4aab47bd3f6,b2a7ab2d2990676b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:35:46.188 INFO 2981023 --- [ocp-schedules-9,8282c83bcdd7db09,21acacb6cb06bc1d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:35:46.188 INFO 2981023 --- [ocp-schedules-20,1a8ecabc25f99146,5e6ccd37462cb755] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:35:46.188 INFO 2981023 --- [ocp-schedules-22,7a2e826dff2e843c,252f8588ee228cc4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:35:46.189 INFO 2981023 --- [ocp-schedules-9,8282c83bcdd7db09,21acacb6cb06bc1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:35:46.191 INFO 2981023 --- [ocp-schedules-21,3180d4aab47bd3f6,b2a7ab2d2990676b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:35:46.191 INFO 2981023 --- [ocp-schedules-20,1a8ecabc25f99146,5e6ccd37462cb755] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:35:46.199 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:35:46.199 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:35:46.206 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:35:46.210 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:35:46.210 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:35:46.218 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:35:46.220 INFO 2981023 --- [ocp-schedules-9,8282c83bcdd7db09,21acacb6cb06bc1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:35:46.238 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:35:46.252 INFO 2981023 --- [ocp-schedules-13,ba3476fd39c9075f,5a5fc393390ee4c8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:35:46.269 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:35:46.271 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:35:46.291 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:35:46.297 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:35:46.311 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.311 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:46.312 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:35:46.328 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.328 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:46.329 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:35:46.335 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.335 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:35:46.336 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:35:46.337 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:35:46.337 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.341 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:35:46.340831+08:00, lastProcessTime=2024-09-11T17:35:36.338030+08:00 2024-09-11 17:35:46.345 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:35:46.351 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:35:46.351 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:35:46.355 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.355 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:46.356 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:35:46.358 INFO 2981023 --- [ocp-schedules-21,3180d4aab47bd3f6,b2a7ab2d2990676b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:35:46.359 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.359 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:46.365 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:35:46.371 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:35:46.373 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:35:46.374 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:35:46.383 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.383 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:46.389 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:35:46.389 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:35:46.391 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:35:46.396 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.397 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:46.405 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:35:46.421 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:35:46.426 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:35:46.430 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.433 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:35:46.434 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.435 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.436 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:35:46.436 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:35:46.436 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.442 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:35:46.447 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:35:46.454 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:35:46.459 INFO 2981023 --- [host-check-host-status3,b9f2760de4bb0f9c,991c7c676cc6f0ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:35:46.460 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:35:46.461 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.461 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.462 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:46.463 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:35:46.463 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:35:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:35:46.468 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:35:46.470 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:35:46.472 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:35:46.482 INFO 2981023 --- [host-check-host-status1,b9f2760de4bb0f9c,7c7aae5263c307c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:35:46.483 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:35:46.485 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:46.485 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:35:46.489 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.493 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.493 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:35:46.501 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.509 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:35:46.509 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:46.516 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:35:46.517 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.519 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.520 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.520 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:46.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:35:46.521 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.522 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:35:46.523 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:35:46.523 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.524 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:35:46.525 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:35:46.525 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.526 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:35:46.529 INFO 2981023 --- [host-check-host-status4,b9f2760de4bb0f9c,99ef1edf46cd5ec2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:35:46.531 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:35:46.532 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:35:46.532 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:35:46.540 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.540 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.541 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:46.542 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.544 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:35:46.544 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:35:46.559 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.560 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.561 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:35:46.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:35:46.566 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:35:46.568 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:46.568 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:35:46.571 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:35:46.591 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:46.592 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.595 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.595 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:35:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.596 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:35:46.596 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.601 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:35:46.615 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:35:46.616 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:35:46.617 INFO 2981023 --- [host-check-host-status2,b9f2760de4bb0f9c,9b3cd05dc0217a12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:35:46.620 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.620 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.621 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:35:46.621 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.621 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:46.622 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:35:46.623 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:35:46.624 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:35:46.631 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:35:46.637 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:46.637 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:35:46.639 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:35:46.640 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:35:46.642 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:35:46.648 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:35:46.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.658 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.659 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:35:46.661 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:35:46.666 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:35:46.666 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:35:46.670 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.671 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.671 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:35:46.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:35:46.672 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:35:46.672 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.680 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.680 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.680 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:35:46.681 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:35:46.683 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:35:46.686 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:35:46.687 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:35:46.696 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:35:46.340831+08:00 2024-09-11 17:35:46.700 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:35:46.717 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:35:46.720 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:35:46.726 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.727 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.727 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:35:46.729 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:35:46.734 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:35:46.734 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:35:46.734 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:35:46.736 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:46.737 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:46.737 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:35:46.741 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:35:46.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:35:46.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:35:46.744 INFO 2981023 --- [ocp-schedules-20,1a8ecabc25f99146,5e6ccd37462cb755] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:35:46.744 INFO 2981023 --- [ocp-schedules-10,b80a60b700547786,79c7166afe5392d6] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:35:46.837 INFO 2981023 --- [ocp-schedules-12,9072cf9a7534510a,87551e66385e40f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8197,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:35:46.838 INFO 2981023 --- [ocp-schedules-12,9072cf9a7534510a,87551e66385e40f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:35:46.891 INFO 2981023 --- [ocp-schedules-15,faba53c82d7b2504,8e514a5fe9dca219] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:35:46.966 INFO 2981023 --- [host-check-clockdiff1,6888e4f0b9ed0aab,84ab245aea806364] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047346 0 0 , error= 2024-09-11 17:35:47.190 INFO 2981023 --- [ocp-async-5,5830fcb39c5d6d1f,b03ad8a2e2eb9c6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:47.213 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:47.213 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,9813b8c88c985514,78b98342b4737994] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,9813b8c88c985514,78b98342b4737994] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:47.753 INFO 2981023 --- [ocp-async-4,ad2c51e24c17d958,7c4d417757d0d862] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:35:49.190 INFO 2981023 --- [ocp-async-1,d2499a35070f168e,08e7f20d98c77026] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:35:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:35:50.213 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:50.215 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:50.216 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.221 INFO 2981023 --- [ocp-schedules-2,73d46d63265cf3bf,11c2e7464f0a6f68] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:35:50.226 INFO 2981023 --- [ocp-schedules-15,6f58ea2a0c3a9f55,f2b07c0ef478aca0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:35:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,ec6b9f5a397dc374,7cfb07541b0bb3b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,ec6b9f5a397dc374,7cfb07541b0bb3b7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:51.190 INFO 2981023 --- [ocp-async-7,dedf00373dbeb859,6d54dfbad938035e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:51.539 INFO 2981023 --- [ocp-schedules-14,ff845b50273ee809,914ec4d6a12657d5] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:35:51.540 INFO 2981023 --- [ocp-schedules-14,ff845b50273ee809,914ec4d6a12657d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:35:51.540 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:35:51.540 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:35:51.540 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:35:51.541 INFO 2981023 --- [ocp-schedules-14,ff845b50273ee809,914ec4d6a12657d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:35:51.911 INFO 2981023 --- [ocp-schedules-6,fd4852e3d7321ddc,8ac4158150c62ab7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:35:51.957 INFO 2981023 --- [ocp-schedules-30,96facf06fbc8ce2a,8b1d3353a3fe32f7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:35:51.963 WARN 2981023 --- [scanner-13511691-1,fd4852e3d7321ddc,b00a8c7294849278] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:35:52.082 INFO 2981023 --- [host-check-clockdiff3,6888e4f0b9ed0aab,efb6d620d19d2638] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047352 1 1 , error= 2024-09-11 17:35:52.358 INFO 2981023 --- [ocp-schedules-27,58a701943fe411b9,76cdbe903176ef45] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:35:52.358+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:35:52.362 INFO 2981023 --- [ocp-schedules-27,58a701943fe411b9,76cdbe903176ef45] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:35:52.499 INFO 2981023 --- [ocp-schedules-27,58a701943fe411b9,76cdbe903176ef45] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 44 to SqlAggregateData 2024-09-11 17:35:52.500 INFO 2981023 --- [ocp-schedules-27,58a701943fe411b9,76cdbe903176ef45] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:35:52.720 INFO 2981023 --- [ocp-schedules-22,ae98841bd6172d05,141a412ef186788d] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:35:52.738 WARN 2981023 --- [ocp-schedules-22,ae98841bd6172d05,141a412ef186788d] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:35:53.089 INFO 2981023 --- [ocp-schedules-21,af66456cd15cb703,794d43f7fec6d65c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:35:53.089 INFO 2981023 --- [ocp-schedules-21,af66456cd15cb703,794d43f7fec6d65c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:35:53.167 INFO 2981023 --- [ocp-schedules-15,ad43a888c934780d,657c9c01b813ef71] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:35:53.168 INFO 2981023 --- [ocp-schedules-15,ad43a888c934780d,657c9c01b813ef71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:35:53.190 INFO 2981023 --- [ocp-async-8,e0e4f37a3d262355,ca88e25d9a524465] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:53.201 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:35:53.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:35:53.202 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:35:53.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.205 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:35:53.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:35:53.207 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:35:53.209 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.209 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.210 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.210 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.211 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.211 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.218 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.218 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.219 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:53.222 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.222 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.222 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.223 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.224 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:53.224 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.225 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:35:53.231 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.231 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:35:53.234 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.234 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.235 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:35:53.235 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2851D6E3-0-0] 2024-09-11 17:35:53.235 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:35:53.236 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.236 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.236 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.236 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.237 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:35:53.237 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:35:53.243 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.243 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.243 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:35:53.244 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:35:53.244 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.245 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:35:53.245 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.246 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:35:53.246 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:35:53.248 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.248 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:35:53.249 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:35:53.256 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.257 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:35:53.258 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.259 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:35:53.262 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.262 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:35:53.262 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:53.263 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:35:53.263 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2851D6ED-0-0] 2024-09-11 17:35:53.263 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:35:53.264 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.264 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.265 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.266 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:35:53.267 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.268 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:35:53.269 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:35:53.269 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.269 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.269 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:35:53.270 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:35:53.270 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.270 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.271 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:35:53.271 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:35:53.271 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.271 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.272 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:35:53.272 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:35:53.272 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.272 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.273 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:35:53.273 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:35:53.274 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.274 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.274 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:35:53.274 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:35:53.275 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:35:53.275 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:35:53.276 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:35:53.276 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:35:53.279 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:35:53.282 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:35:53.282 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:35:53.282 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:35:53.289 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:53.290 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:35:53.290 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.291 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:35:53.293 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.294 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:35:53.300 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:35:53.302 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:35:53.302 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.302 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:35:53.303 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:35:53.305 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:35:53.305 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.305 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.308 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:35:53.310 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:35:53.310 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:53.311 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:35:53.316 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:35:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,f3305786c228af6d,7ba3570b5e9d3c23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:53.317 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:35:53.318 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:35:53.318 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:35:53.318 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:35:53.319 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:35:53.319 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:35:53.319 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:35:53.319 INFO 2981023 --- [ocp-schedules-15,ad43a888c934780d,657c9c01b813ef71] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:35:53.319 INFO 2981023 --- [prometheus-minute-schedule-1,f3305786c228af6d,7ba3570b5e9d3c23] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:53.889 INFO 2981023 --- [ocp-schedules-5,d6f181a4d067d535,560c205a83b18413] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:35:53.897 INFO 2981023 --- [ocp-schedules-5,d6f181a4d067d535,560c205a83b18413] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:35:53.897 INFO 2981023 --- [ocp-schedules-5,d6f181a4d067d535,560c205a83b18413] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:35:54.220 INFO 2981023 --- [ocp-schedules-9,46eab295a6a83c7f,55a980f1b7b1d8e1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:35:55.190 INFO 2981023 --- [ocp-async-10,6720c39afcb3ae71,b5595271b45aea3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:55.677 INFO 2981023 --- [ocp-schedules-20,f24c78294842b9b2,0d8253040c32eb21] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:35:55.677 INFO 2981023 --- [ocp-schedules-20,f24c78294842b9b2,0d8253040c32eb21] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:35:55.678 INFO 2981023 --- [ocp-schedules-20,f24c78294842b9b2,0d8253040c32eb21] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:35:55.678 INFO 2981023 --- [ocp-schedules-20,f24c78294842b9b2,0d8253040c32eb21] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:35:55.679 INFO 2981023 --- [ocp-schedules-20,f24c78294842b9b2,0d8253040c32eb21] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:35:55.679 INFO 2981023 --- [ocp-schedules-20,f24c78294842b9b2,0d8253040c32eb21] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:35:55.763 INFO 2981023 --- [ocp-schedules-9,cb36e260cdbb3fef,a0af5234f612983f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:35:55.769 INFO 2981023 --- [ocp-schedules-9,cb36e260cdbb3fef,a0af5234f612983f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:35:55.769 INFO 2981023 --- [ocp-schedules-9,cb36e260cdbb3fef,a0af5234f612983f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:35:56.169 INFO 2981023 --- [ocp-schedules-12,b690b6706e4263ef,930035788f2d5a12] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:35:56.169 INFO 2981023 --- [ocp-schedules-12,b690b6706e4263ef,930035788f2d5a12] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:35:56.182 INFO 2981023 --- [Collect all tenants major compactions1,b690b6706e4263ef,dd820d748d5487fc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:35:56.183 INFO 2981023 --- [Collect all tenants major compactions4,b690b6706e4263ef,a9be185e8d110066] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:35:56.183 INFO 2981023 --- [Collect all tenants major compactions5,b690b6706e4263ef,1b481dc198d921d0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:35:56.184 INFO 2981023 --- [Collect all tenants major compactions6,b690b6706e4263ef,9875eaaf1fd25eef] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:35:56.184 INFO 2981023 --- [Collect all tenants major compactions3,b690b6706e4263ef,0798a6d72959fae7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:35:56.185 INFO 2981023 --- [Collect all tenants major compactions2,b690b6706e4263ef,dff4e62a6e39df9d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:35:56.196 INFO 2981023 --- [Collect all tenants major compactions4,b690b6706e4263ef,a9be185e8d110066] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:56.196 INFO 2981023 --- [Collect all tenants major compactions3,b690b6706e4263ef,0798a6d72959fae7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:56.196 INFO 2981023 --- [Collect all tenants major compactions2,b690b6706e4263ef,dff4e62a6e39df9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:56.196 INFO 2981023 --- [Collect all tenants major compactions1,b690b6706e4263ef,dd820d748d5487fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:56.196 INFO 2981023 --- [Collect all tenants major compactions5,b690b6706e4263ef,1b481dc198d921d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:56.197 INFO 2981023 --- [Collect all tenants major compactions3,b690b6706e4263ef,0798a6d72959fae7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:35:56.197 INFO 2981023 --- [Collect all tenants major compactions4,b690b6706e4263ef,a9be185e8d110066] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:35:56.197 INFO 2981023 --- [Collect all tenants major compactions5,b690b6706e4263ef,1b481dc198d921d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:35:56.197 INFO 2981023 --- [Collect all tenants major compactions1,b690b6706e4263ef,dd820d748d5487fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:35:56.198 INFO 2981023 --- [Collect all tenants major compactions2,b690b6706e4263ef,dff4e62a6e39df9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:35:56.200 INFO 2981023 --- [Collect all tenants major compactions6,b690b6706e4263ef,9875eaaf1fd25eef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:35:56.201 INFO 2981023 --- [Collect all tenants major compactions2,b690b6706e4263ef,dff4e62a6e39df9d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:35:56.201 INFO 2981023 --- [Collect all tenants major compactions2,b690b6706e4263ef,dff4e62a6e39df9d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:35:56.201 INFO 2981023 --- [Collect all tenants major compactions1,b690b6706e4263ef,dd820d748d5487fc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:35:56.201 INFO 2981023 --- [Collect all tenants major compactions1,b690b6706e4263ef,dd820d748d5487fc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:35:56.201 INFO 2981023 --- [Collect all tenants major compactions6,b690b6706e4263ef,9875eaaf1fd25eef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:35:56.202 INFO 2981023 --- [Collect all tenants major compactions3,b690b6706e4263ef,0798a6d72959fae7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:35:56.202 INFO 2981023 --- [Collect all tenants major compactions4,b690b6706e4263ef,a9be185e8d110066] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:35:56.202 INFO 2981023 --- [Collect all tenants major compactions3,b690b6706e4263ef,0798a6d72959fae7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:35:56.202 INFO 2981023 --- [Collect all tenants major compactions4,b690b6706e4263ef,a9be185e8d110066] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:35:56.202 INFO 2981023 --- [Collect all tenants major compactions5,b690b6706e4263ef,1b481dc198d921d0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:35:56.202 INFO 2981023 --- [Collect all tenants major compactions5,b690b6706e4263ef,1b481dc198d921d0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:35:56.203 INFO 2981023 --- [Collect all tenants major compactions6,b690b6706e4263ef,9875eaaf1fd25eef] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:35:56.203 INFO 2981023 --- [Collect all tenants major compactions6,b690b6706e4263ef,9875eaaf1fd25eef] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:35:56.204 INFO 2981023 --- [ocp-schedules-12,b690b6706e4263ef,930035788f2d5a12] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:35:56.224 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:56.225 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,5a52fa010559fafb,01e2aa72037cf4e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,5a52fa010559fafb,01e2aa72037cf4e7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:35:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:35:56.353078+08:00, lastProcessTime=2024-09-11T17:35:46.340831+08:00 2024-09-11 17:35:56.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:35:56.353078+08:00 2024-09-11 17:35:57.190 INFO 2981023 --- [ocp-async-2,0cbaa9db35d8d4a1,77b02a77f971a740] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:57.753 INFO 2981023 --- [ocp-async-3,bfca5649fa9406e1,e10379e34d10d601] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:35:59.190 INFO 2981023 --- [ocp-async-9,fe7280cb5704c184,6396a098354a8cc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:35:59.225 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:35:59.225 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:35:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,1dd51e16b38c3f42,fbec44703c68469b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047360, interval=60 2024-09-11 17:35:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,1dd51e16b38c3f42,fbec44703c68469b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:01.199 INFO 2981023 --- [ocp-async-10,114303da965f1a3a,0302a9137babdf84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:01.962 INFO 2981023 --- [ocp-schedules-28,190b2e71ad8be5a0,db9a3ac35e12a1ec] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:36:02.115 INFO 2981023 --- [ocp-schedules-11,f7f5f23b028495bf,692a54e5896dc712] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:36:02.151 WARN 2981023 --- [scanner-13511925-1,f7f5f23b028495bf,5cbe7a61a064c6fb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:36:02.175 INFO 2981023 --- [alarm-detect-10,f7f5f23b028495bf,80ab7c03b0e24d52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644179, data is empty. 2024-09-11 17:36:02.181 INFO 2981023 --- [alarm-detect-10,f7f5f23b028495bf,80ab7c03b0e24d52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644223, data is empty. 2024-09-11 17:36:02.186 INFO 2981023 --- [alarm-detect-10,f7f5f23b028495bf,80ab7c03b0e24d52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644223, data is empty. 2024-09-11 17:36:02.205 INFO 2981023 --- [alarm-detect-5,f7f5f23b028495bf,893d647078456bc2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644179, data is empty. 2024-09-11 17:36:02.211 INFO 2981023 --- [alarm-detect-15,f7f5f23b028495bf,8d29afcbba5cb687] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644179, data is empty. 2024-09-11 17:36:02.212 INFO 2981023 --- [alarm-detect-5,f7f5f23b028495bf,893d647078456bc2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644223, data is empty. 2024-09-11 17:36:02.215 INFO 2981023 --- [alarm-detect-15,f7f5f23b028495bf,8d29afcbba5cb687] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644223, data is empty. 2024-09-11 17:36:02.217 INFO 2981023 --- [alarm-detect-5,f7f5f23b028495bf,893d647078456bc2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644223, data is empty. 2024-09-11 17:36:02.219 INFO 2981023 --- [alarm-detect-15,f7f5f23b028495bf,8d29afcbba5cb687] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644223, data is empty. 2024-09-11 17:36:02.225 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:02.226 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,7f85094c05fe4ca9,214de66cd0856278] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:02.322 INFO 2981023 --- [alarm-detect-14,f7f5f23b028495bf,0b6bbe87cae0b5fd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644179, data is empty. 2024-09-11 17:36:02.326 INFO 2981023 --- [prometheus-minute-schedule-1,7f85094c05fe4ca9,214de66cd0856278] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:02.335 INFO 2981023 --- [alarm-detect-14,f7f5f23b028495bf,0b6bbe87cae0b5fd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644223, data is empty. 2024-09-11 17:36:02.344 INFO 2981023 --- [alarm-detect-14,f7f5f23b028495bf,0b6bbe87cae0b5fd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644223, data is empty. 2024-09-11 17:36:02.448 INFO 2981023 --- [ocp-schedules-7,5ec1876e299ec83c,13abfbf51cf5b5d4] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:36:02.449 INFO 2981023 --- [ocp-schedules-7,5ec1876e299ec83c,13abfbf51cf5b5d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:02.457 INFO 2981023 --- [ocp-schedules-7,5ec1876e299ec83c,13abfbf51cf5b5d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:36:03.190 INFO 2981023 --- [ocp-async-10,6d7da7adb342ea0f,5dceafd91b92f9e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:05.191 INFO 2981023 --- [ocp-async-8,0d5097efb6ae2c77,072f615fb2b66588] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:05.226 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:05.228 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f2c93f1a8fefb45d,e877ac9f9acf61ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,f2c93f1a8fefb45d,e877ac9f9acf61ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:36:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:36:05.995 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:36:06.000 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:36:06.005 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:36:06.005 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:36:06.005 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:36:06.008 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:36:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:36:06.010 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:06.010 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:06.010 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:36:06.010 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:36:06.063 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:33:40, start=2024-09-11T17:30:10, end=2024-09-11T17:32:40, updates=135, granule=120000ms, delay=210000ms 2024-09-11 17:36:06.068 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:33:40, target=2024-09-11T17:32:40, granule=120000ms, delay=60000ms) 2024-09-11 17:36:06.068 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:32:40, target=2024-09-11T17:19:41, granule=600000ms, delay=779000ms) 2024-09-11 17:36:06.145 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:33:41, start=2024-09-11T17:30:11, end=2024-09-11T17:32:41, updates=301, granule=120000ms, delay=210000ms 2024-09-11 17:36:06.156 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:33:41, target=2024-09-11T17:32:41, granule=120000ms, delay=60000ms) 2024-09-11 17:36:06.156 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:32:41, target=2024-09-11T17:20:51, granule=600000ms, delay=710000ms) 2024-09-11 17:36:06.156 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=189ms, period=120000ms 2024-09-11 17:36:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:36:06.352328+08:00, lastProcessTime=2024-09-11T17:35:56.353078+08:00 2024-09-11 17:36:06.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:36:06.352328+08:00 2024-09-11 17:36:07.191 INFO 2981023 --- [ocp-async-6,05a6e6de93abeb48,fff996f22c2bfb0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:07.346 INFO 2981023 --- [ocp-schedules-10,899e1d54d1632b07,79d951a898870c2f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:36:07.352 INFO 2981023 --- [ocp-schedules-28,4ba43f3ede7b8f35,f83a30d5a62f15ea] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:36:07.352+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:36:07.353 INFO 2981023 --- [ocp-schedules-28,4ba43f3ede7b8f35,f83a30d5a62f15ea] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:36:07.354 INFO 2981023 --- [ocp-schedules-28,4ba43f3ede7b8f35,f83a30d5a62f15ea] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:36:07.354 INFO 2981023 --- [ocp-schedules-28,4ba43f3ede7b8f35,f83a30d5a62f15ea] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:36:07.753 INFO 2981023 --- [ocp-async-8,caa4fbf4f37ed744,f792d62554972445] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:36:08.228 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:08.230 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,094efb7f9a147ff9,6c8e1718502feae8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:08.317 INFO 2981023 --- [prometheus-minute-schedule-1,094efb7f9a147ff9,6c8e1718502feae8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:09.191 INFO 2981023 --- [ocp-async-2,8a625aa09107e564,3da0d94cd5ea1110] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:10.003 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:36:10.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:36:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:36:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:36:11.211 INFO 2981023 --- [ocp-async-5,f7e445db3b516f1b,0cb543d2efd931ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:11.230 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:11.230 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,d0b127f8cafeeb75,dfda94b1b06d0414] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,d0b127f8cafeeb75,dfda94b1b06d0414] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:11.964 INFO 2981023 --- [ocp-schedules-2,0df895164ac9e3d2,e8b6426eb2b7f2a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:36:12.112 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047300, nodeCount=1000, affectRows=1000 2024-09-11 17:36:12.116 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047300, nodeCount=1000, validCount=1000 2024-09-11 17:36:12.125 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047300, nodeCount=1000, affectRows=1000 2024-09-11 17:36:12.128 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047300, nodeCount=999, validCount=999 2024-09-11 17:36:12.129 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047300, nodeCount=48, affectRows=48 2024-09-11 17:36:12.129 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047300, nodeCount=48, validCount=48 2024-09-11 17:36:12.376 INFO 2981023 --- [ocp-schedules-20,fd7d974ff9aa9717,7191ba5a545436e9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:36:12.423 WARN 2981023 --- [scanner-13513749-1,fd7d974ff9aa9717,7cff9c492195335c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:36:12.623 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047300, nodeCount=480, affectRows=480 2024-09-11 17:36:13.211 INFO 2981023 --- [ocp-async-6,89289f406f7ce792,30540456fb6277bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:13.901 INFO 2981023 --- [ocp-schedules-27,0a99edfbe86ba71c,f6b46bfba81c11e1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:36:13.911 INFO 2981023 --- [ocp-schedules-27,0a99edfbe86ba71c,f6b46bfba81c11e1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:36:13.911 INFO 2981023 --- [ocp-schedules-27,0a99edfbe86ba71c,f6b46bfba81c11e1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:36:14.230 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:14.231 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,72d9362613aed306,83655a73f212e64e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,e5946a260a773547,42452102e86c2daa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:36:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,72d9362613aed306,83655a73f212e64e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:14.364 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047300, nodeCount=480, validCount=480 2024-09-11 17:36:15.211 INFO 2981023 --- [ocp-async-4,8b20af0f82e9e955,cbc60baae8de11a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:16.017 INFO 2981023 --- [ocp-schedules-19,a109d1efc64748f1,600bbe020b68b99d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:36:16.018 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:36:16.018 INFO 2981023 --- [ocp-schedules-5,22aeff41ae57d629,feab17a779ad0880] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:36:16.018 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:36:16.019 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:36:16.020 INFO 2981023 --- [host-check-clockdiff1,22aeff41ae57d629,690467797123fb38] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:36:16.026 INFO 2981023 --- [host-check-clockdiff2,22aeff41ae57d629,b70642a4e7bbde35] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:36:16.028 INFO 2981023 --- [ocp-schedules-22,4a5774a22e2ca5be,f06c39723110f3c7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:36:16.032 INFO 2981023 --- [host-check-clockdiff3,22aeff41ae57d629,5b978d22f3e4b1e7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:36:16.033 INFO 2981023 --- [host-check-clockdiff4,22aeff41ae57d629,1e823080b70f2d93] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:36:16.034 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:36:16.043 INFO 2981023 --- [host-check-agent-status1,4a5774a22e2ca5be,d241c191f9243dd7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:36:16.043 INFO 2981023 --- [host-check-clockdiff1,22aeff41ae57d629,690467797123fb38] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047376 0 0 , error= 2024-09-11 17:36:16.046 INFO 2981023 --- [host-check-agent-status2,4a5774a22e2ca5be,92fbddf032977997] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:36:16.046 INFO 2981023 --- [host-check-agent-status3,4a5774a22e2ca5be,df14696ed5f42ca5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:36:16.047 INFO 2981023 --- [host-check-agent-status4,4a5774a22e2ca5be,5de7e2f05594a395] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:36:16.048 INFO 2981023 --- [host-check-clockdiff4,22aeff41ae57d629,1e823080b70f2d93] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047376 0 0 , error= 2024-09-11 17:36:16.048 INFO 2981023 --- [host-check-clockdiff2,22aeff41ae57d629,b70642a4e7bbde35] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047376 0 0 , error= 2024-09-11 17:36:16.057 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:36:16.062 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:36:16.063 INFO 2981023 --- [host-check-agent-status4,4a5774a22e2ca5be,5de7e2f05594a395] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:36:16.064 INFO 2981023 --- [host-check-agent-status4,4a5774a22e2ca5be,5de7e2f05594a395] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:36:16.065 INFO 2981023 --- [host-check-agent-status1,4a5774a22e2ca5be,d241c191f9243dd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:36:16.066 INFO 2981023 --- [host-check-agent-status1,4a5774a22e2ca5be,d241c191f9243dd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:36:16.070 INFO 2981023 --- [host-check-agent-status2,4a5774a22e2ca5be,92fbddf032977997] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:36:16.084 INFO 2981023 --- [host-check-agent-status2,4a5774a22e2ca5be,92fbddf032977997] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:36:16.190 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:36:16.191 INFO 2981023 --- [ocp-schedules-23,2d5d2fd6f03fb7ee,3f2d48b6aef5874a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:36:16.191 INFO 2981023 --- [ocp-schedules-10,ae47bc4b0ece6028,9e5b831ab4ec57e1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:36:16.191 INFO 2981023 --- [ocp-schedules-29,40cf89cf36e4f998,b5f4bf8d8050ec62] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:36:16.191 INFO 2981023 --- [ocp-schedules-7,994df2b49da6aa5c,2a3866cf618dd7ca] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:36:16.197 INFO 2981023 --- [ocp-schedules-23,2d5d2fd6f03fb7ee,3f2d48b6aef5874a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:16.256 INFO 2981023 --- [ocp-schedules-29,40cf89cf36e4f998,b5f4bf8d8050ec62] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:16.256 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:36:16.256 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:36:16.264 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:36:16.275 INFO 2981023 --- [ocp-schedules-10,ae47bc4b0ece6028,9e5b831ab4ec57e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:16.281 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:36:16.283 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:36:16.286 INFO 2981023 --- [host-check-agent-status3,4a5774a22e2ca5be,df14696ed5f42ca5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:36:16.287 INFO 2981023 --- [host-check-agent-status3,4a5774a22e2ca5be,df14696ed5f42ca5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:36:16.299 INFO 2981023 --- [ocp-schedules-10,ae47bc4b0ece6028,9e5b831ab4ec57e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:36:16.300 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:36:16.303 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:36:16.302633855+08:00 2024-09-11 17:36:16.305 INFO 2981023 --- [host-check-agent-status2,4a5774a22e2ca5be,92fbddf032977997] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:36:16.305 INFO 2981023 --- [ocp-schedules-15,c79462c88714ee4d,b1c12fe7b3036bab] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:36:16.308 INFO 2981023 --- [host-check-agent-status2,4a5774a22e2ca5be,92fbddf032977997] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:36:16.308 INFO 2981023 --- [host-check-agent-status2,4a5774a22e2ca5be,92fbddf032977997] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:36:16.308 INFO 2981023 --- [host-check-agent-status2,4a5774a22e2ca5be,92fbddf032977997] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:36:16.309 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:36:16.303+08:00 2024-09-11 17:36:16.313 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:36:16.303+08:00 and status AVAILABLE 2024-09-11 17:36:16.358 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.359 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:36:16.362 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:36:16.362829420+08:00 2024-09-11 17:36:16.363 INFO 2981023 --- [host-check-agent-status1,4a5774a22e2ca5be,d241c191f9243dd7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:36:16.364 INFO 2981023 --- [host-check-agent-status4,4a5774a22e2ca5be,5de7e2f05594a395] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:36:16.364 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:36:16.363+08:00 2024-09-11 17:36:16.366 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:16.366 INFO 2981023 --- [host-check-agent-status1,4a5774a22e2ca5be,d241c191f9243dd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:36:16.366 INFO 2981023 --- [host-check-agent-status1,4a5774a22e2ca5be,d241c191f9243dd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:36:16.366 INFO 2981023 --- [host-check-agent-status1,4a5774a22e2ca5be,d241c191f9243dd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:36:16.367 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:36:16.363+08:00 and status AVAILABLE 2024-09-11 17:36:16.367 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:36:16.368 INFO 2981023 --- [host-check-agent-status4,4a5774a22e2ca5be,5de7e2f05594a395] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:36:16.368 INFO 2981023 --- [host-check-agent-status4,4a5774a22e2ca5be,5de7e2f05594a395] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:36:16.368 INFO 2981023 --- [host-check-agent-status4,4a5774a22e2ca5be,5de7e2f05594a395] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:36:16.371 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:36:16.372 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:36:16.371951976+08:00 2024-09-11 17:36:16.373 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:36:16.373 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.373 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:36:16.372+08:00 2024-09-11 17:36:16.374 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:16.375 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:36:16.376 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:36:16.377 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:36:16.376688031+08:00 2024-09-11 17:36:16.378 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:36:16.377+08:00 2024-09-11 17:36:16.380 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.381 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:16.382 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:36:16.382 INFO 2981023 --- [ocp-schedules-23,2d5d2fd6f03fb7ee,3f2d48b6aef5874a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:36:16.384 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:36:16.377+08:00 and status AVAILABLE 2024-09-11 17:36:16.385 INFO 2981023 --- [host-check-agent-status3,4a5774a22e2ca5be,df14696ed5f42ca5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:36:16.390 INFO 2981023 --- [host-check-agent-status3,4a5774a22e2ca5be,df14696ed5f42ca5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:36:16.390 INFO 2981023 --- [host-check-agent-status3,4a5774a22e2ca5be,df14696ed5f42ca5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:36:16.390 INFO 2981023 --- [host-check-agent-status3,4a5774a22e2ca5be,df14696ed5f42ca5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:36:16.391 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:36:16.399 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:36:16.398789+08:00, lastProcessTime=2024-09-11T17:36:06.352328+08:00 2024-09-11 17:36:16.400 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:36:16.402 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:36:16.402 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:36:16.402 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.402 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:16.404 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:36:16.372+08:00 and status AVAILABLE 2024-09-11 17:36:16.417 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:36:16.418 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:36:16.418 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:36:16.419 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:36:16.420 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:36:16.454 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:36:16.459 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.459 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:16.461 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:36:16.462 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.464 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:16.466 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:36:16.473 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:36:16.474 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:36:16.482 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:36:16.494 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:36:16.496 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:36:16.501 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:36:16.506 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:36:16.509 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:36:16.520 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:36:16.531 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.534 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.535 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.535 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.535 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:36:16.536 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:36:16.536 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:36:16.537 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:36:16.544 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:36:16.558 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:36:16.559 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:36:16.561 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:36:16.564 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:36:16.569 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:36:16.571 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:36:16.582 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.583 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.583 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:16.584 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.584 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.585 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:16.585 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:36:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.587 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:36:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:36:16.588 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:36:16.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:36:16.590 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:36:16.595 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:36:16.597 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:36:16.599 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:36:16.602 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:36:16.608 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:16.608 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:36:16.609 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:16.609 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:36:16.609 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.610 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.611 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:36:16.611 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:36:16.622 INFO 2981023 --- [host-check-host-status3,a109d1efc64748f1,54096bbb03d0505c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:36:16.623 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:36:16.625 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:16.625 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:36:16.630 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:36:16.637 INFO 2981023 --- [host-check-host-status1,a109d1efc64748f1,d274463d77947577] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:36:16.637 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.637 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:36:16.637 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:16.637 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.638 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:16.639 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:36:16.655 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:36:16.658 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:36:16.660 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:36:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:36:16.663 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.663 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.663 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.664 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.664 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:16.664 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:16.665 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:36:16.665 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:36:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:36:16.669 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:36:16.670 INFO 2981023 --- [host-check-host-status2,a109d1efc64748f1,2b14257fc7f38f21] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:36:16.673 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:36:16.677 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:36:16.679 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:36:16.680 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:36:16.681 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:16.681 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:36:16.683 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:36:16.704 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:36:16.710 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:16.714 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:36:16.719 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.720 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.720 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:16.721 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:36:16.722 INFO 2981023 --- [host-check-host-status4,a109d1efc64748f1,ad53b292022cc369] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:36:16.725 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:36:16.728 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:36:16.729 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:36:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:36:16.753 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:36:16.759 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.759 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.760 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:36:16.764 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:36:16.766 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:36:16.766 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:36:16.770 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.771 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.772 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:36:16.777 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:36:16.778 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:36:16.778 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:36:16.782 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:16.783 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:16.783 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:36:16.789 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:36:16.789 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:36:16.789 INFO 2981023 --- [ocp-schedules-29,40cf89cf36e4f998,b5f4bf8d8050ec62] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:36:16.801 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:36:16.398789+08:00 2024-09-11 17:36:17.211 INFO 2981023 --- [ocp-async-7,9e9155bece44be16,04a4f793424c0dbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:17.232 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:17.232 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,6ce66c81bcf7664c,4f25872038454167] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,6ce66c81bcf7664c,4f25872038454167] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:17.753 INFO 2981023 --- [ocp-async-4,470cd41a2c02c3ee,847cb0130c58f28a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:36:19.211 INFO 2981023 --- [ocp-async-1,da21c6d55aa7d811,791c659408796be9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:20.232 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:20.233 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,0cd729d4814e8011,e73203dfbedfddbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,0cd729d4814e8011,e73203dfbedfddbf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:21.211 INFO 2981023 --- [ocp-async-10,d0e6edf3e6a6f0e1,c52bbcaa35c35eb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:21.967 INFO 2981023 --- [ocp-schedules-14,811db5420ebc2e09,983501645989f457] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:36:22.035 INFO 2981023 --- [host-check-clockdiff3,22aeff41ae57d629,5b978d22f3e4b1e7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047382 1 1 , error= 2024-09-11 17:36:22.602 INFO 2981023 --- [ocp-schedules-15,f8eb571109c2819f,8eee1192526d9e2f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:36:22.645 WARN 2981023 --- [scanner-13513978-1,f8eb571109c2819f,ab82b9a95b58d192] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:36:22.664 INFO 2981023 --- [alarm-detect-17,f8eb571109c2819f,8763a0f4debaff52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644223, data is empty. 2024-09-11 17:36:22.667 INFO 2981023 --- [alarm-detect-17,f8eb571109c2819f,8763a0f4debaff52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644223, data is empty. 2024-09-11 17:36:22.669 INFO 2981023 --- [alarm-detect-17,f8eb571109c2819f,8763a0f4debaff52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644223, data is empty. 2024-09-11 17:36:22.699 INFO 2981023 --- [alarm-detect-15,f8eb571109c2819f,4dba16c33104d89b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644223, data is empty. 2024-09-11 17:36:22.700 INFO 2981023 --- [alarm-detect-6,f8eb571109c2819f,7389c4047f30b8e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644223, data is empty. 2024-09-11 17:36:22.704 INFO 2981023 --- [alarm-detect-6,f8eb571109c2819f,7389c4047f30b8e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644223, data is empty. 2024-09-11 17:36:22.704 INFO 2981023 --- [alarm-detect-15,f8eb571109c2819f,4dba16c33104d89b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644223, data is empty. 2024-09-11 17:36:22.708 INFO 2981023 --- [alarm-detect-6,f8eb571109c2819f,7389c4047f30b8e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644223, data is empty. 2024-09-11 17:36:22.708 INFO 2981023 --- [alarm-detect-15,f8eb571109c2819f,4dba16c33104d89b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644223, data is empty. 2024-09-11 17:36:22.754 INFO 2981023 --- [ocp-schedules-13,37c6a5ce12f56d15,43b93d594fe53f20] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:36:22.754 INFO 2981023 --- [ocp-schedules-13,37c6a5ce12f56d15,43b93d594fe53f20] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:36:22.754 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:36:22.754 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:36:22.757 INFO 2981023 --- [ocp-schedules-13,37c6a5ce12f56d15,43b93d594fe53f20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:36:22.757 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:36:22.758 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:36:22.758 INFO 2981023 --- [ocp-schedules-13,37c6a5ce12f56d15,43b93d594fe53f20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:36:22.759 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:36:22.763 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:36:22.766 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:36:22.767 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:36:22.770 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:36:22.797 INFO 2981023 --- [ocp-schedules-13,37c6a5ce12f56d15,43b93d594fe53f20] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:36:22.797 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:36:22.797 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:36:22.802 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:36:22.814 INFO 2981023 --- [ocp-schedules-13,37c6a5ce12f56d15,43b93d594fe53f20] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:36:22.843 INFO 2981023 --- [alarm-detect-19,f8eb571109c2819f,37fa8584f63bf242] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644223, data is empty. 2024-09-11 17:36:22.851 INFO 2981023 --- [alarm-detect-19,f8eb571109c2819f,37fa8584f63bf242] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644223, data is empty. 2024-09-11 17:36:22.853 INFO 2981023 --- [alarm-detect-19,f8eb571109c2819f,37fa8584f63bf242] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644223, data is empty. 2024-09-11 17:36:23.211 INFO 2981023 --- [ocp-async-7,88ca74950d79a889,37a3011251035ff0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:23.233 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:23.233 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,a5afcd1d179d8933,949c2886f40a5174] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,a5afcd1d179d8933,949c2886f40a5174] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:24.224 INFO 2981023 --- [ocp-schedules-16,53e60161892ab5f7,8212a424fe61ade3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:36:25.211 INFO 2981023 --- [ocp-async-6,160c46ed4861c860,402bbda7cf04a7b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:26.233 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:26.234 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,9376a497e9ffb765,092e5c2de5c21f7e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,9376a497e9ffb765,092e5c2de5c21f7e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:36:26.338103+08:00, lastProcessTime=2024-09-11T17:36:16.398789+08:00 2024-09-11 17:36:26.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:36:26.338103+08:00 2024-09-11 17:36:27.212 INFO 2981023 --- [ocp-async-4,9bd93e041c7ef647,865fcc435b6cc38b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:27.753 INFO 2981023 --- [ocp-async-6,0e8cc928179bc1e9,7f4ea773746b5c79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:36:28.709 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:36:28.709 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:36:28.711 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:36:28.711 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:36:28.711 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:36:28.711 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:36:28.714 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:36:28.714 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:36:28.714 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:36:28.714 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:36:28.737 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:36:28.741 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:36:28.741 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:36:28.741 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:36:28.741 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:36:28.741 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:36:28.744 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:36:29.014 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:36:29.212 INFO 2981023 --- [ocp-async-7,be8c742c9e13d2e5,f061f7c0cd7df943] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:29.234 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:29.234 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:29.272 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.557891326S 2024-09-11 17:36:29.272 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:36:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,1a2c2a972a5d30c8,3352c4b648564c34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,1a2c2a972a5d30c8,3352c4b648564c34] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:29.325 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:36:29.325 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:36:29.340 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:36:29.372 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:36:29.372 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:36:29.372 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:36:29.372 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:36:29.372 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:36:29.372 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:36:29.377 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:36:29.458 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] o.s.boot.SpringApplication : Started application in 0.747 seconds (JVM running for 83189.331) 2024-09-11 17:36:29.458 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.747111652S 2024-09-11 17:36:29.660 INFO 2981023 --- [ocp-schedules-20,faa82e3501fc8983,fd5ee7cc7117f688] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:36:31.212 INFO 2981023 --- [ocp-async-8,4e183a1fb31cdb2c,0660a9fef4694b09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:31.970 INFO 2981023 --- [ocp-schedules-26,a547e6d86bf2ccbd,d9831a98abf7541b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:36:32.234 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:32.235 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,8ad9b39c5654cdd5,8748bde2b13d70a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,8ad9b39c5654cdd5,8748bde2b13d70a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:32.895 INFO 2981023 --- [ocp-schedules-8,b5ff5c6c3b186e10,a5e223566d8cb569] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:36:32.956 WARN 2981023 --- [scanner-13514308-1,b5ff5c6c3b186e10,e7dc36aadba293ca] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:36:33.212 INFO 2981023 --- [ocp-async-3,2cc4ca1dd82e0739,c10f14df072b578f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:33.912 INFO 2981023 --- [ocp-schedules-16,9d390222569c16b5,e9c8794d677f3c46] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:36:33.918 INFO 2981023 --- [ocp-schedules-16,9d390222569c16b5,e9c8794d677f3c46] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:36:33.918 INFO 2981023 --- [ocp-schedules-16,9d390222569c16b5,e9c8794d677f3c46] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:36:35.212 INFO 2981023 --- [ocp-async-10,dc09f2ad943a2afe,5e26af64f844b7f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:35.235 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:35.236 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,a9ad3d0110f9a8fc,74168d5332ae8603] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,a9ad3d0110f9a8fc,74168d5332ae8603] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:36:36.336378+08:00, lastProcessTime=2024-09-11T17:36:26.338103+08:00 2024-09-11 17:36:36.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:36:36.336378+08:00 2024-09-11 17:36:37.212 INFO 2981023 --- [ocp-async-7,bbde0a31cb93ce69,e1810082e49f569f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:37.314 INFO 2981023 --- [prometheus-cache-monitor1,60709511869fbb27,b3a757ebb2becb52] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8197,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:36:37.315 INFO 2981023 --- [prometheus-cache-monitor1,60709511869fbb27,b3a757ebb2becb52] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:36:37.753 INFO 2981023 --- [ocp-async-10,79beb96c2901060c,cc91700711040e50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:36:38.236 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:38.237 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,b25367102fd4f3c5,5a02ac79554aedb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,b25367102fd4f3c5,5a02ac79554aedb2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:39.212 INFO 2981023 --- [ocp-async-1,b0edb71f05a7f829,b171c30bac3ce267] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:39.572 INFO 2981023 --- [ocp-schedules-7,a079936d98bc2606,5ed69981cb07dcb0] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:36:39.591 WARN 2981023 --- [ocp-schedules-7,a079936d98bc2606,5ed69981cb07dcb0] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:36:40.002 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:36:40.002 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:36:41.212 INFO 2981023 --- [ocp-async-9,227a0e1c1b5ccc96,cd60a00004507bb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:41.237 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:41.242 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,fb95606bd4cc3aee,4969461ba5bd30a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,fb95606bd4cc3aee,4969461ba5bd30a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:41.973 INFO 2981023 --- [ocp-schedules-9,c7776231829244ce,f25a0e5114fc925c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:36:43.132 INFO 2981023 --- [ocp-schedules-14,d73200a523804a22,35c4e666b3a321fd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:36:43.170 WARN 2981023 --- [scanner-13514485-1,d73200a523804a22,2bf24be49707b970] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:36:43.194 INFO 2981023 --- [alarm-detect-16,d73200a523804a22,383c4eabdb6d6f9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644223, data is empty. 2024-09-11 17:36:43.197 INFO 2981023 --- [alarm-detect-16,d73200a523804a22,383c4eabdb6d6f9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644223, data is empty. 2024-09-11 17:36:43.200 INFO 2981023 --- [alarm-detect-16,d73200a523804a22,383c4eabdb6d6f9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644223, data is empty. 2024-09-11 17:36:43.235 INFO 2981023 --- [ocp-async-6,e504fef7d0009b80,58e5530fcd90847d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:43.248 INFO 2981023 --- [alarm-detect-12,d73200a523804a22,bd5c904b7f178d9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644223, data is empty. 2024-09-11 17:36:43.251 INFO 2981023 --- [alarm-detect-12,d73200a523804a22,bd5c904b7f178d9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644223, data is empty. 2024-09-11 17:36:43.255 INFO 2981023 --- [alarm-detect-5,d73200a523804a22,1b798dfe0059a940] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644223, data is empty. 2024-09-11 17:36:43.260 INFO 2981023 --- [alarm-detect-5,d73200a523804a22,1b798dfe0059a940] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644223, data is empty. 2024-09-11 17:36:43.262 INFO 2981023 --- [alarm-detect-12,d73200a523804a22,bd5c904b7f178d9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644223, data is empty. 2024-09-11 17:36:43.271 INFO 2981023 --- [alarm-detect-5,d73200a523804a22,1b798dfe0059a940] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644223, data is empty. 2024-09-11 17:36:43.363 INFO 2981023 --- [alarm-detect-7,d73200a523804a22,3bc60d1632e38dfb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644223, data is empty. 2024-09-11 17:36:43.370 INFO 2981023 --- [alarm-detect-7,d73200a523804a22,3bc60d1632e38dfb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644223, data is empty. 2024-09-11 17:36:43.376 INFO 2981023 --- [alarm-detect-7,d73200a523804a22,3bc60d1632e38dfb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644223, data is empty. 2024-09-11 17:36:44.242 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:44.242 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,e03cbd927d89d778,fcc32e8af5f5e5c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:44.317 INFO 2981023 --- [prometheus-minute-schedule-1,e03cbd927d89d778,fcc32e8af5f5e5c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:45.101 INFO 2981023 --- [prometheus-second-schedule-1,8003cb522ff883a2,42e980c2b6607cc3] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:36:45.235 INFO 2981023 --- [ocp-async-4,7e75eac13324359d,84e34176b2de37df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:45.998 INFO 2981023 --- [ocp-schedules-14,d982d700f6847c4c,26a3cf3edd5edb68] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:36:45.998 INFO 2981023 --- [host-check-clockdiff1,d982d700f6847c4c,a6ad48523eca6cd6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:36:45.998 INFO 2981023 --- [host-check-clockdiff2,d982d700f6847c4c,ceb905b98e881003] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:36:45.998 INFO 2981023 --- [host-check-clockdiff3,d982d700f6847c4c,203eed1f1d9c5722] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:36:46.001 INFO 2981023 --- [host-check-clockdiff4,d982d700f6847c4c,f08fb844b56e8215] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:36:46.003 INFO 2981023 --- [ocp-schedules-10,b29626494ba18e4f,35205a84fa025d83] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:36:46.008 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:36:46.008 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:36:46.009 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:36:46.009 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:36:46.009 INFO 2981023 --- [ocp-schedules-13,132961fadc07a174,5db1384ddffbc9eb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:36:46.011 INFO 2981023 --- [host-check-agent-status1,132961fadc07a174,5ad4c47e13a923b4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:36:46.014 INFO 2981023 --- [host-check-clockdiff2,d982d700f6847c4c,ceb905b98e881003] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047406 0 0 , error= 2024-09-11 17:36:46.015 INFO 2981023 --- [host-check-agent-status3,132961fadc07a174,f4440ba92692d310] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:36:46.016 INFO 2981023 --- [host-check-clockdiff4,d982d700f6847c4c,f08fb844b56e8215] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047406 0 0 , error= 2024-09-11 17:36:46.017 INFO 2981023 --- [host-check-agent-status2,132961fadc07a174,9dbb511ae7f51758] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:36:46.017 INFO 2981023 --- [host-check-agent-status4,132961fadc07a174,7c0fd164e0a42df0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:36:46.020 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:36:46.020 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:36:46.021 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:36:46.022 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:36:46.022 INFO 2981023 --- [host-check-agent-status1,132961fadc07a174,5ad4c47e13a923b4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:36:46.022 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:36:46.023 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:36:46.024 INFO 2981023 --- [host-check-agent-status1,132961fadc07a174,5ad4c47e13a923b4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:36:46.024 INFO 2981023 --- [host-check-agent-status4,132961fadc07a174,7c0fd164e0a42df0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:36:46.024 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:36:46.024 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:36:46.025 INFO 2981023 --- [host-check-agent-status2,132961fadc07a174,9dbb511ae7f51758] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:36:46.026 INFO 2981023 --- [host-check-agent-status4,132961fadc07a174,7c0fd164e0a42df0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:36:46.026 INFO 2981023 --- [host-check-agent-status3,132961fadc07a174,f4440ba92692d310] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:36:46.026 INFO 2981023 --- [host-check-agent-status2,132961fadc07a174,9dbb511ae7f51758] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:36:46.029 INFO 2981023 --- [host-check-agent-status3,132961fadc07a174,f4440ba92692d310] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:36:46.048 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:36:46.051 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:36:46.051464577+08:00 2024-09-11 17:36:46.052 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:36:46.051+08:00 2024-09-11 17:36:46.053 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:36:46.053 INFO 2981023 --- [host-check-agent-status1,132961fadc07a174,5ad4c47e13a923b4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:36:46.055 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:36:46.054842398+08:00 2024-09-11 17:36:46.055 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:36:46.055 INFO 2981023 --- [host-check-agent-status4,132961fadc07a174,7c0fd164e0a42df0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:36:46.056 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:36:46.055+08:00 2024-09-11 17:36:46.057 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:36:46.056566375+08:00 2024-09-11 17:36:46.057 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:36:46.051+08:00 and status AVAILABLE 2024-09-11 17:36:46.057 INFO 2981023 --- [host-check-agent-status1,132961fadc07a174,5ad4c47e13a923b4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:36:46.057 INFO 2981023 --- [host-check-agent-status1,132961fadc07a174,5ad4c47e13a923b4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:36:46.057 INFO 2981023 --- [host-check-agent-status1,132961fadc07a174,5ad4c47e13a923b4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:36:46.058 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:36:46.058 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:36:46.058372978+08:00 2024-09-11 17:36:46.059 INFO 2981023 --- [host-check-agent-status4,132961fadc07a174,7c0fd164e0a42df0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:36:46.059 INFO 2981023 --- [host-check-agent-status4,132961fadc07a174,7c0fd164e0a42df0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:36:46.059 INFO 2981023 --- [host-check-agent-status4,132961fadc07a174,7c0fd164e0a42df0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:36:46.059 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:36:46.055+08:00 and status AVAILABLE 2024-09-11 17:36:46.059 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:36:46.057+08:00 2024-09-11 17:36:46.059 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:36:46.058+08:00 2024-09-11 17:36:46.060 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:36:46.057+08:00 and status AVAILABLE 2024-09-11 17:36:46.061 INFO 2981023 --- [host-check-agent-status2,132961fadc07a174,9dbb511ae7f51758] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:36:46.061 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:36:46.058+08:00 and status AVAILABLE 2024-09-11 17:36:46.062 INFO 2981023 --- [host-check-agent-status3,132961fadc07a174,f4440ba92692d310] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:36:46.062 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:36:46.064 INFO 2981023 --- [host-check-agent-status2,132961fadc07a174,9dbb511ae7f51758] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:36:46.064 INFO 2981023 --- [host-check-agent-status2,132961fadc07a174,9dbb511ae7f51758] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:36:46.064 INFO 2981023 --- [host-check-agent-status2,132961fadc07a174,9dbb511ae7f51758] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:36:46.065 INFO 2981023 --- [host-check-agent-status3,132961fadc07a174,f4440ba92692d310] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:36:46.065 INFO 2981023 --- [host-check-agent-status3,132961fadc07a174,f4440ba92692d310] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:36:46.065 INFO 2981023 --- [host-check-agent-status3,132961fadc07a174,f4440ba92692d310] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:36:46.066 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:36:46.066 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:36:46.067 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:36:46.068 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:36:46.069 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:36:46.073 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:36:46.074 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:36:46.074 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:36:46.075 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:36:46.078 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:36:46.080 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:36:46.093 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:36:46.097 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:36:46.097 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:36:46.101 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:36:46.115 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:36:46.121 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:36:46.123 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:36:46.131 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:36:46.132 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:36:46.140 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:36:46.152 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:36:46.157 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:36:46.158 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:36:46.160 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:36:46.167 INFO 2981023 --- [ocp-schedules-11,85a9d20ef5c28498,6c7732c02f3f0d3f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:36:46.175 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:36:46.186 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:36:46.187 INFO 2981023 --- [ocp-schedules-9,8f5d788f290605c6,0c79b8bd28f7fe6b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:36:46.187 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:36:46.187 INFO 2981023 --- [ocp-schedules-29,8d4734a4bbedf1bb,216f1c66c2de765c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:36:46.188 INFO 2981023 --- [ocp-schedules-22,7b4d88736a67489f,914d3df58f36e517] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:36:46.188 INFO 2981023 --- [ocp-schedules-27,7ee5aebbb0ba0332,9eed83747b76e720] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:36:46.190 INFO 2981023 --- [ocp-schedules-9,8f5d788f290605c6,0c79b8bd28f7fe6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:46.194 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:36:46.194 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:36:46.194 INFO 2981023 --- [ocp-schedules-22,7b4d88736a67489f,914d3df58f36e517] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:46.194 INFO 2981023 --- [ocp-schedules-29,8d4734a4bbedf1bb,216f1c66c2de765c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:46.204 INFO 2981023 --- [ocp-schedules-29,8d4734a4bbedf1bb,216f1c66c2de765c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:36:46.214 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:36:46.217 INFO 2981023 --- [ocp-schedules-11,85a9d20ef5c28498,6c7732c02f3f0d3f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:36:46.221 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:36:46.223 INFO 2981023 --- [ocp-schedules-4,02fb6886670f1deb,60f30a55d280a8c6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:36:46.223 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:36:46.225 INFO 2981023 --- [ocp-schedules-4,02fb6886670f1deb,60f30a55d280a8c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:46.230 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:36:46.233 INFO 2981023 --- [ocp-schedules-24,56c869ce5042fea5,25b715b063cd01cc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:36:46.233 INFO 2981023 --- [ocp-schedules-5,77d2063961719982,43fbacd7d344c59b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:36:46.233 INFO 2981023 --- [ocp-schedules-5,77d2063961719982,43fbacd7d344c59b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:36:46.238 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.240 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:36:46.241 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:36:46.241 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.242 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.242 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:46.243 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:36:46.247 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.247 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:46.248 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:36:46.249 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.250 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:46.251 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:36:46.251 INFO 2981023 --- [ocp-schedules-9,8f5d788f290605c6,0c79b8bd28f7fe6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:36:46.257 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.259 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:36:46.259 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:46.259 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.260 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:46.278 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:36:46.284 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.284 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:36:46.287 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:36:46.292 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:36:46.292 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.293 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:36:46.298 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:36:46.301 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,67c497c15f21e70d,2639f2001d0f6a50] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&ObClusterId=1719994443&version=2, client=10.31.35.10, traceId=67c497c15f21e70d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkrtd,1719994443,false,2,] 2024-09-11 17:36:46.301 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,67c497c15f21e70d,2639f2001d0f6a50] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=67c497c15f21e70d, duration=1 ms] 2024-09-11 17:36:46.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,891e12fce727b6a2,5ac8560d05e82256] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=891e12fce727b6a2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:36:46.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,891e12fce727b6a2,5ac8560d05e82256] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=891e12fce727b6a2, duration=0 ms] 2024-09-11 17:36:46.329 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:36:46.329 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.330 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:36:46.336 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:36:46.336 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.337 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:46.338 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.338 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.338 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:36:46.342 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:36:46.343 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.346 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:36:46.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:36:46.350616+08:00, lastProcessTime=2024-09-11T17:36:36.336378+08:00 2024-09-11 17:36:46.354 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.356 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.356 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:36:46.360 INFO 2981023 --- [host-check-host-status3,b29626494ba18e4f,889b9fa17265c13d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:36:46.365 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:36:46.367 INFO 2981023 --- [host-check-host-status2,b29626494ba18e4f,eb782a7e89d69b01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:36:46.368 INFO 2981023 --- [host-check-host-status1,b29626494ba18e4f,44434d9d7a844c65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:36:46.376 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.377 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:36:46.386 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8fe62a1197b7f767,698a3fd2d4f46b51] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=8fe62a1197b7f767, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:36:46.387 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,8fe62a1197b7f767,698a3fd2d4f46b51] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=8fe62a1197b7f767, duration=1 ms] 2024-09-11 17:36:46.388 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.389 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.389 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.389 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:36:46.390 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:36:46.390 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:36:46.391 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.410 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.414 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:36:46.420 INFO 2981023 --- [ocp-schedules-4,02fb6886670f1deb,60f30a55d280a8c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:36:46.423 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:36:46.430 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:36:46.434 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.435 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.435 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:36:46.436 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.437 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:36:46.437 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.438 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:46.438 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:36:46.439 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.442 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:36:46.443 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:36:46.446 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:36:46.451 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.452 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.452 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:36:46.460 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:46.461 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:36:46.478 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.483 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.484 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.485 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:36:46.487 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:46.488 INFO 2981023 --- [pool-10-thread-5,e3c11c39a48f520e,7ccc80ec8c5777ac] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:36:46.488 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.491 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:36:46.493 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:36:46.506 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:46.507 INFO 2981023 --- [pool-10-thread-6,e3c11c39a48f520e,3256de9174d0ccf1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:36:46.512 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.515 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.517 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:36:46.517 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:36:46.519 INFO 2981023 --- [pool-10-thread-8,e3c11c39a48f520e,fffd9e5a83651878] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:36:46.519 INFO 2981023 --- [pool-10-thread-7,e3c11c39a48f520e,a69fbd0be542707d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:36:46.520 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.521 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.523 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.523 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.523 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.524 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:36:46.524 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:46.524 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:36:46.524 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:36:46.525 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:36:46.526 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:36:46.526 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.527 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:36:46.528 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:46.528 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:36:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:36:46.531 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:36:46.535 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:36:46.537 INFO 2981023 --- [pool-10-thread-8,e3c11c39a48f520e,fffd9e5a83651878] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:36:46.537 INFO 2981023 --- [pool-10-thread-8,e3c11c39a48f520e,fffd9e5a83651878] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:36:46.540 INFO 2981023 --- [pool-10-thread-6,e3c11c39a48f520e,3256de9174d0ccf1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:36:46.542 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:36:46.542 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:36:46.544 INFO 2981023 --- [ForkJoinPool-6-worker-3,e3c11c39a48f520e,7885abd11a06af20] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:36:46.544 INFO 2981023 --- [ForkJoinPool-6-worker-3,e3c11c39a48f520e,7885abd11a06af20] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:36:46.545 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:36:46.545 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:36:46.546 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:36:46.563 INFO 2981023 --- [host-check-host-status4,b29626494ba18e4f,df0d8c903e371e6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:36:46.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:46.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:36:46.572 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:36:46.575 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:46.577 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:36:46.584 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:36:46.589 WARN 2981023 --- [ForkJoinPool-6-worker-3,e3c11c39a48f520e,7885abd11a06af20] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:36:46.592 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.593 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.594 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:46.594 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:36:46.596 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.596 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:36:46.597 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.597 INFO 2981023 --- [ForkJoinPool-6-worker-7,e3c11c39a48f520e,e43f5231d1605d6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.597 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.599 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:36:46.599 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:36:46.600 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:36:46.601 INFO 2981023 --- [ForkJoinPool-6-worker-7,e3c11c39a48f520e,e43f5231d1605d6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:36:46.602 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:36:46.608 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:36:46.608 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:36:46.612 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:36:46.614 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.614 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:36:46.615 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:36:46.615 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:36:46.624 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.625 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:36:46, 2024-09-11 17:36:46] 2024-09-11 17:36:46.626 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.626 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.627 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:36:46.627 INFO 2981023 --- [ForkJoinPool-6-worker-12,e3c11c39a48f520e,d283e31aa6a71610] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:36:46.627 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:36:46.628 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:36:46.629 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.629 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:36:46.634 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:36:46.635 INFO 2981023 --- [ForkJoinPool-6-worker-7,e3c11c39a48f520e,e43f5231d1605d6c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:36:46.636 INFO 2981023 --- [ForkJoinPool-6-worker-7,e3c11c39a48f520e,e43f5231d1605d6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.637 INFO 2981023 --- [ForkJoinPool-6-worker-7,e3c11c39a48f520e,e43f5231d1605d6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:36:46.639 INFO 2981023 --- [ForkJoinPool-6-worker-7,e3c11c39a48f520e,e43f5231d1605d6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.640 INFO 2981023 --- [ForkJoinPool-6-worker-7,e3c11c39a48f520e,e43f5231d1605d6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:36:46.641 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.648 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.648 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.648 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:36:46.648 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.648 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:36:46.649 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:36:46.649 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.650 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:36:46.666 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:36:46.666 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:36:46.667 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.668 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:36:46.669 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.669 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.670 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:36:46.671 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:36:46.672 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:36:46.679 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:36:46.679 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:36:46.681 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:36:46.689 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:36:46.690 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.691 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:36:46.692 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.692 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:36:46.693 INFO 2981023 --- [ForkJoinPool-6-worker-5,e3c11c39a48f520e,0e4d7d6a0c0433ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:36:46.694 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:36:46.695 INFO 2981023 --- [pool-10-thread-5,e3c11c39a48f520e,7ccc80ec8c5777ac] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:36:46.696 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.697 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:36:46, 2024-09-11 17:36:46] 2024-09-11 17:36:46.709 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:36:46.710 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.715 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:36:46.715 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.715 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.716 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:36:46.717 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:36:46.718 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:36:46.718 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.718 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.719 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:36:46, 2024-09-11 17:36:46] 2024-09-11 17:36:46.720 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:36:46.720 INFO 2981023 --- [ForkJoinPool-6-worker-10,e3c11c39a48f520e,e538f1835ba116fd] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:36:46.723 INFO 2981023 --- [pool-10-thread-7,e3c11c39a48f520e,a69fbd0be542707d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:36:46.738 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:36:46.745 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:36:46.765 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.768 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.772 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.772 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.772 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:36:46.773 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:36:46.778 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:36:46.779 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:36:46.784 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.785 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:36:46.785 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:36:46.786 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:36:46.786 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:36:46.787 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:46.791 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:36:46.792 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:36:46.798 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:36:46.800 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:36:46.350616+08:00 2024-09-11 17:36:46.801 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:36:46.801 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:36:46.801 INFO 2981023 --- [ocp-schedules-22,7b4d88736a67489f,914d3df58f36e517] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:36:46.806 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:36:46.808 INFO 2981023 --- [ocp-schedules-20,57b9b844e5aaefd0,f241bb07489aa857] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:36:46.859 INFO 2981023 --- [pool-10-thread-6,e3c11c39a48f520e,3256de9174d0ccf1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:36:46.862 INFO 2981023 --- [pool-10-thread-6,e3c11c39a48f520e,3256de9174d0ccf1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:36:47.235 INFO 2981023 --- [ocp-async-9,de766bc66322b334,66f27cef5f673372] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:47.243 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:47.243 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,27fcda0937eeaa22,f528a685afb3f3a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:47.320 INFO 2981023 --- [prometheus-minute-schedule-1,27fcda0937eeaa22,f528a685afb3f3a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:47.341 INFO 2981023 --- [ocp-schedules-21,561eb217d73c39db,b3ab8d0bc8ccd9c5] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:36:47.754 INFO 2981023 --- [ocp-async-1,45adb1babdb92035,36dc9caa0007823f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:36:49.235 INFO 2981023 --- [ocp-async-6,6b553d47d730d053,411bd02b7f695374] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:36:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:36:50.228 INFO 2981023 --- [ocp-schedules-12,fc349424243c044a,4c803a28042daa9a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:36:50.243 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:50.244 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,d38551ff7de500d6,da639a4d283646fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,d38551ff7de500d6,da639a4d283646fe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:51.236 INFO 2981023 --- [ocp-async-2,4692a856113e83ef,53049c4a75612dcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:51.283 INFO 2981023 --- [host-check-clockdiff1,d982d700f6847c4c,a6ad48523eca6cd6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047411 0 0 , error= 2024-09-11 17:36:51.541 INFO 2981023 --- [ocp-schedules-17,5ef21773f3188fb2,35893b742408b8f6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:36:51.542 INFO 2981023 --- [ocp-schedules-17,5ef21773f3188fb2,35893b742408b8f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:51.542 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:36:51.543 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:36:51.543 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:36:51.544 INFO 2981023 --- [ocp-schedules-17,5ef21773f3188fb2,35893b742408b8f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:36:51.954 INFO 2981023 --- [host-check-clockdiff3,d982d700f6847c4c,203eed1f1d9c5722] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047411 1 1 , error= 2024-09-11 17:36:51.976 INFO 2981023 --- [ocp-schedules-16,d0eb6200fbd809e9,2ceb01339097fd07] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:36:52.514 INFO 2981023 --- [ocp-schedules-9,6e0738e8c48fccac,9030545e7783778b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:36:52.514+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:36:52.518 INFO 2981023 --- [ocp-schedules-9,6e0738e8c48fccac,9030545e7783778b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:36:52.637 INFO 2981023 --- [ocp-schedules-9,6e0738e8c48fccac,9030545e7783778b] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 45 to SqlAggregateData 2024-09-11 17:36:52.638 INFO 2981023 --- [ocp-schedules-9,6e0738e8c48fccac,9030545e7783778b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:36:52.765 INFO 2981023 --- [ocp-schedules-30,e0db693ee804957c,3ee447d7aa9bd39e] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:36:52.784 WARN 2981023 --- [ocp-schedules-30,e0db693ee804957c,3ee447d7aa9bd39e] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:36:53.095 INFO 2981023 --- [ocp-schedules-27,6ff198e806fca63a,d475d8e50d7c90d8] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:36:53.095 INFO 2981023 --- [ocp-schedules-27,6ff198e806fca63a,d475d8e50d7c90d8] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:36:53.236 INFO 2981023 --- [ocp-async-7,1b649eb84cbcbe8d,ee131f379b1b82d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:53.244 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:53.244 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,285810fd5fc197c1,b0eeba40ab24e57a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,285810fd5fc197c1,b0eeba40ab24e57a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:53.326 INFO 2981023 --- [ocp-schedules-3,7bc52e7b585c153f,71a2e26935c8a329] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:36:53.328 INFO 2981023 --- [ocp-schedules-3,7bc52e7b585c153f,71a2e26935c8a329] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:36:53.360 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:36:53.360 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.361 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:36:53.361 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.362 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:36:53.362 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.364 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:36:53.364 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:36:53.366 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:36:53.371 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.371 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.371 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.372 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.373 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.374 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.382 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.383 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:36:53.386 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.386 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.394 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.394 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:36:53.398 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.398 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.399 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.400 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.400 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.400 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:36:53.400 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:36:53.400 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2981D724-0-0] 2024-09-11 17:36:53.401 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:36:53.401 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.401 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:36:53.410 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.410 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.410 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:36:53.411 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:36:53.411 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.412 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:36:53.412 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.413 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.413 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:36:53.413 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:36:53.414 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:36:53.415 INFO 2981023 --- [ocp-schedules-10,65350d291aaf2404,a838aa3bb55d77f0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:36:53.421 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:36:53.425 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.426 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:36:53.428 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.428 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.428 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:36:53.429 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:36:53.429 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2981D72E-0-0] 2024-09-11 17:36:53.429 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:36:53.429 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:36:53.430 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.431 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:36:53.432 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.433 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:36:53.434 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.435 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:36:53.435 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.436 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:36:53.436 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.437 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:36:53.438 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.440 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:36:53.441 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.442 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:36:53.443 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:36:53.448 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:36:53.453 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.454 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:36:53.455 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.456 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:36:53.458 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.459 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:36:53.459 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.460 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:53.460 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:36:53.461 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.462 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:36:53.463 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.463 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.464 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.464 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:36:53.465 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:36:53.476 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:36:53.478 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:36:53.482 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:36:53.482 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:36:53.482 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:36:53.482 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:53.486 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.486 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:36:53.494 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:36:53.498 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:36:53.498 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.499 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:36:53.500 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:36:53.504 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:36:53.508 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:36:53.508 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.508 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.509 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.509 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:36:53.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:36:53.518 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:36:53.518 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:36:53.518 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:36:53.521 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:36:53.522 WARN 2981023 --- [scanner-13514829-1,65350d291aaf2404,3f69e3ea600cd841] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:36:53.523 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:36:53.524 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:53.524 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:36:53.536 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:36:53.544 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:36:53.544 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:36:53.544 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:36:53.547 INFO 2981023 --- [ocp-schedules-3,7bc52e7b585c153f,71a2e26935c8a329] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:36:53.591 INFO 2981023 --- [alarm-detect-17,65350d291aaf2404,285d698e4652aa66] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007101] will be resolved. 2024-09-11 17:36:53.591 INFO 2981023 --- [alarm-detect-17,65350d291aaf2404,285d698e4652aa66] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007101] will be resolved. 2024-09-11 17:36:53.646 INFO 2981023 --- [alarm-detect-17,65350d291aaf2404,285d698e4652aa66] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 17:36:53.920 INFO 2981023 --- [ocp-schedules-20,f9e5fb115d707d27,ceb8030a6848d9dc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:36:53.925 INFO 2981023 --- [ocp-schedules-20,f9e5fb115d707d27,ceb8030a6848d9dc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:36:53.925 INFO 2981023 --- [ocp-schedules-20,f9e5fb115d707d27,ceb8030a6848d9dc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:36:54.226 INFO 2981023 --- [ocp-schedules-22,32ff5b71cd931fe9,a4f64ecc8a781366] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:36:54.760 INFO 2981023 --- [ocp-schedules-28,3bdf406e585d6cd5,53ee6daaf2ebf1be] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:36:54.760 INFO 2981023 --- [ocp-schedules-28,3bdf406e585d6cd5,53ee6daaf2ebf1be] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:36:54.763 INFO 2981023 --- [ocp-schedules-28,3bdf406e585d6cd5,53ee6daaf2ebf1be] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:36:54.763 INFO 2981023 --- [ocp-schedules-28,3bdf406e585d6cd5,53ee6daaf2ebf1be] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:36:54.763 INFO 2981023 --- [ocp-schedules-28,3bdf406e585d6cd5,53ee6daaf2ebf1be] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:36:54.766 INFO 2981023 --- [ocp-schedules-28,3bdf406e585d6cd5,53ee6daaf2ebf1be] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:36:54.766 INFO 2981023 --- [ocp-schedules-28,3bdf406e585d6cd5,53ee6daaf2ebf1be] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:36:54.766 INFO 2981023 --- [ocp-schedules-28,3bdf406e585d6cd5,53ee6daaf2ebf1be] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:36:54.776 INFO 2981023 --- [ocp-schedules-28,3bdf406e585d6cd5,53ee6daaf2ebf1be] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:36:55.236 INFO 2981023 --- [ocp-async-5,2bf94620a5b56932,f6c05a7ffbe99541] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:55.679 INFO 2981023 --- [ocp-schedules-25,382aeed22a701715,000e4ad7ee603a80] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:36:55.679 INFO 2981023 --- [ocp-schedules-25,382aeed22a701715,000e4ad7ee603a80] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:36:55.682 INFO 2981023 --- [ocp-schedules-25,382aeed22a701715,000e4ad7ee603a80] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:36:55.682 INFO 2981023 --- [ocp-schedules-25,382aeed22a701715,000e4ad7ee603a80] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:36:55.684 INFO 2981023 --- [ocp-schedules-25,382aeed22a701715,000e4ad7ee603a80] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:36:55.684 INFO 2981023 --- [ocp-schedules-25,382aeed22a701715,000e4ad7ee603a80] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:36:55.770 INFO 2981023 --- [ocp-schedules-22,b8740b1c7eb32b35,1ef52effd94b4ba2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:36:55.776 INFO 2981023 --- [ocp-schedules-22,b8740b1c7eb32b35,1ef52effd94b4ba2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:36:55.776 INFO 2981023 --- [ocp-schedules-22,b8740b1c7eb32b35,1ef52effd94b4ba2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:36:56.204 INFO 2981023 --- [ocp-schedules-17,9af25ab9bd7e4da2,d7a653a194e2115e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:36:56.204 INFO 2981023 --- [ocp-schedules-17,9af25ab9bd7e4da2,d7a653a194e2115e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:36:56.216 INFO 2981023 --- [Collect all tenants major compactions2,9af25ab9bd7e4da2,10a853b860b63681] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:36:56.217 INFO 2981023 --- [Collect all tenants major compactions5,9af25ab9bd7e4da2,4ae418ccf7a4ae02] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:36:56.217 INFO 2981023 --- [Collect all tenants major compactions1,9af25ab9bd7e4da2,b104f66b5ef378ee] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:36:56.219 INFO 2981023 --- [Collect all tenants major compactions6,9af25ab9bd7e4da2,61e8446c5b9ac164] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:36:56.220 INFO 2981023 --- [Collect all tenants major compactions3,9af25ab9bd7e4da2,6e187df0d61f3fab] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:36:56.220 INFO 2981023 --- [Collect all tenants major compactions4,9af25ab9bd7e4da2,be1a1dbe783fb960] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:36:56.230 INFO 2981023 --- [Collect all tenants major compactions2,9af25ab9bd7e4da2,10a853b860b63681] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:56.230 INFO 2981023 --- [Collect all tenants major compactions2,9af25ab9bd7e4da2,10a853b860b63681] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:36:56.232 INFO 2981023 --- [Collect all tenants major compactions3,9af25ab9bd7e4da2,6e187df0d61f3fab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:56.233 INFO 2981023 --- [Collect all tenants major compactions5,9af25ab9bd7e4da2,4ae418ccf7a4ae02] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:56.233 INFO 2981023 --- [Collect all tenants major compactions3,9af25ab9bd7e4da2,6e187df0d61f3fab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:36:56.234 INFO 2981023 --- [Collect all tenants major compactions2,9af25ab9bd7e4da2,10a853b860b63681] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:36:56.234 INFO 2981023 --- [Collect all tenants major compactions5,9af25ab9bd7e4da2,4ae418ccf7a4ae02] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:36:56.235 INFO 2981023 --- [Collect all tenants major compactions3,9af25ab9bd7e4da2,6e187df0d61f3fab] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:36:56.235 INFO 2981023 --- [Collect all tenants major compactions1,9af25ab9bd7e4da2,b104f66b5ef378ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:56.236 INFO 2981023 --- [Collect all tenants major compactions5,9af25ab9bd7e4da2,4ae418ccf7a4ae02] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:36:56.237 INFO 2981023 --- [Collect all tenants major compactions2,9af25ab9bd7e4da2,10a853b860b63681] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:36:56.238 INFO 2981023 --- [Collect all tenants major compactions1,9af25ab9bd7e4da2,b104f66b5ef378ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:36:56.238 INFO 2981023 --- [Collect all tenants major compactions4,9af25ab9bd7e4da2,be1a1dbe783fb960] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:56.240 INFO 2981023 --- [Collect all tenants major compactions4,9af25ab9bd7e4da2,be1a1dbe783fb960] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:36:56.240 INFO 2981023 --- [Collect all tenants major compactions1,9af25ab9bd7e4da2,b104f66b5ef378ee] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:36:56.240 INFO 2981023 --- [Collect all tenants major compactions6,9af25ab9bd7e4da2,61e8446c5b9ac164] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:36:56.240 INFO 2981023 --- [Collect all tenants major compactions1,9af25ab9bd7e4da2,b104f66b5ef378ee] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:36:56.240 INFO 2981023 --- [Collect all tenants major compactions3,9af25ab9bd7e4da2,6e187df0d61f3fab] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:36:56.241 INFO 2981023 --- [Collect all tenants major compactions5,9af25ab9bd7e4da2,4ae418ccf7a4ae02] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:36:56.241 INFO 2981023 --- [Collect all tenants major compactions6,9af25ab9bd7e4da2,61e8446c5b9ac164] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:36:56.242 INFO 2981023 --- [Collect all tenants major compactions4,9af25ab9bd7e4da2,be1a1dbe783fb960] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:36:56.242 INFO 2981023 --- [Collect all tenants major compactions4,9af25ab9bd7e4da2,be1a1dbe783fb960] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:36:56.243 INFO 2981023 --- [Collect all tenants major compactions6,9af25ab9bd7e4da2,61e8446c5b9ac164] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:36:56.243 INFO 2981023 --- [Collect all tenants major compactions6,9af25ab9bd7e4da2,61e8446c5b9ac164] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:36:56.244 INFO 2981023 --- [ocp-schedules-17,9af25ab9bd7e4da2,d7a653a194e2115e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:36:56.244 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:56.245 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,bbd0181ed652aca5,c35b33aa76701f55] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,bbd0181ed652aca5,c35b33aa76701f55] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:36:56.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:36:56.337098+08:00, lastProcessTime=2024-09-11T17:36:46.350616+08:00 2024-09-11 17:36:56.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:36:56.471 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:36:56.337098+08:00 2024-09-11 17:36:57.236 INFO 2981023 --- [ocp-async-1,7b203a01d897d871,69124ce9cae0f496] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:57.754 INFO 2981023 --- [ocp-async-8,10e8ad5ef865288c,24fe614ffa94e7a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:36:59.236 INFO 2981023 --- [ocp-async-7,2a37139c2fae8612,01194f5cee00d6bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:36:59.245 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:36:59.246 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:36:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,a112b0789cc95497,1a1805389417c99c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047420, interval=60 2024-09-11 17:36:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,a112b0789cc95497,1a1805389417c99c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:01.244 INFO 2981023 --- [ocp-async-6,9d28d2ef5f0c5889,d425e88cf6edcc77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:01.978 INFO 2981023 --- [ocp-schedules-2,4b54572596e0342a,daad4825f6ea643f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:37:02.246 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:02.247 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,1df1f788471f7566,b4e6517b4bc09033] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,1df1f788471f7566,b4e6517b4bc09033] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:02.390 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:33:02.014, end=2024-09-11T17:35:02.387} 2024-09-11 17:37:02.396 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClust 2024-09-11 17:37:02.397 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:37:02.398 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:33:02.014, end=2024-09-11T17:35:02.387},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:37:02.398 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:37:02.398 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:37:02.399 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:37:02.400 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:33:02.014, end=2024-09-11T17:35:02.387},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:37:02.400 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:37:02.400 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:37:02.423 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:37:02.423 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:37:02.424 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:33:02.014, end=2024-09-11T17:35:02.387},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:37:02.424 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:37:02.424 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:37:02.434 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:37:02.453 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022453000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.453Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.455 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:37:02.455 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.460 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.460 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.462 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.462 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.467 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.467 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.468 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:37:02.468 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.453, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.468 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:37:02.469 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:33:02.014, end=2024-09-11T17:35:02.387},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:37:02.469 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:37:02.469 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:37:02.470 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:37:02.470 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:33:02.014, end=2024-09-11T17:35:02.387},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:37:02.470 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:37:02.470 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:37:02.471 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:37:02.473 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:33:02.014, end=2024-09-11T17:35:02.387},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:37:02.473 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:37:02.473 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:37:02.524 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022524000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.524Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.525 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022525000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.525Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.525 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022525000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.525Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.528 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022528000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.528Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.531 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022531000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=8A1B851989150B802AEEACA99152D145, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.531Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.532 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022532000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.532Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.534 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022534000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.534Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.535 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022535000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.535Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.536 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 60 2024-09-11 17:37:02.536 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.547 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.547 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.549 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.549 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.549 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 26 2024-09-11 17:37:02.549 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.550 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.550 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.559 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.559 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.560 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.560 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.562 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.562 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.563 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.563 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.567 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:37:02.567 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.568 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:37:02.568 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.569 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.569 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.571 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.571 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.573 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.573 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.573 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.573 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.576 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.576 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.576 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.576 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.577 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.577 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.581 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.581 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.582 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.582 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.583 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.583 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.584 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.584 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.586 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.587 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.591 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.591 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.591 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.591 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.593 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.593 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.593 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.593 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.595 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.595 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.595 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.595 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.599 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.599 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.600 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.600 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.600 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.601 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.601 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.602 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.606 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.525, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.606 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.524, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.607 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.528, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.607 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.525, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.607 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.532, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.608 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.531, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=8A1B851989150B802AEEACA99152D145, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.614 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.614 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.618 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.620 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.620 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.624 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.624 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.534, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.634 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.535, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.641 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022641000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.641Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.642 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022642000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.642Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.647 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022647000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.647Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.648 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022648000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.648Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.648 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022648000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.648Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.650 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022650000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.650Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.654 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.654 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.659 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:37:02.659 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.661 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.661 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.665 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:37:02.665 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.668 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.668 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.669 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:37:02.669 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.672 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.672 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.673 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.673 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.675 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.675 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.678 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.678 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.678 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.678 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.679 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.679 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.681 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.681 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.683 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.683 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.684 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022684000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=C0995E24A01F869BEB311E8C790E6C86, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.684Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.686 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.686 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.686 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022686000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.686Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.687 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.687 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.688 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.688 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.691 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.691 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.693 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.693 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.693 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.693 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.695 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.695 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.697 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.697 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.697 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.697 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.697 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.698 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.698 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.698 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.699 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.700 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.706 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.706 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.711 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.712 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.641, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.714 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.714 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.714 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.648, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.715 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.715 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.716 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.65, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.717 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.717 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.717 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.717 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.718 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.718 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.718 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.647, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.719 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.642, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.719 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.724 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.648, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.727 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.727 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.727 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.727 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.728 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.728 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.733 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.684, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=C0995E24A01F869BEB311E8C790E6C86, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.734 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961022734000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:37:02.734Z, endTime=2024-09-11T09:35:02.387Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:37:02.736 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.686, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.741 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:37:02.741 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:37:02.746 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.746 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.748 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:37:02.748 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:37:02.753 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:37:02.753 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047302387000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:37:02.754 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:37:02.757 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:37:02.734, end=2024-09-11T17:35:02.387}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:37:02.757 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:37:03.236 INFO 2981023 --- [ocp-async-2,4888c13cc2394741,c6b1ff6666c8738f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:03.706 INFO 2981023 --- [ocp-schedules-30,310e0e926cd3e9e7,dd8338953f891c22] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:37:03.749 WARN 2981023 --- [scanner-13515018-1,310e0e926cd3e9e7,3bd6829c4b2c39dc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:37:03.773 INFO 2981023 --- [alarm-detect-14,310e0e926cd3e9e7,dcfe72277f9c903b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644223, data is empty. 2024-09-11 17:37:03.782 INFO 2981023 --- [alarm-detect-14,310e0e926cd3e9e7,dcfe72277f9c903b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644268, data is empty. 2024-09-11 17:37:03.788 INFO 2981023 --- [alarm-detect-14,310e0e926cd3e9e7,dcfe72277f9c903b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644268, data is empty. 2024-09-11 17:37:03.799 INFO 2981023 --- [alarm-detect-19,310e0e926cd3e9e7,57d3aabd7e0fd85a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644223, data is empty. 2024-09-11 17:37:03.800 INFO 2981023 --- [alarm-detect-15,310e0e926cd3e9e7,997d04d761a3e8e1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644223, data is empty. 2024-09-11 17:37:03.802 INFO 2981023 --- [alarm-detect-19,310e0e926cd3e9e7,57d3aabd7e0fd85a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644268, data is empty. 2024-09-11 17:37:03.806 INFO 2981023 --- [alarm-detect-15,310e0e926cd3e9e7,997d04d761a3e8e1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644268, data is empty. 2024-09-11 17:37:03.807 INFO 2981023 --- [alarm-detect-19,310e0e926cd3e9e7,57d3aabd7e0fd85a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644268, data is empty. 2024-09-11 17:37:03.809 INFO 2981023 --- [alarm-detect-15,310e0e926cd3e9e7,997d04d761a3e8e1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644268, data is empty. 2024-09-11 17:37:03.944 INFO 2981023 --- [alarm-detect-8,310e0e926cd3e9e7,1710d0af0143e7bf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644223, data is empty. 2024-09-11 17:37:03.950 INFO 2981023 --- [alarm-detect-8,310e0e926cd3e9e7,1710d0af0143e7bf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644268, data is empty. 2024-09-11 17:37:03.953 INFO 2981023 --- [alarm-detect-8,310e0e926cd3e9e7,1710d0af0143e7bf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644268, data is empty. 2024-09-11 17:37:05.237 INFO 2981023 --- [ocp-async-9,11ea2540c0c4080c,dfac2e754a75f465] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:05.247 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:05.247 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,8117de3e50567920,8089035df4d1925e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,8117de3e50567920,8089035df4d1925e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:37:06.351412+08:00, lastProcessTime=2024-09-11T17:36:56.337098+08:00 2024-09-11 17:37:06.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:37:06.351412+08:00 2024-09-11 17:37:07.237 INFO 2981023 --- [ocp-async-3,9dcf219e38c765a4,f0c4597bebeca164] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:07.359 INFO 2981023 --- [ocp-schedules-21,ca3aa0efb7191af3,5bea18f46517aac7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:37:07.366 INFO 2981023 --- [ocp-schedules-23,f5057d9c3c3befec,614eaec05fd803b8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:37:07.366+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:37:07.367 INFO 2981023 --- [ocp-schedules-23,f5057d9c3c3befec,614eaec05fd803b8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:37:07.369 INFO 2981023 --- [ocp-schedules-23,f5057d9c3c3befec,614eaec05fd803b8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:37:07.369 INFO 2981023 --- [ocp-schedules-23,f5057d9c3c3befec,614eaec05fd803b8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:37:07.754 INFO 2981023 --- [ocp-async-9,3a426d10eac8cb20,75fd7e172d9afd9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:37:08.248 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:08.248 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,419369c51df6f836,2ba3bcd9dbe7a939] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,419369c51df6f836,2ba3bcd9dbe7a939] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:09.237 INFO 2981023 --- [ocp-async-2,a508fdc8e0c5284d,8ba999ea9da56610] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:10.001 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:37:10.002 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:37:10.918 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:37:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:37:11.237 INFO 2981023 --- [ocp-async-10,a054f16cef0ad299,1d0d945d046801d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:11.248 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:11.248 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,50f8bb749d10f000,9c34f7aa1c9736bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,50f8bb749d10f000,9c34f7aa1c9736bf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:11.981 INFO 2981023 --- [ocp-schedules-26,f3d66595129e622a,ae4de8ef7ec415d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:37:12.281 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047360, nodeCount=1000, affectRows=1000 2024-09-11 17:37:12.284 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047360, nodeCount=1000, validCount=1000 2024-09-11 17:37:12.293 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047360, nodeCount=48, affectRows=48 2024-09-11 17:37:12.293 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047360, nodeCount=48, validCount=48 2024-09-11 17:37:12.303 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047360, nodeCount=1000, affectRows=1000 2024-09-11 17:37:12.305 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047360, nodeCount=999, validCount=999 2024-09-11 17:37:12.620 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:37:12.633 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:12.636 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:12.696 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:12.696 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:12.708 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:37:12.708 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:37:12.719 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:12.720 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:12.731 INFO 2981023 --- [ocp-schedules-7,94651f43c225b749,ee58b76fe9a30abc] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=5 2024-09-11 17:37:13.237 INFO 2981023 --- [ocp-async-3,1f7153e788f70c35,465fdb6dff565712] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:13.927 INFO 2981023 --- [ocp-schedules-22,ea570886a73dc42e,e8fafeeb62af95b9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:37:13.934 INFO 2981023 --- [ocp-schedules-22,ea570886a73dc42e,e8fafeeb62af95b9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:37:13.934 INFO 2981023 --- [ocp-schedules-22,ea570886a73dc42e,e8fafeeb62af95b9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:37:13.988 INFO 2981023 --- [ocp-schedules-10,17be7646583a8a22,892b372766e345bc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:37:14.037 WARN 2981023 --- [scanner-13516849-1,17be7646583a8a22,d2649b31fbef14ee] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:37:14.249 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:14.249 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,2472e3cc4b4bf49d,487d98217a11b971] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,289a7c6bd1ca8ab8,6e8e7f400dfd79bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:37:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,2472e3cc4b4bf49d,487d98217a11b971] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:14.449 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047360, nodeCount=480, affectRows=480 2024-09-11 17:37:15.237 INFO 2981023 --- [ocp-async-4,3a0b85ae6ea4934c,89b384015dd61ebf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:16.020 INFO 2981023 --- [ocp-schedules-7,0eb8769260ad0fa0,1f12bd01e1d1c225] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:37:16.021 INFO 2981023 --- [host-check-clockdiff1,0eb8769260ad0fa0,776bcd68d7c47253] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:37:16.021 INFO 2981023 --- [host-check-clockdiff2,0eb8769260ad0fa0,5a2437996fff77d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:37:16.021 INFO 2981023 --- [host-check-clockdiff3,0eb8769260ad0fa0,50b0681e2fb4c8c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:37:16.025 INFO 2981023 --- [host-check-clockdiff4,0eb8769260ad0fa0,98478ff436083df9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:37:16.027 INFO 2981023 --- [ocp-schedules-25,665b0443aa3eda44,e8522ebcf1b9dc45] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:37:16.028 INFO 2981023 --- [ocp-schedules-23,3d67596727ae330a,5480d5564c31dad5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:37:16.028 INFO 2981023 --- [host-check-agent-status1,665b0443aa3eda44,8281a664eecb4038] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:37:16.031 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:37:16.035 INFO 2981023 --- [host-check-agent-status2,665b0443aa3eda44,8353ebd8a8246df6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:37:16.035 INFO 2981023 --- [host-check-agent-status3,665b0443aa3eda44,a997a9d0cffa7ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:37:16.035 INFO 2981023 --- [host-check-agent-status4,665b0443aa3eda44,540fc9a92835bc9e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:37:16.037 INFO 2981023 --- [host-check-clockdiff4,0eb8769260ad0fa0,98478ff436083df9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047436 0 0 , error= 2024-09-11 17:37:16.054 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:37:16.054 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:37:16.064 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:37:16.069 INFO 2981023 --- [host-check-clockdiff1,0eb8769260ad0fa0,776bcd68d7c47253] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047436 0 0 , error= 2024-09-11 17:37:16.070 INFO 2981023 --- [host-check-agent-status3,665b0443aa3eda44,a997a9d0cffa7ad7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:37:16.071 INFO 2981023 --- [host-check-agent-status1,665b0443aa3eda44,8281a664eecb4038] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:16.071 INFO 2981023 --- [host-check-agent-status4,665b0443aa3eda44,540fc9a92835bc9e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:16.071 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:16.072 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:16.072 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:16.073 INFO 2981023 --- [host-check-agent-status3,665b0443aa3eda44,a997a9d0cffa7ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:37:16.073 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:16.073 INFO 2981023 --- [host-check-agent-status4,665b0443aa3eda44,540fc9a92835bc9e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:16.074 INFO 2981023 --- [host-check-agent-status1,665b0443aa3eda44,8281a664eecb4038] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:16.075 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:16.075 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:16.099 INFO 2981023 --- [host-check-agent-status4,665b0443aa3eda44,540fc9a92835bc9e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:37:16.099 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:37:16.102 INFO 2981023 --- [host-check-agent-status2,665b0443aa3eda44,8353ebd8a8246df6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:16.103 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:37:16.103195657+08:00 2024-09-11 17:37:16.104 INFO 2981023 --- [host-check-agent-status2,665b0443aa3eda44,8353ebd8a8246df6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:16.106 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:37:16.106 INFO 2981023 --- [host-check-agent-status4,665b0443aa3eda44,540fc9a92835bc9e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:37:16.106 INFO 2981023 --- [host-check-agent-status4,665b0443aa3eda44,540fc9a92835bc9e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:37:16.106 INFO 2981023 --- [host-check-agent-status4,665b0443aa3eda44,540fc9a92835bc9e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:37:16.107 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:37:16.103+08:00 2024-09-11 17:37:16.109 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:37:16.111 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:37:16.103+08:00 and status AVAILABLE 2024-09-11 17:37:16.119 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:37:16.121 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:37:16.121 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:37:16.121510986+08:00 2024-09-11 17:37:16.122 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:37:16.121+08:00 2024-09-11 17:37:16.124 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:37:16.123251179+08:00 2024-09-11 17:37:16.125 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:37:16.124+08:00 2024-09-11 17:37:16.126 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:37:16.121+08:00 and status AVAILABLE 2024-09-11 17:37:16.128 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:37:16.124+08:00 and status AVAILABLE 2024-09-11 17:37:16.131 INFO 2981023 --- [host-check-agent-status2,665b0443aa3eda44,8353ebd8a8246df6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:37:16.133 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:37:16.133 INFO 2981023 --- [host-check-agent-status2,665b0443aa3eda44,8353ebd8a8246df6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:37:16.133 INFO 2981023 --- [host-check-agent-status2,665b0443aa3eda44,8353ebd8a8246df6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:37:16.133 INFO 2981023 --- [host-check-agent-status2,665b0443aa3eda44,8353ebd8a8246df6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:37:16.135 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:37:16.136 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:37:16.136033350+08:00 2024-09-11 17:37:16.136 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:37:16.136+08:00 2024-09-11 17:37:16.142 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:37:16.136+08:00 and status AVAILABLE 2024-09-11 17:37:16.159 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:37:16.165 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:37:16.166 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:37:16.170 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:37:16.173 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:37:16.181 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:16.183 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:16.190 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:37:16.191 INFO 2981023 --- [ocp-schedules-18,d640d926c47b7351,0030debb804cdb2a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:37:16.191 INFO 2981023 --- [ocp-schedules-20,c8c53c84a0bb0800,25a85248b2ec48fc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:37:16.191 INFO 2981023 --- [ocp-schedules-27,f63e31207b6f89a5,a3d100eff1f3cacb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:37:16.191 INFO 2981023 --- [ocp-schedules-21,6e3277842978127f,37ae535e7dcd013d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:37:16.191 INFO 2981023 --- [host-check-agent-status1,665b0443aa3eda44,8281a664eecb4038] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:37:16.191 INFO 2981023 --- [host-check-agent-status3,665b0443aa3eda44,a997a9d0cffa7ad7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:37:16.195 INFO 2981023 --- [ocp-schedules-20,c8c53c84a0bb0800,25a85248b2ec48fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:37:16.196 INFO 2981023 --- [ocp-schedules-27,f63e31207b6f89a5,a3d100eff1f3cacb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:37:16.198 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:16.198 INFO 2981023 --- [ocp-schedules-18,d640d926c47b7351,0030debb804cdb2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:37:16.201 INFO 2981023 --- [host-check-agent-status1,665b0443aa3eda44,8281a664eecb4038] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:37:16.201 INFO 2981023 --- [host-check-agent-status1,665b0443aa3eda44,8281a664eecb4038] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:37:16.201 INFO 2981023 --- [host-check-agent-status1,665b0443aa3eda44,8281a664eecb4038] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:37:16.202 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:16.202 INFO 2981023 --- [ocp-schedules-20,c8c53c84a0bb0800,25a85248b2ec48fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:37:16.208 INFO 2981023 --- [host-check-agent-status3,665b0443aa3eda44,a997a9d0cffa7ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:37:16.208 INFO 2981023 --- [host-check-agent-status3,665b0443aa3eda44,a997a9d0cffa7ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:37:16.208 INFO 2981023 --- [host-check-agent-status3,665b0443aa3eda44,a997a9d0cffa7ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:37:16.214 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:37:16.218 INFO 2981023 --- [ocp-schedules-6,d05ce514e1f7d050,a478ec7a2d5df245] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:37:16.220 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:37:16.235 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:16.243 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:16.249 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:37:16.257 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:37:16.260 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.260 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:16.261 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:37:16.263 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.264 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:16.264 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:37:16.281 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.282 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:16.283 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:37:16.287 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:37:16.288 INFO 2981023 --- [ocp-schedules-18,d640d926c47b7351,0030debb804cdb2a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:37:16.291 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:37:16.298 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.299 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:16.300 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:37:16.302 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.303 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:16.305 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:37:16.316 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:37:16.320 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:37:16.323 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:37:16.333 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.334 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:16.336 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:37:16.343 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:37:16.342506+08:00, lastProcessTime=2024-09-11T17:37:06.351412+08:00 2024-09-11 17:37:16.352 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:37:16.359 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:37:16.365 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:37:16.372 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.373 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.375 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:37:16.378 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:37:16.385 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:37:16.390 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:37:16.393 INFO 2981023 --- [host-check-host-status3,3d67596727ae330a,e6b644389831563b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:37:16.404 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:37:16.406 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.407 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.408 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:37:16.414 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:37:16.414 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:37:16.418 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047360, nodeCount=480, validCount=480 2024-09-11 17:37:16.420 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:37:16.423 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.424 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:37:16.425 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:16.426 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:37:16.427 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.429 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:37:16.430 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:37:16.446 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:37:16.447 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:16.447 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:37:16.450 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:37:16.452 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.453 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:37:16.453 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.453 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:16.456 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.460 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:37:16.461 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:37:16.471 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:37:16.471 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:16.472 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:37:16.472 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:16.472 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:37:16.482 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.483 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.483 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:37:16.485 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:37:16.485 INFO 2981023 --- [host-check-host-status1,3d67596727ae330a,a4182c28adbf7edc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:37:16.488 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.489 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.489 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:16.490 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:37:16.491 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:16.494 INFO 2981023 --- [host-check-host-status4,3d67596727ae330a,1af80d1b3642e272] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:37:16.499 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:37:16.500 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:37:16.503 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.503 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.504 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:16.505 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:37:16.505 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:16.506 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:16.509 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:37:16.515 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:16.515 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.516 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.516 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:16.517 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:37:16.517 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.519 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:37:16.519 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:16.520 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:37:16.527 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:37:16.528 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:37:16.528 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:16.529 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:37:16.539 INFO 2981023 --- [host-check-host-status2,3d67596727ae330a,2cf89f8d61986f16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:37:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:16.549 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:37:16.553 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.553 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.554 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:16.554 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:37:16.569 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:37:16.578 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:16.580 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:16.591 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:37:16.598 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.598 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.599 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:37:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:37:16.602 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:37:16.342506+08:00 2024-09-11 17:37:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:37:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:37:16.614 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:37:16.619 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.619 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.620 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:37:16.625 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:37:16.625 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:37:16.629 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:37:16.635 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:16.636 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:16.636 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:37:16.642 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:37:16.642 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:37:16.642 INFO 2981023 --- [ocp-schedules-27,f63e31207b6f89a5,a3d100eff1f3cacb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:37:16.970 INFO 2981023 --- [host-check-clockdiff2,0eb8769260ad0fa0,5a2437996fff77d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047436 0 0 , error= 2024-09-11 17:37:17.237 INFO 2981023 --- [ocp-async-9,5aa0e479e017dda7,94cbf563420e1e73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:17.249 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:17.249 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,7d26636029dccf2e,65aef1d60fba74d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:17.315 INFO 2981023 --- [prometheus-minute-schedule-1,7d26636029dccf2e,65aef1d60fba74d6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:17.754 INFO 2981023 --- [ocp-async-4,a6cfb234a16dff7c,32764091fbc78d06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:37:19.237 INFO 2981023 --- [ocp-async-8,448efc17a2abc79d,f620a8e660a2a488] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:20.249 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:20.250 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,6591c595fe21790b,6f07f245083db448] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,6591c595fe21790b,6f07f245083db448] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:21.237 INFO 2981023 --- [ocp-async-4,91979612de1fdac7,ad4c19e5a181e862] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:21.983 INFO 2981023 --- [ocp-schedules-5,4e3c51d1c9c84e27,190a899e461f05a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:37:22.018 INFO 2981023 --- [host-check-clockdiff3,0eb8769260ad0fa0,50b0681e2fb4c8c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047442 1 1 , error= 2024-09-11 17:37:22.817 INFO 2981023 --- [ocp-schedules-21,a234f1ecd71b39ab,94d4f91b8fecad08] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:37:22.817 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:37:22.817 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:37:22.817 INFO 2981023 --- [ocp-schedules-21,a234f1ecd71b39ab,94d4f91b8fecad08] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:37:22.817 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:37:22.821 INFO 2981023 --- [ocp-schedules-21,a234f1ecd71b39ab,94d4f91b8fecad08] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:37:22.821 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:22.822 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:22.822 INFO 2981023 --- [ocp-schedules-21,a234f1ecd71b39ab,94d4f91b8fecad08] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:37:22.822 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:22.823 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:22.823 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:22.824 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:22.839 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:37:22.841 INFO 2981023 --- [ocp-schedules-21,a234f1ecd71b39ab,94d4f91b8fecad08] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:37:22.842 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:37:22.842 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:37:22.845 INFO 2981023 --- [ocp-schedules-21,a234f1ecd71b39ab,94d4f91b8fecad08] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:37:23.237 INFO 2981023 --- [ocp-async-1,e3a689e59017e5f4,1862f46cf7edb960] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:23.250 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:23.251 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,41fa0bfd77961950,5535ad9a87faa629] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,41fa0bfd77961950,5535ad9a87faa629] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:24.182 INFO 2981023 --- [ocp-schedules-18,8dbe6cbebe2b3551,0cb0f1c307972ac3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:37:24.232 INFO 2981023 --- [ocp-schedules-12,36b7ffac93d75fc4,e15708ecf8b3d28a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:37:24.241 WARN 2981023 --- [scanner-13517084-1,8dbe6cbebe2b3551,5dc559c159c75489] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:37:24.257 INFO 2981023 --- [alarm-detect-20,8dbe6cbebe2b3551,c639e6c95fa4fe45] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644268, data is empty. 2024-09-11 17:37:24.260 INFO 2981023 --- [alarm-detect-20,8dbe6cbebe2b3551,c639e6c95fa4fe45] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644268, data is empty. 2024-09-11 17:37:24.264 INFO 2981023 --- [alarm-detect-20,8dbe6cbebe2b3551,c639e6c95fa4fe45] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644268, data is empty. 2024-09-11 17:37:24.279 INFO 2981023 --- [alarm-detect-5,8dbe6cbebe2b3551,b3faedc130c04d6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644268, data is empty. 2024-09-11 17:37:24.284 INFO 2981023 --- [alarm-detect-6,8dbe6cbebe2b3551,015f5492136191b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644268, data is empty. 2024-09-11 17:37:24.284 INFO 2981023 --- [alarm-detect-5,8dbe6cbebe2b3551,b3faedc130c04d6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644268, data is empty. 2024-09-11 17:37:24.289 INFO 2981023 --- [alarm-detect-5,8dbe6cbebe2b3551,b3faedc130c04d6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644268, data is empty. 2024-09-11 17:37:24.292 INFO 2981023 --- [alarm-detect-6,8dbe6cbebe2b3551,015f5492136191b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644268, data is empty. 2024-09-11 17:37:24.295 INFO 2981023 --- [alarm-detect-6,8dbe6cbebe2b3551,015f5492136191b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644268, data is empty. 2024-09-11 17:37:24.401 INFO 2981023 --- [alarm-detect-10,8dbe6cbebe2b3551,1d490d38679d0663] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644268, data is empty. 2024-09-11 17:37:24.406 INFO 2981023 --- [alarm-detect-10,8dbe6cbebe2b3551,1d490d38679d0663] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644268, data is empty. 2024-09-11 17:37:24.410 INFO 2981023 --- [alarm-detect-10,8dbe6cbebe2b3551,1d490d38679d0663] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644268, data is empty. 2024-09-11 17:37:25.238 INFO 2981023 --- [ocp-async-7,5b621fac7d5fbca7,7ab75a105ef9a29b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:26.251 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:26.252 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,a2a9723227527236,a8632e1020e8dd11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,a2a9723227527236,a8632e1020e8dd11] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:37:26.353064+08:00, lastProcessTime=2024-09-11T17:37:16.342506+08:00 2024-09-11 17:37:26.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:37:26.353064+08:00 2024-09-11 17:37:27.238 INFO 2981023 --- [ocp-async-6,faee7d75b11e085f,b165b128828311b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:27.754 INFO 2981023 --- [ocp-async-7,3c21c6102d9e8a53,50ae677ea3b3fe7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:37:29.238 INFO 2981023 --- [ocp-async-1,b7176b4513e9c6a1,0733b6560edf9b6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:29.252 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:29.254 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,9eb0908923728ca7,3a0ddd193ee9bbd5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:29.318 INFO 2981023 --- [prometheus-minute-schedule-1,9eb0908923728ca7,3a0ddd193ee9bbd5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:31.238 INFO 2981023 --- [ocp-async-7,7cdd31da4732c787,02c1186cd56cb683] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:31.986 INFO 2981023 --- [ocp-schedules-13,90c30e3a62f61f03,fd08df604845419d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:37:32.254 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:32.254 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,1cccc28baa4bb1b3,d4a2d281d8726729] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,1cccc28baa4bb1b3,d4a2d281d8726729] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:33.238 INFO 2981023 --- [ocp-async-6,b365519ccff682fe,15f05e92aecca864] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:33.937 INFO 2981023 --- [ocp-schedules-20,d58fcee0952b329f,8fdc3215a2a720a6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:37:34.007 INFO 2981023 --- [ocp-schedules-20,d58fcee0952b329f,8fdc3215a2a720a6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:37:34.007 INFO 2981023 --- [ocp-schedules-20,d58fcee0952b329f,8fdc3215a2a720a6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:37:34.459 INFO 2981023 --- [ocp-schedules-3,418926578606fe99,b8592ca9080e9ad5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:37:34.507 WARN 2981023 --- [scanner-13517421-1,418926578606fe99,8217f901c3b7b8f0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:37:35.238 INFO 2981023 --- [ocp-async-1,0078a1e9cfa311f8,81da3e532a490c4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:35.255 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:35.255 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,4162aea91dc1273f,bc7d443f626299cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,4162aea91dc1273f,bc7d443f626299cd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:36.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:37:36.351154+08:00, lastProcessTime=2024-09-11T17:37:26.353064+08:00 2024-09-11 17:37:36.480 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:37:36.351154+08:00 2024-09-11 17:37:37.238 INFO 2981023 --- [ocp-async-7,83a181ef29b918a9,b804f670a59c9397] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:37.314 INFO 2981023 --- [prometheus-cache-monitor1,589730015251a74e,55f195cde461b123] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:37:37.315 INFO 2981023 --- [prometheus-cache-monitor1,589730015251a74e,55f195cde461b123] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:37:37.754 INFO 2981023 --- [ocp-async-1,2f423599c5595ba3,ecbba0e62d4df463] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:37:38.255 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:38.256 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,fcceaea4b49e0d09,ba77b84358b35b40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,fcceaea4b49e0d09,ba77b84358b35b40] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:39.238 INFO 2981023 --- [ocp-async-2,8bcba618143241bc,a21c4314a21feef2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:39.726 INFO 2981023 --- [ocp-schedules-28,ae6fd1a7904ce201,21578557d0a76c3a] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:37:39.744 WARN 2981023 --- [ocp-schedules-28,ae6fd1a7904ce201,21578557d0a76c3a] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:37:40.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:37:40.003 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:37:41.238 INFO 2981023 --- [ocp-async-9,1b2eb8a7820d6691,a08ceca597574e0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:41.256 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:41.256 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,42b5f3294d4d8a55,28781eb7e75d05db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,42b5f3294d4d8a55,28781eb7e75d05db] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:41.989 INFO 2981023 --- [ocp-schedules-17,5026eb10eeebb31d,4c8f1953586fb0d4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:37:43.239 INFO 2981023 --- [ocp-async-3,8e0145f9476cfc5e,e67d49b8df21220f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:44.256 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:44.257 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,11c5d68c27646930,73f9a731341233c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,11c5d68c27646930,73f9a731341233c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:44.710 INFO 2981023 --- [ocp-schedules-2,00f58dbab5dfd070,e97b5482a17f677f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:37:44.775 WARN 2981023 --- [scanner-13517620-1,00f58dbab5dfd070,d0adb691f4f82482] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:37:44.802 INFO 2981023 --- [alarm-detect-10,00f58dbab5dfd070,72d56965dee6024d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644268, data is empty. 2024-09-11 17:37:44.807 INFO 2981023 --- [alarm-detect-10,00f58dbab5dfd070,72d56965dee6024d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644268, data is empty. 2024-09-11 17:37:44.811 INFO 2981023 --- [alarm-detect-10,00f58dbab5dfd070,72d56965dee6024d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644268, data is empty. 2024-09-11 17:37:44.831 INFO 2981023 --- [alarm-detect-7,00f58dbab5dfd070,ac9492ed69980e9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644268, data is empty. 2024-09-11 17:37:44.835 INFO 2981023 --- [alarm-detect-17,00f58dbab5dfd070,b2506172b49a7c3e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644268, data is empty. 2024-09-11 17:37:44.835 INFO 2981023 --- [alarm-detect-7,00f58dbab5dfd070,ac9492ed69980e9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644268, data is empty. 2024-09-11 17:37:44.839 INFO 2981023 --- [alarm-detect-17,00f58dbab5dfd070,b2506172b49a7c3e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644268, data is empty. 2024-09-11 17:37:44.844 INFO 2981023 --- [alarm-detect-7,00f58dbab5dfd070,ac9492ed69980e9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644268, data is empty. 2024-09-11 17:37:44.844 INFO 2981023 --- [alarm-detect-17,00f58dbab5dfd070,b2506172b49a7c3e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644268, data is empty. 2024-09-11 17:37:44.945 INFO 2981023 --- [alarm-detect-12,00f58dbab5dfd070,fe6a2e52ad7050b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644268, data is empty. 2024-09-11 17:37:44.950 INFO 2981023 --- [alarm-detect-12,00f58dbab5dfd070,fe6a2e52ad7050b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644268, data is empty. 2024-09-11 17:37:44.959 INFO 2981023 --- [alarm-detect-12,00f58dbab5dfd070,fe6a2e52ad7050b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644268, data is empty. 2024-09-11 17:37:45.101 INFO 2981023 --- [prometheus-second-schedule-1,d150a7528359001b,3ee750c78c940877] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:37:45.239 INFO 2981023 --- [ocp-async-5,9ed669d6f6d866dc,aa00c81dd406b784] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:46.020 INFO 2981023 --- [ocp-schedules-12,91ba6874aba1cc93,ce25b2010dea46e0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:37:46.022 INFO 2981023 --- [ocp-schedules-8,9155bcd412f8ceb9,9d37310650384582] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:37:46.025 INFO 2981023 --- [host-check-clockdiff1,91ba6874aba1cc93,f082e5a93305372c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:37:46.027 INFO 2981023 --- [host-check-clockdiff2,91ba6874aba1cc93,5d94ce3716d232e0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:37:46.028 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:37:46.028 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:37:46.028 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:37:46.028 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:37:46.029 INFO 2981023 --- [host-check-clockdiff3,91ba6874aba1cc93,bc4b939693d73470] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:37:46.029 INFO 2981023 --- [ocp-schedules-4,a95594d39acf2945,a23dba0d9d5d8864] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:37:46.029 INFO 2981023 --- [host-check-clockdiff4,91ba6874aba1cc93,05a353c0e7fdf487] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:37:46.045 INFO 2981023 --- [host-check-agent-status1,a95594d39acf2945,34b0b5dcbfb92f6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:37:46.046 INFO 2981023 --- [host-check-agent-status2,a95594d39acf2945,2cc752f219bd6351] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:37:46.047 INFO 2981023 --- [host-check-agent-status3,a95594d39acf2945,11fb1aec98b17ed9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:37:46.066 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:46.067 INFO 2981023 --- [host-check-agent-status4,a95594d39acf2945,8052e97f8d802552] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:37:46.069 INFO 2981023 --- [host-check-agent-status3,a95594d39acf2945,11fb1aec98b17ed9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:37:46.074 INFO 2981023 --- [host-check-clockdiff4,91ba6874aba1cc93,05a353c0e7fdf487] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047466 0 0 , error= 2024-09-11 17:37:46.074 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:46.077 INFO 2981023 --- [host-check-agent-status4,a95594d39acf2945,8052e97f8d802552] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:46.078 INFO 2981023 --- [host-check-agent-status4,a95594d39acf2945,8052e97f8d802552] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:46.078 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:46.079 INFO 2981023 --- [host-check-agent-status3,a95594d39acf2945,11fb1aec98b17ed9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:37:46.081 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:46.082 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:46.087 INFO 2981023 --- [host-check-agent-status1,a95594d39acf2945,34b0b5dcbfb92f6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:46.087 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:46.091 INFO 2981023 --- [host-check-agent-status2,a95594d39acf2945,2cc752f219bd6351] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:46.101 INFO 2981023 --- [host-check-agent-status1,a95594d39acf2945,34b0b5dcbfb92f6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:46.102 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:37:46.112 INFO 2981023 --- [host-check-agent-status2,a95594d39acf2945,2cc752f219bd6351] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:46.114 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:37:46.170 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:37:46.191 INFO 2981023 --- [ocp-schedules-18,da58c2d32ec19ebe,c4896732aa767210] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:37:46.191 INFO 2981023 --- [ocp-schedules-19,2bb354fe841f91ab,fc2c1ca193ca691f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:37:46.193 INFO 2981023 --- [ocp-schedules-16,7f49b86f327b809d,e31715f9ea2b438e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:37:46.193 INFO 2981023 --- [ocp-schedules-20,1bb669f5c4074448,31d74d4745781720] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:37:46.273 INFO 2981023 --- [ocp-schedules-19,2bb354fe841f91ab,fc2c1ca193ca691f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:37:46.288 INFO 2981023 --- [ocp-schedules-16,7f49b86f327b809d,e31715f9ea2b438e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:37:46.288 INFO 2981023 --- [ocp-schedules-17,6fa725248fee14b4,6fd7f138b4a893b7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:37:46.297 INFO 2981023 --- [ocp-schedules-18,da58c2d32ec19ebe,c4896732aa767210] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:37:46.306 INFO 2981023 --- [ocp-schedules-19,2bb354fe841f91ab,fc2c1ca193ca691f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:37:46.383 INFO 2981023 --- [ocp-schedules-17,6fa725248fee14b4,6fd7f138b4a893b7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:37:46.389 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:37:46.390 INFO 2981023 --- [ocp-schedules-14,06e15b6ed0d74ddb,d4395e390c03353f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:37:46.391 INFO 2981023 --- [host-check-agent-status1,a95594d39acf2945,34b0b5dcbfb92f6d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:37:46.392 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:37:46.395 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:37:46.395258936+08:00 2024-09-11 17:37:46.400 INFO 2981023 --- [host-check-agent-status1,a95594d39acf2945,34b0b5dcbfb92f6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:37:46.400 INFO 2981023 --- [host-check-agent-status1,a95594d39acf2945,34b0b5dcbfb92f6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:37:46.400 INFO 2981023 --- [host-check-agent-status1,a95594d39acf2945,34b0b5dcbfb92f6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:37:46.454 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:37:46.395+08:00 2024-09-11 17:37:46.472 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:37:46.395+08:00 and status AVAILABLE 2024-09-11 17:37:46.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:37:46.483493+08:00, lastProcessTime=2024-09-11T17:37:36.351154+08:00 2024-09-11 17:37:46.493 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:37:46.493 INFO 2981023 --- [host-check-agent-status4,a95594d39acf2945,8052e97f8d802552] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:37:46.499 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:37:46.499512531+08:00 2024-09-11 17:37:46.501 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:37:46.501 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:37:46.501554922+08:00 2024-09-11 17:37:46.502 INFO 2981023 --- [host-check-agent-status4,a95594d39acf2945,8052e97f8d802552] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:37:46.502 INFO 2981023 --- [host-check-agent-status4,a95594d39acf2945,8052e97f8d802552] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:37:46.502 INFO 2981023 --- [host-check-agent-status4,a95594d39acf2945,8052e97f8d802552] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:37:46.502 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:37:46.499+08:00 2024-09-11 17:37:46.505 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:37:46.501+08:00 2024-09-11 17:37:46.505 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.506 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:46.506 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:37:46.499+08:00 and status AVAILABLE 2024-09-11 17:37:46.506 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:37:46.506 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:37:46.508 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:37:46.501+08:00 and status AVAILABLE 2024-09-11 17:37:46.508 INFO 2981023 --- [host-check-agent-status3,a95594d39acf2945,11fb1aec98b17ed9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:37:46.511 INFO 2981023 --- [host-check-agent-status3,a95594d39acf2945,11fb1aec98b17ed9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:37:46.511 INFO 2981023 --- [host-check-agent-status3,a95594d39acf2945,11fb1aec98b17ed9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:37:46.511 INFO 2981023 --- [host-check-agent-status3,a95594d39acf2945,11fb1aec98b17ed9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:37:46.562 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.562 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:37:46.562 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:46.563 INFO 2981023 --- [host-check-agent-status2,a95594d39acf2945,2cc752f219bd6351] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:37:46.563 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:37:46.562884476+08:00 2024-09-11 17:37:46.563 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:37:46.563 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:37:46.563+08:00 2024-09-11 17:37:46.565 INFO 2981023 --- [host-check-agent-status2,a95594d39acf2945,2cc752f219bd6351] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:37:46.565 INFO 2981023 --- [host-check-agent-status2,a95594d39acf2945,2cc752f219bd6351] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:37:46.565 INFO 2981023 --- [host-check-agent-status2,a95594d39acf2945,2cc752f219bd6351] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:37:46.566 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:37:46.563+08:00 and status AVAILABLE 2024-09-11 17:37:46.569 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:37:46.570 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.571 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:46.572 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.572 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:37:46.572 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:46.573 INFO 2981023 --- [ocp-schedules-18,da58c2d32ec19ebe,c4896732aa767210] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:37:46.573 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:37:46.576 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.577 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:46.586 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.586 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:37:46.586 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:37:46.587 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:37:46.588 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.592 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:37:46.593 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:46.594 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:46.595 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:37:46.599 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:46.601 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:46.605 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.605 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:46.606 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:37:46.617 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:46.618 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:46.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.619 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.619 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:37:46.660 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:37:46.660 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:37:46.661 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:37:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:46.680 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:37:46.680 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:37:46.680 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.681 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:37:46.683 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:37:46.684 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:37:46.688 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:37:46.696 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:37:46.703 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:37:46.707 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:46.707 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:37:46.712 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:37:46.718 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:37:46.723 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.727 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.728 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:37:46.728 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:37:46.728 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.745 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:37:46.754 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:46.756 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:37:46.757 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.758 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.758 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:37:46.759 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:37:46.760 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:37:46.763 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:37:46.766 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.766 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.767 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:46.768 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:37:46.768 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:37:46.774 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:37:46.779 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:37:46.780 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:37:46.784 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:37:46.787 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.796 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:37:46.800 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.803 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:37:46.804 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.805 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.805 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.805 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.805 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:37:46.806 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:37:46.806 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:37:46.806 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.808 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:37:46.810 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.811 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:37:46.811 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:46.811 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:37:46.811 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.812 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.813 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:37:46.817 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:37:46.819 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:37:46.820 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:37:46.830 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:37:46.837 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:37:46.837 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:46.837 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:37:46.838 INFO 2981023 --- [ocp-schedules-22,45f372acddb44a51,3157090e00077b8d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:37:46.838 INFO 2981023 --- [ocp-schedules-22,45f372acddb44a51,3157090e00077b8d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:37:46.840 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:37:46.841 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:37:46.850 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:37:46.856 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.860 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:37:46.861 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:37:46.861 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:37:46.862 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:37:46.862 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.865 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:37:46.865 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:37:46.866 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:37:46.867 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:37:46.873 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.874 INFO 2981023 --- [host-check-host-status3,9155bcd412f8ceb9,407edff9283ea4e2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:37:46.874 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.874 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:37:46.875 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:37:46.875 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.878 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:37:46.880 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.881 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.882 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:46.882 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:37:46.884 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:46.884 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:37:46.886 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:37:46.890 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:37:46.891 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:37:46.893 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:37:46.896 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:37:46.901 INFO 2981023 --- [host-check-host-status1,9155bcd412f8ceb9,e7389959e02ac610] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:37:46.906 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:37:46.917 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:37:46.919 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:37:46.920 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.920 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:37:46.921 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:37:46.922 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.922 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:37:46.923 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:37:46.924 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:37:46.930 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:37:46.930 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.931 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.931 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:37:46.937 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:37:46.938 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:37:46.940 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:37:46.940 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:37:46.948 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.949 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:37:46.949 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.949 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:37:46.950 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:37:46.950 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.954 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:37:46.967 INFO 2981023 --- [host-check-host-status4,9155bcd412f8ceb9,6016261744da9a6e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:37:46.970 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:37:46.976 INFO 2981023 --- [host-check-clockdiff2,91ba6874aba1cc93,5d94ce3716d232e0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047466 0 0 , error= 2024-09-11 17:37:46.982 INFO 2981023 --- [host-check-host-status2,9155bcd412f8ceb9,b374a3c57cc96309] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:37:46.983 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:37:46.985 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:37:46.483493+08:00 2024-09-11 17:37:46.990 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.990 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:46.991 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:37:46.991 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:37:46.998 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:46.999 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:47.000 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:37:47.000 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:37:47.000 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:37:47.005 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:37:47.006 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:37:47.006 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:37:47.006 INFO 2981023 --- [ocp-schedules-16,7f49b86f327b809d,e31715f9ea2b438e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:37:47.020 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:37:47.026 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:37:47.028 INFO 2981023 --- [ocp-schedules-21,6d45f03d3f256650,314c6234ee1c8909] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:37:47.239 INFO 2981023 --- [ocp-async-7,235b0fef76db00a6,7e3edbba3c6f2b2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:47.257 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:47.257 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,0410cdc520abfe5e,1bfe9241e5e98358] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,0410cdc520abfe5e,1bfe9241e5e98358] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:47.754 INFO 2981023 --- [ocp-async-9,036cfe7bd78cbaf4,da5616aafd05fe7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:37:47.796 INFO 2981023 --- [ocp-schedules-9,b201d1237b443a8c,b1f9f22403cb8266] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:37:47.821 INFO 2981023 --- [host-check-clockdiff1,91ba6874aba1cc93,f082e5a93305372c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047467 0 0 , error= 2024-09-11 17:37:49.239 INFO 2981023 --- [ocp-async-3,205923acad178f53,3a32fd6b7efe30d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:37:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:37:50.230 INFO 2981023 --- [ocp-schedules-7,216ff4eded4a37f2,e2d2e708f7312d8c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:37:50.257 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:50.258 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,cc3ae10557cf660a,99bbd738b80c82a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,cc3ae10557cf660a,99bbd738b80c82a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:51.239 INFO 2981023 --- [ocp-async-10,7195451f48e82454,529c6ec03e67bce4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:51.544 INFO 2981023 --- [ocp-schedules-14,7690e222da1c4755,0b3d8f1edaa12f60] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:37:51.545 INFO 2981023 --- [ocp-schedules-14,7690e222da1c4755,0b3d8f1edaa12f60] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:37:51.545 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:37:51.545 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:37:51.546 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:37:51.546 INFO 2981023 --- [ocp-schedules-14,7690e222da1c4755,0b3d8f1edaa12f60] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:37:51.992 INFO 2981023 --- [ocp-schedules-19,4bdaf7e77b8857d0,ce9d97d72bd14f79] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:37:52.007 INFO 2981023 --- [host-check-clockdiff3,91ba6874aba1cc93,bc4b939693d73470] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047472 1 1 , error= 2024-09-11 17:37:52.651 INFO 2981023 --- [ocp-schedules-18,668089f425c3774a,8df697fa284a756a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:37:52.651+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:37:52.654 INFO 2981023 --- [ocp-schedules-18,668089f425c3774a,8df697fa284a756a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:37:52.809 INFO 2981023 --- [ocp-schedules-18,668089f425c3774a,8df697fa284a756a] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 42 to SqlAggregateData 2024-09-11 17:37:52.810 INFO 2981023 --- [ocp-schedules-18,668089f425c3774a,8df697fa284a756a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:37:52.811 INFO 2981023 --- [ocp-schedules-10,0e292e2a2a530ffe,11e4e4dc447d1e87] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:37:52.831 WARN 2981023 --- [ocp-schedules-10,0e292e2a2a530ffe,11e4e4dc447d1e87] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:37:53.098 INFO 2981023 --- [ocp-schedules-30,f4821bd54d7916cc,a77dc5c6362506ec] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:37:53.098 INFO 2981023 --- [ocp-schedules-30,f4821bd54d7916cc,a77dc5c6362506ec] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:37:53.239 INFO 2981023 --- [ocp-async-2,79d2b3067624a2bb,02510b2afe28ce32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:53.258 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:53.258 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,d0e874129e774353,e5e8d7648118bf93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,d0e874129e774353,e5e8d7648118bf93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:53.547 INFO 2981023 --- [ocp-schedules-5,5bc6c814a47e490b,9b619f6a8e287f34] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:37:53.548 INFO 2981023 --- [ocp-schedules-5,5bc6c814a47e490b,9b619f6a8e287f34] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:37:53.573 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:37:53.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:37:53.573 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.574 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:37:53.574 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.576 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:37:53.577 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:37:53.578 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:37:53.580 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.580 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.582 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.582 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.582 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.582 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.586 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.586 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.592 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.593 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.593 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.593 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:37:53.597 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.597 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.603 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.603 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:37:53.605 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.605 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.606 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:37:53.606 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2AA1D6E7-0-0] 2024-09-11 17:37:53.606 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:37:53.606 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.606 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:37:53.610 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.610 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.610 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:37:53.617 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.618 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:37:53.619 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.619 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:37:53.624 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.625 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:37:53.626 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.626 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:37:53.627 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.627 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:37:53.628 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:37:53.630 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.631 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:37:53.631 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.632 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:37:53.632 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.632 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:37:53.632 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.633 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:37:53.633 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:37:53.633 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:37:53.633 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2AA1D6F1-0-0] 2024-09-11 17:37:53.633 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:37:53.633 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.634 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:37:53.635 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.636 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:37:53.636 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.637 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:37:53.637 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.638 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:37:53.638 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.639 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:37:53.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.640 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:37:53.641 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:37:53.641 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:37:53.641 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.642 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:37:53.642 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:37:53.642 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.643 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:37:53.644 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.644 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:53.644 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:37:53.645 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.646 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.646 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:37:53.646 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.646 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.647 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:37:53.647 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:37:53.649 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:37:53.654 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:53.655 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.655 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:37:53.660 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:37:53.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:37:53.662 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:37:53.663 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.663 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:37:53.663 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:37:53.663 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:37:53.663 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:37:53.663 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:37:53.663 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.664 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:37:53.668 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:37:53.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:37:53.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.670 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.672 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:37:53.674 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:37:53.674 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:53.674 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:37:53.675 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:37:53.676 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:37:53.676 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:37:53.676 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:37:53.682 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:37:53.683 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:37:53.683 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:37:53.683 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:37:53.684 INFO 2981023 --- [ocp-schedules-5,5bc6c814a47e490b,9b619f6a8e287f34] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:37:54.009 INFO 2981023 --- [ocp-schedules-21,a62690e9bd548add,8c4c13d1922b4fa6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:37:54.015 INFO 2981023 --- [ocp-schedules-21,a62690e9bd548add,8c4c13d1922b4fa6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:37:54.015 INFO 2981023 --- [ocp-schedules-21,a62690e9bd548add,8c4c13d1922b4fa6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:37:54.234 INFO 2981023 --- [ocp-schedules-2,cfd091886f73dbae,d11dfa00dcb12bc4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:37:54.991 INFO 2981023 --- [ocp-schedules-24,1a834f0fb810f149,0c39ba1c55f1adc0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:37:55.036 WARN 2981023 --- [scanner-13517951-1,1a834f0fb810f149,0f10ac333efca74d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:37:55.239 INFO 2981023 --- [ocp-async-8,ffbb8105946cae65,a5aaea4cb08ec1b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:55.687 INFO 2981023 --- [ocp-schedules-2,3038fe9151941eac,f485ca23cda13e93] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:37:55.687 INFO 2981023 --- [ocp-schedules-2,3038fe9151941eac,f485ca23cda13e93] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:37:55.688 INFO 2981023 --- [ocp-schedules-2,3038fe9151941eac,f485ca23cda13e93] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:37:55.688 INFO 2981023 --- [ocp-schedules-2,3038fe9151941eac,f485ca23cda13e93] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:37:55.688 INFO 2981023 --- [ocp-schedules-2,3038fe9151941eac,f485ca23cda13e93] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:37:55.688 INFO 2981023 --- [ocp-schedules-2,3038fe9151941eac,f485ca23cda13e93] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:37:55.777 INFO 2981023 --- [ocp-schedules-6,2c9dcc28b914682b,f41cbc17278ba2fb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:37:55.783 INFO 2981023 --- [ocp-schedules-6,2c9dcc28b914682b,f41cbc17278ba2fb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:37:55.784 INFO 2981023 --- [ocp-schedules-6,2c9dcc28b914682b,f41cbc17278ba2fb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:37:56.244 INFO 2981023 --- [ocp-schedules-30,b19cb8515707cfc6,adef1cbd1d30e4d8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:37:56.244 INFO 2981023 --- [ocp-schedules-30,b19cb8515707cfc6,adef1cbd1d30e4d8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:37:56.258 INFO 2981023 --- [Collect all tenants major compactions6,b19cb8515707cfc6,e1c14b183e6bc738] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:37:56.259 INFO 2981023 --- [Collect all tenants major compactions2,b19cb8515707cfc6,5c0c43dc13320181] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:37:56.259 INFO 2981023 --- [Collect all tenants major compactions3,b19cb8515707cfc6,9195b57bb0c4e764] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:37:56.259 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:56.260 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:56.260 INFO 2981023 --- [Collect all tenants major compactions4,b19cb8515707cfc6,1370b5dd4770b26c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:37:56.260 INFO 2981023 --- [Collect all tenants major compactions5,b19cb8515707cfc6,e37bf4c5b8c69939] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:37:56.261 INFO 2981023 --- [Collect all tenants major compactions1,b19cb8515707cfc6,3b275ccbdd9be6ce] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:37:56.271 INFO 2981023 --- [Collect all tenants major compactions4,b19cb8515707cfc6,1370b5dd4770b26c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:56.272 INFO 2981023 --- [Collect all tenants major compactions2,b19cb8515707cfc6,5c0c43dc13320181] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:56.273 INFO 2981023 --- [Collect all tenants major compactions4,b19cb8515707cfc6,1370b5dd4770b26c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:37:56.273 INFO 2981023 --- [Collect all tenants major compactions3,b19cb8515707cfc6,9195b57bb0c4e764] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:56.273 INFO 2981023 --- [Collect all tenants major compactions5,b19cb8515707cfc6,e37bf4c5b8c69939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:56.273 INFO 2981023 --- [Collect all tenants major compactions6,b19cb8515707cfc6,e1c14b183e6bc738] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:56.273 INFO 2981023 --- [Collect all tenants major compactions1,b19cb8515707cfc6,3b275ccbdd9be6ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:37:56.274 INFO 2981023 --- [Collect all tenants major compactions2,b19cb8515707cfc6,5c0c43dc13320181] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:37:56.274 INFO 2981023 --- [Collect all tenants major compactions3,b19cb8515707cfc6,9195b57bb0c4e764] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:37:56.274 INFO 2981023 --- [Collect all tenants major compactions6,b19cb8515707cfc6,e1c14b183e6bc738] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:37:56.274 INFO 2981023 --- [Collect all tenants major compactions1,b19cb8515707cfc6,3b275ccbdd9be6ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:37:56.275 INFO 2981023 --- [Collect all tenants major compactions5,b19cb8515707cfc6,e37bf4c5b8c69939] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:37:56.277 INFO 2981023 --- [Collect all tenants major compactions4,b19cb8515707cfc6,1370b5dd4770b26c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:37:56.277 INFO 2981023 --- [Collect all tenants major compactions2,b19cb8515707cfc6,5c0c43dc13320181] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:37:56.277 INFO 2981023 --- [Collect all tenants major compactions2,b19cb8515707cfc6,5c0c43dc13320181] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:37:56.277 INFO 2981023 --- [Collect all tenants major compactions4,b19cb8515707cfc6,1370b5dd4770b26c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:37:56.278 INFO 2981023 --- [Collect all tenants major compactions3,b19cb8515707cfc6,9195b57bb0c4e764] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:37:56.278 INFO 2981023 --- [Collect all tenants major compactions1,b19cb8515707cfc6,3b275ccbdd9be6ce] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:37:56.278 INFO 2981023 --- [Collect all tenants major compactions1,b19cb8515707cfc6,3b275ccbdd9be6ce] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:37:56.278 INFO 2981023 --- [Collect all tenants major compactions3,b19cb8515707cfc6,9195b57bb0c4e764] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:37:56.278 INFO 2981023 --- [Collect all tenants major compactions6,b19cb8515707cfc6,e1c14b183e6bc738] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:37:56.278 INFO 2981023 --- [Collect all tenants major compactions6,b19cb8515707cfc6,e1c14b183e6bc738] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:37:56.278 INFO 2981023 --- [Collect all tenants major compactions5,b19cb8515707cfc6,e37bf4c5b8c69939] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:37:56.278 INFO 2981023 --- [Collect all tenants major compactions5,b19cb8515707cfc6,e37bf4c5b8c69939] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:37:56.279 INFO 2981023 --- [ocp-schedules-30,b19cb8515707cfc6,adef1cbd1d30e4d8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:37:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,94f2a2b7a65f91ba,a0c8d39f220e6cf1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,94f2a2b7a65f91ba,a0c8d39f220e6cf1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:37:56.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:37:56.336945+08:00, lastProcessTime=2024-09-11T17:37:46.483493+08:00 2024-09-11 17:37:56.471 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:37:56.336945+08:00 2024-09-11 17:37:57.239 INFO 2981023 --- [ocp-async-9,5fd945743c84c0f4,defacf5e3ac6943d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:57.754 INFO 2981023 --- [ocp-async-4,d6c4d82dac09f48c,c39cd824321a3b12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:37:58.355 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.359 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, ob 2024-09-11 17:37:58.365 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.370 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.371 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.376 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.377 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.378 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.378 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.379 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.382 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.384 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.387 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.390 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.393 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.396 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.399 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.401 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.402 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.403 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.403 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.404 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.408 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.411 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.414 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.417 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:31:58.273, end=2024-09-11T17:36:58.352} 2024-09-11 17:37:58.417 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:37:59.240 INFO 2981023 --- [ocp-async-2,f925639b33c1a6d2,57e94b71a411d04e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:37:59.260 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:37:59.260 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:37:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,38c30c1e8561ee09,605a8d1cde6916b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047480, interval=60 2024-09-11 17:37:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,38c30c1e8561ee09,605a8d1cde6916b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:01.243 INFO 2981023 --- [ocp-async-3,9d9b9bdbb2f0176c,e9acabf48df64b1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:01.995 INFO 2981023 --- [ocp-schedules-1,71a52d254cfafe7e,e42df81040878abf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:38:02.027 INFO 2981023 --- [ocp-schedules-28,4bc5d6ecf944bc2d,fcaeb0d694158ac5] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:38:02.027 INFO 2981023 --- [ocp-schedules-28,4bc5d6ecf944bc2d,fcaeb0d694158ac5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:38:02.261 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:02.261 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,ede4343456257891,511f4c2ebaed86ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,ede4343456257891,511f4c2ebaed86ca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:02.457 INFO 2981023 --- [ocp-schedules-20,51752aa6949d94f4,cd3475a213a0ecf4] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:38:02.458 INFO 2981023 --- [ocp-schedules-20,51752aa6949d94f4,cd3475a213a0ecf4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:38:02.467 INFO 2981023 --- [ocp-schedules-20,51752aa6949d94f4,cd3475a213a0ecf4] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:38:03.240 INFO 2981023 --- [ocp-async-9,04b063cebf993680,afd8db0f00bd2f3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:05.184 INFO 2981023 --- [ocp-schedules-25,b62ea5f95c1074a9,e9111fb67c31edb1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:38:05.232 WARN 2981023 --- [scanner-13518166-1,b62ea5f95c1074a9,f3a078aad1ebdfbd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:38:05.240 INFO 2981023 --- [ocp-async-1,c762ad84980e73ff,a6dad9f9e606efc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:05.259 INFO 2981023 --- [alarm-detect-9,b62ea5f95c1074a9,b2f8b2a989817963] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644268, data is empty. 2024-09-11 17:38:05.262 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:05.265 INFO 2981023 --- [alarm-detect-9,b62ea5f95c1074a9,b2f8b2a989817963] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644310, data is empty. 2024-09-11 17:38:05.266 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:05.274 INFO 2981023 --- [alarm-detect-9,b62ea5f95c1074a9,b2f8b2a989817963] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644310, data is empty. 2024-09-11 17:38:05.286 INFO 2981023 --- [alarm-detect-4,b62ea5f95c1074a9,2ee151996a0a6b81] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644268, data is empty. 2024-09-11 17:38:05.286 INFO 2981023 --- [alarm-detect-7,b62ea5f95c1074a9,8c51a59479b30980] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644268, data is empty. 2024-09-11 17:38:05.289 INFO 2981023 --- [alarm-detect-7,b62ea5f95c1074a9,8c51a59479b30980] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644310, data is empty. 2024-09-11 17:38:05.294 INFO 2981023 --- [alarm-detect-4,b62ea5f95c1074a9,2ee151996a0a6b81] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644310, data is empty. 2024-09-11 17:38:05.295 INFO 2981023 --- [alarm-detect-7,b62ea5f95c1074a9,8c51a59479b30980] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644310, data is empty. 2024-09-11 17:38:05.297 INFO 2981023 --- [alarm-detect-4,b62ea5f95c1074a9,2ee151996a0a6b81] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644310, data is empty. 2024-09-11 17:38:05.338 INFO 2981023 --- [prometheus-minute-schedule-1,e69ed616f6701cbf,6ca99668cad01adc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:05.345 INFO 2981023 --- [prometheus-minute-schedule-1,e69ed616f6701cbf,6ca99668cad01adc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:05.422 INFO 2981023 --- [alarm-detect-2,b62ea5f95c1074a9,6f99e40f8189b66d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644268, data is empty. 2024-09-11 17:38:05.426 INFO 2981023 --- [alarm-detect-2,b62ea5f95c1074a9,6f99e40f8189b66d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644310, data is empty. 2024-09-11 17:38:05.430 INFO 2981023 --- [alarm-detect-2,b62ea5f95c1074a9,6f99e40f8189b66d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644310, data is empty. 2024-09-11 17:38:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:38:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:38:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:38:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:38:05.995 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:38:06.000 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:38:06.005 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:38:06.005 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:38:06.005 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:38:06.007 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:06.008 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:06.008 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:38:06.008 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:38:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:06.009 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:38:06.010 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:38:06.050 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:35:40, start=2024-09-11T17:32:40, end=2024-09-11T17:35:10, updates=108, granule=120000ms, delay=180000ms 2024-09-11 17:38:06.055 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:35:40, target=2024-09-11T17:35:10, granule=120000ms, delay=30000ms) 2024-09-11 17:38:06.147 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:35:40, start=2024-09-11T17:32:41, end=2024-09-11T17:35:11, updates=298, granule=120000ms, delay=179000ms 2024-09-11 17:38:06.152 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - source=2024-09-11T17:35:10, start=2024-09-11T17:19:41, end=2024-09-11T17:29:41, updates=294, granule=600000ms, delay=929000ms 2024-09-11 17:38:06.154 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:35:40, target=2024-09-11T17:35:10, granule=120000ms, delay=30000ms) 2024-09-11 17:38:06.156 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:35:10, target=2024-09-11T17:29:31, granule=600000ms, delay=339000ms) 2024-09-11 17:38:06.221 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T17:35:10, start=2024-09-11T17:20:51, end=2024-09-11T17:30:51, updates=396, granule=600000ms, delay=859000ms 2024-09-11 17:38:06.225 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:35:10, target=2024-09-11T17:30:51, granule=600000ms, delay=259000ms) 2024-09-11 17:38:06.225 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=255ms, period=120000ms 2024-09-11 17:38:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:38:06.351912+08:00, lastProcessTime=2024-09-11T17:37:56.336945+08:00 2024-09-11 17:38:06.496 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:38:06.351912+08:00 2024-09-11 17:38:07.240 INFO 2981023 --- [ocp-async-6,3ea08c5599ec7d91,76ee20bc7f29dd54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:07.373 INFO 2981023 --- [ocp-schedules-4,0234666e8e4d5ce5,8a2480a66972870b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:38:07.381 INFO 2981023 --- [ocp-schedules-21,059215345d1384d3,1f8420989ec45c71] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:38:07.381+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:38:07.381 INFO 2981023 --- [ocp-schedules-21,059215345d1384d3,1f8420989ec45c71] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:38:07.383 INFO 2981023 --- [ocp-schedules-21,059215345d1384d3,1f8420989ec45c71] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:38:07.383 INFO 2981023 --- [ocp-schedules-21,059215345d1384d3,1f8420989ec45c71] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:38:07.755 INFO 2981023 --- [ocp-async-2,267ed5b8ca56b4b8,59aa569229a4bdf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:38:08.266 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:08.268 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,96875bffccfc598a,d4a52bae80f9b3d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,96875bffccfc598a,d4a52bae80f9b3d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:09.240 INFO 2981023 --- [ocp-async-5,bed8b03fe4dde59a,86c18bd1d96b6d65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:10.003 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:38:10.003 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:38:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:38:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:38:11.240 INFO 2981023 --- [ocp-async-7,1cae4b7720b195bb,7fc70d2fb888d97b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:11.268 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:11.268 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,f466a1e1fe0e2e2f,b91c4242380696f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,f466a1e1fe0e2e2f,b91c4242380696f4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:11.997 INFO 2981023 --- [ocp-schedules-14,4d9b5dd294c9515e,a227337d6a919f86] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:38:12.423 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047420, nodeCount=1000, affectRows=1000 2024-09-11 17:38:12.424 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047420, nodeCount=1000, affectRows=1000 2024-09-11 17:38:12.426 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047420, nodeCount=999, validCount=999 2024-09-11 17:38:12.426 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047420, nodeCount=1000, validCount=1000 2024-09-11 17:38:12.434 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047420, nodeCount=48, affectRows=48 2024-09-11 17:38:12.435 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047420, nodeCount=48, validCount=48 2024-09-11 17:38:13.241 INFO 2981023 --- [ocp-async-4,11f8198b29fce501,db59d6b0e7f05ee8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:14.018 INFO 2981023 --- [ocp-schedules-22,03d946dec389253d,8fd4642ec6f50026] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:38:14.026 INFO 2981023 --- [ocp-schedules-22,03d946dec389253d,8fd4642ec6f50026] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:38:14.026 INFO 2981023 --- [ocp-schedules-22,03d946dec389253d,8fd4642ec6f50026] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:38:14.268 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:14.269 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,cf309c0cb5654e1d,0753cf8c8714e34e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,c5dd567c7e167bfb,803ff8b60a730860] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:38:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,cf309c0cb5654e1d,0753cf8c8714e34e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:15.241 INFO 2981023 --- [ocp-async-8,ca0f9e4078598ae0,5376a65f9715219d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:15.471 INFO 2981023 --- [ocp-schedules-29,68ed163cd770d8fa,566a7578321b9712] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:38:15.520 WARN 2981023 --- [scanner-13519959-1,68ed163cd770d8fa,766f83a760da6604] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:38:15.726 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047420, nodeCount=480, affectRows=480 2024-09-11 17:38:16.022 INFO 2981023 --- [ocp-schedules-1,dce4ed3f1e640e32,ae6ed9f14b410c40] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:38:16.023 INFO 2981023 --- [ocp-schedules-15,592e0e126204431c,e4ab93fcba96911b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:38:16.023 INFO 2981023 --- [host-check-clockdiff1,dce4ed3f1e640e32,690c0ec04b1c4d65] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:38:16.023 INFO 2981023 --- [host-check-clockdiff2,dce4ed3f1e640e32,ac114aff9d46a8ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:38:16.024 INFO 2981023 --- [host-check-clockdiff3,dce4ed3f1e640e32,47a5d67ddf542189] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:38:16.024 INFO 2981023 --- [host-check-clockdiff4,dce4ed3f1e640e32,9d4c2db280eef83b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:38:16.024 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:38:16.024 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:38:16.025 INFO 2981023 --- [ocp-schedules-14,8882aa20b1109280,cda0a610c6590450] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:38:16.026 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:38:16.027 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:38:16.033 INFO 2981023 --- [host-check-agent-status1,8882aa20b1109280,875ea67ef92b3c44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:38:16.041 INFO 2981023 --- [host-check-agent-status2,8882aa20b1109280,5a9cdea5f19ed0df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:38:16.042 INFO 2981023 --- [host-check-agent-status3,8882aa20b1109280,eeb871ce39501b3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:38:16.042 INFO 2981023 --- [host-check-agent-status4,8882aa20b1109280,f775d0518aab2136] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:38:16.049 INFO 2981023 --- [host-check-clockdiff2,dce4ed3f1e640e32,ac114aff9d46a8ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047496 0 0 , error= 2024-09-11 17:38:16.049 INFO 2981023 --- [host-check-clockdiff4,dce4ed3f1e640e32,9d4c2db280eef83b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047496 0 0 , error= 2024-09-11 17:38:16.064 INFO 2981023 --- [host-check-agent-status4,8882aa20b1109280,f775d0518aab2136] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:38:16.064 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:38:16.065 INFO 2981023 --- [host-check-agent-status3,8882aa20b1109280,eeb871ce39501b3e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:38:16.065 INFO 2981023 --- [host-check-agent-status1,8882aa20b1109280,875ea67ef92b3c44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:38:16.068 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:38:16.068 INFO 2981023 --- [host-check-agent-status1,8882aa20b1109280,875ea67ef92b3c44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:38:16.069 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:38:16.069 INFO 2981023 --- [host-check-agent-status4,8882aa20b1109280,f775d0518aab2136] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:38:16.069 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:38:16.070 INFO 2981023 --- [host-check-agent-status3,8882aa20b1109280,eeb871ce39501b3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:38:16.072 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:38:16.076 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:38:16.076 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:38:16.107 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:38:16.188 INFO 2981023 --- [ocp-schedules-17,467b46606f46d51b,ab01a8830ab604cf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:38:16.188 INFO 2981023 --- [ocp-schedules-19,3a54c309701f3582,c138a88cad178be0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:38:16.188 INFO 2981023 --- [ocp-schedules-25,9f6a59a5dc1d14d6,4350255ebb89068f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:38:16.188 INFO 2981023 --- [ocp-schedules-7,c0c2fa096afb3125,bbfd87e3e52cd8b7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:38:16.205 INFO 2981023 --- [host-check-agent-status2,8882aa20b1109280,5a9cdea5f19ed0df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:38:16.208 INFO 2981023 --- [ocp-schedules-17,467b46606f46d51b,ab01a8830ab604cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:38:16.212 INFO 2981023 --- [host-check-agent-status2,8882aa20b1109280,5a9cdea5f19ed0df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:38:16.257 INFO 2981023 --- [ocp-schedules-19,3a54c309701f3582,c138a88cad178be0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:38:16.257 INFO 2981023 --- [ocp-schedules-25,9f6a59a5dc1d14d6,4350255ebb89068f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:38:16.278 INFO 2981023 --- [ocp-schedules-19,3a54c309701f3582,c138a88cad178be0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:38:16.292 INFO 2981023 --- [ocp-schedules-27,9f579763d822cb74,5136941ed43f800b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:38:16.300 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:38:16.301 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:38:16.303 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:38:16.303671872+08:00 2024-09-11 17:38:16.305 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:38:16.304347182+08:00 2024-09-11 17:38:16.306 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:38:16.306 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:38:16.305+08:00 2024-09-11 17:38:16.307 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:38:16.303+08:00 2024-09-11 17:38:16.308 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:38:16.307705150+08:00 2024-09-11 17:38:16.309 INFO 2981023 --- [host-check-agent-status1,8882aa20b1109280,875ea67ef92b3c44] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:38:16.309 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:38:16.303+08:00 and status AVAILABLE 2024-09-11 17:38:16.311 INFO 2981023 --- [host-check-agent-status1,8882aa20b1109280,875ea67ef92b3c44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:38:16.311 INFO 2981023 --- [host-check-agent-status1,8882aa20b1109280,875ea67ef92b3c44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:38:16.311 INFO 2981023 --- [host-check-agent-status1,8882aa20b1109280,875ea67ef92b3c44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:38:16.314 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:38:16.305+08:00 and status AVAILABLE 2024-09-11 17:38:16.315 INFO 2981023 --- [host-check-agent-status4,8882aa20b1109280,f775d0518aab2136] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:38:16.315 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:38:16.308+08:00 2024-09-11 17:38:16.317 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:38:16.308+08:00 and status AVAILABLE 2024-09-11 17:38:16.320 INFO 2981023 --- [host-check-agent-status3,8882aa20b1109280,eeb871ce39501b3e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:38:16.321 INFO 2981023 --- [host-check-agent-status4,8882aa20b1109280,f775d0518aab2136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:38:16.321 INFO 2981023 --- [host-check-agent-status4,8882aa20b1109280,f775d0518aab2136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:38:16.321 INFO 2981023 --- [host-check-agent-status4,8882aa20b1109280,f775d0518aab2136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:38:16.320 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.322 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:16.322 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:38:16.323 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:38:16.324 INFO 2981023 --- [host-check-agent-status3,8882aa20b1109280,eeb871ce39501b3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:38:16.324 INFO 2981023 --- [host-check-agent-status3,8882aa20b1109280,eeb871ce39501b3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:38:16.324 INFO 2981023 --- [host-check-agent-status3,8882aa20b1109280,eeb871ce39501b3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:38:16.326 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.326 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:16.327 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:38:16.328 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.329 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:16.330 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:38:16.330 INFO 2981023 --- [ocp-schedules-17,467b46606f46d51b,ab01a8830ab604cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:38:16.337 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:38:16.341 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:38:16.342 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:38:16.342 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:38:16.342461723+08:00 2024-09-11 17:38:16.360 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:38:16.342+08:00 2024-09-11 17:38:16.364 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:38:16.342+08:00 and status AVAILABLE 2024-09-11 17:38:16.366 INFO 2981023 --- [host-check-agent-status2,8882aa20b1109280,5a9cdea5f19ed0df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:38:16.367 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:38:16.369 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:38:16.376 INFO 2981023 --- [host-check-agent-status2,8882aa20b1109280,5a9cdea5f19ed0df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:38:16.376 INFO 2981023 --- [host-check-agent-status2,8882aa20b1109280,5a9cdea5f19ed0df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:38:16.376 INFO 2981023 --- [host-check-agent-status2,8882aa20b1109280,5a9cdea5f19ed0df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:38:16.378 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:38:16.379 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.380 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:16.392 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:38:16.401 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:38:16.401 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:38:16.400102+08:00, lastProcessTime=2024-09-11T17:38:06.351912+08:00 2024-09-11 17:38:16.402 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.402 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:38:16.402 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:38:16.403 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:16.403 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:38:16.405 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:38:16.405 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:38:16.412 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:38:16.419 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.419 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:16.437 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:38:16.474 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:38:16.475 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:38:16.480 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:38:16.492 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:38:16.492 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.493 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:38:16.500 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:38:16.510 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:38:16.514 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:38:16.517 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.518 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.519 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:38:16.523 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:38:16.525 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:38:16.529 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:38:16.529 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:38:16.532 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.535 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.536 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:16.537 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:38:16.537 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.540 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:38:16.540 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:38:16.540 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.541 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:38:16.544 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:38:16.553 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.554 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.554 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:16.555 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.558 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:38:16.559 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:38:16.561 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:38:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:16.567 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:38:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:38:16.567 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:38:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.572 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:16.572 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:38:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.573 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:16.573 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:38:16.574 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.576 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:38:16.577 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:38:16.578 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:38:16.582 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:38:16.586 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:16.586 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:38:16.592 INFO 2981023 --- [host-check-host-status3,592e0e126204431c,832012f0bae9fe03] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:38:16.593 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:16.593 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:16.595 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:38:16.599 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:38:16.608 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:16.608 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:38:16.611 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:38:16.614 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.614 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.615 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.615 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:16.616 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.616 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:38:16.617 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.617 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:16.617 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.617 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:38:16.618 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:38:16.618 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:16.618 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:38:16.622 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:38:16.623 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:38:16.623 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:38:16.624 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:38:16.626 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:38:16.628 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:38:16.628 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:38:16.628 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:38:16.630 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:38:16.631 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:38:16.647 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:38:16.648 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:38:16.654 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:38:16.656 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:38:16.657 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:38:16.660 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:38:16.690 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:38:16.691 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:38:16.692 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:38:16.697 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:38:16.698 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.699 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.700 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:38:16.704 INFO 2981023 --- [host-check-host-status1,592e0e126204431c,01a62fc73de3c8a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:38:16.709 INFO 2981023 --- [host-check-host-status4,592e0e126204431c,e59dfad0b1568b6a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:38:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:38:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:38:16.712 INFO 2981023 --- [host-check-host-status2,592e0e126204431c,f4beb41a484a36a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:38:16.722 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:38:16.729 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.730 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.730 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:38:16.734 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:38:16.740 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:38:16.740 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:38:16.742 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:16.743 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:16.744 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:38:16.750 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:38:16.750 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:38:16.751 INFO 2981023 --- [ocp-schedules-25,9f6a59a5dc1d14d6,4350255ebb89068f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:38:16.777 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:38:16.400102+08:00 2024-09-11 17:38:16.969 INFO 2981023 --- [host-check-clockdiff1,dce4ed3f1e640e32,690c0ec04b1c4d65] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047496 0 0 , error= 2024-09-11 17:38:17.241 INFO 2981023 --- [ocp-async-10,82f63ab3f58ed89a,68d0cc62b80ca2fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:17.269 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:17.269 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,70696cfccde8963f,dd62d495690dd67d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,70696cfccde8963f,dd62d495690dd67d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:17.755 INFO 2981023 --- [ocp-async-8,7f77c59bf92838f1,f18dee49e06e0c0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:38:17.807 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047420, nodeCount=480, validCount=480 2024-09-11 17:38:19.241 INFO 2981023 --- [ocp-async-6,796d5e9f6b0eb688,7cc94c53cb272e27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:20.269 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:20.270 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,22999049c60d2ac5,6cfb80f7fd9631e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,22999049c60d2ac5,6cfb80f7fd9631e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:20.532 INFO 2981023 --- [host-check-clockdiff3,dce4ed3f1e640e32,47a5d67ddf542189] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047500 1 1 , error= 2024-09-11 17:38:21.241 INFO 2981023 --- [ocp-async-9,703cb1a303d0a8d7,b49423085d7a69f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:22.002 INFO 2981023 --- [ocp-schedules-3,2f38cb85f0eb5358,029fd395130c7ce2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:38:22.059 INFO 2981023 --- [ocp-schedules-12,bf739e84172271ed,2752574679b2099f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 17:38:22.059 INFO 2981023 --- [ocp-schedules-12,bf739e84172271ed,2752574679b2099f] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 17:38:22.072 INFO 2981023 --- [pool-547-thread-2,bf739e84172271ed,cffe7ef05698aa5a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:38:22.073 INFO 2981023 --- [pool-547-thread-1,bf739e84172271ed,bad08d90a3742cee] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:38:22.073 INFO 2981023 --- [pool-547-thread-3,bf739e84172271ed,a277f36fe63a2277] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:38:22.091 INFO 2981023 --- [pool-547-thread-1,bf739e84172271ed,bad08d90a3742cee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:22.092 INFO 2981023 --- [pool-547-thread-3,bf739e84172271ed,a277f36fe63a2277] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:22.093 INFO 2981023 --- [pool-547-thread-2,bf739e84172271ed,cffe7ef05698aa5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:22.094 INFO 2981023 --- [pool-547-thread-1,bf739e84172271ed,bad08d90a3742cee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046601000000, 1726047502000000] 2024-09-11 17:38:22.095 INFO 2981023 --- [pool-547-thread-3,bf739e84172271ed,a277f36fe63a2277] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046601000000, 1726047502000000] 2024-09-11 17:38:22.095 INFO 2981023 --- [pool-547-thread-2,bf739e84172271ed,cffe7ef05698aa5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726046601000000, 1726047502000000] 2024-09-11 17:38:22.099 INFO 2981023 --- [pool-547-thread-3,bf739e84172271ed,a277f36fe63a2277] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:38:22.177 INFO 2981023 --- [pool-547-thread-1,bf739e84172271ed,bad08d90a3742cee] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:38:22.247 INFO 2981023 --- [pool-547-thread-2,bf739e84172271ed,cffe7ef05698aa5a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:38:22.247 INFO 2981023 --- [ocp-schedules-12,bf739e84172271ed,2752574679b2099f] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 17:38:22.848 INFO 2981023 --- [ocp-schedules-15,6f3369815feb2a87,f8f4dfcc53d0a12f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:38:22.848 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:38:22.848 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:38:22.848 INFO 2981023 --- [ocp-schedules-15,6f3369815feb2a87,f8f4dfcc53d0a12f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:38:22.848 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:38:22.852 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:38:22.853 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:38:22.854 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:38:22.854 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:38:22.855 INFO 2981023 --- [ocp-schedules-15,6f3369815feb2a87,f8f4dfcc53d0a12f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:38:22.855 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:38:22.856 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:38:22.856 INFO 2981023 --- [ocp-schedules-15,6f3369815feb2a87,f8f4dfcc53d0a12f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:38:22.875 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:38:22.878 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:38:22.879 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:38:22.880 INFO 2981023 --- [ocp-schedules-15,6f3369815feb2a87,f8f4dfcc53d0a12f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:38:22.883 INFO 2981023 --- [ocp-schedules-15,6f3369815feb2a87,f8f4dfcc53d0a12f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:38:23.241 INFO 2981023 --- [ocp-async-2,14075c862062bda3,14a25c2ed331a813] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:23.270 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:23.271 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,0fbf1b1d4e5ed133,38576ee603460c18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,0fbf1b1d4e5ed133,38576ee603460c18] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:24.237 INFO 2981023 --- [ocp-schedules-24,5f8eb3ca30663f72,149f5694a330a714] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:38:25.241 INFO 2981023 --- [ocp-async-4,f40da6040c778ec0,acd0c4f7b299f4c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:25.667 INFO 2981023 --- [ocp-schedules-6,d5d65e8b00186e49,dfaba78b5746aaa3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:38:25.682 WARN 2981023 --- [scanner-13520163-1,d5d65e8b00186e49,08f3edd6d2dc219c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:38:25.731 INFO 2981023 --- [alarm-detect-13,d5d65e8b00186e49,bb5ec0429d6f2ac9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644310, data is empty. 2024-09-11 17:38:25.735 INFO 2981023 --- [alarm-detect-13,d5d65e8b00186e49,bb5ec0429d6f2ac9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644310, data is empty. 2024-09-11 17:38:25.739 INFO 2981023 --- [alarm-detect-13,d5d65e8b00186e49,bb5ec0429d6f2ac9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644310, data is empty. 2024-09-11 17:38:25.760 INFO 2981023 --- [alarm-detect-13,d5d65e8b00186e49,c276916148d0f3ba] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644310, data is empty. 2024-09-11 17:38:25.767 INFO 2981023 --- [alarm-detect-17,d5d65e8b00186e49,db255cd981d8b7b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644310, data is empty. 2024-09-11 17:38:25.768 INFO 2981023 --- [alarm-detect-13,d5d65e8b00186e49,c276916148d0f3ba] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644310, data is empty. 2024-09-11 17:38:25.774 INFO 2981023 --- [alarm-detect-17,d5d65e8b00186e49,db255cd981d8b7b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644310, data is empty. 2024-09-11 17:38:25.774 INFO 2981023 --- [alarm-detect-13,d5d65e8b00186e49,c276916148d0f3ba] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644310, data is empty. 2024-09-11 17:38:25.777 INFO 2981023 --- [alarm-detect-17,d5d65e8b00186e49,db255cd981d8b7b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644310, data is empty. 2024-09-11 17:38:25.881 INFO 2981023 --- [alarm-detect-1,d5d65e8b00186e49,60c3539657e3fa1b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644310, data is empty. 2024-09-11 17:38:25.896 INFO 2981023 --- [alarm-detect-1,d5d65e8b00186e49,60c3539657e3fa1b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644310, data is empty. 2024-09-11 17:38:25.900 INFO 2981023 --- [alarm-detect-1,d5d65e8b00186e49,60c3539657e3fa1b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644310, data is empty. 2024-09-11 17:38:26.271 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:26.271 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,e4ab810301438dff,b13b75e4d322b33b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,e4ab810301438dff,b13b75e4d322b33b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:38:26.336970+08:00, lastProcessTime=2024-09-11T17:38:16.400102+08:00 2024-09-11 17:38:26.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:38:26.336970+08:00 2024-09-11 17:38:26.917 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:26.920 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterI 2024-09-11 17:38:26.925 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:26.925 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:38:26.928 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:26.932 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:26.941 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:26.946 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:26.950 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 26 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:26.982 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 6 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:26.990 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 161 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.022 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.024 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.025 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.026 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.027 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.027 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.028 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.028 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:38:27.029 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.029 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.030 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.031 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.031 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.032 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.033 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.033 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.037 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.037 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:38:27.039 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.043 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 4 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.061 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:32:26.7, end=2024-09-11T17:37:26.914} 2024-09-11 17:38:27.061 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 6 dubious sqls 2024-09-11 17:38:27.241 INFO 2981023 --- [ocp-async-8,b266653ef7382db3,c947f43ba36f279d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:27.755 INFO 2981023 --- [ocp-async-4,6115851e17ede939,998b71ff69f1145a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:38:29.241 INFO 2981023 --- [ocp-async-10,a7ea7f35ed641d41,d624939b064ffbaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:29.271 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:29.272 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,fda6bc701afc1bd2,690fa6bf020c3059] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,fda6bc701afc1bd2,690fa6bf020c3059] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:31.241 INFO 2981023 --- [ocp-async-4,2087107c2466cb35,f8bf33dd3cbd03f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:32.004 INFO 2981023 --- [ocp-schedules-16,2606dffb139d33a4,ba4646b8933ae3eb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:38:32.272 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:32.273 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,f69034523962164a,8e1bb5fe3296e3e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,f69034523962164a,8e1bb5fe3296e3e2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:33.242 INFO 2981023 --- [ocp-async-8,8f4af7c9a94e1280,b823eef93c9dec23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:34.028 INFO 2981023 --- [ocp-schedules-29,d764beee0fd9b9e9,c9f0387d810683b5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:38:34.033 INFO 2981023 --- [ocp-schedules-29,d764beee0fd9b9e9,c9f0387d810683b5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:38:34.033 INFO 2981023 --- [ocp-schedules-29,d764beee0fd9b9e9,c9f0387d810683b5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:38:35.242 INFO 2981023 --- [ocp-async-5,a8afff89fdbe4613,85cbdd69544796eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:35.273 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:35.273 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,451e3dd0a2bbcdbc,da143e62a363fd38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,451e3dd0a2bbcdbc,da143e62a363fd38] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:35.940 INFO 2981023 --- [ocp-schedules-29,f64332175f4136fd,ca98ee33e89d9de1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:38:35.991 WARN 2981023 --- [scanner-13520486-1,f64332175f4136fd,ef86813610d99b2f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:38:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:38:36.337393+08:00, lastProcessTime=2024-09-11T17:38:26.336970+08:00 2024-09-11 17:38:36.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:38:36.337393+08:00 2024-09-11 17:38:37.242 INFO 2981023 --- [ocp-async-7,543bbdad30378d40,129d33f104d9338f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:37.314 INFO 2981023 --- [prometheus-cache-monitor1,0cafbd45e6d8d6c6,48d7372ccab18d47] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:38:37.315 INFO 2981023 --- [prometheus-cache-monitor1,0cafbd45e6d8d6c6,48d7372ccab18d47] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:38:37.755 INFO 2981023 --- [ocp-async-5,a09ded53d4e6818f,6cc77958f5192ab0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:38:38.273 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:38.274 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,dd0bf78d6df8453a,377a22f99180db99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,dd0bf78d6df8453a,377a22f99180db99] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:39.242 INFO 2981023 --- [ocp-async-3,de965a6c92c48830,46f8a53a3fb736f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:39.876 INFO 2981023 --- [ocp-schedules-7,cbab6fb410a84dd4,7a434273a758de29] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:38:39.895 WARN 2981023 --- [ocp-schedules-7,cbab6fb410a84dd4,7a434273a758de29] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:38:40.001 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:38:40.002 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:38:41.242 INFO 2981023 --- [ocp-async-2,8d06034bbb19b741,c878609cdaf381c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:41.274 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:41.274 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,ebc9d95ad6111ce2,33192015d8f689fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,ebc9d95ad6111ce2,33192015d8f689fe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:42.007 INFO 2981023 --- [ocp-schedules-1,69f1d13890b95230,142d2816eb082b4c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:38:43.242 INFO 2981023 --- [ocp-async-1,30ac03158084aa65,64a62be9b1d1bde1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:44.274 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:44.275 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,61266b57d355eb99,15f1cc155f1ebe87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,61266b57d355eb99,15f1cc155f1ebe87] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:45.101 INFO 2981023 --- [prometheus-second-schedule-1,de2a7c7bda67773c,69ee51699d06b1bc] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:38:45.242 INFO 2981023 --- [ocp-async-6,71fc1e0beb8ceb51,381d17a9e0a4a80b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:45.997 INFO 2981023 --- [ocp-schedules-18,e103ff2acedf5822,38daf7647e6764f4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:38:45.998 INFO 2981023 --- [host-check-clockdiff1,e103ff2acedf5822,702c2e75b84cadfc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:38:45.998 INFO 2981023 --- [host-check-clockdiff3,e103ff2acedf5822,84aed8ec8e24df53] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:38:45.998 INFO 2981023 --- [host-check-clockdiff4,e103ff2acedf5822,e526de951e738d51] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:38:45.998 INFO 2981023 --- [host-check-clockdiff2,e103ff2acedf5822,8ef91f67d69229a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:38:46.004 INFO 2981023 --- [ocp-schedules-21,ce3e195bf1d21b80,f82f0aca8f65e483] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:38:46.004 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:38:46.005 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:38:46.006 INFO 2981023 --- [ocp-schedules-22,9a04c26bc591ffb8,e8173982361c7273] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:38:46.006 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:38:46.007 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:38:46.009 INFO 2981023 --- [host-check-clockdiff4,e103ff2acedf5822,e526de951e738d51] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047526 0 0 , error= 2024-09-11 17:38:46.009 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:38:46.011 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:38:46.011 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:38:46.012 INFO 2981023 --- [host-check-agent-status2,9a04c26bc591ffb8,dc4bbaee12f46f18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:38:46.013 INFO 2981023 --- [host-check-agent-status3,9a04c26bc591ffb8,8cc97bcdbed2d91e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:38:46.013 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:38:46.013 INFO 2981023 --- [host-check-agent-status1,9a04c26bc591ffb8,be28a72e9b1a7525] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:38:46.017 INFO 2981023 --- [host-check-clockdiff2,e103ff2acedf5822,8ef91f67d69229a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047526 0 0 , error= 2024-09-11 17:38:46.017 INFO 2981023 --- [host-check-agent-status4,9a04c26bc591ffb8,1e440d160c1075b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:38:46.018 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:38:46.020 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:38:46.021 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:38:46.022 INFO 2981023 --- [host-check-agent-status2,9a04c26bc591ffb8,dc4bbaee12f46f18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:38:46.023 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:38:46.024 INFO 2981023 --- [host-check-agent-status1,9a04c26bc591ffb8,be28a72e9b1a7525] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:38:46.025 INFO 2981023 --- [host-check-agent-status3,9a04c26bc591ffb8,8cc97bcdbed2d91e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:38:46.025 INFO 2981023 --- [host-check-agent-status1,9a04c26bc591ffb8,be28a72e9b1a7525] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:38:46.027 INFO 2981023 --- [host-check-agent-status3,9a04c26bc591ffb8,8cc97bcdbed2d91e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:38:46.029 INFO 2981023 --- [host-check-agent-status2,9a04c26bc591ffb8,dc4bbaee12f46f18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:38:46.041 INFO 2981023 --- [host-check-agent-status4,9a04c26bc591ffb8,1e440d160c1075b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:38:46.042 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:38:46.043 INFO 2981023 --- [host-check-agent-status4,9a04c26bc591ffb8,1e440d160c1075b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:38:46.044 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:38:46.044198851+08:00 2024-09-11 17:38:46.045 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:38:46.044+08:00 2024-09-11 17:38:46.047 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:38:46.044+08:00 and status AVAILABLE 2024-09-11 17:38:46.049 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:38:46.052 INFO 2981023 --- [host-check-agent-status1,9a04c26bc591ffb8,be28a72e9b1a7525] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:38:46.052 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:38:46.053 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:38:46.052982266+08:00 2024-09-11 17:38:46.054 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:38:46.053+08:00 2024-09-11 17:38:46.054 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:38:46.054374714+08:00 2024-09-11 17:38:46.055 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:38:46.054+08:00 2024-09-11 17:38:46.055 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:38:46.056 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:38:46.053+08:00 and status AVAILABLE 2024-09-11 17:38:46.056 INFO 2981023 --- [host-check-agent-status1,9a04c26bc591ffb8,be28a72e9b1a7525] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:38:46.056 INFO 2981023 --- [host-check-agent-status1,9a04c26bc591ffb8,be28a72e9b1a7525] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:38:46.056 INFO 2981023 --- [host-check-agent-status1,9a04c26bc591ffb8,be28a72e9b1a7525] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:38:46.058 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:38:46.054+08:00 and status AVAILABLE 2024-09-11 17:38:46.060 INFO 2981023 --- [host-check-agent-status3,9a04c26bc591ffb8,8cc97bcdbed2d91e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:38:46.060 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:38:46.061 INFO 2981023 --- [host-check-agent-status2,9a04c26bc591ffb8,dc4bbaee12f46f18] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:38:46.062 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:38:46.062571050+08:00 2024-09-11 17:38:46.063 INFO 2981023 --- [host-check-agent-status3,9a04c26bc591ffb8,8cc97bcdbed2d91e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:38:46.063 INFO 2981023 --- [host-check-agent-status3,9a04c26bc591ffb8,8cc97bcdbed2d91e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:38:46.063 INFO 2981023 --- [host-check-agent-status3,9a04c26bc591ffb8,8cc97bcdbed2d91e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:38:46.063 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:38:46.062+08:00 2024-09-11 17:38:46.063 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:38:46.064 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:38:46.064 INFO 2981023 --- [host-check-agent-status2,9a04c26bc591ffb8,dc4bbaee12f46f18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:38:46.064 INFO 2981023 --- [host-check-agent-status2,9a04c26bc591ffb8,dc4bbaee12f46f18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:38:46.064 INFO 2981023 --- [host-check-agent-status2,9a04c26bc591ffb8,dc4bbaee12f46f18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:38:46.065 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:38:46.065 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:38:46.062+08:00 and status AVAILABLE 2024-09-11 17:38:46.069 INFO 2981023 --- [host-check-agent-status4,9a04c26bc591ffb8,1e440d160c1075b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:38:46.070 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:38:46.071 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:38:46.075 INFO 2981023 --- [host-check-agent-status4,9a04c26bc591ffb8,1e440d160c1075b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:38:46.075 INFO 2981023 --- [host-check-agent-status4,9a04c26bc591ffb8,1e440d160c1075b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:38:46.075 INFO 2981023 --- [host-check-agent-status4,9a04c26bc591ffb8,1e440d160c1075b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:38:46.075 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:38:46.082 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:38:46.083 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:38:46.085 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:38:46.086 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:38:46.096 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:38:46.097 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:38:46.102 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:38:46.102 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:38:46.112 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:38:46.121 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:38:46.124 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:38:46.126 INFO 2981023 --- [ocp-schedules-28,6e8bfaab932fce24,4c24e631ddac95be] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:38:46.127 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:38:46.142 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:38:46.143 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:38:46.145 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:38:46.157 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:38:46.158 INFO 2981023 --- [ocp-schedules-5,9b36e3e18877f2ad,efba6679fa7c7493] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:38:46.164 INFO 2981023 --- [ocp-schedules-5,9b36e3e18877f2ad,efba6679fa7c7493] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:38:46.172 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:38:46.175 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:38:46.187 INFO 2981023 --- [ocp-schedules-2,f9ad2f3e1a9f5be5,1c562b8c59ff9487] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:38:46.187 INFO 2981023 --- [ocp-schedules-9,779bf5648faef892,9ceb90494d7f7be4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:38:46.188 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:38:46.190 INFO 2981023 --- [ocp-schedules-9,779bf5648faef892,9ceb90494d7f7be4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:38:46.190 INFO 2981023 --- [ocp-schedules-2,f9ad2f3e1a9f5be5,1c562b8c59ff9487] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:38:46.190 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:38:46.192 INFO 2981023 --- [ocp-schedules-14,65739e2f640779f3,512b466e5eb0cb85] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:38:46.192 INFO 2981023 --- [ocp-schedules-19,d3012815e92eeecb,30ef93bd81f6ae82] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:38:46.195 INFO 2981023 --- [ocp-schedules-14,65739e2f640779f3,512b466e5eb0cb85] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:38:46.201 INFO 2981023 --- [ocp-schedules-7,a0a969b9f0aa268a,9bced5b1e6d8ff2e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:38:46.205 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.205 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:38:46.205 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:38:46.206 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:38:46.206 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.209 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:38:46.209 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:38:46.216 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:38:46.220 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:38:46.220 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:38:46.224 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:38:46.229 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:38:46.237 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:38:46.242 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:38:46.258 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:38:46.269 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:38:46.270 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:38:46.284 INFO 2981023 --- [host-check-host-status3,ce3e195bf1d21b80,8965b2935d7bb118] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:38:46.290 INFO 2981023 --- [host-check-host-status1,ce3e195bf1d21b80,88cae6ce615695a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:38:46.292 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:38:46.295 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.301 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:38:46.314 INFO 2981023 --- [host-check-host-status2,ce3e195bf1d21b80,1174c9659e0d0737] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:38:46.315 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.316 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:38:46.316 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:38:46.317 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.328 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:38:46.339 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:38:46.339033+08:00, lastProcessTime=2024-09-11T17:38:36.337393+08:00 2024-09-11 17:38:46.340 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:38:46.357 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:38:46.380 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.380 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:38:46.398 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.399 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:38:46.399 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.400 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:38:46.400 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:38:46.401 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.401 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:38:46.402 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:38:46.403 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.409 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:38:46.409 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.410 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:46.411 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:38:46.421 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.422 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:46.424 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:38:46.428 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.428 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:46.433 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.433 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:46.434 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.434 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:46.438 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:38:46.453 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:38:46.468 INFO 2981023 --- [ocp-schedules-9,779bf5648faef892,9ceb90494d7f7be4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:38:46.475 WARN 2981023 --- [scanner-13520749-1,6e8bfaab932fce24,0d9030da1dd2ca5f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:38:46.484 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.485 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.485 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:38:46.486 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:38:46.486 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.495 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:38:46.505 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:38:46.507 INFO 2981023 --- [host-check-host-status4,ce3e195bf1d21b80,ff485198fcd55206] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:38:46.510 INFO 2981023 --- [alarm-detect-11,6e8bfaab932fce24,ac5c09a722e38fa2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644310, data is empty. 2024-09-11 17:38:46.514 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.515 INFO 2981023 --- [alarm-detect-11,6e8bfaab932fce24,ac5c09a722e38fa2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644310, data is empty. 2024-09-11 17:38:46.516 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.516 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:38:46.517 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.519 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:38:46.521 INFO 2981023 --- [alarm-detect-11,6e8bfaab932fce24,ac5c09a722e38fa2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644310, data is empty. 2024-09-11 17:38:46.523 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.525 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:46.526 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:38:46.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.540 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:38:46.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.543 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:38:46.558 INFO 2981023 --- [alarm-detect-20,6e8bfaab932fce24,710a43470cdd4701] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644310, data is empty. 2024-09-11 17:38:46.561 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.561 INFO 2981023 --- [alarm-detect-2,6e8bfaab932fce24,62a87bb58e979db4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644310, data is empty. 2024-09-11 17:38:46.563 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:46.567 INFO 2981023 --- [alarm-detect-20,6e8bfaab932fce24,710a43470cdd4701] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644310, data is empty. 2024-09-11 17:38:46.568 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:38:46.568 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.571 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:38:46.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:38:46.574 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.575 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.575 INFO 2981023 --- [alarm-detect-2,6e8bfaab932fce24,62a87bb58e979db4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644310, data is empty. 2024-09-11 17:38:46.576 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:46.577 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.578 INFO 2981023 --- [alarm-detect-20,6e8bfaab932fce24,710a43470cdd4701] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644310, data is empty. 2024-09-11 17:38:46.580 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:38:46.581 INFO 2981023 --- [alarm-detect-2,6e8bfaab932fce24,62a87bb58e979db4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644310, data is empty. 2024-09-11 17:38:46.581 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:38:46.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:46.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:38:46.590 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.591 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.591 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.591 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.591 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:38:46.592 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:38:46.592 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:38:46.592 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.593 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:38:46.594 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.595 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:38:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:38:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:38:46.605 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:46.605 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:38:46.612 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:46.617 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:46.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.620 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.620 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:46.620 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:38:46.621 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:38:46.625 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:38:46.628 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.629 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:38:46.630 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:38:46.631 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.631 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.631 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:46.632 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.632 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:38:46.632 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:38:46.633 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:38:46.646 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:38:46.650 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:38:46.658 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:38:46.658 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:38:46.663 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:38:46.665 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:38:46.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:38:46.667 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:38:46.676 INFO 2981023 --- [ocp-schedules-2,f9ad2f3e1a9f5be5,1c562b8c59ff9487] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:38:46.685 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:38:46.696 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:38:46.705 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:38:46.717 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:38:46.748 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:38:46.750 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:38:46.758 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:38:46.762 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.762 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.762 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.763 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.763 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:38:46.763 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:38:46.770 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:46.771 INFO 2981023 --- [ocp-schedules-24,8049721054b502cc,adcc21afe802ecee] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:38:46.772 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:46.772 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:38:46.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:38:46.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:38:46.780 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:38:46.780 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:38:46.811 INFO 2981023 --- [alarm-detect-3,6e8bfaab932fce24,853d0802c97f5c7c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644310, data is empty. 2024-09-11 17:38:46.814 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:38:46.814 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:38:46.818 INFO 2981023 --- [alarm-detect-3,6e8bfaab932fce24,853d0802c97f5c7c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644310, data is empty. 2024-09-11 17:38:46.824 INFO 2981023 --- [alarm-detect-3,6e8bfaab932fce24,853d0802c97f5c7c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644310, data is empty. 2024-09-11 17:38:46.829 INFO 2981023 --- [ocp-schedules-14,65739e2f640779f3,512b466e5eb0cb85] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:38:46.835 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:38:46.339033+08:00 2024-09-11 17:38:47.242 INFO 2981023 --- [ocp-async-2,054ebdd917b06e01,5cdfdbd7c5be5c6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:47.275 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:47.275 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,a0ba95f3617882b7,85b8074cdd44f5db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,a0ba95f3617882b7,85b8074cdd44f5db] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:47.755 INFO 2981023 --- [ocp-async-3,7d8c9216c407830e,d2643b04fb606d23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:38:48.262 INFO 2981023 --- [ocp-schedules-30,b1c79c4528e05a30,90adb5cb8cdf4e41] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:38:49.181 INFO 2981023 --- [host-check-clockdiff1,e103ff2acedf5822,702c2e75b84cadfc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047529 0 0 , error= 2024-09-11 17:38:49.242 INFO 2981023 --- [ocp-async-1,e33dd63463919d30,85f770d4994b9bcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:38:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:38:50.224 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.229 INFO 2981023 --- [ocp-schedules-9,b9b8d6793b63326a,e6a2ea2da74a8cf8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:38:50.231 INFO 2981023 --- [ocp-schedules-27,9c86b0f14497c006,2b703ab529a8c7b7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:38:50.276 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:50.276 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,db181e1896d66054,b174681d4fac6490] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,db181e1896d66054,b174681d4fac6490] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:51.242 INFO 2981023 --- [ocp-async-9,e8c3c32d1e19593a,ebd3ecb141a76c01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:51.546 INFO 2981023 --- [ocp-schedules-23,da9780977c0b69e6,4f4f7638319692cc] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:38:51.547 INFO 2981023 --- [ocp-schedules-23,da9780977c0b69e6,4f4f7638319692cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:38:51.547 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:38:51.547 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:38:51.547 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:38:51.548 INFO 2981023 --- [ocp-schedules-23,da9780977c0b69e6,4f4f7638319692cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:38:51.969 INFO 2981023 --- [host-check-clockdiff3,e103ff2acedf5822,84aed8ec8e24df53] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047531 1 1 , error= 2024-09-11 17:38:52.009 INFO 2981023 --- [ocp-schedules-16,95edb39de7accee2,eca509734e4e7bc1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:38:52.822 INFO 2981023 --- [ocp-schedules-22,83fd4972e3e3928d,5adbe420fc61e514] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:38:52.822+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:38:52.826 INFO 2981023 --- [ocp-schedules-22,83fd4972e3e3928d,5adbe420fc61e514] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:38:52.861 INFO 2981023 --- [ocp-schedules-24,e81cd90624d372d2,b386840349cb6f00] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:38:52.881 WARN 2981023 --- [ocp-schedules-24,e81cd90624d372d2,b386840349cb6f00] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:38:52.937 INFO 2981023 --- [ocp-schedules-22,83fd4972e3e3928d,5adbe420fc61e514] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 43 to SqlAggregateData 2024-09-11 17:38:52.939 INFO 2981023 --- [ocp-schedules-22,83fd4972e3e3928d,5adbe420fc61e514] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:38:53.006 WARN 2981023 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=10.31.35.10:8080, expireInSeconds=30, createTime=2024-09-11T16:01:52+08:00, updateTime=2024-09-11T17:38:22+08:00, expireTime=null, isLeader=false} 2024-09-11 17:38:53.008 WARN 2981023 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2024-09-11 17:38:53.102 INFO 2981023 --- [ocp-schedules-9,ee5f65ccfbd7124c,1cbd50511d4172c7] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:38:53.102 INFO 2981023 --- [ocp-schedules-9,ee5f65ccfbd7124c,1cbd50511d4172c7] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:38:53.243 INFO 2981023 --- [ocp-async-5,f74d8cbd0742c345,17099c02ac844090] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:53.276 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:53.277 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,7c46e05ec80fd7d4,98ea873d671ee679] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,7c46e05ec80fd7d4,98ea873d671ee679] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:53.684 INFO 2981023 --- [ocp-schedules-19,83f13df09c7fa7ee,1434b661d1fe9493] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:38:53.685 INFO 2981023 --- [ocp-schedules-19,83f13df09c7fa7ee,1434b661d1fe9493] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:38:53.709 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:38:53.710 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.710 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:38:53.711 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.712 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:38:53.712 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.713 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:38:53.715 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:38:53.716 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:38:53.716 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.717 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.719 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.719 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.720 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.721 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.724 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.724 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.729 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:38:53.730 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.732 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.732 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.736 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.736 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:38:53.741 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.741 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.743 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.744 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:38:53.745 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.748 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.748 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.748 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:38:53.748 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2BB1D6DA-0-0] 2024-09-11 17:38:53.749 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:38:53.749 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:38:53.749 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.749 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.749 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:38:53.749 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.750 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:38:53.750 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:38:53.750 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.751 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:38:53.751 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:38:53.755 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:38:53.762 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.762 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.762 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:38:53.763 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:38:53.763 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.764 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:38:53.764 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.764 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:38:53.765 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.765 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:38:53.766 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.766 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:53.767 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:38:53.767 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.768 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.768 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:38:53.768 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.768 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.769 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:38:53.769 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.769 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:38:53.770 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.770 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:38:53.771 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:38:53.772 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:38:53.774 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.775 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:38:53.775 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:38:53.775 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2BB1D6E4-0-0] 2024-09-11 17:38:53.775 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:38:53.782 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.782 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:38:53.783 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.783 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:38:53.784 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:53.784 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.784 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:38:53.785 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.785 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:38:53.785 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.785 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:38:53.786 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:38:53.786 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:38:53.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.788 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:38:53.788 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:38:53.788 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:38:53.788 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:38:53.788 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:38:53.789 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:38:53.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:38:53.792 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:38:53.792 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.793 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:38:53.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:38:53.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:38:53.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.799 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:38:53.800 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:38:53.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:38:53.805 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:38:53.805 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:38:53.805 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:38:53.809 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:38:53.810 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:38:53.810 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:53.811 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:38:53.818 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:38:53.819 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:38:53.819 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:38:53.819 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:38:53.820 INFO 2981023 --- [ocp-schedules-19,83f13df09c7fa7ee,1434b661d1fe9493] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:38:54.035 INFO 2981023 --- [ocp-schedules-11,46d0d2ddef250574,9b45dcb97ffd2d9c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:38:54.040 INFO 2981023 --- [ocp-schedules-11,46d0d2ddef250574,9b45dcb97ffd2d9c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:38:54.040 INFO 2981023 --- [ocp-schedules-11,46d0d2ddef250574,9b45dcb97ffd2d9c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:38:54.239 INFO 2981023 --- [ocp-schedules-2,ca19194513f5196c,e3e0390a321d2399] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:38:54.586 INFO 2981023 --- [ocp-schedules-24,bd1b43df14c3c3b6,6ff4968bc12337be] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T17:38:54.586+08:00, tenantSize=6 2024-09-11 17:38:54.588 INFO 2981023 --- [ocp-schedules-24,bd1b43df14c3c3b6,6ff4968bc12337be] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726047534586000 2024-09-11 17:38:54.588 INFO 2981023 --- [ocp-schedules-24,bd1b43df14c3c3b6,6ff4968bc12337be] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726047234559000 2024-09-11 17:38:54.602 INFO 2981023 --- [ocp-schedules-24,bd1b43df14c3c3b6,6ff4968bc12337be] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 17:38:55.243 INFO 2981023 --- [ocp-async-7,1f7a4a9b16339157,a0871992a3c6ecc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:55.688 INFO 2981023 --- [ocp-schedules-6,fdc80f0433fab78b,6d809edacd14decc] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:38:55.688 INFO 2981023 --- [ocp-schedules-6,fdc80f0433fab78b,6d809edacd14decc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:38:55.689 INFO 2981023 --- [ocp-schedules-6,fdc80f0433fab78b,6d809edacd14decc] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:38:55.689 INFO 2981023 --- [ocp-schedules-6,fdc80f0433fab78b,6d809edacd14decc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:38:55.689 INFO 2981023 --- [ocp-schedules-6,fdc80f0433fab78b,6d809edacd14decc] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:38:55.689 INFO 2981023 --- [ocp-schedules-6,fdc80f0433fab78b,6d809edacd14decc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:38:55.784 INFO 2981023 --- [ocp-schedules-2,1dd73965136af626,e91901aca6decf61] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:38:55.790 INFO 2981023 --- [ocp-schedules-2,1dd73965136af626,e91901aca6decf61] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:38:55.790 INFO 2981023 --- [ocp-schedules-2,1dd73965136af626,e91901aca6decf61] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:38:56.277 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:56.277 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:56.279 INFO 2981023 --- [ocp-schedules-13,24813e2617dfdc59,9635f385b2fc4fb3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:38:56.279 INFO 2981023 --- [ocp-schedules-13,24813e2617dfdc59,9635f385b2fc4fb3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:38:56.292 INFO 2981023 --- [Collect all tenants major compactions1,24813e2617dfdc59,284d2ee9a8f56648] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:38:56.293 INFO 2981023 --- [Collect all tenants major compactions3,24813e2617dfdc59,935d94acafb0b2bf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:38:56.293 INFO 2981023 --- [Collect all tenants major compactions6,24813e2617dfdc59,0567b567e567ac90] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:38:56.296 INFO 2981023 --- [Collect all tenants major compactions2,24813e2617dfdc59,85f0ea919e958018] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:38:56.297 INFO 2981023 --- [Collect all tenants major compactions5,24813e2617dfdc59,ec9c361959eeba1f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:38:56.297 INFO 2981023 --- [Collect all tenants major compactions4,24813e2617dfdc59,244556c63a0d7173] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:38:56.304 INFO 2981023 --- [Collect all tenants major compactions3,24813e2617dfdc59,935d94acafb0b2bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:56.305 INFO 2981023 --- [Collect all tenants major compactions1,24813e2617dfdc59,284d2ee9a8f56648] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:56.305 INFO 2981023 --- [Collect all tenants major compactions6,24813e2617dfdc59,0567b567e567ac90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:56.306 INFO 2981023 --- [Collect all tenants major compactions2,24813e2617dfdc59,85f0ea919e958018] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:56.306 INFO 2981023 --- [Collect all tenants major compactions3,24813e2617dfdc59,935d94acafb0b2bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:38:56.306 INFO 2981023 --- [Collect all tenants major compactions6,24813e2617dfdc59,0567b567e567ac90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:38:56.307 INFO 2981023 --- [Collect all tenants major compactions1,24813e2617dfdc59,284d2ee9a8f56648] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:38:56.307 INFO 2981023 --- [Collect all tenants major compactions2,24813e2617dfdc59,85f0ea919e958018] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:38:56.308 INFO 2981023 --- [Collect all tenants major compactions5,24813e2617dfdc59,ec9c361959eeba1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:56.310 INFO 2981023 --- [Collect all tenants major compactions5,24813e2617dfdc59,ec9c361959eeba1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:38:56.311 INFO 2981023 --- [Collect all tenants major compactions2,24813e2617dfdc59,85f0ea919e958018] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:38:56.311 INFO 2981023 --- [Collect all tenants major compactions3,24813e2617dfdc59,935d94acafb0b2bf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:38:56.311 INFO 2981023 --- [Collect all tenants major compactions2,24813e2617dfdc59,85f0ea919e958018] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:38:56.311 INFO 2981023 --- [Collect all tenants major compactions3,24813e2617dfdc59,935d94acafb0b2bf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:38:56.312 INFO 2981023 --- [Collect all tenants major compactions6,24813e2617dfdc59,0567b567e567ac90] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:38:56.312 INFO 2981023 --- [Collect all tenants major compactions6,24813e2617dfdc59,0567b567e567ac90] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:38:56.312 INFO 2981023 --- [Collect all tenants major compactions4,24813e2617dfdc59,244556c63a0d7173] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:38:56.312 INFO 2981023 --- [Collect all tenants major compactions1,24813e2617dfdc59,284d2ee9a8f56648] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:38:56.312 INFO 2981023 --- [Collect all tenants major compactions1,24813e2617dfdc59,284d2ee9a8f56648] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:38:56.313 INFO 2981023 --- [Collect all tenants major compactions4,24813e2617dfdc59,244556c63a0d7173] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:38:56.313 INFO 2981023 --- [Collect all tenants major compactions5,24813e2617dfdc59,ec9c361959eeba1f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:38:56.313 INFO 2981023 --- [Collect all tenants major compactions5,24813e2617dfdc59,ec9c361959eeba1f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:38:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,cfd3f04eb2c36762,b7672f33364e08b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:56.315 INFO 2981023 --- [Collect all tenants major compactions4,24813e2617dfdc59,244556c63a0d7173] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:38:56.315 INFO 2981023 --- [Collect all tenants major compactions4,24813e2617dfdc59,244556c63a0d7173] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:38:56.315 INFO 2981023 --- [ocp-schedules-13,24813e2617dfdc59,9635f385b2fc4fb3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:38:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,cfd3f04eb2c36762,b7672f33364e08b2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:38:56.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:38:56.337590+08:00, lastProcessTime=2024-09-11T17:38:46.339033+08:00 2024-09-11 17:38:56.470 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:38:56.337590+08:00 2024-09-11 17:38:56.859 INFO 2981023 --- [ocp-schedules-29,8872a9c9aa049d75,969a544ba18fde64] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:38:56.905 WARN 2981023 --- [scanner-13521043-1,8872a9c9aa049d75,2487da68052bd44e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:38:57.243 INFO 2981023 --- [ocp-async-6,06d1d8531d2a0ced,9a078c64db503a6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:57.755 INFO 2981023 --- [ocp-async-4,5ee5215d8621d9d1,83525c0d4981ce58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:38:59.243 INFO 2981023 --- [ocp-async-2,5023ff0ca8a0f5b0,c9277916725a5ff3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:38:59.277 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:38:59.279 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:38:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,e9ead101a8c7b6cd,2db0d295beaad270] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047540, interval=60 2024-09-11 17:38:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,e9ead101a8c7b6cd,2db0d295beaad270] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:01.244 INFO 2981023 --- [ocp-async-1,12508945129a0494,ac6b762d122b98ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:02.012 INFO 2981023 --- [ocp-schedules-25,b56548b404333123,267048bbcb852f59] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:39:02.279 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:02.279 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,14a8a1ed87fecf6c,99c9eeabadd31bb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,14a8a1ed87fecf6c,99c9eeabadd31bb7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:02.769 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:35:02.387, end=2024-09-11T17:37:02.766} 2024-09-11 17:39:02.775 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obCl 2024-09-11 17:39:02.775 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:39:02.777 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:35:02.387, end=2024-09-11T17:37:02.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:39:02.780 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:39:02.780 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:39:02.798 INFO 2981023 --- [plan_change_detector18,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-631A71F47F2D363706648DCCB228809F 2024-09-11 17:39:02.798 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:39:02.798 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:39:02.799 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:35:02.387, end=2024-09-11T17:37:02.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:39:02.799 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:39:02.799 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:39:02.800 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:39:02.802 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:35:02.387, end=2024-09-11T17:37:02.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:39:02.802 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:39:02.802 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:39:02.813 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:39:02.832 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961142832000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:02.832Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:02.835 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:39:02.835 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:02.840 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.840 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.841 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:02.841 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:02.846 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.846 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.847 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:39:02.847 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:02.832, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:02.847 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:39:02.848 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:35:02.387, end=2024-09-11T17:37:02.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:39:02.848 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:39:02.848 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:39:02.849 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:39:02.851 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:35:02.387, end=2024-09-11T17:37:02.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:39:02.851 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:39:02.851 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:39:02.895 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961142895000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:02.895Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:02.897 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961142896000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:02.896Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:02.898 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961142898000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:02.898Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:02.900 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961142900000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:02.900Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:02.900 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961142900000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:02.900Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:02.901 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961142901000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:02.901Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:02.903 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961142903000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:02.903Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:02.904 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961142904000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:02.904Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:02.916 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:02.916 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:02.916 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:39:02.916 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:02.949 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:02.949 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:02.950 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.950 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.951 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:02.951 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:02.951 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:39:02.951 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:02.953 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 61 2024-09-11 17:39:02.953 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:02.956 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:02.956 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:02.958 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:02.958 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:02.959 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.959 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.961 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:39:02.961 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:02.963 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:02.963 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:02.967 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.967 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.967 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.967 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.967 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.967 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.970 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.970 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.970 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:02.970 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.970 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.972 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.972 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.973 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.973 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.976 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:02.976 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:02.977 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:02.977 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:02.978 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:02.978 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:02.978 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:02.978 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:02.978 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:02.978 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:02.979 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:02.979 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:02.982 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.982 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.984 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:02.895, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:02.984 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:02.993 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:02.896, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:02.996 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.996 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.997 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.997 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:02.999 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:02.999 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.001 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.001 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.002 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.005 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.005 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.007 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.007 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.007 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:02.898, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.009 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.009 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.009 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.010 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.010 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:02.9, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.011 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:02.903, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.011 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.016 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:02.9, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.017 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:02.901, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.018 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:02.904, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.035 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143035000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.035Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.039 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143039000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.039Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.046 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143046000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.046Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.049 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143049000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.049Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.053 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:39:03.053 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.053 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143053000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.053Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.056 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143056000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.056Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.060 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.060 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.071 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:03.071 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.073 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143073000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.073Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.078 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.078 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.086 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:03.086 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.096 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:03.096 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.100 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.100 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.102 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.102 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.103 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143090000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.090Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.104 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.104 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.108 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:39:03.108 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.112 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.112 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.113 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.113 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.113 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:03.113 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.116 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:03.116 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.117 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.117 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.120 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.120 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.120 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.120 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:03.120 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.121 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.121 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.126 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.126 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.127 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.127 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.127 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.127 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.128 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.035, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.130 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.130 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.131 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.131 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.131 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.132 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.132 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.133 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.135 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.135 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.137 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.049, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.137 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.137 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.139 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.046, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.139 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.139 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.140 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.140 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.140 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.140 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.142 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.144 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.147 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.147 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.148 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.053, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.148 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.149 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.149 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.149 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.039, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.151 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.151 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.151 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.153 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.154 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.056, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.157 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.09, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.159 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.073, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.161 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143161000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.161Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.166 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961143166000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:39:03.166Z, endTime=2024-09-11T09:37:02.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:39:03.168 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:03.168 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.173 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:39:03.173 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:39:03.179 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.179 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.179 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.179 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.180 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.180 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.181 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:39:03.181 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:39:03.186 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.186 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.187 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:39:03.187 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047422766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:39:03.188 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.188 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:39:03.191 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.161, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.191 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:39:03.166, end=2024-09-11T17:37:02.766}, clusterId=2, tenantId=6, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:39:03.192 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:39:03.192 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:35:02.387, end=2024-09-11T17:37:02.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:39:03.192 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:39:03.192 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:39:03.193 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:39:03.243 INFO 2981023 --- [ocp-async-1,ca1afed8278a2d4e,924872fc1b2ab8be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:05.243 INFO 2981023 --- [ocp-async-10,7b4f7cbfe5fa6026,ce5ffdf94fdedb15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:05.279 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:05.281 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,53d61239d5a43cb3,b87bf6e315cc08d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,53d61239d5a43cb3,b87bf6e315cc08d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:39:06.352221+08:00, lastProcessTime=2024-09-11T17:38:56.337590+08:00 2024-09-11 17:39:06.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:39:06.352221+08:00 2024-09-11 17:39:07.056 INFO 2981023 --- [ocp-schedules-11,734eda0656aa56a5,d2cd5bb9b0c4afe1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:39:07.105 WARN 2981023 --- [scanner-13521236-1,734eda0656aa56a5,3e879fbff1400887] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:39:07.123 INFO 2981023 --- [alarm-detect-2,734eda0656aa56a5,a3ab53cb487c646c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644310, data is empty. 2024-09-11 17:39:07.127 INFO 2981023 --- [alarm-detect-2,734eda0656aa56a5,a3ab53cb487c646c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644353, data is empty. 2024-09-11 17:39:07.129 INFO 2981023 --- [alarm-detect-2,734eda0656aa56a5,a3ab53cb487c646c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644353, data is empty. 2024-09-11 17:39:07.151 INFO 2981023 --- [alarm-detect-13,734eda0656aa56a5,7d42f5aae6cb1c0b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644310, data is empty. 2024-09-11 17:39:07.154 INFO 2981023 --- [alarm-detect-13,734eda0656aa56a5,7d42f5aae6cb1c0b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644353, data is empty. 2024-09-11 17:39:07.155 INFO 2981023 --- [alarm-detect-13,734eda0656aa56a5,7d42f5aae6cb1c0b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644353, data is empty. 2024-09-11 17:39:07.170 INFO 2981023 --- [alarm-detect-13,734eda0656aa56a5,1fce6578351c5348] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644310, data is empty. 2024-09-11 17:39:07.176 INFO 2981023 --- [alarm-detect-13,734eda0656aa56a5,1fce6578351c5348] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644353, data is empty. 2024-09-11 17:39:07.181 INFO 2981023 --- [alarm-detect-13,734eda0656aa56a5,1fce6578351c5348] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644353, data is empty. 2024-09-11 17:39:07.243 INFO 2981023 --- [ocp-async-4,b2fe466b40cbf5e9,d59ef3702e6e1d6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:07.287 INFO 2981023 --- [alarm-detect-12,734eda0656aa56a5,7993b5ff1dfff420] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644310, data is empty. 2024-09-11 17:39:07.294 INFO 2981023 --- [alarm-detect-12,734eda0656aa56a5,7993b5ff1dfff420] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644353, data is empty. 2024-09-11 17:39:07.299 INFO 2981023 --- [alarm-detect-12,734eda0656aa56a5,7993b5ff1dfff420] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644353, data is empty. 2024-09-11 17:39:07.386 INFO 2981023 --- [ocp-schedules-21,a4ed53eb22308e75,f1c6eff88a4fbbff] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:39:07.395 INFO 2981023 --- [ocp-schedules-5,5dbe9405d86409b5,03bab9ecc965b60b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:39:07.395+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:39:07.396 INFO 2981023 --- [ocp-schedules-5,5dbe9405d86409b5,03bab9ecc965b60b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:39:07.397 INFO 2981023 --- [ocp-schedules-5,5dbe9405d86409b5,03bab9ecc965b60b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:39:07.398 INFO 2981023 --- [ocp-schedules-5,5dbe9405d86409b5,03bab9ecc965b60b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:39:07.755 INFO 2981023 --- [ocp-async-10,e4e8552efb89a07c,59c02f74e5a90af9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:39:08.281 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:08.282 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,0e916d91df1ceb51,41525d350c4bd36b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,0e916d91df1ceb51,41525d350c4bd36b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:09.244 INFO 2981023 --- [ocp-async-1,4eaced6c9eb0c766,bd5002aa164f45a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:10.003 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:39:10.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:39:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:39:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:39:11.244 INFO 2981023 --- [ocp-async-10,3de8229689f0f4ca,eea587780d0a19e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:11.282 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:11.282 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,7b6a9921ea7d3dce,826583e653709cb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,7b6a9921ea7d3dce,826583e653709cb0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:12.014 INFO 2981023 --- [ocp-schedules-22,c15d7a9b6b102bb1,58a3b6c2f914177f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:39:12.545 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047480, nodeCount=1000, affectRows=1000 2024-09-11 17:39:12.548 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047480, nodeCount=999, validCount=999 2024-09-11 17:39:12.555 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047480, nodeCount=1000, affectRows=1000 2024-09-11 17:39:12.558 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047480, nodeCount=1000, validCount=1000 2024-09-11 17:39:12.558 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047480, nodeCount=48, affectRows=48 2024-09-11 17:39:12.559 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047480, nodeCount=48, validCount=48 2024-09-11 17:39:12.895 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047480, nodeCount=480, affectRows=480 2024-09-11 17:39:13.244 INFO 2981023 --- [ocp-async-4,b407988b2211da99,a3c894c0d1328be8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:14.043 INFO 2981023 --- [ocp-schedules-14,d39676ea5c7700a4,0613d9a75d9a546b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:39:14.051 INFO 2981023 --- [ocp-schedules-14,d39676ea5c7700a4,0613d9a75d9a546b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:39:14.051 INFO 2981023 --- [ocp-schedules-14,d39676ea5c7700a4,0613d9a75d9a546b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:39:14.282 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:14.283 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,dc9387210740e03e,2c9162a67932b8f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,12d9342a2266648e,5c739d43025a623c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:39:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,dc9387210740e03e,2c9162a67932b8f5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:14.512 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047480, nodeCount=480, validCount=480 2024-09-11 17:39:15.244 INFO 2981023 --- [ocp-async-6,294096bebb6c75f0,c44bc9f36308c55b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:16.022 INFO 2981023 --- [ocp-schedules-27,1f347fbe1ce962b8,aaecb4c3f60f7d30] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:39:16.022 INFO 2981023 --- [ocp-schedules-10,3bd64a811bf4e7c8,8d50ed33378d0319] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:39:16.022 INFO 2981023 --- [host-check-clockdiff1,3bd64a811bf4e7c8,1bd177aacbc9f92f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:39:16.023 INFO 2981023 --- [host-check-clockdiff2,3bd64a811bf4e7c8,3c30267c96b578fa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:39:16.023 INFO 2981023 --- [host-check-agent-status1,1f347fbe1ce962b8,4d5d1239160aba0a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:39:16.024 INFO 2981023 --- [host-check-agent-status2,1f347fbe1ce962b8,d437a30a109a0323] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:39:16.025 INFO 2981023 --- [host-check-agent-status3,1f347fbe1ce962b8,eac9efa3d09f6b79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:39:16.025 INFO 2981023 --- [host-check-agent-status4,1f347fbe1ce962b8,3135862f778a5129] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:39:16.025 INFO 2981023 --- [host-check-clockdiff3,3bd64a811bf4e7c8,e83483eae6ef0a9c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:39:16.026 INFO 2981023 --- [ocp-schedules-7,dcb3503ce0993e3a,32f93a95661eb275] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:39:16.027 INFO 2981023 --- [host-check-clockdiff4,3bd64a811bf4e7c8,667064cdc76a377a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:39:16.043 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:39:16.044 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:39:16.044 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:39:16.045 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:39:16.048 INFO 2981023 --- [host-check-clockdiff4,3bd64a811bf4e7c8,667064cdc76a377a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047556 0 0 , error= 2024-09-11 17:39:16.065 INFO 2981023 --- [host-check-agent-status1,1f347fbe1ce962b8,4d5d1239160aba0a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:39:16.066 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:39:16.072 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:39:16.078 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:39:16.079 INFO 2981023 --- [host-check-agent-status2,1f347fbe1ce962b8,d437a30a109a0323] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:39:16.079 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:39:16.080 INFO 2981023 --- [host-check-agent-status2,1f347fbe1ce962b8,d437a30a109a0323] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:39:16.080 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:39:16.081 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:39:16.081 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:39:16.082 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:39:16.094 INFO 2981023 --- [host-check-agent-status1,1f347fbe1ce962b8,4d5d1239160aba0a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:39:16.173 INFO 2981023 --- [host-check-agent-status4,1f347fbe1ce962b8,3135862f778a5129] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:39:16.174 INFO 2981023 --- [host-check-agent-status3,1f347fbe1ce962b8,eac9efa3d09f6b79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:39:16.192 INFO 2981023 --- [ocp-schedules-21,719f1a32b2f86b5b,9b90e70e119cd48f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:39:16.192 INFO 2981023 --- [ocp-schedules-12,7e0d7a743212a93f,e0a73976c6b57d4a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:39:16.192 INFO 2981023 --- [ocp-schedules-30,a950338943648806,e51a2bdadf2af88d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:39:16.192 INFO 2981023 --- [ocp-schedules-16,280c41f04fbc0afa,a5abfbe3352afdf5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:39:16.264 INFO 2981023 --- [ocp-schedules-30,a950338943648806,e51a2bdadf2af88d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:39:16.264 INFO 2981023 --- [ocp-schedules-12,7e0d7a743212a93f,e0a73976c6b57d4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:39:16.265 INFO 2981023 --- [host-check-agent-status3,1f347fbe1ce962b8,eac9efa3d09f6b79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:39:16.265 INFO 2981023 --- [host-check-agent-status4,1f347fbe1ce962b8,3135862f778a5129] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:39:16.271 INFO 2981023 --- [ocp-schedules-21,719f1a32b2f86b5b,9b90e70e119cd48f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:39:16.340 INFO 2981023 --- [ocp-schedules-12,7e0d7a743212a93f,e0a73976c6b57d4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:39:16.358 INFO 2981023 --- [ocp-schedules-5,898fac000c87171a,614583355b2aefc5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:39:16.372 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:39:16.373 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:39:16.374 INFO 2981023 --- [host-check-agent-status1,1f347fbe1ce962b8,4d5d1239160aba0a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:39:16.375 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:39:16.374814120+08:00 2024-09-11 17:39:16.376 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:39:16.376 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:39:16.374742507+08:00 2024-09-11 17:39:16.376 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:39:16.376 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:39:16.375+08:00 2024-09-11 17:39:16.377 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:39:16.376+08:00 2024-09-11 17:39:16.378 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:39:16.375+08:00 and status AVAILABLE 2024-09-11 17:39:16.379 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:39:16.378582062+08:00 2024-09-11 17:39:16.379 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:39:16.378199675+08:00 2024-09-11 17:39:16.379 INFO 2981023 --- [host-check-agent-status1,1f347fbe1ce962b8,4d5d1239160aba0a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:39:16.379 INFO 2981023 --- [host-check-agent-status1,1f347fbe1ce962b8,4d5d1239160aba0a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:39:16.379 INFO 2981023 --- [host-check-agent-status1,1f347fbe1ce962b8,4d5d1239160aba0a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:39:16.379 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:39:16.376+08:00 and status AVAILABLE 2024-09-11 17:39:16.379 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:39:16.379+08:00 2024-09-11 17:39:16.381 INFO 2981023 --- [host-check-agent-status3,1f347fbe1ce962b8,eac9efa3d09f6b79] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:39:16.382 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:39:16.379+08:00 and status AVAILABLE 2024-09-11 17:39:16.383 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:39:16.379+08:00 2024-09-11 17:39:16.384 INFO 2981023 --- [host-check-agent-status3,1f347fbe1ce962b8,eac9efa3d09f6b79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:39:16.384 INFO 2981023 --- [host-check-agent-status3,1f347fbe1ce962b8,eac9efa3d09f6b79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:39:16.384 INFO 2981023 --- [host-check-agent-status3,1f347fbe1ce962b8,eac9efa3d09f6b79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:39:16.387 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:39:16.379+08:00 and status AVAILABLE 2024-09-11 17:39:16.388 INFO 2981023 --- [host-check-agent-status2,1f347fbe1ce962b8,d437a30a109a0323] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:39:16.388 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:39:16.387249+08:00, lastProcessTime=2024-09-11T17:39:06.352221+08:00 2024-09-11 17:39:16.390 INFO 2981023 --- [host-check-agent-status2,1f347fbe1ce962b8,d437a30a109a0323] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:39:16.390 INFO 2981023 --- [host-check-agent-status2,1f347fbe1ce962b8,d437a30a109a0323] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:39:16.390 INFO 2981023 --- [host-check-agent-status2,1f347fbe1ce962b8,d437a30a109a0323] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:39:16.391 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:39:16.396 INFO 2981023 --- [host-check-agent-status4,1f347fbe1ce962b8,3135862f778a5129] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:39:16.397 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.397 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:16.398 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:39:16.400 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:39:16.400 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.401 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:39:16.402 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:39:16.402 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:16.402 INFO 2981023 --- [host-check-agent-status4,1f347fbe1ce962b8,3135862f778a5129] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:39:16.403 INFO 2981023 --- [host-check-agent-status4,1f347fbe1ce962b8,3135862f778a5129] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:39:16.403 INFO 2981023 --- [host-check-agent-status4,1f347fbe1ce962b8,3135862f778a5129] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:39:16.403 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:39:16.460 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:39:16.461 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:39:16.465 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:39:16.466 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:39:16.467 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.467 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.467 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:16.467 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:16.470 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.470 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:16.471 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:39:16.472 INFO 2981023 --- [ocp-schedules-21,719f1a32b2f86b5b,9b90e70e119cd48f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:39:16.475 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:39:16.476 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:39:16.478 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:39:16.479 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:39:16.485 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.485 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:39:16.485 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:16.501 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:39:16.501 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:39:16.511 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.511 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.511 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:39:16.513 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:39:16.513 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:39:16.526 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:39:16.530 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:39:16.533 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:39:16.535 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:39:16.561 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:39:16.562 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:39:16.565 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:39:16.569 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.570 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.570 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:16.571 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:39:16.571 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:39:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:39:16.579 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:39:16.583 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:39:16.592 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.593 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.593 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:16.593 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:39:16.593 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:39:16.598 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:39:16.601 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:39:16.611 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:16.614 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:39:16.620 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:39:16.623 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:39:16.627 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.627 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.627 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:39:16.628 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.628 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.628 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:16.629 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:39:16.630 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:16.631 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.634 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:39:16.634 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:39:16.635 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:39:16.635 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:39:16.647 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:39:16.655 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:16.655 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:39:16.656 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:39:16.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:39:16.659 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:39:16.659 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:39:16.666 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:39:16.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.671 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.673 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:39:16.674 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:16.676 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:39:16.678 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:39:16.680 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.681 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.681 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:16.682 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:39:16.685 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:39:16.688 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:39:16.689 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:39:16.690 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:39:16.694 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:39:16.695 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.696 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:39:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:16.697 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:39:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.700 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:39:16.700 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:39:16.702 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:39:16.704 INFO 2981023 --- [host-check-host-status2,dcb3503ce0993e3a,04ddd8075055a7f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:39:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:39:16.713 INFO 2981023 --- [host-check-host-status3,dcb3503ce0993e3a,413c22495d95411b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:39:16.713 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:39:16.721 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:39:16.725 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:16.725 INFO 2981023 --- [host-check-host-status4,dcb3503ce0993e3a,6259b095bb272a1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:39:16.727 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:39:16.735 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.737 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.737 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:16.737 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:39:16.742 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:39:16.743 INFO 2981023 --- [host-check-host-status1,dcb3503ce0993e3a,464688cb72f1e9e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:39:16.747 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:39:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:39:16.765 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:39:16.785 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:39:16.789 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:39:16.792 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.793 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.793 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:39:16.796 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.796 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.796 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:39:16.797 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:39:16.802 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:39:16.802 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:39:16.804 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:16.805 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:16.805 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:39:16.805 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:39:16.805 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:39:16.806 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:39:16.387249+08:00 2024-09-11 17:39:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:39:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:39:16.811 INFO 2981023 --- [ocp-schedules-30,a950338943648806,e51a2bdadf2af88d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:39:16.969 INFO 2981023 --- [host-check-clockdiff1,3bd64a811bf4e7c8,1bd177aacbc9f92f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047556 0 0 , error= 2024-09-11 17:39:17.244 INFO 2981023 --- [ocp-async-9,f01210654ce6656f,54eab0c8f884276d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:17.283 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:17.283 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,413b45866a86c66c,6aeca925c0508bbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:17.315 INFO 2981023 --- [prometheus-minute-schedule-1,413b45866a86c66c,6aeca925c0508bbc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:17.334 INFO 2981023 --- [ocp-schedules-2,3fbf0be4f30e6590,b07f4d7e9218f279] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:39:17.383 WARN 2981023 --- [scanner-13523048-1,3fbf0be4f30e6590,1dd4a38d3f064884] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:39:17.755 INFO 2981023 --- [ocp-async-6,33e5ca00434cc878,14aed05cd8084f9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:39:17.817 INFO 2981023 --- [host-check-clockdiff2,3bd64a811bf4e7c8,3c30267c96b578fa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047557 0 0 , error= 2024-09-11 17:39:19.244 INFO 2981023 --- [ocp-async-4,b7c78aa8939332e8,cae0457afd8eb013] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:20.283 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:20.284 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,7b80e41d2dbbe969,7557a1569e90a21e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:20.315 INFO 2981023 --- [prometheus-minute-schedule-1,7b80e41d2dbbe969,7557a1569e90a21e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:21.244 INFO 2981023 --- [ocp-async-6,0f44c8e3034d5217,303c8efe8a14b3b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:22.007 INFO 2981023 --- [host-check-clockdiff3,3bd64a811bf4e7c8,e83483eae6ef0a9c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047562 1 1 , error= 2024-09-11 17:39:22.017 INFO 2981023 --- [ocp-schedules-24,8607e03fce1b7a4a,20b26d988d37b172] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:39:22.887 INFO 2981023 --- [ocp-schedules-20,b890bdd883c9f859,a52a54c8036be571] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:39:22.887 INFO 2981023 --- [ocp-schedules-20,b890bdd883c9f859,a52a54c8036be571] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:39:22.887 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:39:22.887 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:39:22.887 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:39:22.892 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:39:22.892 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:39:22.892 INFO 2981023 --- [ocp-schedules-20,b890bdd883c9f859,a52a54c8036be571] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:39:22.892 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:39:22.893 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:39:22.893 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:39:22.893 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:39:22.893 INFO 2981023 --- [ocp-schedules-20,b890bdd883c9f859,a52a54c8036be571] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:39:22.909 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:39:22.910 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:39:22.911 INFO 2981023 --- [ocp-schedules-20,b890bdd883c9f859,a52a54c8036be571] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:39:22.911 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:39:22.913 INFO 2981023 --- [ocp-schedules-20,b890bdd883c9f859,a52a54c8036be571] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:39:23.244 INFO 2981023 --- [ocp-async-9,4abfb003c3b546b3,21eeabfe87ae2339] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:23.284 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:23.284 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,c7e26b10fe5e1dca,4933e590181bf447] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,c7e26b10fe5e1dca,4933e590181bf447] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:24.242 INFO 2981023 --- [ocp-schedules-18,637e3328f270ec97,ba63dbef0b2543ab] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:39:25.244 INFO 2981023 --- [ocp-async-8,7ec61f6f2cf37fa0,fdf4f7d26dfded57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:26.284 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:26.285 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,9e3c3d536344ba96,f08372ad572bb42f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,9e3c3d536344ba96,f08372ad572bb42f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:39:26.337633+08:00, lastProcessTime=2024-09-11T17:39:16.387249+08:00 2024-09-11 17:39:26.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:39:26.337633+08:00 2024-09-11 17:39:27.245 INFO 2981023 --- [ocp-async-7,f937aca416ffe7cd,a7e0644948bdb318] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:27.536 INFO 2981023 --- [ocp-schedules-18,bd89326c6ff5deab,11454823fbbb27a3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:39:27.586 WARN 2981023 --- [scanner-13523287-1,bd89326c6ff5deab,45e8cd15ab86f3a8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:39:27.602 INFO 2981023 --- [alarm-detect-4,bd89326c6ff5deab,e556c96c8d932723] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644353, data is empty. 2024-09-11 17:39:27.607 INFO 2981023 --- [alarm-detect-4,bd89326c6ff5deab,e556c96c8d932723] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644353, data is empty. 2024-09-11 17:39:27.613 INFO 2981023 --- [alarm-detect-4,bd89326c6ff5deab,e556c96c8d932723] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644353, data is empty. 2024-09-11 17:39:27.638 INFO 2981023 --- [alarm-detect-17,bd89326c6ff5deab,94fd6e3c48d18da3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644353, data is empty. 2024-09-11 17:39:27.641 INFO 2981023 --- [alarm-detect-1,bd89326c6ff5deab,61b7f2b5a01f0e60] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644353, data is empty. 2024-09-11 17:39:27.642 INFO 2981023 --- [alarm-detect-17,bd89326c6ff5deab,94fd6e3c48d18da3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644353, data is empty. 2024-09-11 17:39:27.644 INFO 2981023 --- [alarm-detect-1,bd89326c6ff5deab,61b7f2b5a01f0e60] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644353, data is empty. 2024-09-11 17:39:27.646 INFO 2981023 --- [alarm-detect-17,bd89326c6ff5deab,94fd6e3c48d18da3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644353, data is empty. 2024-09-11 17:39:27.648 INFO 2981023 --- [alarm-detect-1,bd89326c6ff5deab,61b7f2b5a01f0e60] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644353, data is empty. 2024-09-11 17:39:27.756 INFO 2981023 --- [ocp-async-8,a0487d0334f04400,a662ca3029b71b9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:39:27.784 INFO 2981023 --- [alarm-detect-2,bd89326c6ff5deab,f35b8530343b078e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644353, data is empty. 2024-09-11 17:39:27.790 INFO 2981023 --- [alarm-detect-2,bd89326c6ff5deab,f35b8530343b078e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644353, data is empty. 2024-09-11 17:39:27.794 INFO 2981023 --- [alarm-detect-2,bd89326c6ff5deab,f35b8530343b078e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644353, data is empty. 2024-09-11 17:39:29.245 INFO 2981023 --- [ocp-async-9,d7b56e79e326a1b1,1336ef6115b597f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:29.285 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:29.286 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,d19199e5b181a657,dca1eaa3d81f28ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,d19199e5b181a657,dca1eaa3d81f28ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:29.660 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:39:29.660 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:39:29.663 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:39:29.663 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:39:29.663 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:39:29.663 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:39:29.666 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:39:29.666 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:39:29.666 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:39:29.666 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:39:29.690 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:39:29.694 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:39:29.694 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:39:29.694 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:39:29.694 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:39:29.694 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:39:29.698 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:39:29.932 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:39:30.194 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.528512032S 2024-09-11 17:39:30.194 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:39:30.243 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:39:30.243 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:39:30.257 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:39:30.294 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:39:30.294 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:39:30.294 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:39:30.294 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:39:30.294 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:39:30.295 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:39:30.301 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:39:30.380 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] o.s.boot.SpringApplication : Started application in 0.716 seconds (JVM running for 83370.253) 2024-09-11 17:39:30.380 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.717055642S 2024-09-11 17:39:30.601 INFO 2981023 --- [ocp-schedules-23,43a69c281eac8af8,ffcc054c31c1c438] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:39:31.245 INFO 2981023 --- [ocp-async-8,380d102c8a29567c,fe333a030cccd6f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:32.020 INFO 2981023 --- [ocp-schedules-29,ad09053eeac5df74,5a2c4c009a5d4690] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:39:32.286 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:32.286 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,17861a773f5fcfe3,aec0becf905eb1ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,17861a773f5fcfe3,aec0becf905eb1ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:33.245 INFO 2981023 --- [ocp-async-7,746200ff7c74c240,8f5b1945d597c809] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:34.053 INFO 2981023 --- [ocp-schedules-7,d76d4118981ea652,71878215a6a54f84] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:39:34.069 INFO 2981023 --- [ocp-schedules-7,d76d4118981ea652,71878215a6a54f84] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:39:34.069 INFO 2981023 --- [ocp-schedules-7,d76d4118981ea652,71878215a6a54f84] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:39:35.245 INFO 2981023 --- [ocp-async-1,16238dae58e95d55,ab80386e8cd80eb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:35.286 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:35.287 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,addd0abb0f323a19,bf59c542ab783325] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,addd0abb0f323a19,bf59c542ab783325] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:39:36.337531+08:00, lastProcessTime=2024-09-11T17:39:26.337633+08:00 2024-09-11 17:39:36.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:39:36.337531+08:00 2024-09-11 17:39:37.245 INFO 2981023 --- [ocp-async-2,dd32dac5182d8b46,081fe0cb7047cca3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:37.314 INFO 2981023 --- [prometheus-cache-monitor1,4d7a591ffd92e2ef,199c629f6fcd268b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8192,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:39:37.315 INFO 2981023 --- [prometheus-cache-monitor1,4d7a591ffd92e2ef,199c629f6fcd268b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:39:37.756 INFO 2981023 --- [ocp-async-1,59f5d173980829c5,1150cd74f1be9ba4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:39:37.831 INFO 2981023 --- [ocp-schedules-3,3c6e25b04e261c90,78604908dc1bbc38] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:39:37.875 WARN 2981023 --- [scanner-13523628-1,3c6e25b04e261c90,4f4391561b8da107] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:39:38.287 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:38.287 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,6e3eb1e2f7efb944,41d7b39d09d29122] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,6e3eb1e2f7efb944,41d7b39d09d29122] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:39.245 INFO 2981023 --- [ocp-async-3,f1d9d51456b08241,27368fb187841d62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:40.002 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:39:40.002 INFO 2981023 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:39:40.027 INFO 2981023 --- [ocp-schedules-18,db41fa5646aa7280,c2a57ca30085ddeb] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:39:40.046 WARN 2981023 --- [ocp-schedules-18,db41fa5646aa7280,c2a57ca30085ddeb] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:39:41.245 INFO 2981023 --- [ocp-async-5,1ca5c13f3d457d01,498f20f6dba895bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:41.287 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:41.288 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,f0613b4353b4c19e,da7c5323b4eff4fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,f0613b4353b4c19e,da7c5323b4eff4fb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:42.022 INFO 2981023 --- [ocp-schedules-11,7416e4a2148e52a4,78c3ba7146670e30] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:39:43.245 INFO 2981023 --- [ocp-async-10,b79d60c5772db403,cbf14523c767c57d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:44.288 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:44.288 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,526957e5c37ce023,a847228915b224cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,526957e5c37ce023,a847228915b224cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:45.101 INFO 2981023 --- [prometheus-second-schedule-1,9f82a07bb2916e02,eaab63addc589154] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:39:45.245 INFO 2981023 --- [ocp-async-4,8613669d60cb8033,4abad81a3c3eb141] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:45.998 INFO 2981023 --- [ocp-schedules-7,1d5157399e203d5d,886889121eae8832] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:39:45.998 INFO 2981023 --- [host-check-clockdiff1,1d5157399e203d5d,fa85d51ff7c4d50d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:39:45.998 INFO 2981023 --- [host-check-clockdiff2,1d5157399e203d5d,152cc53161848539] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:39:45.999 INFO 2981023 --- [host-check-clockdiff3,1d5157399e203d5d,24a3f9f4b4787a6d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:39:46.001 INFO 2981023 --- [host-check-clockdiff4,1d5157399e203d5d,2d016b2ff951f103] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:39:46.004 INFO 2981023 --- [ocp-schedules-9,955e7a1aecada377,01b19afd90d07844] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:39:46.007 INFO 2981023 --- [host-check-clockdiff1,1d5157399e203d5d,fa85d51ff7c4d50d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047586 0 0 , error= 2024-09-11 17:39:46.008 INFO 2981023 --- [ocp-schedules-11,4e6d1b2438ceba36,e275ec1a9d8a0516] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:39:46.012 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:39:46.012 INFO 2981023 --- [host-check-agent-status2,4e6d1b2438ceba36,eeb6f284e2e21a2c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:39:46.013 INFO 2981023 --- [host-check-agent-status1,4e6d1b2438ceba36,2c3b06c78cf23307] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:39:46.013 INFO 2981023 --- [host-check-agent-status3,4e6d1b2438ceba36,cd528f876114f33c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:39:46.014 INFO 2981023 --- [host-check-agent-status4,4e6d1b2438ceba36,62720bd2f6ff6156] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:39:46.014 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:39:46.016 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:39:46.017 INFO 2981023 --- [host-check-agent-status1,4e6d1b2438ceba36,2c3b06c78cf23307] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:39:46.018 INFO 2981023 --- [host-check-agent-status2,4e6d1b2438ceba36,eeb6f284e2e21a2c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:39:46.020 INFO 2981023 --- [host-check-clockdiff4,1d5157399e203d5d,2d016b2ff951f103] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047586 0 0 , error= 2024-09-11 17:39:46.021 INFO 2981023 --- [host-check-agent-status2,4e6d1b2438ceba36,eeb6f284e2e21a2c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:39:46.022 INFO 2981023 --- [host-check-agent-status3,4e6d1b2438ceba36,cd528f876114f33c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:39:46.022 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:39:46.022 INFO 2981023 --- [host-check-agent-status1,4e6d1b2438ceba36,2c3b06c78cf23307] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:39:46.023 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:39:46.023 INFO 2981023 --- [host-check-agent-status3,4e6d1b2438ceba36,cd528f876114f33c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:39:46.025 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:39:46.027 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:39:46.027 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:39:46.028 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:39:46.028 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:39:46.032 INFO 2981023 --- [host-check-agent-status4,4e6d1b2438ceba36,62720bd2f6ff6156] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:39:46.032 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:39:46.034 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:39:46.035 INFO 2981023 --- [host-check-agent-status4,4e6d1b2438ceba36,62720bd2f6ff6156] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:39:46.050 INFO 2981023 --- [host-check-agent-status1,4e6d1b2438ceba36,2c3b06c78cf23307] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:39:46.052 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:39:46.054 INFO 2981023 --- [host-check-agent-status1,4e6d1b2438ceba36,2c3b06c78cf23307] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:39:46.054 INFO 2981023 --- [host-check-agent-status1,4e6d1b2438ceba36,2c3b06c78cf23307] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:39:46.054 INFO 2981023 --- [host-check-agent-status1,4e6d1b2438ceba36,2c3b06c78cf23307] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:39:46.054 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:39:46.054869528+08:00 2024-09-11 17:39:46.055 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:39:46.056 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:39:46.055+08:00 2024-09-11 17:39:46.058 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:39:46.058401801+08:00 2024-09-11 17:39:46.060 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:39:46.058+08:00 2024-09-11 17:39:46.062 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:39:46.055+08:00 and status AVAILABLE 2024-09-11 17:39:46.062 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:39:46.058+08:00 and status AVAILABLE 2024-09-11 17:39:46.062 INFO 2981023 --- [host-check-agent-status2,4e6d1b2438ceba36,eeb6f284e2e21a2c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:39:46.066 INFO 2981023 --- [host-check-agent-status3,4e6d1b2438ceba36,cd528f876114f33c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:39:46.066 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:39:46.066 INFO 2981023 --- [host-check-agent-status2,4e6d1b2438ceba36,eeb6f284e2e21a2c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:39:46.066 INFO 2981023 --- [host-check-agent-status2,4e6d1b2438ceba36,eeb6f284e2e21a2c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:39:46.066 INFO 2981023 --- [host-check-agent-status2,4e6d1b2438ceba36,eeb6f284e2e21a2c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:39:46.066 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:39:46.067 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:39:46.067008083+08:00 2024-09-11 17:39:46.068 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:39:46.067+08:00 2024-09-11 17:39:46.069 INFO 2981023 --- [host-check-agent-status3,4e6d1b2438ceba36,cd528f876114f33c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:39:46.069 INFO 2981023 --- [host-check-agent-status3,4e6d1b2438ceba36,cd528f876114f33c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:39:46.069 INFO 2981023 --- [host-check-agent-status3,4e6d1b2438ceba36,cd528f876114f33c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:39:46.069 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:39:46.067+08:00 and status AVAILABLE 2024-09-11 17:39:46.071 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:39:46.071 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:39:46.071278954+08:00 2024-09-11 17:39:46.072 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:39:46.071+08:00 2024-09-11 17:39:46.072 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:39:46.073 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:39:46.071+08:00 and status AVAILABLE 2024-09-11 17:39:46.075 INFO 2981023 --- [host-check-agent-status4,4e6d1b2438ceba36,62720bd2f6ff6156] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:39:46.078 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:39:46.079 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:39:46.079 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:39:46.080 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:39:46.081 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:39:46.085 INFO 2981023 --- [host-check-agent-status4,4e6d1b2438ceba36,62720bd2f6ff6156] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:39:46.085 INFO 2981023 --- [host-check-agent-status4,4e6d1b2438ceba36,62720bd2f6ff6156] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:39:46.085 INFO 2981023 --- [host-check-agent-status4,4e6d1b2438ceba36,62720bd2f6ff6156] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:39:46.087 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:39:46.088 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:39:46.097 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:39:46.099 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:39:46.114 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:39:46.115 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:39:46.116 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:39:46.121 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:39:46.133 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:39:46.142 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:39:46.142 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:39:46.150 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:39:46.150 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:39:46.158 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:39:46.167 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:39:46.178 INFO 2981023 --- [ocp-schedules-15,2c3fecc7d73a0c60,85dc0afeb337ab15] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:39:46.187 INFO 2981023 --- [ocp-schedules-15,2c3fecc7d73a0c60,85dc0afeb337ab15] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:39:46.187 INFO 2981023 --- [ocp-schedules-15,a62b89106254c1af,f9da5666d3bdf379] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:39:46.188 INFO 2981023 --- [ocp-schedules-23,076f28a4927ad7f6,06d7bb01a80be815] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:39:46.188 INFO 2981023 --- [ocp-schedules-25,22b71b11d85d9793,bc596c6d3f3aaf10] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:39:46.188 INFO 2981023 --- [ocp-schedules-12,168d88dc1358fb9f,33deca3c17d40496] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:39:46.189 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:39:46.189 INFO 2981023 --- [ocp-schedules-23,076f28a4927ad7f6,06d7bb01a80be815] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:39:46.189 INFO 2981023 --- [ocp-schedules-15,a62b89106254c1af,f9da5666d3bdf379] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:39:46.189 INFO 2981023 --- [ocp-schedules-25,22b71b11d85d9793,bc596c6d3f3aaf10] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:39:46.192 INFO 2981023 --- [ocp-schedules-23,076f28a4927ad7f6,06d7bb01a80be815] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:39:46.221 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:39:46.227 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.231 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:46.233 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:39:46.234 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:39:46.238 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:39:46.240 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:39:46.241 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:39:46.250 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:39:46.252 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.254 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:46.260 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:39:46.261 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.262 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:46.263 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:39:46.264 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.264 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:46.269 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:39:46.270 INFO 2981023 --- [ocp-schedules-17,30d968b7ced1fed0,46de6fbf4e69205a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:39:46.278 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:39:46.278 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:39:46.279 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:39:46.281 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:39:46.304 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.304 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.310 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:39:46.310 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.310 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.312 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:39:46.312 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:39:46.312 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:39:46.313 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.316 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.316 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:46.317 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:39:46.320 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.320 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:39:46.321 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.321 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:39:46.323 INFO 2981023 --- [ocp-schedules-15,a62b89106254c1af,f9da5666d3bdf379] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:39:46.328 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:39:46.330 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.331 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:46.334 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:39:46.339 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:39:46.339 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.340 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.340 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:46.344 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.344 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:39:46.348 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:39:46.349 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:39:46.349 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:39:46.349114+08:00, lastProcessTime=2024-09-11T17:39:36.337531+08:00 2024-09-11 17:39:46.350 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:39:46.363 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:39:46.371 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:39:46.374 INFO 2981023 --- [host-check-host-status3,955e7a1aecada377,07e2c4d2f7b07e0b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:39:46.379 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:39:46.379 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:46.379 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:39:46.387 INFO 2981023 --- [host-check-host-status1,955e7a1aecada377,4e71cbe720f15034] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:39:46.398 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.400 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.400 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:46.402 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:39:46.402 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.403 INFO 2981023 --- [host-check-host-status2,955e7a1aecada377,54e9a0633083c813] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:39:46.405 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:39:46.405 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:39:46.410 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.412 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.412 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:39:46.413 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:39:46.413 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.415 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:39:46.422 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:46.422 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:39:46.442 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:39:46.444 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:46.447 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:46.448 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.449 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.449 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:39:46.454 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:39:46.456 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.457 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.457 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:46.458 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.458 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:39:46.458 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:46.459 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:39:46.459 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:39:46.462 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:39:46.464 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:39:46.468 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:39:46.469 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:39:46.469 INFO 2981023 --- [host-check-host-status4,955e7a1aecada377,d3ea07c08fac7de3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:39:46.469 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:39:46.470 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:39:46.474 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.485 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.485 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.486 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.486 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.486 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:39:46.486 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:39:46.486 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:39:46.487 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:39:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.487 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:39:46.489 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:39:46.489 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:39:46.490 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:39:46.490 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.490 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:39:46.491 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:39:46.491 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.496 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:39:46.501 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:46.501 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:39:46.516 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:39:46.518 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:39:46.526 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.526 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.527 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:39:46.527 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:39:46.533 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:39:46.535 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.537 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.538 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:39:46.538 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:39:46.538 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.540 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:39:46.541 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:39:46.552 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:39:46.553 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:39:46.560 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.562 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:39:46.563 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.564 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:39:46.569 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:39:46.572 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:39:46.572 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:39:46.574 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:39:46.576 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.576 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.577 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:39:46.583 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:39:46.583 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726046989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:39:46.588 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.589 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.589 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:39:46.590 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:39:46.590 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.591 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:39:46.597 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:46.598 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:46.598 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:39:46.604 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:39:46.604 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047012660000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:39:46.605 INFO 2981023 --- [ocp-schedules-25,22b71b11d85d9793,bc596c6d3f3aaf10] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:39:46.622 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:39:46.349114+08:00 2024-09-11 17:39:46.635 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:39:46.651 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:39:46.657 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:39:46.660 INFO 2981023 --- [ocp-schedules-21,dcba107c22c9ed19,b57d99000b025426] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:39:46.838 INFO 2981023 --- [ocp-schedules-26,47d8661d142663aa,1ecca5248791c1eb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8192,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:39:46.838 INFO 2981023 --- [ocp-schedules-26,47d8661d142663aa,1ecca5248791c1eb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:39:46.948 INFO 2981023 --- [host-check-clockdiff2,1d5157399e203d5d,152cc53161848539] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047586 0 0 , error= 2024-09-11 17:39:47.246 INFO 2981023 --- [ocp-async-1,75a3a095565d3ccd,9738b8624f2b2cb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:47.289 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:47.289 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,3c2939923db6b317,8ce9711bebbe44a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,3c2939923db6b317,8ce9711bebbe44a6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:47.756 INFO 2981023 --- [ocp-async-3,da1e37202227a676,4f4b87faf44243f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:39:48.024 INFO 2981023 --- [ocp-schedules-5,33c517d06b6f9c28,41ff3707fd93bb45] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:39:48.150 WARN 2981023 --- [scanner-13523837-1,33c517d06b6f9c28,3f12fe993bd465dc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:39:48.220 INFO 2981023 --- [alarm-detect-16,33c517d06b6f9c28,e531197fa085c124] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644353, data is empty. 2024-09-11 17:39:48.222 INFO 2981023 --- [alarm-detect-16,33c517d06b6f9c28,e531197fa085c124] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644353, data is empty. 2024-09-11 17:39:48.224 INFO 2981023 --- [alarm-detect-16,33c517d06b6f9c28,e531197fa085c124] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644353, data is empty. 2024-09-11 17:39:48.304 INFO 2981023 --- [alarm-detect-20,33c517d06b6f9c28,9da680d424a3ea20] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644353, data is empty. 2024-09-11 17:39:48.312 INFO 2981023 --- [alarm-detect-20,33c517d06b6f9c28,9da680d424a3ea20] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644353, data is empty. 2024-09-11 17:39:48.315 INFO 2981023 --- [alarm-detect-20,33c517d06b6f9c28,9da680d424a3ea20] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644353, data is empty. 2024-09-11 17:39:48.326 INFO 2981023 --- [alarm-detect-14,33c517d06b6f9c28,c2a59da00407e138] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644353, data is empty. 2024-09-11 17:39:48.330 INFO 2981023 --- [alarm-detect-14,33c517d06b6f9c28,c2a59da00407e138] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644353, data is empty. 2024-09-11 17:39:48.334 INFO 2981023 --- [alarm-detect-14,33c517d06b6f9c28,c2a59da00407e138] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644353, data is empty. 2024-09-11 17:39:48.726 INFO 2981023 --- [alarm-detect-15,33c517d06b6f9c28,922627e338370394] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644353, data is empty. 2024-09-11 17:39:48.728 INFO 2981023 --- [alarm-detect-15,33c517d06b6f9c28,922627e338370394] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644353, data is empty. 2024-09-11 17:39:48.730 INFO 2981023 --- [alarm-detect-15,33c517d06b6f9c28,922627e338370394] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644353, data is empty. 2024-09-11 17:39:48.743 INFO 2981023 --- [ocp-schedules-23,f790ba57f83b3f51,89239a66c1c8946b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:39:49.246 INFO 2981023 --- [ocp-async-10,4583a1a8915d3f89,baf1dd09fdf411c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:49.295 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,2323ae3006712c25,546b063d1e43f251] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&version=2&ObClusterId=1719994437, client=10.31.35.10, traceId=2323ae3006712c25, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkprdocp,1719994437,2,{"ObRegion":"linkprdocp","ObCluster":"linkprdocp","ObRegionId":1719994437,"ObClusterId":1719994437,"Type":"PRIMARY","timestamp":1726047589294079,"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:39:49.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,2323ae3006712c25,546b063d1e43f251] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkprdocp, obClusterId=1719994437, rootServerJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, groupName=linkprdocp, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:39:49.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,2323ae3006712c25,546b063d1e43f251] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:39:49.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,2323ae3006712c25,546b063d1e43f251] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkprdocp, clusterId=1719994437, affectRows=1 2024-09-11 17:39:49.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,2323ae3006712c25,546b063d1e43f251] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.10, traceId=2323ae3006712c25, duration=10 ms] 2024-09-11 17:39:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:39:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:39:50.233 INFO 2981023 --- [ocp-schedules-8,ccf791065bc340de,e5eeb8ea7e57a9db] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:39:50.289 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:50.290 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,d46c3312f4b5f549,807c2496ed0313cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,d46c3312f4b5f549,807c2496ed0313cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:51.246 INFO 2981023 --- [ocp-async-4,4df6e1e38056a85d,38c1f4de74c839ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:51.548 INFO 2981023 --- [ocp-schedules-30,81c26cde6411a3a6,3c92a831201ad541] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:39:51.549 INFO 2981023 --- [ocp-schedules-30,81c26cde6411a3a6,3c92a831201ad541] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:39:51.549 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:39:51.549 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:39:51.549 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:39:51.549 INFO 2981023 --- [ocp-schedules-30,81c26cde6411a3a6,3c92a831201ad541] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:39:52.002 INFO 2981023 --- [host-check-clockdiff3,1d5157399e203d5d,24a3f9f4b4787a6d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047591 1 1 , error= 2024-09-11 17:39:52.025 INFO 2981023 --- [ocp-schedules-8,41b770cb0ac35148,538b4bf06910d77c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:39:52.906 INFO 2981023 --- [ocp-schedules-14,66bfc0d6ac573280,6fb9cf6529f764d4] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:39:52.923 WARN 2981023 --- [ocp-schedules-14,66bfc0d6ac573280,6fb9cf6529f764d4] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:39:52.951 INFO 2981023 --- [ocp-schedules-19,eb73d1ad37cc7a07,da71923277fa3ca5] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:39:52.951+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:39:52.954 INFO 2981023 --- [ocp-schedules-19,eb73d1ad37cc7a07,da71923277fa3ca5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:39:53.007 WARN 2981023 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=10.31.35.10:8080, expireInSeconds=30, createTime=2024-09-11T16:02:52+08:00, updateTime=2024-09-11T17:39:22+08:00, expireTime=null, isLeader=false} 2024-09-11 17:39:53.010 WARN 2981023 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2024-09-11 17:39:53.104 INFO 2981023 --- [ocp-schedules-22,0a250bfa5bee9533,5e38941007246214] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:39:53.104 INFO 2981023 --- [ocp-schedules-22,0a250bfa5bee9533,5e38941007246214] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:39:53.106 INFO 2981023 --- [ocp-schedules-19,eb73d1ad37cc7a07,da71923277fa3ca5] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 41 to SqlAggregateData 2024-09-11 17:39:53.108 INFO 2981023 --- [ocp-schedules-19,eb73d1ad37cc7a07,da71923277fa3ca5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:39:53.246 INFO 2981023 --- [ocp-async-2,83c45c49ad8c6c82,f9616e49e70d2153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:53.290 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:53.290 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,5f4ded373de3d1dc,93453ced04726ac3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:53.315 INFO 2981023 --- [prometheus-minute-schedule-1,5f4ded373de3d1dc,93453ced04726ac3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:53.820 INFO 2981023 --- [ocp-schedules-25,b13bab739b62d77b,2f302f117c04d7f7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:39:53.821 INFO 2981023 --- [ocp-schedules-25,b13bab739b62d77b,2f302f117c04d7f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:39:53.850 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:39:53.850 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.850 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:39:53.851 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.851 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:39:53.853 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:39:53.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.853 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:39:53.856 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:39:53.861 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.862 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.863 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.863 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.864 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.864 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.871 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.871 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.872 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.873 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.876 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.876 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.878 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.878 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:39:53.888 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.889 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:39:53.890 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.890 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.890 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.890 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.891 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:39:53.891 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2CF1D6DA-0-0] 2024-09-11 17:39:53.891 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:39:53.891 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.892 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:39:53.895 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.895 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:39:53.900 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.900 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:39:53.901 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.901 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:39:53.902 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.902 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:39:53.902 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.902 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:39:53.903 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:39:53.904 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.904 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:39:53.910 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:39:53.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.916 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:39:53.916 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.916 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.917 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:39:53.917 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:39:53.917 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.917 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:39:53.917 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2CF1D6E4-0-0] 2024-09-11 17:39:53.918 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:39:53.918 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:39:53.918 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.919 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:39:53.919 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.920 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:39:53.920 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:39:53.922 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.922 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:53.922 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:39:53.923 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.923 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.923 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.924 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.924 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:39:53.925 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.925 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:39:53.926 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.926 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:39:53.926 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:39:53.927 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.927 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.927 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:39:53.927 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:39:53.928 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:39:53.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:39:53.939 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:53.940 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:39:53.941 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.941 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.941 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:39:53.941 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:39:53.942 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.942 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:39:53.943 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:39:53.943 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:39:53.943 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:39:53.943 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:39:53.944 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:39:53.948 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:39:53.949 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:39:53.949 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.950 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:39:53.955 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:39:53.955 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:39:53.957 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.957 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:39:53.957 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:39:53.957 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.957 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.962 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:39:53.963 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:39:53.963 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:39:53.963 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:39:53.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:39:53.967 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:39:53.967 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:53.967 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:39:53.975 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:39:53.976 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:39:53.976 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:39:53.976 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:39:53.976 INFO 2981023 --- [ocp-schedules-25,b13bab739b62d77b,2f302f117c04d7f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:39:54.071 INFO 2981023 --- [ocp-schedules-6,879332710dc2f058,353b21b5a797eca6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:39:54.080 INFO 2981023 --- [ocp-schedules-6,879332710dc2f058,353b21b5a797eca6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:39:54.080 INFO 2981023 --- [ocp-schedules-6,879332710dc2f058,353b21b5a797eca6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:39:54.244 INFO 2981023 --- [ocp-schedules-5,878b37b65b7086fa,4564a61a1471ad68] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:39:54.776 INFO 2981023 --- [ocp-schedules-16,36f13ad59ee486da,45c4359a076682c0] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:39:54.776 INFO 2981023 --- [ocp-schedules-16,36f13ad59ee486da,45c4359a076682c0] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:39:54.779 INFO 2981023 --- [ocp-schedules-16,36f13ad59ee486da,45c4359a076682c0] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:39:54.779 INFO 2981023 --- [ocp-schedules-16,36f13ad59ee486da,45c4359a076682c0] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:39:54.779 INFO 2981023 --- [ocp-schedules-16,36f13ad59ee486da,45c4359a076682c0] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:39:54.782 INFO 2981023 --- [ocp-schedules-16,36f13ad59ee486da,45c4359a076682c0] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:39:54.782 INFO 2981023 --- [ocp-schedules-16,36f13ad59ee486da,45c4359a076682c0] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:39:54.782 INFO 2981023 --- [ocp-schedules-16,36f13ad59ee486da,45c4359a076682c0] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:39:54.790 INFO 2981023 --- [ocp-schedules-16,36f13ad59ee486da,45c4359a076682c0] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:39:55.246 INFO 2981023 --- [ocp-async-7,6897c0194be67b39,f7c7d975e04f2b25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:55.690 INFO 2981023 --- [ocp-schedules-9,04b33cfbf03fa24b,944c49509ec5c290] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:39:55.690 INFO 2981023 --- [ocp-schedules-9,04b33cfbf03fa24b,944c49509ec5c290] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:39:55.693 INFO 2981023 --- [ocp-schedules-9,04b33cfbf03fa24b,944c49509ec5c290] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:39:55.693 INFO 2981023 --- [ocp-schedules-9,04b33cfbf03fa24b,944c49509ec5c290] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:39:55.695 INFO 2981023 --- [ocp-schedules-9,04b33cfbf03fa24b,944c49509ec5c290] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:39:55.695 INFO 2981023 --- [ocp-schedules-9,04b33cfbf03fa24b,944c49509ec5c290] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:39:55.790 INFO 2981023 --- [ocp-schedules-5,711372acccdbeac2,e34ea1837e846d03] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:39:55.797 INFO 2981023 --- [ocp-schedules-5,711372acccdbeac2,e34ea1837e846d03] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:39:55.797 INFO 2981023 --- [ocp-schedules-5,711372acccdbeac2,e34ea1837e846d03] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:39:56.290 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:56.291 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,f79d5f5dcf1a11fc,3bb44cc21d1d9e66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:56.316 INFO 2981023 --- [ocp-schedules-29,e85718f315a28354,0b0e2ebf27c0d966] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:39:56.316 INFO 2981023 --- [ocp-schedules-29,e85718f315a28354,0b0e2ebf27c0d966] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:39:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,f79d5f5dcf1a11fc,3bb44cc21d1d9e66] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:39:56.336 INFO 2981023 --- [Collect all tenants major compactions1,e85718f315a28354,41e86d316c875187] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:39:56.337 INFO 2981023 --- [Collect all tenants major compactions2,e85718f315a28354,42c0814c1fdb3f61] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:39:56.339 INFO 2981023 --- [Collect all tenants major compactions3,e85718f315a28354,7eb5eac7355cdcf4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:39:56.340 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:39:56.340239+08:00, lastProcessTime=2024-09-11T17:39:46.349114+08:00 2024-09-11 17:39:56.340 INFO 2981023 --- [Collect all tenants major compactions5,e85718f315a28354,c4a676e578d0fb03] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:39:56.342 INFO 2981023 --- [Collect all tenants major compactions6,e85718f315a28354,fa1e765b094b8ad6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:39:56.342 INFO 2981023 --- [Collect all tenants major compactions4,e85718f315a28354,a952e29cdf9b1959] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:39:56.355 INFO 2981023 --- [Collect all tenants major compactions1,e85718f315a28354,41e86d316c875187] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:56.356 INFO 2981023 --- [Collect all tenants major compactions1,e85718f315a28354,41e86d316c875187] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:39:56.357 INFO 2981023 --- [Collect all tenants major compactions5,e85718f315a28354,c4a676e578d0fb03] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:56.357 INFO 2981023 --- [Collect all tenants major compactions3,e85718f315a28354,7eb5eac7355cdcf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:56.358 INFO 2981023 --- [Collect all tenants major compactions5,e85718f315a28354,c4a676e578d0fb03] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:39:56.358 INFO 2981023 --- [Collect all tenants major compactions2,e85718f315a28354,42c0814c1fdb3f61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:56.359 INFO 2981023 --- [Collect all tenants major compactions4,e85718f315a28354,a952e29cdf9b1959] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:56.360 INFO 2981023 --- [Collect all tenants major compactions2,e85718f315a28354,42c0814c1fdb3f61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:39:56.360 INFO 2981023 --- [Collect all tenants major compactions4,e85718f315a28354,a952e29cdf9b1959] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:39:56.361 INFO 2981023 --- [Collect all tenants major compactions1,e85718f315a28354,41e86d316c875187] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:39:56.362 INFO 2981023 --- [Collect all tenants major compactions3,e85718f315a28354,7eb5eac7355cdcf4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:39:56.362 INFO 2981023 --- [Collect all tenants major compactions2,e85718f315a28354,42c0814c1fdb3f61] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:39:56.362 INFO 2981023 --- [Collect all tenants major compactions6,e85718f315a28354,fa1e765b094b8ad6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:39:56.363 INFO 2981023 --- [Collect all tenants major compactions6,e85718f315a28354,fa1e765b094b8ad6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:39:56.364 INFO 2981023 --- [Collect all tenants major compactions5,e85718f315a28354,c4a676e578d0fb03] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:39:56.365 INFO 2981023 --- [Collect all tenants major compactions1,e85718f315a28354,41e86d316c875187] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:39:56.366 INFO 2981023 --- [Collect all tenants major compactions3,e85718f315a28354,7eb5eac7355cdcf4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:39:56.366 INFO 2981023 --- [Collect all tenants major compactions3,e85718f315a28354,7eb5eac7355cdcf4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:39:56.366 INFO 2981023 --- [Collect all tenants major compactions2,e85718f315a28354,42c0814c1fdb3f61] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:39:56.367 INFO 2981023 --- [Collect all tenants major compactions6,e85718f315a28354,fa1e765b094b8ad6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:39:56.367 INFO 2981023 --- [Collect all tenants major compactions6,e85718f315a28354,fa1e765b094b8ad6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:39:56.368 INFO 2981023 --- [Collect all tenants major compactions5,e85718f315a28354,c4a676e578d0fb03] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:39:56.369 INFO 2981023 --- [Collect all tenants major compactions4,e85718f315a28354,a952e29cdf9b1959] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:39:56.369 INFO 2981023 --- [Collect all tenants major compactions4,e85718f315a28354,a952e29cdf9b1959] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:39:56.369 INFO 2981023 --- [ocp-schedules-29,e85718f315a28354,0b0e2ebf27c0d966] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:39:56.480 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:39:56.340239+08:00 2024-09-11 17:39:57.246 INFO 2981023 --- [ocp-async-6,773a9d3e5c1e8c1b,f792ab4338a97c4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:57.756 INFO 2981023 --- [ocp-async-8,4ba7dd775553303f,c8fb80e384df43fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:39:58.957 INFO 2981023 --- [ocp-schedules-20,c9787ed6f4eff21a,02b9b24b2614d4e8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:39:59.000 WARN 2981023 --- [scanner-13524121-1,c9787ed6f4eff21a,acbccd6508a17cd6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:39:59.246 INFO 2981023 --- [ocp-async-1,f927449c0e1d73dd,fd1f4fc77b5771fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:39:59.291 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:39:59.292 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:39:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,eacb4e577303cad2,315c61db712b8d28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047600, interval=60 2024-09-11 17:39:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,eacb4e577303cad2,315c61db712b8d28] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:00.718 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:35:00.0, nextRun=2024-09-11 17:40:00.0, currentDate=Wed Sep 11 17:40:00 HKT 2024 2024-09-11 17:40:00.718 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:35:00.0, nextRunTime=2024-09-11 17:40:00.0} 2024-09-11 17:40:00.724 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:40:00.724 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:40:00.724 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:40:00.724+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:40:00.738 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:35:00.0, nextRun=2024-09-11 17:40:00.0, currentDate=Wed Sep 11 17:40:00 HKT 2024 2024-09-11 17:40:00.738 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:35:00.0, nextRunTime=2024-09-11 17:40:00.0} 2024-09-11 17:40:00.756 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2, 1000006] 2024-09-11 17:40:00.765 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:40:00.766 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:40:00.766 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:40:00.766 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,154e1fac041f1ab6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2,1000006]]}}, createTime=null, startTime=2024-09-11T17:40:00.766+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:40:01.216 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,afd649d391fde0cc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048167, operation:EXECUTE, state: RUNNING 2024-09-11 17:40:01.249 INFO 2981023 --- [ocp-async-7,32dbc4c05d7fff39,35d882710b9cdcec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:01.302 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,afd649d391fde0cc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048168, operation:EXECUTE, state: RUNNING 2024-09-11 17:40:01.340 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048167, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048187, task_operation=execute, latest_execution_start_time=2024-09-11T17:40:01.181+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048167}, listMap={}}, executor=10.31.35.10 2024-09-11 17:40:01.354 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048168, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048188, task_operation=execute, latest_execution_start_time=2024-09-11T17:40:01.281+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048168}, listMap={cluster_ids=[[1,2,1000006]]}}, executor=10.31.35.10 2024-09-11 17:40:01.356 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:40:01.391 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:40:01.407 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:40:01.413 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:40:01.427 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:40:01.428 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:40:01.428 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:40:01.445 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:40:01.457 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.perf.sql.internal.SqlParseService : parse 4 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:40:01.511 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:40:01.557 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1000006 2024-09-11 17:40:01.573 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e1d97f84f602a683] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048168, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:40:01.578 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:40:01.594 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:40:01.613 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:40:01.619 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:40:01.619 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:40:01.623 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:40:01.718 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:40:01.770 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:40:01.793 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:40:01.809 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048188, status:SUCCESS 2024-09-11 17:40:01.813 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:40:01.817 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048188, status:SUCCESS 2024-09-11 17:40:01.856 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:40:01.869 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:40:01.881 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:40:01.887 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:40:01.887 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:40:01.892 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:40:01.955 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:40:01.987 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:40:01.999 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:40:02.006 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:40:02.016 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:40:02.021 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:40:02.027 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:40:02.029 INFO 2981023 --- [ocp-schedules-27,bea141e03a692edd,a15ff966eec14d9f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:40:02.030 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:40:02.031 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:40:02.052 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:40:02.061 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:40:02.073 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,009fd21ffedde126] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048167, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:40:02.292 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:02.293 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,89ea1d56db86f60d,7cce1fc83382cfc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,89ea1d56db86f60d,7cce1fc83382cfc6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:02.467 INFO 2981023 --- [ocp-schedules-9,91385f3764ace87a,2e20072db7005fc1] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:40:02.468 INFO 2981023 --- [ocp-schedules-9,91385f3764ace87a,2e20072db7005fc1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:40:02.476 INFO 2981023 --- [ocp-schedules-9,91385f3764ace87a,2e20072db7005fc1] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:40:02.827 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048187, status:SUCCESS 2024-09-11 17:40:02.828 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048187, status:SUCCESS 2024-09-11 17:40:03.246 INFO 2981023 --- [ocp-async-3,74ccaf13a1e35072,a28577f10b7ab032] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:04.872 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a1d81a099ec5e202] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:30:04.0, nextRun=2024-09-11 17:40:04.0, currentDate=Wed Sep 11 17:40:04 HKT 2024 2024-09-11 17:40:04.872 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a1d81a099ec5e202] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-09-11 17:30:04.0, nextRunTime=2024-09-11 17:40:04.0} 2024-09-11 17:40:04.878 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a1d81a099ec5e202] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-09-11 17:40:04.878 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a1d81a099ec5e202] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:40:04.878 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,a1d81a099ec5e202] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:40:04.878+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:40:05.247 INFO 2981023 --- [ocp-async-5,ec05ab9fe285bb66,081ea6450825afc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:05.293 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:05.295 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,1139182229ea376c,a5a197d0e994194a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,1139182229ea376c,a5a197d0e994194a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:05.344 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,05fa2c8b4f225363] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048169, operation:EXECUTE, state: RUNNING 2024-09-11 17:40:05.367 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048169, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048189, task_operation=execute, latest_execution_start_time=2024-09-11T17:40:05.342+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=4048169}, listMap={}}, executor=10.31.35.10 2024-09-11 17:40:05.373 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 17:40:05.373 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 17:40:05.374 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.374 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:40:05.375 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.375 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.376 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.376 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 17:40:05.387 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.387 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.388 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.388 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 17:40:05.397 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.398 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.398 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.399 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 17:40:05.407 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.407 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.408 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.408 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 17:40:05.417 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.417 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.418 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.418 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 17:40:05.421 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.422 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.422 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.422 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 17:40:05.425 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.426 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.426 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.426 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 17:40:05.429 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.429 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.430 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.430 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 17:40:05.434 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.434 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.434 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.435 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 17:40:05.438 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.438 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.439 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.439 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 17:40:05.442 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.442 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.443 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.443 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 17:40:05.449 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.449 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.450 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.450 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 17:40:05.459 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.459 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.460 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.460 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 17:40:05.464 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.464 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.465 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.465 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 17:40:05.469 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.470 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.470 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.470 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 17:40:05.473 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.474 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.474 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.475 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 17:40:05.478 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.478 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.478 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.479 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 17:40:05.481 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.482 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.482 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.482 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 17:40:05.485 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.485 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.486 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.486 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 17:40:05.489 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.490 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.490 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.490 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 17:40:05.493 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.493 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.493 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.494 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 17:40:05.497 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.497 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.497 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.498 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 17:40:05.500 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.500 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.501 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.501 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 17:40:05.504 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.504 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.505 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.505 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 17:40:05.507 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.508 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.508 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.509 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 17:40:05.511 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.512 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.512 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.512 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 17:40:05.516 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.516 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.517 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.517 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 17:40:05.521 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.521 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.522 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.522 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 17:40:05.525 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.526 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.526 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.527 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 17:40:05.530 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.531 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.531 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.532 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 17:40:05.535 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.536 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.536 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.536 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 17:40:05.540 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.541 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.541 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.542 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 17:40:05.545 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.545 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.546 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.546 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 17:40:05.550 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.550 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.551 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.551 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 17:40:05.558 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.558 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.559 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.559 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 17:40:05.564 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.564 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.564 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.565 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 17:40:05.568 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.569 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.569 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.569 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 17:40:05.574 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.574 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.575 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.575 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 17:40:05.577 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.578 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.578 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.579 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 17:40:05.582 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.582 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.583 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.583 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 17:40:05.586 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.587 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.588 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.588 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 17:40:05.591 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.591 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.592 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.592 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 17:40:05.601 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.602 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.602 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.603 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 17:40:05.618 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.619 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.619 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.620 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 17:40:05.624 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.625 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.625 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.625 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 17:40:05.631 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.631 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.631 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.632 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 17:40:05.637 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.637 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.637 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.638 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 17:40:05.640 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.640 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.641 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.641 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 17:40:05.643 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.644 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.644 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.644 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 17:40:05.647 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 17:40:05.647 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 17:40:05.651 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.652 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:40:05.653 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.653 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.654 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.654 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 17:40:05.658 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.658 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:05.659 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.659 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 17:40:05.661 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 17:40:05.671 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,5a897cfec4db66ab] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048169, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:40:05.847 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048189, status:SUCCESS 2024-09-11 17:40:05.847 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048189, status:SUCCESS 2024-09-11 17:40:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:40:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:40:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:40:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:40:05.995 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:40:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:40:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:40:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:40:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:40:06.007 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:06.008 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:40:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:40:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:06.010 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:06.010 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:40:06.060 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:37:40, start=2024-09-11T17:35:10, end=2024-09-11T17:37:40, updates=129, granule=120000ms, delay=150000ms 2024-09-11 17:40:06.065 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:37:40, target=2024-09-11T17:37:40, granule=120000ms, delay=0ms) 2024-09-11 17:40:06.065 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:37:40, target=2024-09-11T17:29:31, granule=600000ms, delay=489000ms) 2024-09-11 17:40:06.142 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:37:41, start=2024-09-11T17:35:10, end=2024-09-11T17:37:40, updates=318, granule=120000ms, delay=151000ms 2024-09-11 17:40:06.149 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:37:41, target=2024-09-11T17:37:40, granule=120000ms, delay=1000ms) 2024-09-11 17:40:06.149 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:37:40, target=2024-09-11T17:30:51, granule=600000ms, delay=409000ms) 2024-09-11 17:40:06.149 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=179ms, period=120000ms 2024-09-11 17:40:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:40:06.351796+08:00, lastProcessTime=2024-09-11T17:39:56.340239+08:00 2024-09-11 17:40:06.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:40:06.351796+08:00 2024-09-11 17:40:07.247 INFO 2981023 --- [ocp-async-8,e88e69fb9ace4f1c,07056ba9050a1058] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:07.398 INFO 2981023 --- [ocp-schedules-21,f128e8b81e4e5d0c,5b109b7c23e8da07] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:40:07.410 INFO 2981023 --- [ocp-schedules-21,e6a39356f4d51437,b48b172f37ad8a64] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:40:07.410+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:40:07.410 INFO 2981023 --- [ocp-schedules-21,e6a39356f4d51437,b48b172f37ad8a64] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:40:07.412 INFO 2981023 --- [ocp-schedules-21,e6a39356f4d51437,b48b172f37ad8a64] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:40:07.412 INFO 2981023 --- [ocp-schedules-21,e6a39356f4d51437,b48b172f37ad8a64] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:40:07.756 INFO 2981023 --- [ocp-async-5,04e3904b159335ec,913f88d02687f998] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:40:08.295 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:08.295 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,cce0fe4c4bee9029,eb21f6f2e43746e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,cce0fe4c4bee9029,eb21f6f2e43746e4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:09.172 INFO 2981023 --- [ocp-schedules-28,7241b52f3a2aca19,bf9df0858a40a9d4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:40:09.191 WARN 2981023 --- [scanner-13524516-1,7241b52f3a2aca19,5a3eba7be167d59f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:40:09.247 INFO 2981023 --- [ocp-async-6,5634e8d3a58c049e,121e1f385a8919ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:09.255 INFO 2981023 --- [alarm-detect-14,7241b52f3a2aca19,2b739b14bf95a2be] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644353, data is empty. 2024-09-11 17:40:09.258 INFO 2981023 --- [alarm-detect-14,7241b52f3a2aca19,2b739b14bf95a2be] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644394, data is empty. 2024-09-11 17:40:09.263 INFO 2981023 --- [alarm-detect-14,7241b52f3a2aca19,2b739b14bf95a2be] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644394, data is empty. 2024-09-11 17:40:09.279 INFO 2981023 --- [alarm-detect-10,7241b52f3a2aca19,1bc8565d55b752f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644353, data is empty. 2024-09-11 17:40:09.281 INFO 2981023 --- [alarm-detect-16,7241b52f3a2aca19,f3f61202ac07aafb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644353, data is empty. 2024-09-11 17:40:09.285 INFO 2981023 --- [alarm-detect-10,7241b52f3a2aca19,1bc8565d55b752f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644394, data is empty. 2024-09-11 17:40:09.288 INFO 2981023 --- [alarm-detect-16,7241b52f3a2aca19,f3f61202ac07aafb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644394, data is empty. 2024-09-11 17:40:09.291 INFO 2981023 --- [alarm-detect-10,7241b52f3a2aca19,1bc8565d55b752f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644394, data is empty. 2024-09-11 17:40:09.293 INFO 2981023 --- [alarm-detect-16,7241b52f3a2aca19,f3f61202ac07aafb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644394, data is empty. 2024-09-11 17:40:09.390 INFO 2981023 --- [alarm-detect-18,7241b52f3a2aca19,72e08224aba8d58a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644353, data is empty. 2024-09-11 17:40:09.395 INFO 2981023 --- [alarm-detect-18,7241b52f3a2aca19,72e08224aba8d58a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644394, data is empty. 2024-09-11 17:40:09.399 INFO 2981023 --- [alarm-detect-18,7241b52f3a2aca19,72e08224aba8d58a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644394, data is empty. 2024-09-11 17:40:10.003 INFO 2981023 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:40:10.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:40:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:40:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:40:11.247 INFO 2981023 --- [ocp-async-2,7742c1ef98739cd6,d97bbdef5d71423f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:11.295 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:11.296 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,5a0de9c3c084ef44,675a2090e8ad8006] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,5a0de9c3c084ef44,675a2090e8ad8006] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:12.037 INFO 2981023 --- [ocp-schedules-5,f8c6ba1857dfc015,2856629e43b88ccc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:40:12.665 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,33751ad3d6bb1ec0,bec7b4689201119a] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&version=2&ObClusterId=1718689631, client=10.31.35.5, traceId=33751ad3d6bb1ec0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=rtdob,1718689631,2,{"ObRegion":"rtdob","ObCluster":"rtdob","ObRegionId":1718689631,"ObClusterId":1718689631,"Type":"PRIMARY","timestamp":1726047612658561,"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:40:12.675 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,33751ad3d6bb1ec0,bec7b4689201119a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=rtdob, obClusterId=1718689631, rootServerJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, groupName=rtdob, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:40:12.675 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,33751ad3d6bb1ec0,bec7b4689201119a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:40:12.675 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,33751ad3d6bb1ec0,bec7b4689201119a] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=rtdob, clusterId=1718689631, affectRows=1 2024-09-11 17:40:12.676 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,33751ad3d6bb1ec0,bec7b4689201119a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.5, traceId=33751ad3d6bb1ec0, duration=12 ms] 2024-09-11 17:40:12.681 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047540, nodeCount=1000, affectRows=1000 2024-09-11 17:40:12.683 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047540, nodeCount=1000, validCount=1000 2024-09-11 17:40:12.702 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047540, nodeCount=48, affectRows=48 2024-09-11 17:40:12.702 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047540, nodeCount=48, validCount=48 2024-09-11 17:40:12.705 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047540, nodeCount=1000, affectRows=1000 2024-09-11 17:40:12.707 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047540, nodeCount=999, validCount=999 2024-09-11 17:40:13.247 INFO 2981023 --- [ocp-async-9,7e60aa72b982144b,9769d755390e92c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:14.083 INFO 2981023 --- [ocp-schedules-19,6a9130fa24e9dd78,0a3e8f24cbb8d3ce] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:40:14.093 INFO 2981023 --- [ocp-schedules-19,6a9130fa24e9dd78,0a3e8f24cbb8d3ce] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:40:14.093 INFO 2981023 --- [ocp-schedules-19,6a9130fa24e9dd78,0a3e8f24cbb8d3ce] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:40:14.296 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:14.297 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,b1b05730af63fded,26047e483f3b6f68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,7052c74aafa9c4ca,70f42886972492c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:40:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,b1b05730af63fded,26047e483f3b6f68] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:14.610 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047540, nodeCount=480, affectRows=480 2024-09-11 17:40:15.247 INFO 2981023 --- [ocp-async-3,0f9975a8cff0e2f0,addfc861757bec31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:16.022 INFO 2981023 --- [ocp-schedules-8,ba9f0270e732dc8d,b792a3ef8b8d27c8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:40:16.022 INFO 2981023 --- [host-check-clockdiff1,ba9f0270e732dc8d,1552e8eaeb572573] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:40:16.023 INFO 2981023 --- [ocp-schedules-10,7c3fa77affb34b86,a04b3f44e77fa344] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:40:16.023 INFO 2981023 --- [host-check-clockdiff2,ba9f0270e732dc8d,998697fcd4f4e3df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:40:16.023 INFO 2981023 --- [host-check-clockdiff3,ba9f0270e732dc8d,9ff360e658fc9688] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:40:16.023 INFO 2981023 --- [host-check-agent-status1,7c3fa77affb34b86,4f05fa0551418854] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:40:16.023 INFO 2981023 --- [host-check-agent-status3,7c3fa77affb34b86,ba5cc271a4dcdeb5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:40:16.023 INFO 2981023 --- [host-check-agent-status4,7c3fa77affb34b86,888a22ff06701efd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:40:16.024 INFO 2981023 --- [host-check-clockdiff4,ba9f0270e732dc8d,125e20c56beb75b9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:40:16.024 INFO 2981023 --- [host-check-agent-status2,7c3fa77affb34b86,17292f990139779c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:40:16.025 INFO 2981023 --- [ocp-schedules-1,db6a856df63a1668,064187af973bafc9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:40:16.032 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:40:16.032 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:40:16.032 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:40:16.033 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:40:16.046 INFO 2981023 --- [host-check-clockdiff4,ba9f0270e732dc8d,125e20c56beb75b9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047616 0 0 , error= 2024-09-11 17:40:16.055 INFO 2981023 --- [host-check-agent-status2,7c3fa77affb34b86,17292f990139779c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:40:16.056 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:40:16.058 INFO 2981023 --- [host-check-agent-status3,7c3fa77affb34b86,ba5cc271a4dcdeb5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:40:16.059 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:40:16.059 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:40:16.060 INFO 2981023 --- [host-check-agent-status2,7c3fa77affb34b86,17292f990139779c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:40:16.060 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:40:16.060 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:40:16.063 INFO 2981023 --- [host-check-agent-status4,7c3fa77affb34b86,888a22ff06701efd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:40:16.064 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:40:16.064 INFO 2981023 --- [host-check-agent-status1,7c3fa77affb34b86,4f05fa0551418854] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:40:16.064 INFO 2981023 --- [host-check-agent-status3,7c3fa77affb34b86,ba5cc271a4dcdeb5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:40:16.065 INFO 2981023 --- [host-check-agent-status1,7c3fa77affb34b86,4f05fa0551418854] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:40:16.066 INFO 2981023 --- [host-check-agent-status4,7c3fa77affb34b86,888a22ff06701efd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:40:16.196 INFO 2981023 --- [ocp-schedules-3,6e3b316b5aca6a6d,610e6cf912192cdb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:40:16.196 INFO 2981023 --- [ocp-schedules-2,80693805cb1c10ad,427b56e6cdaf3509] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:40:16.197 INFO 2981023 --- [ocp-schedules-25,eff6b6eb23260ecc,ebf89ab1ff36c777] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:40:16.197 INFO 2981023 --- [ocp-schedules-24,7cb188cf0a59b8ca,72136b91b562c2d6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:40:16.264 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:40:16.271 INFO 2981023 --- [ocp-schedules-2,80693805cb1c10ad,427b56e6cdaf3509] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:40:16.271 INFO 2981023 --- [ocp-schedules-25,eff6b6eb23260ecc,ebf89ab1ff36c777] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:40:16.277 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:40:16.278 INFO 2981023 --- [ocp-schedules-3,6e3b316b5aca6a6d,610e6cf912192cdb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:40:16.292 INFO 2981023 --- [ocp-schedules-2,80693805cb1c10ad,427b56e6cdaf3509] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:40:16.368 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:40:16.364952+08:00, lastProcessTime=2024-09-11T17:40:06.351796+08:00 2024-09-11 17:40:16.369 INFO 2981023 --- [ocp-schedules-30,19fd4544b7d79e73,c40a48a30aa6d9f2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:40:16.395 INFO 2981023 --- [host-check-agent-status1,7c3fa77affb34b86,4f05fa0551418854] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:40:16.398 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:40:16.400 INFO 2981023 --- [host-check-agent-status1,7c3fa77affb34b86,4f05fa0551418854] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:40:16.400 INFO 2981023 --- [host-check-agent-status1,7c3fa77affb34b86,4f05fa0551418854] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:40:16.400 INFO 2981023 --- [host-check-agent-status1,7c3fa77affb34b86,4f05fa0551418854] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:40:16.400 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.401 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:16.401 INFO 2981023 --- [host-check-agent-status3,7c3fa77affb34b86,ba5cc271a4dcdeb5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:40:16.401 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.401 INFO 2981023 --- [host-check-agent-status2,7c3fa77affb34b86,17292f990139779c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:40:16.401 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:40:16.401290846+08:00 2024-09-11 17:40:16.402 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:16.402 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:40:16.401+08:00 2024-09-11 17:40:16.402 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:40:16.403 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:40:16.404 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:40:16.401+08:00 and status AVAILABLE 2024-09-11 17:40:16.404 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:40:16.405 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:40:16.404778376+08:00 2024-09-11 17:40:16.405 INFO 2981023 --- [host-check-agent-status2,7c3fa77affb34b86,17292f990139779c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:40:16.405 INFO 2981023 --- [host-check-agent-status2,7c3fa77affb34b86,17292f990139779c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:40:16.405 INFO 2981023 --- [host-check-agent-status2,7c3fa77affb34b86,17292f990139779c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:40:16.405 INFO 2981023 --- [host-check-agent-status3,7c3fa77affb34b86,ba5cc271a4dcdeb5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:40:16.405 INFO 2981023 --- [host-check-agent-status3,7c3fa77affb34b86,ba5cc271a4dcdeb5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:40:16.405 INFO 2981023 --- [host-check-agent-status3,7c3fa77affb34b86,ba5cc271a4dcdeb5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:40:16.406 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:40:16.405+08:00 2024-09-11 17:40:16.407 INFO 2981023 --- [host-check-agent-status4,7c3fa77affb34b86,888a22ff06701efd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:40:16.409 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:40:16.405+08:00 and status AVAILABLE 2024-09-11 17:40:16.409 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:40:16.411 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:40:16.411109568+08:00 2024-09-11 17:40:16.412 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:40:16.411+08:00 2024-09-11 17:40:16.412 INFO 2981023 --- [host-check-agent-status4,7c3fa77affb34b86,888a22ff06701efd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:40:16.412 INFO 2981023 --- [host-check-agent-status4,7c3fa77affb34b86,888a22ff06701efd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:40:16.412 INFO 2981023 --- [host-check-agent-status4,7c3fa77affb34b86,888a22ff06701efd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:40:16.413 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:40:16.413 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:40:16.414 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.414 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:40:16.411+08:00 and status AVAILABLE 2024-09-11 17:40:16.414 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:40:16.413860745+08:00 2024-09-11 17:40:16.414 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:16.416 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:40:16.416 INFO 2981023 --- [ocp-schedules-3,6e3b316b5aca6a6d,610e6cf912192cdb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:40:16.417 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:40:16.414+08:00 2024-09-11 17:40:16.453 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:40:16.455 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:40:16.414+08:00 and status AVAILABLE 2024-09-11 17:40:16.456 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:40:16.457 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.458 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.459 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:16.460 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:16.474 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:40:16.475 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:40:16.480 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.481 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:16.486 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:40:16.492 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:40:16.493 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:40:16.494 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:40:16.496 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:40:16.496 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:40:16.500 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:40:16.502 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:40:16.516 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:40:16.554 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:40:16.556 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:40:16.558 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:40:16.564 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:40:16.570 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:40:16.579 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:40:16.587 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:40:16.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.588 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:16.589 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:40:16.590 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.592 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:40:16.593 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:40:16.594 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:40:16.599 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:40:16.600 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:40:16.607 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.610 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.610 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:40:16.612 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:16.612 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:40:16.613 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:40:16.615 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:40:16.634 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:40:16.635 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.636 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.636 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:40:16.636 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:16.639 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:16.645 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.648 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:40:16.648 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:40:16.651 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.651 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.652 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:16.652 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:40:16.652 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:40:16.656 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:40:16.662 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:40:16.663 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:40:16.664 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:40:16.666 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:40:16.667 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:16.667 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:40:16.670 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:40:16.678 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:40:16.681 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:16.682 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:40:16.685 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:40:16.688 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.689 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.689 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:40:16.689 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:16.690 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:40:16.693 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:40:16.694 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:40:16.697 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:40:16.698 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:40:16.698 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:40:16.699 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:40:16.708 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.709 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:40:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.710 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:40:16.711 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:40:16.717 INFO 2981023 --- [host-check-host-status3,db6a856df63a1668,e44618f713c26cd9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:40:16.722 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:40:16.741 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:40:16.742 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047540, nodeCount=480, validCount=480 2024-09-11 17:40:16.744 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:40:16.749 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.751 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.751 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:16.752 INFO 2981023 --- [host-check-host-status4,db6a856df63a1668,89b7c3f818e0fd22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:40:16.752 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:40:16.752 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.755 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:40:16.756 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:40:16.757 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:40:16.771 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:40:16.771 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:40:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:40:16.779 INFO 2981023 --- [host-check-host-status1,db6a856df63a1668,39ed0463a267adb3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:40:16.782 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.783 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.784 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:40:16.786 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:40:16.786 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:40:16.791 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:40:16.791 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:40:16.792 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:16.794 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.795 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.795 INFO 2981023 --- [host-check-host-status2,db6a856df63a1668,812da918655de2a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:40:16.795 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:40:16.798 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:40:16.364952+08:00 2024-09-11 17:40:16.801 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.801 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.801 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:16.802 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:40:16.802 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:40:16.802 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:40:16.804 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:40:16.807 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:40:16.808 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:40:16.819 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:40:16.853 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:40:16.859 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:16.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:16.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:40:16.868 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:40:16.868 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:40:16.868 INFO 2981023 --- [ocp-schedules-25,eff6b6eb23260ecc,ebf89ab1ff36c777] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:40:17.247 INFO 2981023 --- [ocp-async-5,f7f417c0820f9666,1734673c141ece3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:17.297 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:17.297 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,a785fb80c567a3de,808699caf536d02b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,a785fb80c567a3de,808699caf536d02b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:17.756 INFO 2981023 --- [ocp-async-3,1487a6d20609621c,9e9e711f88334db2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:40:17.831 INFO 2981023 --- [host-check-clockdiff2,ba9f0270e732dc8d,998697fcd4f4e3df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047617 0 0 , error= 2024-09-11 17:40:19.248 INFO 2981023 --- [ocp-async-8,a83db1304b819830,f27c97ed8489d1db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:19.443 INFO 2981023 --- [ocp-schedules-1,c9207f2d9cd70cc5,5b06ab1363e3e068] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:40:19.491 WARN 2981023 --- [scanner-13526333-1,c9207f2d9cd70cc5,9870140d1425766d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:40:20.297 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:20.298 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,10cd2b21e9bd780e,d06c40db6caea8c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,10cd2b21e9bd780e,d06c40db6caea8c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:21.248 INFO 2981023 --- [ocp-async-3,3b3e93d591cecf0c,6338c1963354ede8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:21.317 INFO 2981023 --- [host-check-clockdiff1,ba9f0270e732dc8d,1552e8eaeb572573] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047621 0 0 , error= 2024-09-11 17:40:22.007 INFO 2981023 --- [host-check-clockdiff3,ba9f0270e732dc8d,9ff360e658fc9688] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047622 1 1 , error= 2024-09-11 17:40:22.040 INFO 2981023 --- [ocp-schedules-23,23b9813d288cca75,8e1411c5783b919a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:40:22.916 INFO 2981023 --- [ocp-schedules-24,55020bb1af04e010,1a1910c7f4560bf0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:40:22.916 INFO 2981023 --- [ocp-schedules-24,55020bb1af04e010,1a1910c7f4560bf0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:40:22.916 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:40:22.916 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:40:22.917 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:40:22.921 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:40:22.921 INFO 2981023 --- [ocp-schedules-24,55020bb1af04e010,1a1910c7f4560bf0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:40:22.922 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:40:22.922 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:40:22.922 INFO 2981023 --- [ocp-schedules-24,55020bb1af04e010,1a1910c7f4560bf0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:40:22.923 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:40:22.923 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:40:22.924 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:40:22.937 INFO 2981023 --- [ocp-schedules-24,55020bb1af04e010,1a1910c7f4560bf0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:40:22.937 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:40:22.951 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:40:22.953 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:40:22.954 INFO 2981023 --- [ocp-schedules-24,55020bb1af04e010,1a1910c7f4560bf0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:40:23.006 WARN 2981023 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=10.31.35.10:8080, expireInSeconds=30, createTime=2024-09-11T16:04:52+08:00, updateTime=2024-09-11T17:39:52+08:00, expireTime=null, isLeader=false} 2024-09-11 17:40:23.007 WARN 2981023 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2024-09-11 17:40:23.248 INFO 2981023 --- [ocp-async-5,4e628b1fabebe969,5844e9dae9bc2c69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:23.298 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:23.298 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,ead6aa3917ce94b4,34c808f2a2980de1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,ead6aa3917ce94b4,34c808f2a2980de1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:24.247 INFO 2981023 --- [ocp-schedules-19,d430632189646717,a6ca8010ee3d676c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:40:25.248 INFO 2981023 --- [ocp-async-9,1a0235ec107045a3,189ae0fff1a1bac2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:26.298 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:26.299 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,320e220b46545448,1aa1429d64e3d877] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:26.318 INFO 2981023 --- [prometheus-minute-schedule-1,320e220b46545448,1aa1429d64e3d877] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:40:26.338565+08:00, lastProcessTime=2024-09-11T17:40:16.364952+08:00 2024-09-11 17:40:26.468 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:40:26.338565+08:00 2024-09-11 17:40:27.248 INFO 2981023 --- [ocp-async-10,602fe02229b04e3c,e4cd525c74404a8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:27.756 INFO 2981023 --- [ocp-async-9,b33046f8f249221c,7a3eb3cfcbd3c3e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:40:29.248 INFO 2981023 --- [ocp-async-4,90b0188e81d5c1e4,29b44a5ba606ace2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:29.299 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:29.300 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,6bfe1d525a644c03,5790b0b44eb29c4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,6bfe1d525a644c03,5790b0b44eb29c4f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:29.646 INFO 2981023 --- [ocp-schedules-24,636658e52fafdb79,00d86e6ee4eb1b74] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:40:29.707 WARN 2981023 --- [scanner-13526601-1,636658e52fafdb79,71ef904823054889] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:40:29.730 INFO 2981023 --- [alarm-detect-5,636658e52fafdb79,374b58c17fff0afb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644394, data is empty. 2024-09-11 17:40:29.734 INFO 2981023 --- [alarm-detect-5,636658e52fafdb79,374b58c17fff0afb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644394, data is empty. 2024-09-11 17:40:29.740 INFO 2981023 --- [alarm-detect-5,636658e52fafdb79,374b58c17fff0afb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644394, data is empty. 2024-09-11 17:40:29.749 INFO 2981023 --- [alarm-detect-6,636658e52fafdb79,d67cfdace9443586] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644394, data is empty. 2024-09-11 17:40:29.755 INFO 2981023 --- [alarm-detect-6,636658e52fafdb79,d67cfdace9443586] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644394, data is empty. 2024-09-11 17:40:29.756 INFO 2981023 --- [alarm-detect-20,636658e52fafdb79,1b8acb6fe7359a43] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644394, data is empty. 2024-09-11 17:40:29.758 INFO 2981023 --- [alarm-detect-6,636658e52fafdb79,d67cfdace9443586] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644394, data is empty. 2024-09-11 17:40:29.759 INFO 2981023 --- [alarm-detect-20,636658e52fafdb79,1b8acb6fe7359a43] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644394, data is empty. 2024-09-11 17:40:29.764 INFO 2981023 --- [alarm-detect-20,636658e52fafdb79,1b8acb6fe7359a43] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644394, data is empty. 2024-09-11 17:40:29.904 INFO 2981023 --- [alarm-detect-20,636658e52fafdb79,02f66106d7604e38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644394, data is empty. 2024-09-11 17:40:29.910 INFO 2981023 --- [alarm-detect-20,636658e52fafdb79,02f66106d7604e38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644394, data is empty. 2024-09-11 17:40:29.915 INFO 2981023 --- [alarm-detect-20,636658e52fafdb79,02f66106d7604e38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644394, data is empty. 2024-09-11 17:40:31.248 INFO 2981023 --- [ocp-async-1,b401f953a1984c93,c14ad56967a0a1a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:32.042 INFO 2981023 --- [ocp-schedules-29,911ee84ae393ef4a,5cb1666cb3c11153] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:40:32.300 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:32.300 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,551c3b748d42866d,fee2c29aaff5a56e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,551c3b748d42866d,fee2c29aaff5a56e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:33.248 INFO 2981023 --- [ocp-async-7,5e466d6279a7b1ef,21fd04ce24114f18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:34.095 INFO 2981023 --- [ocp-schedules-13,2b8f1bc452516a91,0a7bfcfc878608bf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:40:34.101 INFO 2981023 --- [ocp-schedules-13,2b8f1bc452516a91,0a7bfcfc878608bf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:40:34.101 INFO 2981023 --- [ocp-schedules-13,2b8f1bc452516a91,0a7bfcfc878608bf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:40:35.248 INFO 2981023 --- [ocp-async-6,0bc8ea56056cab82,cf00b9ebadac7c3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:35.300 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:35.301 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,bf2b025a6cc9ace4,d25c05c49f483a11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,bf2b025a6cc9ace4,d25c05c49f483a11] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:40:36.336312+08:00, lastProcessTime=2024-09-11T17:40:26.338565+08:00 2024-09-11 17:40:36.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:40:36.336312+08:00 2024-09-11 17:40:37.248 INFO 2981023 --- [ocp-async-2,8c6d75292a48cd91,106c6d47133c7a63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:37.314 INFO 2981023 --- [prometheus-cache-monitor1,4eb45c692b034bfe,415db21313113b25] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8192,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:40:37.315 INFO 2981023 --- [prometheus-cache-monitor1,4eb45c692b034bfe,415db21313113b25] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:40:37.756 INFO 2981023 --- [ocp-async-6,13e453e1c430c265,5b396e65b7722f29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:40:38.301 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:38.302 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,c4466e39c94bbc12,1bab40faac1ce791] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,c4466e39c94bbc12,1bab40faac1ce791] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:39.249 INFO 2981023 --- [ocp-async-10,a2896b553ed11759,d721480093373417] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:39.972 INFO 2981023 --- [ocp-schedules-12,697c1ff26c58d6f8,b621449c04d00fce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:40:40.004 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:40:40.005 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:40:40.015 WARN 2981023 --- [scanner-13526907-1,697c1ff26c58d6f8,30e4b623e12065b7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:40:40.271 INFO 2981023 --- [ocp-schedules-23,b428a70d8d30db0f,7e2ef5e8e80d6cba] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:40:40.288 WARN 2981023 --- [ocp-schedules-23,b428a70d8d30db0f,7e2ef5e8e80d6cba] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:40:41.249 INFO 2981023 --- [ocp-async-5,0697b21035729407,091b4e7eb51ac4b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:41.302 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:41.302 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,ace7af0af080d930,227f9bd711eb7479] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,ace7af0af080d930,227f9bd711eb7479] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:42.045 INFO 2981023 --- [ocp-schedules-12,6d900eb72a62439a,9e18ff6a6674b694] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:40:43.249 INFO 2981023 --- [ocp-async-4,026a91e5e6170252,df71da382d788b71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:44.302 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:44.303 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,0828293f886abced,03f2c48ba3bf660b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,0828293f886abced,03f2c48ba3bf660b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:45.101 INFO 2981023 --- [prometheus-second-schedule-1,39d86acd8fa5c838,a9a35b79f0aa7c9a] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:40:45.249 INFO 2981023 --- [ocp-async-1,8f83e99373832de9,ae5596b6c6fba7b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:45.998 INFO 2981023 --- [ocp-schedules-25,a74f57a1ccda818a,65cfcde621726670] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:40:45.998 INFO 2981023 --- [host-check-clockdiff1,a74f57a1ccda818a,6d0f9516391f3b61] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:40:45.998 INFO 2981023 --- [host-check-clockdiff3,a74f57a1ccda818a,824acb8c3865d276] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:40:45.998 INFO 2981023 --- [host-check-clockdiff4,a74f57a1ccda818a,79b594af146eb3fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:40:45.999 INFO 2981023 --- [host-check-clockdiff2,a74f57a1ccda818a,d8fc4e4e8cf7f633] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:40:46.003 INFO 2981023 --- [ocp-schedules-24,90241a0d0cb1fcb4,7b1b5da1073d4d57] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:40:46.004 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:40:46.004 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:40:46.007 INFO 2981023 --- [ocp-schedules-4,ba797d1de121ef44,ae2a65e82d1af617] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:40:46.007 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:40:46.007 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:40:46.010 INFO 2981023 --- [host-check-agent-status1,ba797d1de121ef44,0b76e114e737bd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:40:46.010 INFO 2981023 --- [host-check-agent-status2,ba797d1de121ef44,2655d6795d3dadfb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:40:46.010 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:40:46.011 INFO 2981023 --- [host-check-agent-status3,ba797d1de121ef44,780b61a0ec5c8ae0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:40:46.012 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:40:46.012 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:40:46.017 INFO 2981023 --- [host-check-agent-status4,ba797d1de121ef44,84e34a4e49e2bf5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:40:46.020 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:40:46.020 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:40:46.021 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:40:46.021 INFO 2981023 --- [host-check-clockdiff4,a74f57a1ccda818a,79b594af146eb3fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047646 0 0 , error= 2024-09-11 17:40:46.022 INFO 2981023 --- [host-check-clockdiff2,a74f57a1ccda818a,d8fc4e4e8cf7f633] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047646 0 0 , error= 2024-09-11 17:40:46.022 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:40:46.024 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:40:46.025 INFO 2981023 --- [host-check-agent-status2,ba797d1de121ef44,2655d6795d3dadfb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:40:46.026 INFO 2981023 --- [host-check-agent-status1,ba797d1de121ef44,0b76e114e737bd5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:40:46.027 INFO 2981023 --- [host-check-agent-status3,ba797d1de121ef44,780b61a0ec5c8ae0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:40:46.028 INFO 2981023 --- [host-check-agent-status2,ba797d1de121ef44,2655d6795d3dadfb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:40:46.031 INFO 2981023 --- [host-check-agent-status1,ba797d1de121ef44,0b76e114e737bd5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:40:46.031 INFO 2981023 --- [host-check-agent-status3,ba797d1de121ef44,780b61a0ec5c8ae0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:40:46.034 INFO 2981023 --- [host-check-agent-status4,ba797d1de121ef44,84e34a4e49e2bf5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:40:46.036 INFO 2981023 --- [host-check-agent-status4,ba797d1de121ef44,84e34a4e49e2bf5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:40:46.050 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:40:46.052 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:40:46.053 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:40:46.052523236+08:00 2024-09-11 17:40:46.053 INFO 2981023 --- [host-check-agent-status2,ba797d1de121ef44,2655d6795d3dadfb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:40:46.054 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:40:46.053+08:00 2024-09-11 17:40:46.054 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:40:46.053366731+08:00 2024-09-11 17:40:46.054 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:40:46.055 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:40:46.054+08:00 2024-09-11 17:40:46.056 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:40:46.053+08:00 and status AVAILABLE 2024-09-11 17:40:46.056 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:40:46.054+08:00 and status AVAILABLE 2024-09-11 17:40:46.057 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:40:46.056967648+08:00 2024-09-11 17:40:46.057 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:40:46.057+08:00 2024-09-11 17:40:46.058 INFO 2981023 --- [host-check-agent-status2,ba797d1de121ef44,2655d6795d3dadfb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:40:46.058 INFO 2981023 --- [host-check-agent-status2,ba797d1de121ef44,2655d6795d3dadfb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:40:46.058 INFO 2981023 --- [host-check-agent-status2,ba797d1de121ef44,2655d6795d3dadfb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:40:46.058 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:40:46.059 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:40:46.057+08:00 and status AVAILABLE 2024-09-11 17:40:46.061 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:40:46.061396862+08:00 2024-09-11 17:40:46.061 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:40:46.061+08:00 2024-09-11 17:40:46.063 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:40:46.064 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:40:46.061+08:00 and status AVAILABLE 2024-09-11 17:40:46.064 INFO 2981023 --- [host-check-agent-status4,ba797d1de121ef44,84e34a4e49e2bf5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:40:46.068 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:40:46.070 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:40:46.070 INFO 2981023 --- [host-check-agent-status3,ba797d1de121ef44,780b61a0ec5c8ae0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:40:46.071 INFO 2981023 --- [host-check-agent-status4,ba797d1de121ef44,84e34a4e49e2bf5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:40:46.071 INFO 2981023 --- [host-check-agent-status4,ba797d1de121ef44,84e34a4e49e2bf5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:40:46.071 INFO 2981023 --- [host-check-agent-status4,ba797d1de121ef44,84e34a4e49e2bf5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:40:46.071 INFO 2981023 --- [host-check-agent-status1,ba797d1de121ef44,0b76e114e737bd5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:40:46.074 INFO 2981023 --- [host-check-agent-status1,ba797d1de121ef44,0b76e114e737bd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:40:46.074 INFO 2981023 --- [host-check-agent-status1,ba797d1de121ef44,0b76e114e737bd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:40:46.074 INFO 2981023 --- [host-check-agent-status1,ba797d1de121ef44,0b76e114e737bd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:40:46.075 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:40:46.076 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:40:46.076 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:40:46.077 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:40:46.080 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:40:46.082 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:40:46.093 INFO 2981023 --- [host-check-agent-status3,ba797d1de121ef44,780b61a0ec5c8ae0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:40:46.093 INFO 2981023 --- [host-check-agent-status3,ba797d1de121ef44,780b61a0ec5c8ae0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:40:46.093 INFO 2981023 --- [host-check-agent-status3,ba797d1de121ef44,780b61a0ec5c8ae0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:40:46.099 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:40:46.105 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:40:46.106 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:40:46.106 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:40:46.107 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:40:46.108 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:40:46.121 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:40:46.137 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:40:46.138 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:40:46.143 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:40:46.147 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:40:46.153 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:40:46.160 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:40:46.162 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:40:46.168 INFO 2981023 --- [ocp-schedules-5,8e59f6feb8b65a18,71e622d4949a71e4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:40:46.179 INFO 2981023 --- [ocp-schedules-5,8e59f6feb8b65a18,71e622d4949a71e4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:40:46.180 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:40:46.186 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:40:46.188 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:40:46.189 INFO 2981023 --- [ocp-schedules-6,65bfce22cb305f4b,90e72b7c3de186e3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:40:46.189 INFO 2981023 --- [ocp-schedules-16,784a18e79de74e78,f9e0a0b832e72a1a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:40:46.189 INFO 2981023 --- [ocp-schedules-17,1461ea9bed85673e,de5d4dacf4be4683] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:40:46.190 INFO 2981023 --- [ocp-schedules-20,e2d3ad7d2094748b,2bb3167dd4c2d91f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:40:46.192 INFO 2981023 --- [ocp-schedules-17,1461ea9bed85673e,de5d4dacf4be4683] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:40:46.194 INFO 2981023 --- [ocp-schedules-6,65bfce22cb305f4b,90e72b7c3de186e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:40:46.196 INFO 2981023 --- [ocp-schedules-16,784a18e79de74e78,f9e0a0b832e72a1a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:40:46.210 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:40:46.214 INFO 2981023 --- [ocp-schedules-10,e72ec69426e7643d,305c00984ab5d7b8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:40:46.223 INFO 2981023 --- [ocp-schedules-16,784a18e79de74e78,f9e0a0b832e72a1a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:40:46.224 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:40:46.225 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:40:46.237 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.238 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:46.239 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:40:46.240 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:40:46.245 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.246 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:46.248 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:40:46.250 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.250 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:40:46.251 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:46.251 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:40:46.252 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:40:46.255 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:40:46.256 INFO 2981023 --- [ocp-schedules-6,65bfce22cb305f4b,90e72b7c3de186e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:40:46.258 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:40:46.258 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.258 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:40:46.259 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:40:46.259 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.267 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:40:46.281 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:40:46.282 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:40:46.283 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.283 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:46.299 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:40:46.299 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:40:46.307 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:40:46.319 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:40:46.327 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.327 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:46.328 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:40:46.328 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:40:46.332 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:40:46.347 INFO 2981023 --- [host-check-host-status1,90241a0d0cb1fcb4,3d06c9531a65ca0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:40:46.348 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.348 INFO 2981023 --- [host-check-host-status2,90241a0d0cb1fcb4,6231ff3208d2bbfc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:40:46.349 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.349 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:40:46.350 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:40:46.350 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.357 INFO 2981023 --- [host-check-host-status3,90241a0d0cb1fcb4,0525932c8b0085cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:40:46.358 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.358 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:46.382 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:40:46.384 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.384 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.384 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:40:46.385 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:40:46.388 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:40:46.387363+08:00, lastProcessTime=2024-09-11T17:40:36.336312+08:00 2024-09-11 17:40:46.390 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.391 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.392 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:40:46.396 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:40:46.407 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.407 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.409 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:40:46.413 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:40:46.416 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.416 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.416 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:46.417 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:40:46.417 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.419 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:40:46.420 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:40:46.426 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.429 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:46.429 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:40:46.436 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.437 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.439 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:46.441 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.443 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:40:46.444 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.444 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.445 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:40:46.445 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.445 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:40:46.445 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:40:46.446 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:40:46.446 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.446 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.446 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:40:46.447 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:40:46.448 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:40:46.449 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.449 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:40:46.449 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:46.450 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:40:46.450 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:40:46.450 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.454 INFO 2981023 --- [host-check-host-status4,90241a0d0cb1fcb4,2a94fe73d762dad8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:40:46.454 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:46.454 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:40:46.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:46.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:40:46.457 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:40:46.462 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.463 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.463 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:46.464 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:40:46.466 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:40:46.467 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:46.471 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:40:46.471 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:40:46.472 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:40:46.475 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.475 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:40:46.475 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.476 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:46.476 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:40:46.482 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:40:46.484 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.485 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.485 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:40:46.486 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:40:46.486 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:40:46.487 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:40:46.491 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:40:46.492 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:40:46.495 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.496 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.496 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:40:46.497 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:40:46.497 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:40:46.497 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.501 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:40:46.507 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:40:46.511 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:40:46.523 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:40:46.528 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:40:46.529 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:40:46.531 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.531 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.532 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:40:46.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:40:46.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:40:46.546 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.547 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.547 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:40:46.548 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:40:46.548 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.564 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:40:46.571 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.572 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.572 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:40:46.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:40:46.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:40:46.589 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:40:46.597 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:46.597 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:46.598 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:40:46.600 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:40:46.604 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:40:46.604 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:40:46.604 INFO 2981023 --- [ocp-schedules-17,1461ea9bed85673e,de5d4dacf4be4683] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:40:46.617 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:40:46.620 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:40:46.387363+08:00 2024-09-11 17:40:46.623 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:40:46.625 INFO 2981023 --- [ocp-schedules-18,5601a6e2ea127094,9112559abfa83cfe] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:40:47.249 INFO 2981023 --- [ocp-async-5,4dafbde7c4b1b293,7e521aa18f149dbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:47.303 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:47.303 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,0890d2e5c5c677ee,123fa5c6d9223841] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,0890d2e5c5c677ee,123fa5c6d9223841] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:47.756 INFO 2981023 --- [ocp-async-10,b9bad4923013fdda,19797a77163a7003] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:40:47.791 INFO 2981023 --- [host-check-clockdiff1,a74f57a1ccda818a,6d0f9516391f3b61] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047647 0 0 , error= 2024-09-11 17:40:49.249 INFO 2981023 --- [ocp-async-4,770d62cac50579a7,196b6f6862fe3433] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:49.270 INFO 2981023 --- [ocp-schedules-30,ffed35f2ed18fb1e,53aa6f33db497304] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:40:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:40:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:40:50.191 INFO 2981023 --- [ocp-schedules-8,257354b7fd117407,633773677106f643] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:40:50.236 INFO 2981023 --- [ocp-schedules-20,42cbd1a5b4b4c8ad,4391bbff9ec33c40] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:40:50.246 WARN 2981023 --- [scanner-13527141-1,257354b7fd117407,d500f2d15dc675b4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:40:50.261 INFO 2981023 --- [alarm-detect-20,257354b7fd117407,6106c652ca727b9f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644394, data is empty. 2024-09-11 17:40:50.268 INFO 2981023 --- [alarm-detect-20,257354b7fd117407,6106c652ca727b9f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644394, data is empty. 2024-09-11 17:40:50.271 INFO 2981023 --- [alarm-detect-20,257354b7fd117407,6106c652ca727b9f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644394, data is empty. 2024-09-11 17:40:50.303 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:50.304 INFO 2981023 --- [alarm-detect-16,257354b7fd117407,e14582d751288d2b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644394, data is empty. 2024-09-11 17:40:50.304 INFO 2981023 --- [alarm-detect-9,257354b7fd117407,a04557dd6b10d3b1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644394, data is empty. 2024-09-11 17:40:50.309 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:50.311 INFO 2981023 --- [alarm-detect-9,257354b7fd117407,a04557dd6b10d3b1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644394, data is empty. 2024-09-11 17:40:50.311 INFO 2981023 --- [alarm-detect-16,257354b7fd117407,e14582d751288d2b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644394, data is empty. 2024-09-11 17:40:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,b289ed3f7352d611,8adf8bbbb75f9350] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:50.316 INFO 2981023 --- [alarm-detect-16,257354b7fd117407,e14582d751288d2b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644394, data is empty. 2024-09-11 17:40:50.317 INFO 2981023 --- [alarm-detect-9,257354b7fd117407,a04557dd6b10d3b1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644394, data is empty. 2024-09-11 17:40:50.319 INFO 2981023 --- [prometheus-minute-schedule-1,b289ed3f7352d611,8adf8bbbb75f9350] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:50.454 INFO 2981023 --- [alarm-detect-16,257354b7fd117407,c1509d8bea18f4a5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644394, data is empty. 2024-09-11 17:40:50.458 INFO 2981023 --- [alarm-detect-16,257354b7fd117407,c1509d8bea18f4a5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644394, data is empty. 2024-09-11 17:40:50.463 INFO 2981023 --- [alarm-detect-16,257354b7fd117407,c1509d8bea18f4a5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644394, data is empty. 2024-09-11 17:40:51.249 INFO 2981023 --- [ocp-async-1,c45ba3e276e80bfd,e010bc2163abc80f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:51.550 INFO 2981023 --- [ocp-schedules-23,8422df19aefc9d0f,f790759c52ad4dee] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:40:51.551 INFO 2981023 --- [ocp-schedules-23,8422df19aefc9d0f,f790759c52ad4dee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:40:51.551 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:40:51.551 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:40:51.551 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:40:51.551 INFO 2981023 --- [ocp-schedules-23,8422df19aefc9d0f,f790759c52ad4dee] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:40:51.949 INFO 2981023 --- [host-check-clockdiff3,a74f57a1ccda818a,824acb8c3865d276] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047651 1 1 , error= 2024-09-11 17:40:52.048 INFO 2981023 --- [ocp-schedules-16,587782a27de65869,aa16e708aff94744] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:40:52.952 INFO 2981023 --- [ocp-schedules-14,9912ff0326e5661d,cf8133c660fa3318] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:40:52.972 WARN 2981023 --- [ocp-schedules-14,9912ff0326e5661d,cf8133c660fa3318] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:40:53.108 INFO 2981023 --- [ocp-schedules-20,41f5970f76e644d9,cd2b8c5da2da769d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:40:53.108 INFO 2981023 --- [ocp-schedules-20,41f5970f76e644d9,cd2b8c5da2da769d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:40:53.121 INFO 2981023 --- [ocp-schedules-28,63a7c1f0f6e4b2c9,a84e3583270a2d03] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:40:53.121+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:40:53.126 INFO 2981023 --- [ocp-schedules-28,63a7c1f0f6e4b2c9,a84e3583270a2d03] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:40:53.249 INFO 2981023 --- [ocp-async-5,3703120c55dc911d,ff3815b55526f835] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:53.258 INFO 2981023 --- [ocp-schedules-28,63a7c1f0f6e4b2c9,a84e3583270a2d03] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 18 to SqlAggregateData 2024-09-11 17:40:53.259 INFO 2981023 --- [ocp-schedules-28,63a7c1f0f6e4b2c9,a84e3583270a2d03] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:40:53.309 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:53.309 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,2def63871d95ac49,e30f67d5a743b025] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,2def63871d95ac49,e30f67d5a743b025] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:53.976 INFO 2981023 --- [ocp-schedules-10,936bc76d027f2d2b,c56bd76f62270bbe] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:40:53.978 INFO 2981023 --- [ocp-schedules-10,936bc76d027f2d2b,c56bd76f62270bbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:40:54.003 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:40:54.003 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:40:54.003 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.003 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.003 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:40:54.004 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.006 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:40:54.007 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:40:54.007 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:40:54.010 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.010 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.010 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.011 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.013 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.021 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.022 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:40:54.022 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.027 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.028 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.031 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.032 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:40:54.036 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.037 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.038 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.038 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:40:54.042 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.043 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.048 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.048 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.049 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:40:54.049 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.049 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:40:54.049 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2E11D6DA-0-0] 2024-09-11 17:40:54.050 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:40:54.050 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.050 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:40:54.051 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.052 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:40:54.052 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.053 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:40:54.053 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.054 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:40:54.054 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:40:54.058 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:40:54.060 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.061 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:40:54.062 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.062 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.062 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:40:54.063 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:40:54.063 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.063 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:40:54.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:40:54.065 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:40:54.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.067 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:40:54.068 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.068 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:40:54.068 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:54.069 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.069 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:40:54.070 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.070 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.070 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.070 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:40:54.071 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:40:54.072 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:40:54.075 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.075 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:40:54.076 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:40:54.079 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2E11D6E4-0-0] 2024-09-11 17:40:54.079 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:40:54.083 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:40:54.084 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:54.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:40:54.085 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.086 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:40:54.086 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:40:54.086 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:40:54.086 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:40:54.086 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.087 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:40:54.087 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.088 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:40:54.088 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.089 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:40:54.089 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.090 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:40:54.090 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.091 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:40:54.091 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:40:54.092 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:40:54.094 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:40:54.095 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:40:54.095 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.096 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:40:54.101 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:40:54.103 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:40:54.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:40:54.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.104 INFO 2981023 --- [ocp-schedules-2,a74440fa8d658032,b2db580f4f7ace1e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:40:54.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.105 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.105 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:40:54.111 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:40:54.115 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:40:54.116 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:40:54.116 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:40:54.116 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:40:54.117 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:40:54.117 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:54.118 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:40:54.119 INFO 2981023 --- [ocp-schedules-2,a74440fa8d658032,b2db580f4f7ace1e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:40:54.119 INFO 2981023 --- [ocp-schedules-2,a74440fa8d658032,b2db580f4f7ace1e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:40:54.127 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:40:54.128 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:40:54.128 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:40:54.128 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:40:54.128 INFO 2981023 --- [ocp-schedules-10,936bc76d027f2d2b,c56bd76f62270bbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:40:54.249 INFO 2981023 --- [ocp-schedules-21,b6e53a45bc6164d8,e3b3d1a57651db5e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:40:55.250 INFO 2981023 --- [ocp-async-2,931b1fbc5db33fee,c28fa327a275bfc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:55.698 INFO 2981023 --- [ocp-schedules-7,6b070b492b07cc28,bc02c4f5c870fc24] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:40:55.698 INFO 2981023 --- [ocp-schedules-7,6b070b492b07cc28,bc02c4f5c870fc24] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:40:55.700 INFO 2981023 --- [ocp-schedules-7,6b070b492b07cc28,bc02c4f5c870fc24] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:40:55.700 INFO 2981023 --- [ocp-schedules-7,6b070b492b07cc28,bc02c4f5c870fc24] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:40:55.702 INFO 2981023 --- [ocp-schedules-7,6b070b492b07cc28,bc02c4f5c870fc24] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:40:55.702 INFO 2981023 --- [ocp-schedules-7,6b070b492b07cc28,bc02c4f5c870fc24] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:40:55.797 INFO 2981023 --- [ocp-schedules-21,5da481666cdb1dfb,104f9e79f43b7160] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:40:55.803 INFO 2981023 --- [ocp-schedules-21,5da481666cdb1dfb,104f9e79f43b7160] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:40:55.803 INFO 2981023 --- [ocp-schedules-21,5da481666cdb1dfb,104f9e79f43b7160] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:40:56.309 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:56.310 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,dee68da3d1f3e515,12ce8e4ce4013c7e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:56.315 INFO 2981023 --- [prometheus-minute-schedule-1,dee68da3d1f3e515,12ce8e4ce4013c7e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:40:56.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:40:56.337704+08:00, lastProcessTime=2024-09-11T17:40:46.387363+08:00 2024-09-11 17:40:56.369 INFO 2981023 --- [ocp-schedules-25,b6df66805ef49625,fc02b8a0be36af6e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:40:56.369 INFO 2981023 --- [ocp-schedules-25,b6df66805ef49625,fc02b8a0be36af6e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:40:56.384 INFO 2981023 --- [Collect all tenants major compactions1,b6df66805ef49625,314de8c725f14edc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:40:56.385 INFO 2981023 --- [Collect all tenants major compactions2,b6df66805ef49625,d6c9dddb8b2323a3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:40:56.385 INFO 2981023 --- [Collect all tenants major compactions3,b6df66805ef49625,ec622dc43ecf4108] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:40:56.386 INFO 2981023 --- [Collect all tenants major compactions6,b6df66805ef49625,fc8ac7fa7fd3ec99] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:40:56.387 INFO 2981023 --- [Collect all tenants major compactions5,b6df66805ef49625,4c86d9297702c5a8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:40:56.388 INFO 2981023 --- [Collect all tenants major compactions4,b6df66805ef49625,1d7ec43ccff6eb71] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:40:56.397 INFO 2981023 --- [Collect all tenants major compactions1,b6df66805ef49625,314de8c725f14edc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:56.397 INFO 2981023 --- [Collect all tenants major compactions2,b6df66805ef49625,d6c9dddb8b2323a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:56.398 INFO 2981023 --- [Collect all tenants major compactions2,b6df66805ef49625,d6c9dddb8b2323a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:40:56.399 INFO 2981023 --- [Collect all tenants major compactions3,b6df66805ef49625,ec622dc43ecf4108] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:56.400 INFO 2981023 --- [Collect all tenants major compactions1,b6df66805ef49625,314de8c725f14edc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:40:56.400 INFO 2981023 --- [Collect all tenants major compactions5,b6df66805ef49625,4c86d9297702c5a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:56.400 INFO 2981023 --- [Collect all tenants major compactions3,b6df66805ef49625,ec622dc43ecf4108] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:40:56.401 INFO 2981023 --- [Collect all tenants major compactions4,b6df66805ef49625,1d7ec43ccff6eb71] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:56.401 INFO 2981023 --- [Collect all tenants major compactions5,b6df66805ef49625,4c86d9297702c5a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:40:56.402 INFO 2981023 --- [Collect all tenants major compactions4,b6df66805ef49625,1d7ec43ccff6eb71] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:40:56.403 INFO 2981023 --- [Collect all tenants major compactions2,b6df66805ef49625,d6c9dddb8b2323a3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:40:56.403 INFO 2981023 --- [Collect all tenants major compactions2,b6df66805ef49625,d6c9dddb8b2323a3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:40:56.403 INFO 2981023 --- [Collect all tenants major compactions5,b6df66805ef49625,4c86d9297702c5a8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:40:56.403 INFO 2981023 --- [Collect all tenants major compactions5,b6df66805ef49625,4c86d9297702c5a8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:40:56.404 INFO 2981023 --- [Collect all tenants major compactions3,b6df66805ef49625,ec622dc43ecf4108] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:40:56.404 INFO 2981023 --- [Collect all tenants major compactions3,b6df66805ef49625,ec622dc43ecf4108] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:40:56.404 INFO 2981023 --- [Collect all tenants major compactions1,b6df66805ef49625,314de8c725f14edc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:40:56.404 INFO 2981023 --- [Collect all tenants major compactions1,b6df66805ef49625,314de8c725f14edc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:40:56.404 INFO 2981023 --- [Collect all tenants major compactions4,b6df66805ef49625,1d7ec43ccff6eb71] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:40:56.404 INFO 2981023 --- [Collect all tenants major compactions4,b6df66805ef49625,1d7ec43ccff6eb71] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:40:56.405 INFO 2981023 --- [Collect all tenants major compactions6,b6df66805ef49625,fc8ac7fa7fd3ec99] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:40:56.406 INFO 2981023 --- [Collect all tenants major compactions6,b6df66805ef49625,fc8ac7fa7fd3ec99] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:40:56.408 INFO 2981023 --- [Collect all tenants major compactions6,b6df66805ef49625,fc8ac7fa7fd3ec99] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:40:56.408 INFO 2981023 --- [Collect all tenants major compactions6,b6df66805ef49625,fc8ac7fa7fd3ec99] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:40:56.408 INFO 2981023 --- [ocp-schedules-25,b6df66805ef49625,fc02b8a0be36af6e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:40:56.484 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:40:56.337704+08:00 2024-09-11 17:40:57.250 INFO 2981023 --- [ocp-async-1,90085f5a489c509c,a8d3792d37554229] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:57.757 INFO 2981023 --- [ocp-async-4,6a6fa3c0bed98361,a5f0748df3657e51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:40:59.250 INFO 2981023 --- [ocp-async-5,8d320de3f568b2f2,f90fb614734df481] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:40:59.310 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:40:59.311 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:40:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,49f1927333c58bbe,e5f96269466d6d0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047660, interval=60 2024-09-11 17:40:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,49f1927333c58bbe,e5f96269466d6d0f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:00.518 INFO 2981023 --- [ocp-schedules-24,62dedad86bb7bb29,8f8d01f699f5789c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:41:00.533 WARN 2981023 --- [scanner-13527403-1,62dedad86bb7bb29,d4a937cffce02fa0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:41:01.257 INFO 2981023 --- [ocp-async-4,7e5eda17de8bf9e0,192c029cb781fe96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:02.051 INFO 2981023 --- [ocp-schedules-4,54e6a04dc9f0c0af,7f6eba0648b674ac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:41:02.311 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:02.312 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,699d9c64a1cd216d,8c1337b317ba2981] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,699d9c64a1cd216d,8c1337b317ba2981] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:03.224 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:37:02.766, end=2024-09-11T17:39:03.22} 2024-09-11 17:41:03.232 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId 2024-09-11 17:41:03.233 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:41:03.238 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:02.766, end=2024-09-11T17:39:03.22},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:41:03.238 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:41:03.238 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:41:03.256 INFO 2981023 --- [ocp-async-1,a67fc1ce3c5dfec6,de084ab8ffbb783c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:03.280 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:41:03.297 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263297000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.297Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.300 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:41:03.300 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.305 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.305 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.309 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.309 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.314 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.314 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.315 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:41:03.315 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.297, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.315 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:41:03.316 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:02.766, end=2024-09-11T17:39:03.22},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:41:03.316 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:41:03.316 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:41:03.317 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:41:03.322 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:02.766, end=2024-09-11T17:39:03.22},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:41:03.326 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:41:03.326 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:41:03.369 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263368000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.368Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.371 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263371000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.371Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.375 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263375000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.375Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.376 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263376000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.376Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.378 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263378000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.378Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.386 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263386000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=8A1B851989150B802AEEACA99152D145, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.386Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.387 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263387000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.387Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.388 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263388000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.388Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.407 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.407 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.407 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.407 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.415 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.415 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.416 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 27 2024-09-11 17:41:03.416 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.417 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:41:03.417 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.419 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.419 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.426 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.426 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.426 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:41:03.426 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.427 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:41:03.427 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.429 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 61 2024-09-11 17:41:03.429 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.430 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.430 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.430 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.430 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.431 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.431 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.435 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.435 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.437 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.437 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.440 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.440 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.440 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.440 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.442 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.442 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.443 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.444 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.444 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.444 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.444 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.445 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.445 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.446 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.446 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.447 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.447 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.447 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.449 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.449 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.452 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.452 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.453 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.368, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.456 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.456 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.459 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.459 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.459 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.459 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.461 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.461 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.461 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.462 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.464 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.371, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.464 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.467 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.375, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.467 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.467 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.469 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.378, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.470 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.471 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.471 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.473 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.473 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.475 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.386, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=8A1B851989150B802AEEACA99152D145, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.476 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.476 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.476 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.376, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.481 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.387, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.483 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263483000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.483Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.486 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.388, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.495 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.495 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.504 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263504000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.504Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.509 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.509 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.513 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.513 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.518 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:41:03.518 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.518 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263518000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.518Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.520 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263520000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.520Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.523 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263523000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.523Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.523 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263523000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.523Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.525 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.525 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.528 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:41:03.528 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.529 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.529 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263529000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.529Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.531 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.531 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.536 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.536 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.537 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.537 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.537 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.537 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.538 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263538000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.538Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.540 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.540 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.541 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.541 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.544 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.544 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.545 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.545 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.545 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.483, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.551 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.551 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.551 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.551 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.556 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.556 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.556 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.556 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.558 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.558 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.559 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.559 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.561 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.561 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.561 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.562 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.562 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.562 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.562 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.565 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.565 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.566 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.566 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.571 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.574 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.574 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.575 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.575 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.577 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.579 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.579 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.580 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.580 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.581 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.581 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.582 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.584 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.585 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.504, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.585 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.52, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.585 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961263585000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:41:03.585Z, endTime=2024-09-11T09:39:03.220Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:41:03.586 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.523, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.586 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.518, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.586 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.586 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.588 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.590 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.523, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.593 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.529, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.594 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.594 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.594 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:41:03.594 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:41:03.595 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.598 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.538, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.600 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.600 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.601 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:41:03.601 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:41:03.606 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:41:03.606 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047543220000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 17:41:03.607 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:41:03.610 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:41:03.585, end=2024-09-11T17:39:03.22}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:41:03.610 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:41:03.611 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:02.766, end=2024-09-11T17:39:03.22},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:41:03.611 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:41:03.611 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:41:03.612 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:41:03.616 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:02.766, end=2024-09-11T17:39:03.22},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:41:03.616 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:41:03.616 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:41:03.626 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:41:03.626 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:41:03.626 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:02.766, end=2024-09-11T17:39:03.22},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:41:03.626 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:41:03.626 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:41:03.627 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:41:05.256 INFO 2981023 --- [ocp-async-9,c42892b16b120e80,83126f9267e8b40a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:05.312 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,28a75f6905beda07,da2f1c0f595c448e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:05.313 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,28a75f6905beda07,da2f1c0f595c448e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:41:06.352186+08:00, lastProcessTime=2024-09-11T17:40:56.337704+08:00 2024-09-11 17:41:06.488 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:41:06.352186+08:00 2024-09-11 17:41:07.256 INFO 2981023 --- [ocp-async-7,c1ec80fda5de7788,79643edbd17d48d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:07.412 INFO 2981023 --- [ocp-schedules-9,9151b1dd19b04f9c,b08842a4b6ea1cb8] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:41:07.423 INFO 2981023 --- [ocp-schedules-6,6bd00750703c7988,9317e41ee148f21f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:41:07.423+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:41:07.425 INFO 2981023 --- [ocp-schedules-6,6bd00750703c7988,9317e41ee148f21f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:41:07.429 INFO 2981023 --- [ocp-schedules-6,6bd00750703c7988,9317e41ee148f21f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:41:07.429 INFO 2981023 --- [ocp-schedules-6,6bd00750703c7988,9317e41ee148f21f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:41:07.757 INFO 2981023 --- [ocp-async-9,a672b25bafa343cb,5f1e187b22014d57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:41:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,b87b13a778ba6c65,46a351cc65155fdc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:08.313 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:08.315 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:08.317 INFO 2981023 --- [prometheus-minute-schedule-1,b87b13a778ba6c65,46a351cc65155fdc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:09.256 INFO 2981023 --- [ocp-async-1,0ddbe4bc353d4af4,9d6a024ac98aa2ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:10.003 INFO 2981023 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:41:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:41:10.732 INFO 2981023 --- [ocp-schedules-23,916b7e4310c17adf,9e9074e989350a73] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:41:10.767 WARN 2981023 --- [scanner-13527633-1,916b7e4310c17adf,5336ba125f78aa24] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:41:10.796 INFO 2981023 --- [alarm-detect-11,916b7e4310c17adf,56e8baf3dd52da80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644394, data is empty. 2024-09-11 17:41:10.805 INFO 2981023 --- [alarm-detect-11,916b7e4310c17adf,56e8baf3dd52da80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644412, data is empty. 2024-09-11 17:41:10.809 INFO 2981023 --- [alarm-detect-11,916b7e4310c17adf,56e8baf3dd52da80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644412, data is empty. 2024-09-11 17:41:10.839 INFO 2981023 --- [alarm-detect-5,916b7e4310c17adf,3cffc5bb5b464063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644394, data is empty. 2024-09-11 17:41:10.846 INFO 2981023 --- [alarm-detect-6,916b7e4310c17adf,ce018cbec7c340fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644394, data is empty. 2024-09-11 17:41:10.849 INFO 2981023 --- [alarm-detect-5,916b7e4310c17adf,3cffc5bb5b464063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644412, data is empty. 2024-09-11 17:41:10.852 INFO 2981023 --- [alarm-detect-6,916b7e4310c17adf,ce018cbec7c340fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644412, data is empty. 2024-09-11 17:41:10.854 INFO 2981023 --- [alarm-detect-5,916b7e4310c17adf,3cffc5bb5b464063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644412, data is empty. 2024-09-11 17:41:10.856 INFO 2981023 --- [alarm-detect-6,916b7e4310c17adf,ce018cbec7c340fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644412, data is empty. 2024-09-11 17:41:10.930 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:41:10.938 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:41:10.961 INFO 2981023 --- [alarm-detect-13,916b7e4310c17adf,12a6b5592beaf492] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644394, data is empty. 2024-09-11 17:41:10.968 INFO 2981023 --- [alarm-detect-13,916b7e4310c17adf,12a6b5592beaf492] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644412, data is empty. 2024-09-11 17:41:10.971 INFO 2981023 --- [alarm-detect-13,916b7e4310c17adf,12a6b5592beaf492] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644412, data is empty. 2024-09-11 17:41:11.257 INFO 2981023 --- [ocp-async-3,df6fbe5bed2fdfbc,95bf1fdd58378b19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,e4f5f5cff8eee394,b96be2d114bccf4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:11.316 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:11.316 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:11.320 INFO 2981023 --- [prometheus-minute-schedule-1,e4f5f5cff8eee394,b96be2d114bccf4c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:11.508 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047600, nodeCount=465, affectRows=465 2024-09-11 17:41:11.748 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047600, nodeCount=15, affectRows=15 2024-09-11 17:41:11.833 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047600, nodeCount=15, validCount=15 2024-09-11 17:41:12.055 INFO 2981023 --- [ocp-schedules-29,42fcec71cc5cdd6b,1a51cb72ed472b6f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:41:12.879 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047600, nodeCount=1000, affectRows=1000 2024-09-11 17:41:12.882 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047600, nodeCount=1000, validCount=1000 2024-09-11 17:41:12.882 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047600, nodeCount=1000, affectRows=1000 2024-09-11 17:41:12.885 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047600, nodeCount=999, validCount=999 2024-09-11 17:41:12.899 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047600, nodeCount=50, affectRows=50 2024-09-11 17:41:12.900 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047600, nodeCount=50, validCount=50 2024-09-11 17:41:13.257 INFO 2981023 --- [ocp-async-7,8817589edaed61ab,d9b265849eabd4f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:13.781 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047600, nodeCount=465, validCount=465 2024-09-11 17:41:14.122 INFO 2981023 --- [ocp-schedules-23,40872a716aedeab6,68dadd5c1823f2df] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:41:14.131 INFO 2981023 --- [ocp-schedules-23,40872a716aedeab6,68dadd5c1823f2df] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:41:14.131 INFO 2981023 --- [ocp-schedules-23,40872a716aedeab6,68dadd5c1823f2df] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:41:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,77d80d13150ff941,be64e8f2f857e8e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,1db9d34128357ab7,38f71178a0c19b43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:41:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,77d80d13150ff941,be64e8f2f857e8e1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:14.317 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:14.317 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:15.257 INFO 2981023 --- [ocp-async-4,1b59b6f1119fc57e,363b69e0d1b3e3a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:16.030 INFO 2981023 --- [ocp-schedules-23,3ec7606f61b869e7,a3694d6e79b0dd1b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:41:16.031 INFO 2981023 --- [host-check-clockdiff1,3ec7606f61b869e7,b680861c5b369274] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:41:16.032 INFO 2981023 --- [host-check-clockdiff2,3ec7606f61b869e7,5a433fee95100e7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:41:16.032 INFO 2981023 --- [host-check-clockdiff3,3ec7606f61b869e7,ebd7218a288b2254] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:41:16.034 INFO 2981023 --- [ocp-schedules-21,31b45ec4a803c9fa,22654e7772694960] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:41:16.034 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:41:16.034 INFO 2981023 --- [host-check-clockdiff4,3ec7606f61b869e7,7909096eeb31eb52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:41:16.035 INFO 2981023 --- [ocp-schedules-12,6bb16aa9eeeb659e,fb1028749bfc4fa4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:41:16.036 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:41:16.037 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:41:16.037 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:41:16.060 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:41:16.061 INFO 2981023 --- [host-check-agent-status1,6bb16aa9eeeb659e,4840a3bbf2ac95d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:41:16.061 INFO 2981023 --- [host-check-agent-status2,6bb16aa9eeeb659e,d06f2a7a16213945] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:41:16.062 INFO 2981023 --- [host-check-agent-status3,6bb16aa9eeeb659e,e0b7d7cd0879fd8f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:41:16.062 INFO 2981023 --- [host-check-agent-status4,6bb16aa9eeeb659e,5b265e88f41aba8c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:41:16.077 INFO 2981023 --- [host-check-clockdiff4,3ec7606f61b869e7,7909096eeb31eb52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047676 0 0 , error= 2024-09-11 17:41:16.078 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:41:16.079 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:41:16.081 INFO 2981023 --- [host-check-agent-status3,6bb16aa9eeeb659e,e0b7d7cd0879fd8f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:16.082 INFO 2981023 --- [host-check-agent-status3,6bb16aa9eeeb659e,e0b7d7cd0879fd8f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:16.082 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:41:16.085 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:41:16.105 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:41:16.105 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:16.108 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:16.187 INFO 2981023 --- [ocp-schedules-3,afcbd7ced0161c1e,9e9c151cc31ea73c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:41:16.189 INFO 2981023 --- [ocp-schedules-14,125b893d04ee46d9,40d1080f964d1aa2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:41:16.196 INFO 2981023 --- [ocp-schedules-10,09655704ab547dce,103ecf19954f3f05] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:41:16.196 INFO 2981023 --- [ocp-schedules-28,0053e87e33b76624,02bc69ee76b2afa3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:41:16.198 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:41:16.202 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:41:16.201576800+08:00 2024-09-11 17:41:16.253 INFO 2981023 --- [host-check-agent-status2,6bb16aa9eeeb659e,d06f2a7a16213945] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:41:16.254 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:41:16.202+08:00 2024-09-11 17:41:16.257 INFO 2981023 --- [ocp-schedules-3,afcbd7ced0161c1e,9e9c151cc31ea73c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:41:16.258 INFO 2981023 --- [ocp-schedules-14,125b893d04ee46d9,40d1080f964d1aa2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:41:16.258 INFO 2981023 --- [host-check-agent-status2,6bb16aa9eeeb659e,d06f2a7a16213945] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:41:16.263 INFO 2981023 --- [host-check-agent-status1,6bb16aa9eeeb659e,4840a3bbf2ac95d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:41:16.265 INFO 2981023 --- [host-check-agent-status1,6bb16aa9eeeb659e,4840a3bbf2ac95d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:41:16.266 INFO 2981023 --- [ocp-schedules-14,125b893d04ee46d9,40d1080f964d1aa2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:41:16.267 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:41:16.202+08:00 and status AVAILABLE 2024-09-11 17:41:16.274 INFO 2981023 --- [host-check-agent-status4,6bb16aa9eeeb659e,5b265e88f41aba8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:41:16.275 INFO 2981023 --- [host-check-agent-status4,6bb16aa9eeeb659e,5b265e88f41aba8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:41:16.296 INFO 2981023 --- [ocp-schedules-10,09655704ab547dce,103ecf19954f3f05] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:41:16.299 INFO 2981023 --- [ocp-schedules-4,fe5534d9c5b01e1d,122fa6b6675b5c0d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:41:16.320 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:41:16.355 INFO 2981023 --- [host-check-agent-status3,6bb16aa9eeeb659e,e0b7d7cd0879fd8f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:41:16.360 INFO 2981023 --- [host-check-agent-status3,6bb16aa9eeeb659e,e0b7d7cd0879fd8f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:41:16.360 INFO 2981023 --- [host-check-agent-status3,6bb16aa9eeeb659e,e0b7d7cd0879fd8f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:41:16.360 INFO 2981023 --- [host-check-agent-status3,6bb16aa9eeeb659e,e0b7d7cd0879fd8f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:41:16.386 INFO 2981023 --- [host-check-agent-status4,6bb16aa9eeeb659e,5b265e88f41aba8c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:41:16.388 INFO 2981023 --- [host-check-agent-status2,6bb16aa9eeeb659e,d06f2a7a16213945] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:41:16.389 INFO 2981023 --- [host-check-agent-status4,6bb16aa9eeeb659e,5b265e88f41aba8c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:41:16.389 INFO 2981023 --- [host-check-agent-status4,6bb16aa9eeeb659e,5b265e88f41aba8c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:41:16.389 INFO 2981023 --- [host-check-agent-status4,6bb16aa9eeeb659e,5b265e88f41aba8c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:41:16.391 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:41:16.391 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:41:16.392 INFO 2981023 --- [host-check-agent-status2,6bb16aa9eeeb659e,d06f2a7a16213945] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:41:16.392 INFO 2981023 --- [host-check-agent-status2,6bb16aa9eeeb659e,d06f2a7a16213945] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:41:16.392 INFO 2981023 --- [host-check-agent-status2,6bb16aa9eeeb659e,d06f2a7a16213945] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:41:16.393 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:41:16.392856155+08:00 2024-09-11 17:41:16.394 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:41:16.394 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:41:16.395 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:41:16.393+08:00 2024-09-11 17:41:16.396 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:41:16.396133218+08:00 2024-09-11 17:41:16.398 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:41:16.393+08:00 and status AVAILABLE 2024-09-11 17:41:16.398 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:41:16.396+08:00 2024-09-11 17:41:16.400 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:41:16.401 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:41:16.396+08:00 and status AVAILABLE 2024-09-11 17:41:16.402 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:41:16.401896688+08:00 2024-09-11 17:41:16.403 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.404 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:41:16.402+08:00 2024-09-11 17:41:16.404 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:16.405 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:41:16.454 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:41:16.456 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.456 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:16.457 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:41:16.402+08:00 and status AVAILABLE 2024-09-11 17:41:16.458 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:41:16.459 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:41:16.460 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:41:16.459336+08:00, lastProcessTime=2024-09-11T17:41:06.352186+08:00 2024-09-11 17:41:16.460 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:16.461 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:41:16.462 INFO 2981023 --- [ocp-schedules-3,afcbd7ced0161c1e,9e9c151cc31ea73c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:41:16.465 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:41:16.466 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:41:16.467 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:41:16.467 INFO 2981023 --- [host-check-agent-status1,6bb16aa9eeeb659e,4840a3bbf2ac95d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:41:16.469 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.469 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:16.470 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.470 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:16.471 INFO 2981023 --- [host-check-agent-status1,6bb16aa9eeeb659e,4840a3bbf2ac95d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:41:16.471 INFO 2981023 --- [host-check-agent-status1,6bb16aa9eeeb659e,4840a3bbf2ac95d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:41:16.471 INFO 2981023 --- [host-check-agent-status1,6bb16aa9eeeb659e,4840a3bbf2ac95d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:41:16.473 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:41:16.474 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.474 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:41:16.474 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:16.479 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:41:16.481 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:16.482 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:16.491 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:41:16.506 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:41:16.512 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:41:16.517 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:41:16.520 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:41:16.522 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.522 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.523 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:41:16.526 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:41:16.532 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:41:16.537 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:41:16.539 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:41:16.558 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:41:16.562 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:41:16.569 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.570 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.571 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:41:16.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.573 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:16.575 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:41:16.576 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.577 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:41:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:41:16.579 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:41:16.579 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:41:16.582 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:41:16.595 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:41:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:41:16.605 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:41:16.609 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:41:16.613 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:41:16.614 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.614 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.615 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:16.615 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:41:16.616 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.618 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:41:16.620 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:41:16.620 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:41:16.621 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:41:16.623 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:16.625 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:41:16.630 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:41:16.634 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:16.634 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:41:16.636 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.636 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.639 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:16.640 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:41:16.655 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:41:16.659 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:41:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:41:16.662 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:41:16.666 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:16.667 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:41:16.667 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:41:16.674 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:41:16.675 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.676 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.676 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:16.677 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:41:16.679 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:41:16.680 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:41:16.684 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:41:16.686 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:41:16.687 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:41:16.690 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:41:16.691 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:41:16.699 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.701 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.701 INFO 2981023 --- [host-check-host-status2,31b45ec4a803c9fa,8abdea9cf5b78381] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:41:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:16.703 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:41:16.704 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.706 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:41:16.707 INFO 2981023 --- [host-check-host-status1,31b45ec4a803c9fa,d980bd44d3e8e3a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:41:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:41:16.711 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:41:16.719 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:41:16.720 INFO 2981023 --- [host-check-host-status3,31b45ec4a803c9fa,bbe2a637ac4fa8e6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:41:16.720 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:41:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:41:16.738 INFO 2981023 --- [host-check-host-status4,31b45ec4a803c9fa,d64f76812eb2fb05] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:41:16.739 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:16.747 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.749 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:16.749 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:41:16.754 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:41:16.758 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:41:16.759 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:41:16.768 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:41:16.776 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.776 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:41:16.777 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.777 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:41:16.778 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:41:16.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:41:16.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:41:16.785 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.786 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.786 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:41:16.792 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:41:16.792 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047084472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:41:16.805 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:41:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:41:16.813 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:41:16.459336+08:00 2024-09-11 17:41:16.818 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:41:16.818 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:41:16.818 INFO 2981023 --- [ocp-schedules-10,09655704ab547dce,103ecf19954f3f05] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:41:16.979 INFO 2981023 --- [host-check-clockdiff2,3ec7606f61b869e7,5a433fee95100e7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047676 0 0 , error= 2024-09-11 17:41:17.257 INFO 2981023 --- [ocp-async-9,6c444c2a4537d7e7,3e07631cde5ff23b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,fb18c0082fdd1ad2,43bae6b3bc8eaf83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,fb18c0082fdd1ad2,43bae6b3bc8eaf83] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:17.317 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:17.319 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:17.757 INFO 2981023 --- [ocp-async-4,dd655b0135db7ed4,67b5c9a0f40ea1f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:41:17.843 INFO 2981023 --- [host-check-clockdiff1,3ec7606f61b869e7,b680861c5b369274] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047677 0 0 , error= 2024-09-11 17:41:19.257 INFO 2981023 --- [ocp-async-8,6e6ccf494301da40,87372037c55cf88a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,2cac7de2bed5fcc2,290dafa7eb1eec9c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,2cac7de2bed5fcc2,290dafa7eb1eec9c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:20.320 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:20.325 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:21.034 INFO 2981023 --- [ocp-schedules-2,ed7a0cf195f51393,f33b2df5c074344f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:41:21.080 WARN 2981023 --- [scanner-13529441-1,ed7a0cf195f51393,a5205faa412e812f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:41:21.257 INFO 2981023 --- [ocp-async-6,39931dad096ff6fd,1786b6da6d0a32ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:21.996 INFO 2981023 --- [host-check-clockdiff3,3ec7606f61b869e7,ebd7218a288b2254] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047681 1 1 , error= 2024-09-11 17:41:22.058 INFO 2981023 --- [ocp-schedules-30,8b5af2ade9a86a60,80b3720f28509273] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:41:22.957 INFO 2981023 --- [ocp-schedules-4,79612a8722ed01ca,b270b7558cc954fb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:41:22.957 INFO 2981023 --- [ocp-schedules-4,79612a8722ed01ca,b270b7558cc954fb] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000008 2024-09-11 17:41:22.958 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:41:22.958 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:41:22.958 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:41:22.961 INFO 2981023 --- [ocp-schedules-4,79612a8722ed01ca,b270b7558cc954fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:22.962 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:41:22.962 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:41:22.963 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:41:22.963 INFO 2981023 --- [ocp-schedules-4,79612a8722ed01ca,b270b7558cc954fb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:22.963 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:41:22.964 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:41:22.965 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:41:22.979 INFO 2981023 --- [ocp-schedules-4,79612a8722ed01ca,b270b7558cc954fb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:41:22.979 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:41:22.981 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:41:22.982 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:41:22.983 INFO 2981023 --- [ocp-schedules-4,79612a8722ed01ca,b270b7558cc954fb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:41:23.257 INFO 2981023 --- [ocp-async-10,cd7830cf8723f8cd,9f2e1416ef844c66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,e20ed065e3020a3e,5e6b0e05ec245e62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,e20ed065e3020a3e,5e6b0e05ec245e62] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:23.325 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:23.325 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:24.252 INFO 2981023 --- [ocp-schedules-13,f6c5726930a68f2c,43de3265194e7ac1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:41:24.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,baf0b153b107476d,4b206923090de462] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&version=2&ObClusterId=1719994443, client=10.31.35.6, traceId=baf0b153b107476d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkrtd,1719994443,2,{"ObRegion":"linkrtd","ObCluster":"linkrtd","ObRegionId":1719994443,"ObClusterId":1719994443,"Type":"PRIMARY","timestamp":1726047684469722,"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:41:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,baf0b153b107476d,4b206923090de462] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkrtd, obClusterId=1719994443, rootServerJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, groupName=linkrtd, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:41:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,baf0b153b107476d,4b206923090de462] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:41:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,baf0b153b107476d,4b206923090de462] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkrtd, clusterId=1719994443, affectRows=1 2024-09-11 17:41:24.480 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,baf0b153b107476d,4b206923090de462] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.6, traceId=baf0b153b107476d, duration=9 ms] 2024-09-11 17:41:25.257 INFO 2981023 --- [ocp-async-5,428212d4bcee25d7,260732e18009ee62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,0054b67f94eb49e1,0d936f911f5bbfc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,0054b67f94eb49e1,0d936f911f5bbfc7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:26.326 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:26.326 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:41:26.337187+08:00, lastProcessTime=2024-09-11T17:41:16.459336+08:00 2024-09-11 17:41:26.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:41:26.337187+08:00 2024-09-11 17:41:27.257 INFO 2981023 --- [ocp-async-4,b8ec6909abeb612a,fc8c7f9368174022] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:27.757 INFO 2981023 --- [ocp-async-5,28ec4fa62d828f50,9b2b0177fa05e071] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:41:29.257 INFO 2981023 --- [ocp-async-10,e9ae2820c6a9c4a1,3a3c2f2e5f73b35e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,695137e954adde45,cc805259a17d988f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,695137e954adde45,cc805259a17d988f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:29.326 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:29.327 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:31.237 INFO 2981023 --- [ocp-schedules-15,cd67b7eeb8063013,1a44f53bbb7946b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:41:31.263 INFO 2981023 --- [ocp-async-3,9b5ccc0e480f4d47,bf98a9ebcaf4eda1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:31.361 WARN 2981023 --- [scanner-13529689-1,cd67b7eeb8063013,f48dd8cc9da451e0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:41:31.378 INFO 2981023 --- [alarm-detect-16,cd67b7eeb8063013,16870fe627d8f7a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644412, data is empty. 2024-09-11 17:41:31.383 INFO 2981023 --- [alarm-detect-16,cd67b7eeb8063013,16870fe627d8f7a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644412, data is empty. 2024-09-11 17:41:31.388 INFO 2981023 --- [alarm-detect-16,cd67b7eeb8063013,16870fe627d8f7a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644412, data is empty. 2024-09-11 17:41:31.401 INFO 2981023 --- [alarm-detect-16,cd67b7eeb8063013,fed1b52d3f73c991] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644412, data is empty. 2024-09-11 17:41:31.404 INFO 2981023 --- [alarm-detect-15,cd67b7eeb8063013,c7db905e04a05bef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644412, data is empty. 2024-09-11 17:41:31.409 INFO 2981023 --- [alarm-detect-16,cd67b7eeb8063013,fed1b52d3f73c991] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644412, data is empty. 2024-09-11 17:41:31.413 INFO 2981023 --- [alarm-detect-16,cd67b7eeb8063013,fed1b52d3f73c991] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644412, data is empty. 2024-09-11 17:41:31.414 INFO 2981023 --- [alarm-detect-15,cd67b7eeb8063013,c7db905e04a05bef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644412, data is empty. 2024-09-11 17:41:31.420 INFO 2981023 --- [alarm-detect-15,cd67b7eeb8063013,c7db905e04a05bef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644412, data is empty. 2024-09-11 17:41:31.557 INFO 2981023 --- [alarm-detect-11,cd67b7eeb8063013,ae3fbd6ebc8bb125] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644412, data is empty. 2024-09-11 17:41:31.560 INFO 2981023 --- [alarm-detect-11,cd67b7eeb8063013,ae3fbd6ebc8bb125] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644412, data is empty. 2024-09-11 17:41:31.562 INFO 2981023 --- [alarm-detect-11,cd67b7eeb8063013,ae3fbd6ebc8bb125] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644412, data is empty. 2024-09-11 17:41:32.061 INFO 2981023 --- [ocp-schedules-7,f7d07af8efd586b6,171c75e09376f90b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:41:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,f00847bf7af8ed12,1bcbe4808dd3cc48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,f00847bf7af8ed12,1bcbe4808dd3cc48] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:32.327 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:32.328 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:33.258 INFO 2981023 --- [ocp-async-2,31ea0bc387adc825,f6bfe196133b28c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:34.133 INFO 2981023 --- [ocp-schedules-7,14b80bf9d62752ef,e41701fcabdee9da] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:41:34.147 INFO 2981023 --- [ocp-schedules-7,14b80bf9d62752ef,e41701fcabdee9da] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:41:34.147 INFO 2981023 --- [ocp-schedules-7,14b80bf9d62752ef,e41701fcabdee9da] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:41:35.258 INFO 2981023 --- [ocp-async-1,71055bb0aa293d0a,9775266ffcf1cb28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,a726ab99c95e0a9b,3fb4060c228985c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,a726ab99c95e0a9b,3fb4060c228985c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:35.328 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:35.328 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:41:36.337088+08:00, lastProcessTime=2024-09-11T17:41:26.337187+08:00 2024-09-11 17:41:36.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:41:36.337088+08:00 2024-09-11 17:41:37.258 INFO 2981023 --- [ocp-async-10,8212ca6ab086ba2d,8b6f030609eb5b36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:37.314 INFO 2981023 --- [prometheus-cache-monitor1,8d216652a9a6e64a,3c5658ede31d87fe] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:41:37.315 INFO 2981023 --- [prometheus-cache-monitor1,8d216652a9a6e64a,3c5658ede31d87fe] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:41:37.757 INFO 2981023 --- [ocp-async-6,5063664043f7798e,f3227a93e5085823] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:41:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,da435729f6fe43d1,8065fb08ae5b54cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,da435729f6fe43d1,8065fb08ae5b54cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:38.328 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:38.329 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:39.258 INFO 2981023 --- [ocp-async-8,fbe5968f9a29b052,2523b6c11d2ced9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:40.002 INFO 2981023 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:41:40.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:41:40.422 INFO 2981023 --- [ocp-schedules-25,3d1480023c51c545,aa2c61ec8d4395e7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:41:40.441 WARN 2981023 --- [ocp-schedules-25,3d1480023c51c545,aa2c61ec8d4395e7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:41:41.258 INFO 2981023 --- [ocp-async-1,ae8740e2325b3c3c,06cd1a4b81bef1a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,4ba10ff3be33bb6b,09622fe600bdbaa3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,4ba10ff3be33bb6b,09622fe600bdbaa3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:41.329 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:41.330 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:41.614 INFO 2981023 --- [ocp-schedules-10,bdc133b486e0bf21,e3ead0dd68bb673f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:41:41.768 WARN 2981023 --- [scanner-13529998-1,bdc133b486e0bf21,4417370ff5d2e996] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:41:42.063 INFO 2981023 --- [ocp-schedules-9,67d15a75c72cda93,d410e95b1b4db280] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:41:43.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,922d9baa2aecae93,10ffaef5a0d9e18c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=922d9baa2aecae93, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 17:41:43.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,922d9baa2aecae93,10ffaef5a0d9e18c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=922d9baa2aecae93, duration=12 ms] 2024-09-11 17:41:43.194 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,006f42ee75dd0812,d21f2cd831042531] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=006f42ee75dd0812, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 17:41:43.204 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,7e2f1050485c55b6,e93de667b8193c05] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=7e2f1050485c55b6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 17:41:43.207 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,7e2f1050485c55b6,e93de667b8193c05] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=7e2f1050485c55b6, duration=5 ms] 2024-09-11 17:41:43.209 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ac20161baf055c61,a829d523fb365479] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=ac20161baf055c61, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 17:41:43.210 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ac20161baf055c61,a829d523fb365479] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=ac20161baf055c61, duration=4 ms] 2024-09-11 17:41:43.218 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e86907f3544b4f68,d4afca21a5c0ab37] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=e86907f3544b4f68, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 17:41:43.218 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e86907f3544b4f68,d4afca21a5c0ab37] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=e86907f3544b4f68, duration=2 ms] 2024-09-11 17:41:43.229 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,006f42ee75dd0812,d21f2cd831042531] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=006f42ee75dd0812, duration=40 ms] 2024-09-11 17:41:43.233 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3f0077c2d6a5d6ea,5c0ebfe55890a9eb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_TENANT&type=TOP&maxPoints=360, client=10.31.33.41, traceId=3f0077c2d6a5d6ea, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_TENANT,] 2024-09-11 17:41:43.243 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3f0077c2d6a5d6ea,5c0ebfe55890a9eb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=3f0077c2d6a5d6ea, duration=13 ms] 2024-09-11 17:41:43.251 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,621f1e09d04235b7,f21be9c940fdc2ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_ODP&type=TOP&maxPoints=360, client=10.31.33.41, traceId=621f1e09d04235b7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_ODP,] 2024-09-11 17:41:43.254 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,621f1e09d04235b7,f21be9c940fdc2ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=621f1e09d04235b7, duration=5 ms] 2024-09-11 17:41:43.258 INFO 2981023 --- [ocp-async-5,ee0f4925b87ab756,d6849bd168d6a595] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:43.334 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,eea77de207ed306e,a2fc60286f9d96ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_cpu_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=eea77de207ed306e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_cpu_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 17:41:43.335 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,3f43427eb184511e,532b3f0cf326723b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_disk_used_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=3f43427eb184511e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_disk_used_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 17:41:43.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,eea77de207ed306e,a2fc60286f9d96ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=eea77de207ed306e, duration=7 ms] 2024-09-11 17:41:43.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,3f43427eb184511e,532b3f0cf326723b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=3f43427eb184511e, duration=5 ms] 2024-09-11 17:41:43.341 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a63414ee300c0123,609cb3112b9d6ff8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_HOST&type=TOP&maxPoints=360, client=10.31.33.41, traceId=a63414ee300c0123, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_HOST,] 2024-09-11 17:41:43.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,2362705ea5b60c3d,5ca0914bb2b9b044] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_memory_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=2362705ea5b60c3d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_memory_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 17:41:43.346 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a63414ee300c0123,609cb3112b9d6ff8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=a63414ee300c0123, duration=8 ms] 2024-09-11 17:41:43.348 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,2362705ea5b60c3d,5ca0914bb2b9b044] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=2362705ea5b60c3d, duration=5 ms] 2024-09-11 17:41:43.372 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,61bcd1975ad8ad5a,5907ff3a54c98835] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_memory_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=61bcd1975ad8ad5a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_memory_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 17:41:43.376 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,61bcd1975ad8ad5a,5907ff3a54c98835] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=61bcd1975ad8ad5a, duration=6 ms] 2024-09-11 17:41:43.382 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,d7c1f6bff16891ff,4def7f3ad64c6b31] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_cpu_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=d7c1f6bff16891ff, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_cpu_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 17:41:43.386 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,d7c1f6bff16891ff,4def7f3ad64c6b31] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=d7c1f6bff16891ff, duration=6 ms] 2024-09-11 17:41:43.445 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,d1b15bd085ab4925,344be290d8fdb39a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_tenant_disk_usage_v4&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=d1b15bd085ab4925, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_tenant_disk_usage_v4,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 17:41:43.449 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,d1b15bd085ab4925,344be290d8fdb39a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=d1b15bd085ab4925, duration=6 ms] 2024-09-11 17:41:43.464 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,93d28838df1a92c4,cd8f5c94dd4acdf5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?foldStandby=false, client=10.31.33.41, traceId=93d28838df1a92c4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 17:41:43.470 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,93d28838df1a92c4,cd8f5c94dd4acdf5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=93d28838df1a92c4, duration=8 ms] 2024-09-11 17:41:43.472 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,abc30c2a75c6ae7e,0d06e0373ee9b96d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.31.33.41, traceId=abc30c2a75c6ae7e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-09-11 17:41:43.475 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,abc30c2a75c6ae7e,0d06e0373ee9b96d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.31.33.41, traceId=abc30c2a75c6ae7e, duration=5 ms] 2024-09-11 17:41:43.477 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,53af78f8c237a9d2,70c51eeb742f0960] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.31.33.41, traceId=53af78f8c237a9d2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-09-11 17:41:43.477 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,53af78f8c237a9d2,70c51eeb742f0960] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.31.33.41, traceId=53af78f8c237a9d2, duration=2 ms] 2024-09-11 17:41:43.506 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0b0034d57939ac7a,cfa63670745bfba5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.31.33.41, traceId=0b0034d57939ac7a, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-09-11 17:41:43.506 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0b0034d57939ac7a,cfa63670745bfba5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.31.33.41, traceId=0b0034d57939ac7a, duration=2 ms] 2024-09-11 17:41:43.515 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f9d91f853539b6a9,9fa92aea335c50bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.31.33.41, traceId=f9d91f853539b6a9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-09-11 17:41:43.515 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f9d91f853539b6a9,9fa92aea335c50bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.31.33.41, traceId=f9d91f853539b6a9, duration=1 ms] 2024-09-11 17:41:43.588 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,570eae104d081a3c,00b1dd9aa27293c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.31.33.41, traceId=570eae104d081a3c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-09-11 17:41:43.590 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,570eae104d081a3c,00b1dd9aa27293c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.31.33.41, traceId=570eae104d081a3c, duration=4 ms] 2024-09-11 17:41:43.593 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,4ab840ee9ba28c2f,7d0ff589c40001c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.31.33.41, traceId=4ab840ee9ba28c2f, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-09-11 17:41:43.593 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,4ab840ee9ba28c2f,7d0ff589c40001c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.31.33.41, traceId=4ab840ee9ba28c2f, duration=2 ms] 2024-09-11 17:41:43.636 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,da1a7ff0d6d1e625,ee5c499afb0fa274] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.31.33.41, traceId=da1a7ff0d6d1e625, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:41:43.677 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 17:41:43.677 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, client=10.31.33.41, traceId=] 2024-09-11 17:41:43.677 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 17:41:43.679 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 17:41:43.679 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, client=10.31.33.41, traceId=] 2024-09-11 17:41:43.679 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 17:41:43.793 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,da1a7ff0d6d1e625,ee5c499afb0fa274] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.31.33.41, traceId=da1a7ff0d6d1e625, duration=159 ms] 2024-09-11 17:41:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,2ecb436a3e023cb6,54fcb506dcda1939] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,2ecb436a3e023cb6,54fcb506dcda1939] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:44.330 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:44.331 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:45.101 INFO 2981023 --- [prometheus-second-schedule-1,967ae1f2ec7076a4,14bce515128cae08] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:41:45.258 INFO 2981023 --- [ocp-async-3,6661b47417ed8764,ebd50df2186ab351] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:45.998 INFO 2981023 --- [ocp-schedules-21,ca5c77b0ac0affe0,1a31945763b154b8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:41:45.998 INFO 2981023 --- [host-check-clockdiff1,ca5c77b0ac0affe0,20185ab96933ce0a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:41:45.998 INFO 2981023 --- [host-check-clockdiff2,ca5c77b0ac0affe0,3d7dfb48f8908016] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:41:45.999 INFO 2981023 --- [host-check-clockdiff3,ca5c77b0ac0affe0,afbc5633f417dfb0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:41:45.999 INFO 2981023 --- [host-check-clockdiff4,ca5c77b0ac0affe0,21984b1aba1f3819] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:41:46.005 INFO 2981023 --- [ocp-schedules-18,90690fc3f0bacce8,9dc36178dd32ce4a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:41:46.007 INFO 2981023 --- [ocp-schedules-13,b29fd0fd527764c3,0e07e800de7099ad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.7, 10.31.35.10] 2024-09-11 17:41:46.015 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:41:46.015 INFO 2981023 --- [host-check-agent-status1,90690fc3f0bacce8,080b1f6680f0a0c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:41:46.015 INFO 2981023 --- [host-check-agent-status2,90690fc3f0bacce8,30135758ea983023] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:41:46.015 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:41:46.015 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:41:46.018 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:41:46.022 INFO 2981023 --- [host-check-agent-status3,90690fc3f0bacce8,41f6a665c11a6ab8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000008, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:41:46.029 INFO 2981023 --- [host-check-clockdiff4,ca5c77b0ac0affe0,21984b1aba1f3819] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047706 0 0 , error= 2024-09-11 17:41:46.029 INFO 2981023 --- [host-check-agent-status4,90690fc3f0bacce8,062630c527cdcae6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:41:46.030 INFO 2981023 --- [host-check-agent-status1,90690fc3f0bacce8,080b1f6680f0a0c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:41:46.032 INFO 2981023 --- [host-check-agent-status1,90690fc3f0bacce8,080b1f6680f0a0c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:41:46.032 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:41:46.032 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:46.032 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:41:46.033 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:41:46.033 INFO 2981023 --- [host-check-agent-status2,90690fc3f0bacce8,30135758ea983023] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:41:46.033 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:46.034 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:41:46.034 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:41:46.034 INFO 2981023 --- [host-check-agent-status4,90690fc3f0bacce8,062630c527cdcae6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:41:46.035 INFO 2981023 --- [host-check-agent-status2,90690fc3f0bacce8,30135758ea983023] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:41:46.035 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:41:46.036 INFO 2981023 --- [host-check-agent-status4,90690fc3f0bacce8,062630c527cdcae6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:41:46.036 INFO 2981023 --- [host-check-agent-status3,90690fc3f0bacce8,41f6a665c11a6ab8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:46.038 INFO 2981023 --- [host-check-agent-status3,90690fc3f0bacce8,41f6a665c11a6ab8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:46.063 INFO 2981023 --- [host-check-agent-status4,90690fc3f0bacce8,062630c527cdcae6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:41:46.064 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:41:46.065 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:41:46.066 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:41:46.065748149+08:00 2024-09-11 17:41:46.066 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:41:46.065718191+08:00 2024-09-11 17:41:46.066 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:41:46.066 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:41:46.066+08:00 2024-09-11 17:41:46.066 INFO 2981023 --- [host-check-agent-status2,90690fc3f0bacce8,30135758ea983023] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:41:46.067 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:41:46.066+08:00 2024-09-11 17:41:46.067 INFO 2981023 --- [host-check-agent-status4,90690fc3f0bacce8,062630c527cdcae6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:41:46.067 INFO 2981023 --- [host-check-agent-status4,90690fc3f0bacce8,062630c527cdcae6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:41:46.067 INFO 2981023 --- [host-check-agent-status4,90690fc3f0bacce8,062630c527cdcae6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:41:46.068 INFO 2981023 --- [host-check-agent-status1,90690fc3f0bacce8,080b1f6680f0a0c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:41:46.068 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000008, ip=10.31.35.7, timestamp=2024-09-11T17:41:46.068599406+08:00 2024-09-11 17:41:46.068 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:41:46.066+08:00 and status AVAILABLE 2024-09-11 17:41:46.069 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:41:46.066+08:00 and status AVAILABLE 2024-09-11 17:41:46.069 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:41:46.069 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000008, lastAvailableTime=2024-09-11T17:41:46.068+08:00 2024-09-11 17:41:46.069 INFO 2981023 --- [host-check-agent-status3,90690fc3f0bacce8,41f6a665c11a6ab8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:41:46.070 INFO 2981023 --- [host-check-agent-status2,90690fc3f0bacce8,30135758ea983023] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:41:46.070 INFO 2981023 --- [host-check-agent-status2,90690fc3f0bacce8,30135758ea983023] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:41:46.070 INFO 2981023 --- [host-check-agent-status2,90690fc3f0bacce8,30135758ea983023] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:41:46.070 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:41:46.070467256+08:00 2024-09-11 17:41:46.070 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000008 last available time 2024-09-11T17:41:46.068+08:00 and status AVAILABLE 2024-09-11 17:41:46.071 INFO 2981023 --- [host-check-agent-status1,90690fc3f0bacce8,080b1f6680f0a0c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:41:46.071 INFO 2981023 --- [host-check-agent-status1,90690fc3f0bacce8,080b1f6680f0a0c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:41:46.071 INFO 2981023 --- [host-check-agent-status1,90690fc3f0bacce8,080b1f6680f0a0c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:41:46.072 INFO 2981023 --- [host-check-agent-status3,90690fc3f0bacce8,41f6a665c11a6ab8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:41:46.072 INFO 2981023 --- [host-check-agent-status3,90690fc3f0bacce8,41f6a665c11a6ab8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:41:46.072 INFO 2981023 --- [host-check-agent-status3,90690fc3f0bacce8,41f6a665c11a6ab8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:41:46.073 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:41:46.070+08:00 2024-09-11 17:41:46.075 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:41:46.070+08:00 and status AVAILABLE 2024-09-11 17:41:46.081 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:41:46.081 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:41:46.082 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000008 2024-09-11 17:41:46.083 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:41:46.091 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:41:46.092 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:41:46.093 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:41:46.095 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:41:46.095 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:46.097 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:46.105 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:41:46.110 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:41:46.119 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:41:46.121 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:41:46.122 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:41:46.144 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000046 2024-09-11 17:41:46.158 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:41:46.159 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:41:46.181 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000046 2024-09-11 17:41:46.187 INFO 2981023 --- [ocp-schedules-3,1bd0389832ea28aa,45b562d0ec5e3e09] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:41:46.188 INFO 2981023 --- [ocp-schedules-10,e2d772bc7010a275,78aacbe38a6e7107] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:41:46.188 INFO 2981023 --- [ocp-schedules-29,b6e1bc5f872bc802,03d3480d2f0eb423] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:41:46.191 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:41:46.194 INFO 2981023 --- [ocp-schedules-29,b6e1bc5f872bc802,03d3480d2f0eb423] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:41:46.195 INFO 2981023 --- [ocp-schedules-10,e2d772bc7010a275,78aacbe38a6e7107] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:41:46.195 INFO 2981023 --- [ocp-schedules-25,4fc14537987b48b0,7730550e934ecc25] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:41:46.195 INFO 2981023 --- [ocp-schedules-14,9053e6d7413412c4,2bf648c9e7c06989] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:41:46.202 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:41:46.208 INFO 2981023 --- [ocp-schedules-25,4fc14537987b48b0,7730550e934ecc25] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:41:46.226 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:41:46.229 INFO 2981023 --- [ocp-schedules-3,1bd0389832ea28aa,45b562d0ec5e3e09] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:41:46.232 INFO 2981023 --- [ocp-schedules-29,b6e1bc5f872bc802,03d3480d2f0eb423] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:41:46.234 INFO 2981023 --- [ocp-schedules-22,7e41d61f3cbff6d2,1f2ad9f935cc6645] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:41:46.234 INFO 2981023 --- [ocp-schedules-22,7e41d61f3cbff6d2,1f2ad9f935cc6645] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:41:46.236 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:41:46.252 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:41:46.255 INFO 2981023 --- [ocp-schedules-5,9597bf986a30bb16,644a5496dcc2be23] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:41:46.255 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:41:46.257 INFO 2981023 --- [ocp-schedules-5,9597bf986a30bb16,644a5496dcc2be23] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:41:46.261 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:41:46.264 INFO 2981023 --- [ocp-schedules-16,eb1b1b14c90a6574,09d40db4d9bb9639] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:41:46.279 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.279 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:46.281 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:41:46.281 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:41:46.285 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:41:46.298 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:41:46.304 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:41:46.304 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.305 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:46.308 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.310 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.311 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:41:46.311 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:41:46.311 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:46.311 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:41:46.311 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.312 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:41:46.314 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.314 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:41:46.319 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000047 2024-09-11 17:41:46.320 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:41:46.321 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.321 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:41:46.323 INFO 2981023 --- [ocp-schedules-10,e2d772bc7010a275,78aacbe38a6e7107] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:41:46.323 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.325 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:41:46.326 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:41:46.326 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,2ae8f69157b2e431,2a49b23204106590] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=2ae8f69157b2e431, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:41:46.326 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,2ae8f69157b2e431,2a49b23204106590] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=2ae8f69157b2e431, duration=0 ms] 2024-09-11 17:41:46.333 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,f9fd5c2dc5dabd57,2613e5b78e67b12c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&ObClusterId=1719994443&version=2, client=10.31.35.10, traceId=f9fd5c2dc5dabd57, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkrtd,1719994443,false,2,] 2024-09-11 17:41:46.333 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,f9fd5c2dc5dabd57,2613e5b78e67b12c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=f9fd5c2dc5dabd57, duration=0 ms] 2024-09-11 17:41:46.337 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.338 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:46.339 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:41:46.348 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.349 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:41:46.350 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.354 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.356 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c9cf1c6149bbda69,4c9725832f580ead] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=c9cf1c6149bbda69, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:41:46.356 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c9cf1c6149bbda69,4c9725832f580ead] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=c9cf1c6149bbda69, duration=0 ms] 2024-09-11 17:41:46.358 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.358 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:46.360 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:41:46.354 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:46.361 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.361 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:41:46.361 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:41:46.362 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000047 2024-09-11 17:41:46.366 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.368 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.369 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:41:46.369 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:41:46.371 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:41:46.373 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.376 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:41:46.381 INFO 2981023 --- [ocp-schedules-5,9597bf986a30bb16,644a5496dcc2be23] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:41:46.386 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:41:46.384669+08:00, lastProcessTime=2024-09-11T17:41:36.337088+08:00 2024-09-11 17:41:46.387 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:41:46.388 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:41:46.399 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:41:46.400 INFO 2981023 --- [host-check-host-status1,b29fd0fd527764c3,45d42bc5c1104109] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:41:46.406 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:41:46.420 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000048 2024-09-11 17:41:46.420 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:41:46.431 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.432 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.432 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:41:46.433 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:41:46.433 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:41:46.433 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.444 INFO 2981023 --- [host-check-host-status3,b29fd0fd527764c3,eda44ad06bbcf55a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000048 2024-09-11 17:41:46.445 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.445 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.445 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:41:46.448 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:41:46.459 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:41:46.460 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.460 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.461 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:41:46.461 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:41:46.468 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.471 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:41:46.472 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:41:46.484 INFO 2981023 --- [host-check-host-status2,b29fd0fd527764c3,8d2568f50cc7f880] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:41:46.488 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:41:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:46.499 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:41:46.499 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.502 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:41:46.502 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:41:46.509 INFO 2981023 --- [pool-10-thread-10,b31a20a845f80b02,ff2f46d98acdbf0b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:41:46.509 INFO 2981023 --- [pool-10-thread-9,b31a20a845f80b02,b2a9c430696fd13e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:41:46.509 INFO 2981023 --- [pool-10-thread-9,b31a20a845f80b02,b2a9c430696fd13e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:41:46.510 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.511 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.511 INFO 2981023 --- [host-check-host-status4,b29fd0fd527764c3,532075fdf136a64e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:41:46.512 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:46.512 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:41:46.513 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.514 INFO 2981023 --- [pool-10-thread-11,b31a20a845f80b02,e54c75c66f4d6d3a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:41:46.516 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:41:46.517 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:41:46.517 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.517 INFO 2981023 --- [pool-10-thread-12,b31a20a845f80b02,6688383783b93800] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:41:46.518 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.519 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:41:46.521 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.523 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:41:46.523 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:46.523 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:41:46.524 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.524 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:41:46.533 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:41:46.533 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:41:46.538 INFO 2981023 --- [pool-10-thread-11,b31a20a845f80b02,e54c75c66f4d6d3a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:41:46.538 INFO 2981023 --- [pool-10-thread-11,b31a20a845f80b02,e54c75c66f4d6d3a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:41:46.544 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:41:46.544 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:41:46.545 INFO 2981023 --- [ForkJoinPool-6-worker-13,b31a20a845f80b02,d26cf079f0457b42] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:41:46.545 INFO 2981023 --- [ForkJoinPool-6-worker-13,b31a20a845f80b02,d26cf079f0457b42] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:41:46.545 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.545 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.546 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:41:46.546 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:41:46.546 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.547 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:41:46.547 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:41:46.547 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.548 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:46.548 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:41:46.553 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:41:46.559 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:46.559 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:41:46.577 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:46.588 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:46.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:41:46.594 WARN 2981023 --- [ForkJoinPool-6-worker-13,b31a20a845f80b02,d26cf079f0457b42] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:41:46.596 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:41:46.598 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.598 INFO 2981023 --- [ForkJoinPool-6-worker-11,b31a20a845f80b02,0728965e0b527d4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.598 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:46.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:41:46.599 INFO 2981023 --- [ForkJoinPool-6-worker-11,b31a20a845f80b02,0728965e0b527d4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:41:46.601 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.601 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.602 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:41:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:46.603 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.603 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.603 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:41:46.603 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:41:46.603 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:41:46.604 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.605 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:41:46.606 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:41:46.606 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.609 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:41:46.611 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:41:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.614 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:41:46.616 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:41:46.618 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:41:46.619 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:41:46.622 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:41:46.629 INFO 2981023 --- [ForkJoinPool-6-worker-11,b31a20a845f80b02,0728965e0b527d4e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:41:46.630 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:41:46.634 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:41:46.635 INFO 2981023 --- [ForkJoinPool-6-worker-11,b31a20a845f80b02,0728965e0b527d4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.636 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.637 INFO 2981023 --- [ForkJoinPool-6-worker-11,b31a20a845f80b02,0728965e0b527d4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:41:46.637 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:41:46.637 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:41:46.637 INFO 2981023 --- [ForkJoinPool-6-worker-11,b31a20a845f80b02,0728965e0b527d4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.638 INFO 2981023 --- [ForkJoinPool-6-worker-11,b31a20a845f80b02,0728965e0b527d4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:41:46.638 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:41:46, 2024-09-11 17:41:46] 2024-09-11 17:41:46.639 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:41:46.640 INFO 2981023 --- [ForkJoinPool-6-worker-6,b31a20a845f80b02,110d7d941215b9aa] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:41:46.641 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:41:46.642 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.643 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:41:46.647 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.648 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:41:46.648 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:41:46.649 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:41:46.656 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:41:46.663 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.663 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.664 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:41:46.664 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:41:46.664 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.672 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.674 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:41:46.676 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:41:46.679 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:41:46.680 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.680 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.681 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:41:46.682 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:41:46.683 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.683 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:41:46.684 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:41:46.687 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:41:46.692 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:41:46.694 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.695 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:41:46.697 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.698 INFO 2981023 --- [ForkJoinPool-6-worker-9,b31a20a845f80b02,92b69c45b9b93ef7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:41:46.699 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:41:46.699 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.699 INFO 2981023 --- [pool-10-thread-10,b31a20a845f80b02,ff2f46d98acdbf0b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:41:46.700 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:41:46, 2024-09-11 17:41:46] 2024-09-11 17:41:46.707 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:41:46.719 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:41:46.721 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:41:46.727 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:41:46.728 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:41:46, 2024-09-11 17:41:46] 2024-09-11 17:41:46.731 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.732 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.732 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:41:46.732 INFO 2981023 --- [ForkJoinPool-6-worker-4,b31a20a845f80b02,7d81f9dd52629b05] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:41:46.734 INFO 2981023 --- [pool-10-thread-12,b31a20a845f80b02,6688383783b93800] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:41:46.736 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.736 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:41:46.738 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.738 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:41:46.744 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.750 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.754 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:41:46.754 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:41:46.755 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:41:46.757 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:41:46.757 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:41:46.761 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.762 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:41:46.763 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.763 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:41:46.764 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:41:46.764 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.781 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:46.781 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:46.782 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:41:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:41:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:41:46.790 INFO 2981023 --- [ocp-schedules-25,4fc14537987b48b0,7730550e934ecc25] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:41:46.840 INFO 2981023 --- [ocp-schedules-11,35570bb17d856414,33dec5ea5ff52a14] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:41:46.840 INFO 2981023 --- [ocp-schedules-11,35570bb17d856414,33dec5ea5ff52a14] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:41:46.856 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:41:46.871 INFO 2981023 --- [pool-10-thread-9,b31a20a845f80b02,b2a9c430696fd13e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:41:46.872 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:41:46.384669+08:00 2024-09-11 17:41:46.873 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:41:46.874 INFO 2981023 --- [pool-10-thread-9,b31a20a845f80b02,b2a9c430696fd13e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:41:46.882 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:41:46.884 INFO 2981023 --- [ocp-schedules-15,36ea947e1450dda5,8337d6c5e39e4867] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:41:46.949 INFO 2981023 --- [host-check-clockdiff1,ca5c77b0ac0affe0,20185ab96933ce0a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047706 0 0 , error= 2024-09-11 17:41:47.259 INFO 2981023 --- [ocp-async-1,ecd227091dc0b946,c93f5e4b87ee7822] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,eb60b0710f90f83a,077b6d5ee65edb6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,eb60b0710f90f83a,077b6d5ee65edb6a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:47.331 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:47.332 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:47.426 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,82c8ec32d0aa3340,a527a5dc87cc9c48] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.31.33.41, traceId=82c8ec32d0aa3340, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-09-11 17:41:47.431 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,82c8ec32d0aa3340,a527a5dc87cc9c48] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.31.33.41, traceId=82c8ec32d0aa3340, duration=7 ms] 2024-09-11 17:41:47.757 INFO 2981023 --- [ocp-async-8,5b112f7f983866d3,858e14305e742f18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:41:47.787 INFO 2981023 --- [host-check-clockdiff2,ca5c77b0ac0affe0,3d7dfb48f8908016] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047707 0 0 , error= 2024-09-11 17:41:48.015 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7290a033a7c6ba54,9962a21940c0f5f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.31.33.41, traceId=7290a033a7c6ba54, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:41:48.018 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7290a033a7c6ba54,9962a21940c0f5f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.31.33.41, traceId=7290a033a7c6ba54, duration=5 ms] 2024-09-11 17:41:48.027 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,59db0f0835ad2867,9ef26a6be6e2e065] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/regions?null, client=10.31.33.41, traceId=59db0f0835ad2867, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listRegions(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:41:48.027 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4ee048a911f3d165,6aa79b99d7ec4fa6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=4ee048a911f3d165, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:41:48.032 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4ee048a911f3d165,6aa79b99d7ec4fa6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=4ee048a911f3d165, duration=7 ms] 2024-09-11 17:41:48.037 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,59db0f0835ad2867,9ef26a6be6e2e065] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/regions client=10.31.33.41, traceId=59db0f0835ad2867, duration=12 ms] 2024-09-11 17:41:48.072 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e3db202e77487c0c,800b62af44e35f4c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=e3db202e77487c0c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:41:48.074 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e3db202e77487c0c,800b62af44e35f4c] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:41:48.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e3db202e77487c0c,800b62af44e35f4c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=e3db202e77487c0c, duration=35 ms] 2024-09-11 17:41:49.259 INFO 2981023 --- [ocp-async-4,e9143afb4c7cd4c9,98082ddfc3b8b2ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:49.718 INFO 2981023 --- [ocp-schedules-15,d703d772a234271c,73fc0e5e522eba82] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:41:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:41:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:41:50.233 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:41:50.238 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-7,30d1a24e9b434a6f,12a7890ab77b262e] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:41:50.239 INFO 2981023 --- [ocp-schedules-27,ffe6dd18d5cf4ee4,7353a875e7bf08b9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:41:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,f7300e657e52804e,0afe421be51ebf67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,f7300e657e52804e,0afe421be51ebf67] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:50.332 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:50.333 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:51.259 INFO 2981023 --- [ocp-async-8,a525f17d23e43968,a52204418a589860] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:51.551 INFO 2981023 --- [ocp-schedules-26,2a3b1c2533a16d35,a6f87c0d1f5e951a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:41:51.553 INFO 2981023 --- [ocp-schedules-26,2a3b1c2533a16d35,a6f87c0d1f5e951a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:41:51.553 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:41:51.553 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:41:51.553 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:41:51.553 INFO 2981023 --- [ocp-schedules-26,2a3b1c2533a16d35,a6f87c0d1f5e951a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:41:51.927 INFO 2981023 --- [ocp-schedules-12,2c0ac75f93d8d6c1,1ed1e7a3cb1a8d91] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:41:51.944 WARN 2981023 --- [scanner-13530218-1,2c0ac75f93d8d6c1,4dd18fd5777e43f5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:41:51.999 INFO 2981023 --- [alarm-detect-4,2c0ac75f93d8d6c1,a7e042e5e20e0428] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644412, data is empty. 2024-09-11 17:41:52.004 INFO 2981023 --- [alarm-detect-4,2c0ac75f93d8d6c1,a7e042e5e20e0428] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644412, data is empty. 2024-09-11 17:41:52.008 INFO 2981023 --- [host-check-clockdiff3,ca5c77b0ac0affe0,afbc5633f417dfb0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047712 1 1 , error= 2024-09-11 17:41:52.016 INFO 2981023 --- [alarm-detect-17,2c0ac75f93d8d6c1,518788698e6b5a14] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644412, data is empty. 2024-09-11 17:41:52.020 INFO 2981023 --- [alarm-detect-17,2c0ac75f93d8d6c1,518788698e6b5a14] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644412, data is empty. 2024-09-11 17:41:52.023 INFO 2981023 --- [alarm-detect-7,2c0ac75f93d8d6c1,7c7171c79243a3bd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644412, data is empty. 2024-09-11 17:41:52.024 INFO 2981023 --- [alarm-detect-17,2c0ac75f93d8d6c1,518788698e6b5a14] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644412, data is empty. 2024-09-11 17:41:52.025 INFO 2981023 --- [alarm-detect-7,2c0ac75f93d8d6c1,7c7171c79243a3bd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644412, data is empty. 2024-09-11 17:41:52.028 INFO 2981023 --- [alarm-detect-7,2c0ac75f93d8d6c1,7c7171c79243a3bd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644412, data is empty. 2024-09-11 17:41:52.045 INFO 2981023 --- [alarm-detect-4,2c0ac75f93d8d6c1,a7e042e5e20e0428] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644412, data is empty. 2024-09-11 17:41:52.066 INFO 2981023 --- [ocp-schedules-9,8800c381023979a9,5a3a92a76b0461f9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:41:52.135 INFO 2981023 --- [alarm-detect-18,2c0ac75f93d8d6c1,23d28d944ccb04f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644412, data is empty. 2024-09-11 17:41:52.139 INFO 2981023 --- [alarm-detect-18,2c0ac75f93d8d6c1,23d28d944ccb04f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644412, data is empty. 2024-09-11 17:41:52.141 INFO 2981023 --- [alarm-detect-18,2c0ac75f93d8d6c1,23d28d944ccb04f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644412, data is empty. 2024-09-11 17:41:52.998 INFO 2981023 --- [ocp-schedules-4,91764f7b9e20ac5e,1e31e7119fab06fa] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:41:53.016 WARN 2981023 --- [ocp-schedules-4,91764f7b9e20ac5e,1e31e7119fab06fa] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:41:53.111 INFO 2981023 --- [ocp-schedules-16,adedb6ca0835c49f,71e82f0f38b8af64] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:41:53.111 INFO 2981023 --- [ocp-schedules-16,adedb6ca0835c49f,71e82f0f38b8af64] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:41:53.259 INFO 2981023 --- [ocp-async-1,c3baded03bbce552,e769fa5f46496139] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:53.291 INFO 2981023 --- [ocp-schedules-14,4817396f26358e59,f24c45365db80c87] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:41:53.291+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:41:53.296 INFO 2981023 --- [ocp-schedules-14,4817396f26358e59,f24c45365db80c87] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:41:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,08988e7b77085e11,6a22da51321765da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,08988e7b77085e11,6a22da51321765da] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:53.333 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:53.334 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:53.448 INFO 2981023 --- [ocp-schedules-14,4817396f26358e59,f24c45365db80c87] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 51 to SqlAggregateData 2024-09-11 17:41:53.450 INFO 2981023 --- [ocp-schedules-14,4817396f26358e59,f24c45365db80c87] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:41:54.128 INFO 2981023 --- [ocp-schedules-19,d5c4494dbdfb88ce,a610ad042ac82b81] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:41:54.130 INFO 2981023 --- [ocp-schedules-19,d5c4494dbdfb88ce,a610ad042ac82b81] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:41:54.150 INFO 2981023 --- [ocp-schedules-15,071117631a7453c6,0c9ea2e48fbf4329] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:41:54.161 INFO 2981023 --- [ocp-schedules-15,071117631a7453c6,0c9ea2e48fbf4329] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:41:54.162 INFO 2981023 --- [ocp-schedules-15,071117631a7453c6,0c9ea2e48fbf4329] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:41:54.170 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:41:54.170 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:41:54.171 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.171 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.171 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:41:54.171 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.173 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:41:54.175 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:41:54.175 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:41:54.177 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.177 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.179 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.179 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.180 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.180 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.182 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.182 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.189 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:41:54.190 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.193 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.193 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.196 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.196 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:41:54.199 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.201 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.201 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:41:54.201 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2F31D6EB-0-0] 2024-09-11 17:41:54.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:41:54.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:41:54.203 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.204 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:41:54.206 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.207 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.213 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.213 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.213 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:41:54.214 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:41:54.214 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.214 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.214 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:41:54.215 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:41:54.216 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.216 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:41:54.217 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:41:54.221 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:41:54.227 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.227 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.227 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:41:54.227 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:41:54.228 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:41:54.228 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE2F31D6F5-0-0] 2024-09-11 17:41:54.228 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:41:54.228 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.229 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:41:54.233 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.234 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:54.234 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:41:54.235 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.235 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:41:54.236 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.236 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.236 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.237 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:41:54.237 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.238 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.238 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:41:54.238 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:41:54.238 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.239 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.239 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:41:54.239 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:41:54.239 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.240 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.240 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:41:54.242 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:41:54.242 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.243 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:41:54.243 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.244 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:41:54.244 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.244 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:41:54.245 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:41:54.245 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.246 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:41:54.246 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:41:54.246 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:41:54.248 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:41:54.252 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:41:54.256 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:41:54.256 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:41:54.256 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:41:54.256 INFO 2981023 --- [ocp-schedules-5,558c582a036d30af,ee8ddf5be2a36253] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:41:54.259 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:54.261 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.261 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:41:54.263 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:41:54.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.268 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:41:54.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:41:54.269 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:41:54.270 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.270 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:41:54.276 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:41:54.278 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:41:54.278 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.278 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.279 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:41:54.280 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:41:54.281 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:54.281 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:41:54.284 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:41:54.285 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:41:54.285 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:41:54.285 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:41:54.291 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:41:54.292 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:41:54.292 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:41:54.292 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:41:54.292 INFO 2981023 --- [ocp-schedules-19,d5c4494dbdfb88ce,a610ad042ac82b81] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:41:55.259 INFO 2981023 --- [ocp-async-5,48db594cf6247802,023c759aa3c47c19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:55.704 INFO 2981023 --- [ocp-schedules-5,d35754dce348114c,a4d77230c35daf5e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:41:55.704 INFO 2981023 --- [ocp-schedules-5,d35754dce348114c,a4d77230c35daf5e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:41:55.706 INFO 2981023 --- [ocp-schedules-5,d35754dce348114c,a4d77230c35daf5e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:41:55.706 INFO 2981023 --- [ocp-schedules-5,d35754dce348114c,a4d77230c35daf5e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:41:55.706 INFO 2981023 --- [ocp-schedules-5,d35754dce348114c,a4d77230c35daf5e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:41:55.706 INFO 2981023 --- [ocp-schedules-5,d35754dce348114c,a4d77230c35daf5e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:41:55.803 INFO 2981023 --- [ocp-schedules-4,2ca75612efef5a42,aa33a6daf71630c7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:41:55.810 INFO 2981023 --- [ocp-schedules-4,2ca75612efef5a42,aa33a6daf71630c7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:41:55.810 INFO 2981023 --- [ocp-schedules-4,2ca75612efef5a42,aa33a6daf71630c7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:41:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,d707bdc96e60f020,af940a6467e3a93e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,d707bdc96e60f020,af940a6467e3a93e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:56.334 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:56.335 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:41:56.351507+08:00, lastProcessTime=2024-09-11T17:41:46.384669+08:00 2024-09-11 17:41:56.409 INFO 2981023 --- [ocp-schedules-7,a5fd9d570506724e,240e0485ec7ff67c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:41:56.409 INFO 2981023 --- [ocp-schedules-7,a5fd9d570506724e,240e0485ec7ff67c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:41:56.422 INFO 2981023 --- [Collect all tenants major compactions2,a5fd9d570506724e,5393e21754b57ecf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:41:56.424 INFO 2981023 --- [Collect all tenants major compactions1,a5fd9d570506724e,f8a0cd1e6464cd3b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:41:56.424 INFO 2981023 --- [Collect all tenants major compactions3,a5fd9d570506724e,4e0000c3adcaced6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:41:56.425 INFO 2981023 --- [Collect all tenants major compactions4,a5fd9d570506724e,915803b963989522] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:41:56.427 INFO 2981023 --- [Collect all tenants major compactions5,a5fd9d570506724e,29560398ba374b25] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:41:56.428 INFO 2981023 --- [Collect all tenants major compactions6,a5fd9d570506724e,dc636f48d8cb83d2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:41:56.436 INFO 2981023 --- [Collect all tenants major compactions2,a5fd9d570506724e,5393e21754b57ecf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:56.436 INFO 2981023 --- [Collect all tenants major compactions1,a5fd9d570506724e,f8a0cd1e6464cd3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:56.437 INFO 2981023 --- [Collect all tenants major compactions1,a5fd9d570506724e,f8a0cd1e6464cd3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:41:56.440 INFO 2981023 --- [Collect all tenants major compactions4,a5fd9d570506724e,915803b963989522] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:56.440 INFO 2981023 --- [Collect all tenants major compactions5,a5fd9d570506724e,29560398ba374b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:56.441 INFO 2981023 --- [Collect all tenants major compactions5,a5fd9d570506724e,29560398ba374b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:41:56.441 INFO 2981023 --- [Collect all tenants major compactions4,a5fd9d570506724e,915803b963989522] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:41:56.442 INFO 2981023 --- [Collect all tenants major compactions1,a5fd9d570506724e,f8a0cd1e6464cd3b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:41:56.442 INFO 2981023 --- [Collect all tenants major compactions1,a5fd9d570506724e,f8a0cd1e6464cd3b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:41:56.443 INFO 2981023 --- [Collect all tenants major compactions5,a5fd9d570506724e,29560398ba374b25] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:41:56.443 INFO 2981023 --- [Collect all tenants major compactions5,a5fd9d570506724e,29560398ba374b25] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:41:56.444 INFO 2981023 --- [Collect all tenants major compactions4,a5fd9d570506724e,915803b963989522] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:41:56.444 INFO 2981023 --- [Collect all tenants major compactions4,a5fd9d570506724e,915803b963989522] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:41:56.445 INFO 2981023 --- [Collect all tenants major compactions6,a5fd9d570506724e,dc636f48d8cb83d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:56.446 INFO 2981023 --- [Collect all tenants major compactions6,a5fd9d570506724e,dc636f48d8cb83d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:41:56.446 INFO 2981023 --- [Collect all tenants major compactions2,a5fd9d570506724e,5393e21754b57ecf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:41:56.448 INFO 2981023 --- [Collect all tenants major compactions6,a5fd9d570506724e,dc636f48d8cb83d2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:41:56.448 INFO 2981023 --- [Collect all tenants major compactions2,a5fd9d570506724e,5393e21754b57ecf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:41:56.448 INFO 2981023 --- [Collect all tenants major compactions6,a5fd9d570506724e,dc636f48d8cb83d2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:41:56.448 INFO 2981023 --- [Collect all tenants major compactions2,a5fd9d570506724e,5393e21754b57ecf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:41:56.449 INFO 2981023 --- [Collect all tenants major compactions3,a5fd9d570506724e,4e0000c3adcaced6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:41:56.450 INFO 2981023 --- [Collect all tenants major compactions3,a5fd9d570506724e,4e0000c3adcaced6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:41:56.452 INFO 2981023 --- [Collect all tenants major compactions3,a5fd9d570506724e,4e0000c3adcaced6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:41:56.452 INFO 2981023 --- [Collect all tenants major compactions3,a5fd9d570506724e,4e0000c3adcaced6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:41:56.453 INFO 2981023 --- [ocp-schedules-7,a5fd9d570506724e,240e0485ec7ff67c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:41:56.511 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:41:56.351507+08:00 2024-09-11 17:41:57.259 INFO 2981023 --- [ocp-async-9,178dfac8c60d80fe,cccd8b100667ea5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:57.268 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [DELETE /api/v2/compute/hosts/2000008, client 10.31.33.41, traceId a0ab82240d59badd] 2024-09-11 17:41:57.269 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.s.c.trace.RequestTracingAspect : API: [DELETE /api/v2/compute/hosts/2000008?forceDelete=false, client=10.31.33.41, traceId=a0ab82240d59badd, method=NoDataResponse com.oceanbase.ocp.server.common.controller.ComputeController.deleteHost(long,Boolean), args=2000008,false,] 2024-09-11 17:41:57.295 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-2000008, method=deleteHost 2024-09-11 17:41:57.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.ocp.service.compute.HostDaoManager : Updated host status: host [2000008:10.31.35.7], from status [AVAILABLE] to [DELETING] 2024-09-11 17:41:57.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=DELETING) 2024-09-11 17:41:57.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.c.agent.HostAgentServiceImpl : Getting OCP agent: hostId=2000008 2024-09-11 17:41:57.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:57.344 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:57.418 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Delete host, nodeCount=4, taskCount=4 2024-09-11 17:41:57.418 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:41:57.418 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Delete host', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=null, tenantId=null, hostId=2000008, creator='admin', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={host_ip=10.31.35.7, prohibit_rollback=false, ocp_agent_origin_version=4.3.1-20240805192406, host_name=vm-eas-prd-rtd-mid05, host_id=2000008}, listMap={}}, createTime=null, startTime=2024-09-11T17:41:57.418+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Delete host info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.DeleteHostInfoTask, seriesId=3, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Uninstall ocp agent, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UninstallOcpAgentTask, seriesId=0, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update uninstall agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateUninstallAgentInfoTask, seriesId=2, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Prepare delete host, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.PrepareDeleteHostTask, seriesId=1, startTime=null, endTime=null}], template=null, traceId='a0ab82240d59badd', operation=EXECUTE} 2024-09-11 17:41:57.431 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=4048190, taskName=Delete host, argument=Argument{parallelIdx=-1, stringMap={host_ip=10.31.35.7, prohibit_rollback=false, ocp_agent_origin_version=4.3.1-20240805192406, host_name=vm-eas-prd-rtd-mid05, host_id=2000008}, listMap={}} 2024-09-11 17:41:57.440 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.c.host.manager.HostDeletingHelper : Deleting host: [2000008:10.31.35.7], from status [AVAILABLE] to [DELETING] 2024-09-11 17:41:57.465 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007102 2024-09-11 17:41:57.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"activeAt":"2024-09-11T09:41:57.456Z","alarmType":"host_operation","appType":"OCP","id":6007102,"labels":{"alarm_name":"伺服器運維操作提醒","alarm_status_zh_tw":"警示中","app_type":"OCP","svr_ip":"10.31.35.7","alarm_status_native":"Active","alarm_level":"5","alarm_level_zh_tw":"提醒","operator":"admin","alarm_summary_zh_tw":"alarm_template_id=0:host=10.31.35.7:operation_name=Delete host SUCCESS","alarm_level_native":"5","alarm_scope_en_us":"Host","alarm_summary_en_us":"alarm_template_id=0:host=10.31.35.7:operation_name=Delete host SUCCESS","alarm_level_en_us":"Info","alarm_type":"host_operation","auto_resolved":"1","alarm_status":"Active","alarm_group_zh_tw":"提示","host":"10.31.35.7","alarm_scope":"Host","alarm_active_at":"2024-09-11T17:41:57.456+08:00","task_result":"success","alarm_target":"alarm_template_id=0:host=10.31.35.7:operation_name=Delete host","alarm_scope_native":"Host","alarm_description_zh_tw":"伺服器：10.31.35.7，警示：Delete host SUCCESS, 操作者 admin，運行結果：success。","alarm_name_zh_tw":"伺服器運維操作提醒","app":"OCP","host_ip":"10.31.35.7","alarm_name_en_us":"Host operation info","alarm_name_zh_cn":"主机运维操作提醒","operation_result":"SUCCESS","alarm_status_en_us":"Active","message_type":"Alarm","alarm_status_zh_cn":"告警中","alarm_scope_zh_tw":"主機","alarm_template_id":"0","alarm_group_en_us":"info","operation_name":"Delete host","alarm_description_en_us":"Host: 10.31.35.7, alert: Delete host SUCCESS, operator was admin, result was success.","__host.id":"2000008","alarm_scope_zh_cn":"主机","alarm_level_zh_cn":"提醒","alarm_description_zh_cn":"服务器：10.31.35.7，告警：Delete host SUCCESS, 操作者 admin，运行结果：success。","alarm_summary_zh_cn":"alarm_template_id=0:host=10.31.35.7:operation_name=Delete host SUCCESS","alarm_updated_at":"2024-09-11T17:41:57.456+08:00","ocp_ip":"10.31.35.10","host_name":"vm-eas-prd-rtd-mid05","alarm_group_zh_cn":"提示"},"level":5,"scope":"Host","status":"Active","statusUpdatedAt":"2024-09-11T09:41:57.459Z","target":"alarm_template_id=0:host=10.31.35.7:operation_name=Delete host","updatedAt":"2024-09-11 2024-09-11 17:41:57.482 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=DELETE_HOSTS, eventType=COMPUTE_HOST_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ComputeController.deleteHost, resourceType=COMPUTE_HOST, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=2000008, resourceName=10.31.35.7, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=null, eventInfo={"forceDelete":false,"id":2000008}, executionType=NEW, requestUrl=http://pms-cas.linkreit.com:15002/api/v2/compute/hosts/2000008, clientIp=10.31.33.41, serverIp=10.31.35.10, startTime=2024-09-11T17:41:57.293+08:00, endTime=2024-09-11T17:41:57.481+08:00, role=null, aliUid=null) 2024-09-11 17:41:57.498 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a0ab82240d59badd,39fbe4ad5ec13869] c.o.o.s.c.trace.RequestTracingAspect : API OK: [DELETE /api/v2/compute/hosts/2000008 client=10.31.33.41, traceId=a0ab82240d59badd, duration=234 ms] 2024-09-11 17:41:57.647 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,be469ea366639248] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048173, operation:EXECUTE, state: RUNNING 2024-09-11 17:41:57.671 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,e697ae41e7e5f106] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048173, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, task_instance_id=4048190, task_operation=execute, latest_execution_start_time=2024-09-11T17:41:57.642+08:00, sub_task_instance_name=Prepare delete host, ocp_agent_origin_version=4.3.1-20240805192406, sub_task_instance_id=4048173, host_name=vm-eas-prd-rtd-mid05, host_id=2000008}, listMap={}}, executor=10.31.35.10 2024-09-11 17:41:57.683 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,e697ae41e7e5f106] c.o.ocp.service.compute.HostDaoManager : Updated host status: host [2000008:10.31.35.7], from status [DELETING] to [DELETING] 2024-09-11 17:41:57.687 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,e697ae41e7e5f106] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=DELETING) 2024-09-11 17:41:57.694 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,222727ef70818e62,b3e529454273bbbe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=222727ef70818e62, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:41:57.695 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-4,222727ef70818e62,b3e529454273bbbe] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:41:57.701 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,e697ae41e7e5f106] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048173, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:41:57.737 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,222727ef70818e62,b3e529454273bbbe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=222727ef70818e62, duration=46 ms] 2024-09-11 17:41:57.757 INFO 2981023 --- [ocp-async-4,1ad659a56fda500a,25034e45fdb2be9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:41:58.456 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048171] 2024-09-11 17:41:58.666 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,872898bf5e924619] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048171, operation:EXECUTE, state: RUNNING 2024-09-11 17:41:58.686 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048171, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, task_instance_id=4048190, task_operation=execute, latest_execution_start_time=2024-09-11T17:41:58.664+08:00, sub_task_instance_name=Uninstall ocp agent, ocp_agent_origin_version=4.3.1-20240805192406, sub_task_instance_id=4048171, host_name=vm-eas-prd-rtd-mid05, host_id=2000008}, listMap={}}, executor=10.31.35.10 2024-09-11 17:41:58.690 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.s.t.b.host.UninstallOcpAgentTask : [UninstallOcpAgentTask] started 2024-09-11 17:41:58.695 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:58.699 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:58.716 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:58.721 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:58.735 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:41:58.740 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:41:58.752 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-09-11 17:41:58.756 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.service.task.util.AgentTaskUtils : Get credential from vault. 2024-09-11 17:41:58.771 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2024-09-11 17:41:58.782 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2024-09-11 17:41:58.793 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.0 2024-09-11 17:41:58.805 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 10.31.35.7:22 success 2024-09-11 17:41:58.867 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PASSWORD, host: root@10.31.35.7 2024-09-11 17:41:58.871 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=10.31.35.7, httpPort=null, sshPort=22, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=password, passwordAuthConfig=PasswordAuthConfig(username=root), privateKeyAuthConfig=null)), proxy=null), estimated size:9 2024-09-11 17:41:58.921 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:41:58.945 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:41:58.986 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.31.35.7 2024-09-11 17:41:59.259 INFO 2981023 --- [ocp-async-1,9b0506500dc9844f,aa448610f9f8666d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:41:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,49635844cac86bbe,09408aeb40ceb379] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047720, interval=60 2024-09-11 17:41:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,49635844cac86bbe,09408aeb40ceb379] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:41:59.335 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:41:59.335 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:41:59.774 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.1-20240805192406.el7.x86_64, err=, extOut=null, exitStatus=0) 2024-09-11 17:41:59.778 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.1-20240805192406.el7.x86_64, err=, extOut=null, exitStatus=0) 2024-09-11 17:41:59.816 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:41:59.838 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:41:59.880 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/tmp/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"'' on 10.31.35.7 2024-09-11 17:41:59.919 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/tmp/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"'' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'rm -rf '"'"'/tmp/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:41:59.926 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/tmp/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"'' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'rm -rf '"'"'/tmp/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:41:59.961 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:41:59.983 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:00.024 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:00.059 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:00.064 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:00.101 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:00.124 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:00.166 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.31.35.7 2024-09-11 17:42:00.954 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:00.960 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:00.981 WARN 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2024-09-11 17:42:00.986 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2024-09-11 17:42:00.996 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:01.022 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:01.064 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.31.35.7 2024-09-11 17:42:01.265 INFO 2981023 --- [ocp-async-5,b6aa2359e3af3220,f9c376cd00f874a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:01.852 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:01.856 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:01.860 WARN 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2024-09-11 17:42:01.863 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2024-09-11 17:42:01.893 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:01.916 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:01.958 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:01.989 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:01.993 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.031 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:02.050 INFO 2981023 --- [ocp-schedules-8,6af361d76275fa59,1838ba2db5da0551] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:42:02.051 INFO 2981023 --- [ocp-schedules-8,6af361d76275fa59,1838ba2db5da0551] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:42:02.054 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.070 INFO 2981023 --- [ocp-schedules-20,cbc4cd5aec8b85a4,533ba427ac20a488] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:42:02.096 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:02.126 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.130 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.168 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:02.182 INFO 2981023 --- [ocp-schedules-5,4fd5d16fff5aa10f,d37c1a49269c0bf3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:42:02.193 WARN 2981023 --- [scanner-13530523-1,4fd5d16fff5aa10f,fc359ed28609f320] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:42:02.197 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.233 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:02.267 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.276 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,b479861214fd33cd,9a3a0de1af61f587] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:02.309 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:02.323 INFO 2981023 --- [prometheus-minute-schedule-1,b479861214fd33cd,9a3a0de1af61f587] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:42:02.335 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:02.345 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:02.340 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.382 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9' on 10.31.35.7 2024-09-11 17:42:02.434 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.437 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.476 INFO 2981023 --- [ocp-schedules-19,caec117a7a6de695,ec242a142dac075e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:42:02.477 INFO 2981023 --- [ocp-schedules-19,caec117a7a6de695,ec242a142dac075e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:02.476 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:02.486 INFO 2981023 --- [ocp-schedules-19,caec117a7a6de695,ec242a142dac075e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:42:02.499 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.541 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:02.571 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.575 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.614 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:02.638 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.680 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:02.711 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.715 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.753 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:02.776 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.818 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:02.851 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.855 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.893 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:02.915 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.957 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:02.987 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:02.991 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.029 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:03.056 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.098 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:03.130 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.134 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.172 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:03.194 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.236 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:03.259 INFO 2981023 --- [ocp-async-8,e97e694d06741ed1,7939eac8681bb426] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:03.266 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.281 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.308 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:03.330 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.372 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:03.404 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.408 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.446 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:03.468 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:03.510 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7 2024-09-11 17:42:04.296 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.301 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.304 WARN 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-09-11 17:42:04.307 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-09-11 17:42:04.338 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:04.361 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.402 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:04.432 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.443 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.474 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:04.496 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.537 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:04.567 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.573 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.609 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:04.636 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.678 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9' on 10.31.35.7 2024-09-11 17:42:04.722 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.726 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.764 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:04.787 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.828 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:42:04.864 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.868 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.906 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:04.928 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:04.969 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9' on 10.31.35.7 2024-09-11 17:42:05.012 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:05.017 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:05.021 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-09-11 17:42:05.054 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:05.077 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:05.118 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7 2024-09-11 17:42:05.260 INFO 2981023 --- [ocp-async-7,c3c1de576c9c5f3e,6d9f61a37bfd05f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,a8e2b0e04029c7ac,f94d782dcd202ec3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:05.318 INFO 2981023 --- [prometheus-minute-schedule-1,a8e2b0e04029c7ac,f94d782dcd202ec3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:42:05.319 INFO 2981023 --- [metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://10.31.35.7:62888/metrics/stat, collectAt=1726047725, message=java.net.ConnectException: Connection refused: /10.31.35.7:62888, rootCause=ConnectException: Connection refused 2024-09-11 17:42:05.319 INFO 2981023 --- [metric-parse-3,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://10.31.35.7:62889/metrics/stat, collectAt=1726047725, message=java.net.ConnectException: Connection refused: /10.31.35.7:62889, rootCause=ConnectException: Connection refused 2024-09-11 17:42:05.345 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:05.345 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:05.907 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:05.911 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:05.915 WARN 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-09-11 17:42:05.920 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-09-11 17:42:05.923 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-09-11 17:42:05.948 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:42:05.970 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:05.989 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:42:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:42:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:42:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:42:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:42:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:42:06.009 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T17:39:41, target=2024-09-11T17:37:40, granule=120000ms, delay=121000ms) 2024-09-11 17:42:06.009 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:37:40, target=2024-09-11T17:30:51, granule=600000ms, delay=409000ms) 2024-09-11 17:42:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T17:39:41, target=2024-09-11T17:37:40, granule=120000ms, delay=121000ms) 2024-09-11 17:42:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:37:40, target=2024-09-11T17:29:31, granule=600000ms, delay=489000ms) 2024-09-11 17:42:06.009 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=42ms, period=120000ms 2024-09-11 17:42:06.012 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.31.35.7 2024-09-11 17:42:06.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:42:06.353562+08:00, lastProcessTime=2024-09-11T17:41:56.351507+08:00 2024-09-11 17:42:06.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:42:06.500 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:42:06.353562+08:00 2024-09-11 17:42:06.803 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.1-20240805192406.el7.x86_64, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:06.810 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.1-20240805192406.el7.x86_64, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:06.845 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:06.867 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:06.909 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/456a81e364504d5b on 10.31.35.7 2024-09-11 17:42:06.932 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/456a81e364504d5b on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=mkdir -p /tmp/456a81e364504d5b, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:06.937 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/456a81e364504d5b on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=mkdir -p /tmp/456a81e364504d5b, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:06.974 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:06.995 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:07.038 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/456a81e364504d5b/uninstall_package.sh && chmod +x /tmp/456a81e364504d5b/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/456a81e364504d5b/uninstall_package.sh on 10.31.35.7 2024-09-11 17:42:07.065 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/456a81e364504d5b/uninstall_package.sh && chmod +x /tmp/456a81e364504d5b/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/456a81e364504d5b/uninstall_package.sh on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=touch /tmp/456a81e364504d5b/uninstall_package.sh && chmod +x /tmp/456a81e364504d5b/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/456a81e364504d5b/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:07.069 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/456a81e364504d5b/uninstall_package.sh && chmod +x /tmp/456a81e364504d5b/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/456a81e364504d5b/uninstall_package.sh on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=touch /tmp/456a81e364504d5b/uninstall_package.sh && chmod +x /tmp/456a81e364504d5b/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package 'ocp-agent-ce'' > /tmp/456a81e364504d5b/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:07.107 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:07.130 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:07.172 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/456a81e364504d5b/uninstall_package.sh on 10.31.35.7 2024-09-11 17:42:07.260 INFO 2981023 --- [ocp-async-4,a988f9b9fdb0a4e4,770366dd9b1998a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:07.424 INFO 2981023 --- [ocp-schedules-8,5057086192dc4899,84591d83f7b8fa11] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:42:07.441 INFO 2981023 --- [ocp-schedules-7,0a4496d48ff9150d,95c1a0278828a4b3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:42:07.441+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:42:07.442 INFO 2981023 --- [ocp-schedules-7,0a4496d48ff9150d,95c1a0278828a4b3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:42:07.444 INFO 2981023 --- [ocp-schedules-7,0a4496d48ff9150d,95c1a0278828a4b3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:42:07.444 INFO 2981023 --- [ocp-schedules-7,0a4496d48ff9150d,95c1a0278828a4b3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:42:07.757 INFO 2981023 --- [ocp-async-7,be40e0b77e666ef0,b789eec3466e6ca3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:42:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,de955f8349819273,0c72e1372d2c4ad7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:08.317 INFO 2981023 --- [prometheus-minute-schedule-1,de955f8349819273,0c72e1372d2c4ad7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:42:08.319 INFO 2981023 --- [metric-parse-6,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://10.31.35.7:62888/metrics/stat, collectAt=1726047728, message=java.net.ConnectException: Connection refused: /10.31.35.7:62888, rootCause=ConnectException: Connection refused 2024-09-11 17:42:08.319 INFO 2981023 --- [metric-parse-12,,] c.o.o.m.s.OcpMetricCollectServiceImpl : Collect failed, exporter=http://10.31.35.7:62889/metrics/stat, collectAt=1726047728, message=java.net.ConnectException: Connection refused: /10.31.35.7:62889, rootCause=ConnectException: Connection refused 2024-09-11 17:42:08.345 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:08.347 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:08.391 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/456a81e364504d5b/uninstall_package.sh on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=/tmp/456a81e364504d5b/uninstall_package.sh, out=, err=warning: /home/admin/ocp_agent/conf/config_properties/ob_logtailer.yaml saved as /home/admin/ocp_agent/conf/config_properties/ob_logtailer.yaml.rpmsave warning: /home/admin/ocp_agent/conf/config_properties/ob_logcleaner.yaml saved as /home/admin/ocp_agent/conf/config_properties/ob_logcleaner.yaml.rpmsave warning: /home/admin/ocp_agent/conf/config_properties/monagent_pipeline.yaml saved as /home/admin/ocp_agent/conf/config_properties/monagent_pipeline.yaml.rpmsave warning: /home/admin/ocp_agent/conf/config_properties/mgragent_pipeline.yaml saved as /home/admin/ocp_agent/conf/config_properties/mgragent_pipeline.yaml.rpmsave warning: /home/admin/ocp_agent/conf/config_properties/basic_auth.yaml saved as /home/admin/ocp_agent/conf/config_properties/basic_auth.yaml.rpmsave , extOut=null, exitStatus=0) 2024-09-11 17:42:08.397 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/456a81e364504d5b/uninstall_package.sh on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=/tmp/456a81e364504d5b/uninstall_package.sh, out=, err=warning: /home/admin/ocp_agent/conf/config_properties/ob_logtailer.yaml saved as /home/admin/ocp_agent/conf/config_properties/ob_logtailer.yaml.rpmsave warning: /home/admin/ocp_agent/conf/config_properties/ob_logcleaner.yaml saved as /home/admin/ocp_agent/conf/config_properties/ob_logcleaner.yaml.rpmsave warning: /home/admin/ocp_agent/conf/config_properties/monagent_pipeline.yaml saved as /home/admin/ocp_agent/conf/config_properties/monagent_pipeline.yaml.rpmsave warning: /home/admin/ocp_agent/conf/config_properties/mgragent_pipeline.yaml saved as /home/admin/ocp_agent/conf/config_properties/mgragent_pipeline.yaml.rpmsave warning: /home/admin/ocp_agent/conf/config_properties/basic_auth.yaml saved as /home/admin/ocp_agent/conf/config_properties/basic_auth.yaml.rpmsave , extOut=null, exitStatus=0) 2024-09-11 17:42:08.433 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:08.457 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:08.498 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: rm -rf /tmp/456a81e364504d5b on 10.31.35.7 2024-09-11 17:42:08.522 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: rm -rf /tmp/456a81e364504d5b on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=rm -rf /tmp/456a81e364504d5b, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:08.526 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: rm -rf /tmp/456a81e364504d5b on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=rm -rf /tmp/456a81e364504d5b, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:08.529 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.ocp.executor.executor.SshExecutor : uninstall complete, rpmName=ocp-agent-ce, result= 2024-09-11 17:42:08.563 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:42:08.585 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:08.627 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.31.35.7 2024-09-11 17:42:08.673 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:08.677 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:42:08.682 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.s.t.b.host.UninstallOcpAgentTask : [UninstallOcpAgentTask] completed 2024-09-11 17:42:08.695 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6216210a761b86be] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048171, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:42:09.260 INFO 2981023 --- [ocp-async-9,07d037fd9a68d332,91bbbfbe085ee1ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:09.542 INFO 2981023 --- [ocp-schedules-27,de7977b0763b5e01,3939523b2c3be39a] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-09-11 17:42:09.542 INFO 2981023 --- [ocp-schedules-27,de7977b0763b5e01,3939523b2c3be39a] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-09-11 17:42:09.542 INFO 2981023 --- [ocp-schedules-27,de7977b0763b5e01,3939523b2c3be39a] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=11634 2024-09-11 17:42:09.544 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048172] 2024-09-11 17:42:09.707 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,18f32b1299a3e58f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048172, operation:EXECUTE, state: RUNNING 2024-09-11 17:42:09.730 INFO 2981023 --- [ocp-schedules-27,de7977b0763b5e01,3939523b2c3be39a] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=11634 2024-09-11 17:42:09.739 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048172, context=Context{parallelIdx=-1, stringMap={ocp_manager_agent_http_port_origin=62888, host_ip=10.31.35.7, ocp_monitor_agent_http_origin=62889, ocp_agent_install_home_origin=/home/admin/ocp_agent, task_instance_id=4048190, task_operation=execute, latest_execution_start_time=2024-09-11T17:42:09.705+08:00, sub_task_instance_name=Update uninstall agent info, ocp_agent_origin_version=4.3.1-20240805192406, sub_task_instance_id=4048172, host_name=vm-eas-prd-rtd-mid05, host_id=2000008}, listMap={}}, executor=10.31.35.10 2024-09-11 17:42:09.742 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] o.o.s.t.b.h.UpdateUninstallAgentInfoTask : [UpdateUninstallAgentInfoTask] started 2024-09-11 17:42:09.746 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] c.o.o.c.agent.HostAgentServiceImpl : Offlining all exporters: hostId=2000008 2024-09-11 17:42:09.767 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] c.o.o.s.c.AgentInstallationTaskService : Deleting host agent records: hostId=2000008 2024-09-11 17:42:09.771 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] c.o.o.c.agent.HostAgentServiceImpl : Deleting host agent and processes: hostId=2000008 2024-09-11 17:42:09.777 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000008 2024-09-11 17:42:09.784 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000008 2024-09-11 17:42:09.842 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Deleting all OCP agent process: hostId=2000008 2024-09-11 17:42:09.870 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] o.o.s.t.b.h.UpdateUninstallAgentInfoTask : [UpdateUninstallAgentInfoTask] completed 2024-09-11 17:42:09.879 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,7bb77821efb3a5bb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048172, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:42:10.002 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:42:10.003 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:42:10.557 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048170] 2024-09-11 17:42:10.734 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,2b041d55bf6640e0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048170, operation:EXECUTE, state: RUNNING 2024-09-11 17:42:10.754 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048170, context=Context{parallelIdx=-1, stringMap={ocp_manager_agent_http_port_origin=62888, host_ip=10.31.35.7, ocp_monitor_agent_http_origin=62889, ocp_agent_install_home_origin=/home/admin/ocp_agent, task_instance_id=4048190, task_operation=execute, latest_execution_start_time=2024-09-11T17:42:10.732+08:00, sub_task_instance_name=Delete host info, ocp_agent_origin_version=4.3.1-20240805192406, sub_task_instance_id=4048170, host_name=vm-eas-prd-rtd-mid05, host_id=2000008}, listMap={}}, executor=10.31.35.10 2024-09-11 17:42:10.757 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.s.t.b.host.DeleteHostInfoTask : [DeleteHostInfoTask] begin 2024-09-11 17:42:10.796 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.c.host.service.HostServiceImpl : Removing host master data and credential accesses: hostId=2000008 2024-09-11 17:42:10.800 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.s.c.CredentialServiceImpl : Deleting HOST credentials access by resource ids [2000008] 2024-09-11 17:42:10.812 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.s.c.CredentialServiceImpl : Success to delete HOST credentials access by resource ids [2000008] 2024-09-11 17:42:10.816 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.s.c.CredentialServiceImpl : Deleting AGENT credentials access by resource ids [2000008] 2024-09-11 17:42:10.823 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.s.c.CredentialServiceImpl : Success to delete AGENT credentials access by resource ids [2000008] 2024-09-11 17:42:10.827 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.c.agent.HostAgentServiceImpl : Deleting OCP Agent host 2000008 2024-09-11 17:42:10.844 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.s.t.b.host.DeleteHostInfoTask : [DeleteHostInfoTask] end 2024-09-11 17:42:10.858 INFO 2981023 --- [manual-subtask-executor16,a0ab82240d59badd,6b6956bc6af675db] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048170, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:42:10.913 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:42:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:42:11.260 INFO 2981023 --- [ocp-async-7,67953dd3bb8bdba7,132a68cc77f986be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,b662e65b1123ea16,82538e1ec83f93f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,b662e65b1123ea16,82538e1ec83f93f1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:11.347 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:11.348 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:11.626 INFO 2981023 --- [task-coordinator2,,] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"activeAt":"2024-09-11T09:41:57Z","alarmTemplateId":0,"alarmType":"host_operation","appType":"OCP","containSelfcure":false,"description":"Host: 10.31.35.7, alert: Delete host SUCCESS, operator was admin, result was success, tips:host not found.","gmtCreate":"2024-09-11T09:41:57Z","gmtModified":"2024-09-11T09:41:57.464335Z","id":6007102,"labels":{"app_type":"OCP","svr_ip":"10.31.35.7","alarm_status_native":"Active","alarm_level":"5","operator":"admin","alarm_summary_zh_tw":"alarm_template_id=0:host=10.31.35.7:operation_name=Delete host SUCCESS","alarm_summary_en_us":"alarm_template_id=0:host=10.31.35.7:operation_name=Delete host SUCCESS","alarm_type":"host_operation","alarm_status":"Active","alarm_group_zh_tw":"提示","host":"10.31.35.7","alarm_scope":"Host","task_result":"success, tips:host not found","alarm_scope_native":"Host","alarm_description_zh_tw":"伺服器：10.31.35.7，警示：Delete host SUCCESS, 操作者 admin，運行結果：success, tips:host not found。","app":"OCP","alarm_duration":"14","alarm_name_en_us":"Host operation info","operation_result":"SUCCESS","message_type":"Alarm","alarm_status_zh_cn":"告警中","alarm_scope_zh_tw":"主機","alarm_group_en_us":"info","alarm_description_en_us":"Host: 10.31.35.7, alert: Delete host SUCCESS, operator was admin, result was success, tips:host not found.","__host.id":"2000008","alarm_scope_zh_cn":"主机","alarm_level_zh_cn":"提醒","alarm_summary_zh_cn":"alarm_template_id=0:host=10.31.35.7:operation_name=Delete host SUCCESS","alarm_description":"Host: 10.31.35.7, alert: Delete host SUCCESS, operator was admin, result was success, tips:host not found.","ocp_ip":"10.31.35.10","alarm_name":"Host operation info","alarm_status_zh_tw":"警示中","task_id":"4048190","alarm_level_zh_tw":"提醒","alarm_level_native":"5","alarm_scope_en_us":"Host","alarm_level_en_us":"Info","auto_resolved":"1","alarm_summary":"alarm_template_id=0:host=10.31.35.7:operation_name=Delete host SUCCESS","alarm_active_at":"2024-09-11T09:41:57Z","alarm_target":"alarm_template_id=0:host=10.31.35.7:operation_name=Dele 2024-09-11 17:42:11.633 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048190, status:SUCCESS 2024-09-11 17:42:11.637 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048190, status:SUCCESS 2024-09-11 17:42:11.946 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047660, nodeCount=435, affectRows=435 2024-09-11 17:42:12.073 INFO 2981023 --- [ocp-schedules-24,76ccc6b1cc83b38c,0553fb81adcca435] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:42:12.415 INFO 2981023 --- [ocp-schedules-11,86c4e5cf75966ce0,dea3f6b7276d5a3a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:42:12.466 WARN 2981023 --- [scanner-13532232-1,86c4e5cf75966ce0,e7b4cebff19dc9f1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:42:12.486 INFO 2981023 --- [alarm-detect-16,86c4e5cf75966ce0,6bb7607d4b96f3e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644412, data is empty. 2024-09-11 17:42:12.496 INFO 2981023 --- [alarm-detect-16,86c4e5cf75966ce0,6bb7607d4b96f3e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644463, data is empty. 2024-09-11 17:42:12.500 INFO 2981023 --- [alarm-detect-16,86c4e5cf75966ce0,6bb7607d4b96f3e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644463, data is empty. 2024-09-11 17:42:12.524 INFO 2981023 --- [alarm-detect-15,86c4e5cf75966ce0,a4c9de9f4dd8d46d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644412, data is empty. 2024-09-11 17:42:12.526 INFO 2981023 --- [alarm-detect-5,86c4e5cf75966ce0,22a95a60b398f211] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644412, data is empty. 2024-09-11 17:42:12.532 INFO 2981023 --- [alarm-detect-15,86c4e5cf75966ce0,a4c9de9f4dd8d46d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644463, data is empty. 2024-09-11 17:42:12.535 INFO 2981023 --- [alarm-detect-5,86c4e5cf75966ce0,22a95a60b398f211] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644463, data is empty. 2024-09-11 17:42:12.539 INFO 2981023 --- [alarm-detect-5,86c4e5cf75966ce0,22a95a60b398f211] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644463, data is empty. 2024-09-11 17:42:12.539 INFO 2981023 --- [alarm-detect-15,86c4e5cf75966ce0,a4c9de9f4dd8d46d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644463, data is empty. 2024-09-11 17:42:12.642 INFO 2981023 --- [alarm-detect-3,86c4e5cf75966ce0,faecbf65ffa80e2b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644412, data is empty. 2024-09-11 17:42:12.648 INFO 2981023 --- [alarm-detect-3,86c4e5cf75966ce0,faecbf65ffa80e2b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644463, data is empty. 2024-09-11 17:42:12.653 INFO 2981023 --- [alarm-detect-3,86c4e5cf75966ce0,faecbf65ffa80e2b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644463, data is empty. 2024-09-11 17:42:12.731 INFO 2981023 --- [ocp-schedules-13,6e98e86e6e192182,219fae3fb2aca533] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:42:12.744 INFO 2981023 --- [ocp-schedules-13,6e98e86e6e192182,219fae3fb2aca533] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:12.745 INFO 2981023 --- [ocp-schedules-13,6e98e86e6e192182,219fae3fb2aca533] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:12.756 INFO 2981023 --- [ocp-schedules-13,6e98e86e6e192182,219fae3fb2aca533] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:12.757 INFO 2981023 --- [ocp-schedules-13,6e98e86e6e192182,219fae3fb2aca533] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:12.769 INFO 2981023 --- [ocp-schedules-13,6e98e86e6e192182,219fae3fb2aca533] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:12.769 INFO 2981023 --- [ocp-schedules-13,6e98e86e6e192182,219fae3fb2aca533] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:12.780 INFO 2981023 --- [ocp-schedules-13,6e98e86e6e192182,219fae3fb2aca533] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=5 2024-09-11 17:42:13.110 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047660, nodeCount=1000, affectRows=1000 2024-09-11 17:42:13.113 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047660, nodeCount=1000, affectRows=1000 2024-09-11 17:42:13.113 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047660, nodeCount=999, validCount=999 2024-09-11 17:42:13.115 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047660, nodeCount=1000, validCount=1000 2024-09-11 17:42:13.126 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047660, nodeCount=48, affectRows=48 2024-09-11 17:42:13.126 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047660, nodeCount=48, validCount=48 2024-09-11 17:42:13.260 INFO 2981023 --- [ocp-async-10,f25615527980999e,273d1a574bd126cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:13.803 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047660, nodeCount=45, affectRows=45 2024-09-11 17:42:13.894 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047660, nodeCount=435, validCount=435 2024-09-11 17:42:13.980 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047660, nodeCount=45, validCount=45 2024-09-11 17:42:14.164 INFO 2981023 --- [ocp-schedules-14,b571eba446b73af7,01dff04d8cd83bec] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:42:14.174 INFO 2981023 --- [ocp-schedules-14,b571eba446b73af7,01dff04d8cd83bec] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:42:14.174 INFO 2981023 --- [ocp-schedules-14,b571eba446b73af7,01dff04d8cd83bec] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:42:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,f224cf1e3c0d3b68,61c7bd961d141c82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,11f35a498245f458,76a4a38e6e6d16c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:42:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,f224cf1e3c0d3b68,61c7bd961d141c82] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:14.348 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:14.348 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:15.260 INFO 2981023 --- [ocp-async-2,d6dfa993dbc9c25e,3eae982f2c4242cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:16.021 INFO 2981023 --- [ocp-schedules-8,16308754f48cb927,44fe125947f4e3b8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:42:16.022 INFO 2981023 --- [host-check-agent-status1,16308754f48cb927,3fcd4666b768b136] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:42:16.022 INFO 2981023 --- [host-check-agent-status2,16308754f48cb927,db84ad63da1a98de] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:42:16.022 INFO 2981023 --- [host-check-agent-status3,16308754f48cb927,b3e84e2cac8b5d6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:42:16.022 INFO 2981023 --- [ocp-schedules-5,5a1aef54051ce7cc,a81e7ff4c2e6cafc] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:42:16.023 INFO 2981023 --- [ocp-schedules-17,ec8233cf1da2e91f,d5cefd64e415c312] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:42:16.023 INFO 2981023 --- [host-check-clockdiff1,5a1aef54051ce7cc,6a23bffee9e38dca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:42:16.024 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:42:16.024 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:42:16.024 INFO 2981023 --- [host-check-clockdiff2,5a1aef54051ce7cc,d2a3f680abf15f7e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:42:16.024 INFO 2981023 --- [host-check-clockdiff3,5a1aef54051ce7cc,b748c477d210863f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:42:16.025 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:42:16.027 INFO 2981023 --- [host-check-agent-status1,16308754f48cb927,3fcd4666b768b136] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:16.028 INFO 2981023 --- [host-check-agent-status1,16308754f48cb927,3fcd4666b768b136] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:16.032 INFO 2981023 --- [host-check-agent-status3,16308754f48cb927,b3e84e2cac8b5d6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:16.033 INFO 2981023 --- [host-check-agent-status2,16308754f48cb927,db84ad63da1a98de] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:16.035 INFO 2981023 --- [host-check-clockdiff3,5a1aef54051ce7cc,b748c477d210863f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047736 0 0 , error= 2024-09-11 17:42:16.035 INFO 2981023 --- [host-check-agent-status2,16308754f48cb927,db84ad63da1a98de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:16.036 INFO 2981023 --- [host-check-agent-status3,16308754f48cb927,b3e84e2cac8b5d6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:16.037 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:16.038 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:16.040 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:16.041 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:16.042 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:16.044 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:16.054 INFO 2981023 --- [host-check-agent-status1,16308754f48cb927,3fcd4666b768b136] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:42:16.058 INFO 2981023 --- [host-check-agent-status1,16308754f48cb927,3fcd4666b768b136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:42:16.058 INFO 2981023 --- [host-check-agent-status1,16308754f48cb927,3fcd4666b768b136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:42:16.058 INFO 2981023 --- [host-check-agent-status1,16308754f48cb927,3fcd4666b768b136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:42:16.059 INFO 2981023 --- [host-check-agent-status2,16308754f48cb927,db84ad63da1a98de] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:42:16.060 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:42:16.060 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:42:16.060317866+08:00 2024-09-11 17:42:16.061 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:42:16.060+08:00 2024-09-11 17:42:16.062 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:42:16.063 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:42:16.060+08:00 and status AVAILABLE 2024-09-11 17:42:16.063 INFO 2981023 --- [host-check-agent-status2,16308754f48cb927,db84ad63da1a98de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:42:16.063 INFO 2981023 --- [host-check-agent-status2,16308754f48cb927,db84ad63da1a98de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:42:16.063 INFO 2981023 --- [host-check-agent-status2,16308754f48cb927,db84ad63da1a98de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:42:16.065 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:42:16.064873622+08:00 2024-09-11 17:42:16.065 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:42:16.065+08:00 2024-09-11 17:42:16.066 INFO 2981023 --- [host-check-agent-status3,16308754f48cb927,b3e84e2cac8b5d6d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:42:16.067 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:42:16.065+08:00 and status AVAILABLE 2024-09-11 17:42:16.067 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:42:16.069 INFO 2981023 --- [host-check-agent-status3,16308754f48cb927,b3e84e2cac8b5d6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:42:16.069 INFO 2981023 --- [host-check-agent-status3,16308754f48cb927,b3e84e2cac8b5d6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:42:16.069 INFO 2981023 --- [host-check-agent-status3,16308754f48cb927,b3e84e2cac8b5d6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:42:16.070 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:42:16.070027200+08:00 2024-09-11 17:42:16.070 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:42:16.070 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:42:16.070+08:00 2024-09-11 17:42:16.072 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:42:16.070+08:00 and status AVAILABLE 2024-09-11 17:42:16.076 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:42:16.093 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:16.094 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:16.099 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:16.099 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:42:16.100 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:16.112 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:42:16.115 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:42:16.128 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:16.129 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:16.141 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:42:16.143 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:42:16.155 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:42:16.174 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:42:16.181 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:42:16.187 INFO 2981023 --- [ocp-schedules-20,4a8675e119527777,78b146aef9b25cc4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:42:16.187 INFO 2981023 --- [ocp-schedules-6,4bbe96531804cefb,b0f17296e4cabacc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:42:16.188 INFO 2981023 --- [ocp-schedules-3,448ea173e288e888,8e658b1d15380d75] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:42:16.188 INFO 2981023 --- [ocp-schedules-2,2399230714fcd82a,ab3442e37ba4f35c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:42:16.191 INFO 2981023 --- [ocp-schedules-20,4a8675e119527777,78b146aef9b25cc4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:16.195 INFO 2981023 --- [ocp-schedules-6,4bbe96531804cefb,b0f17296e4cabacc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:16.199 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:42:16.202 INFO 2981023 --- [ocp-schedules-3,448ea173e288e888,8e658b1d15380d75] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:16.226 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:42:16.229 INFO 2981023 --- [ocp-schedules-16,58ec12dfbbd9484c,324c793d20c0de84] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:42:16.234 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:42:16.242 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:42:16.246 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.247 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:16.248 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:42:16.249 INFO 2981023 --- [ocp-schedules-6,4bbe96531804cefb,b0f17296e4cabacc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:42:16.252 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.253 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:16.254 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:42:16.265 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:42:16.271 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:42:16.277 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:42:16.293 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.293 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:16.294 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:42:16.294 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:42:16.295 INFO 2981023 --- [ocp-schedules-20,4a8675e119527777,78b146aef9b25cc4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:42:16.295 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.295 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:16.299 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.299 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:16.308 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:42:16.309 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:42:16.327 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:42:16.333 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:42:16.335 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.335 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:16.350 INFO 2981023 --- [host-check-host-status1,ec8233cf1da2e91f,efde10a96511063e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:42:16.361 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.362 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.362 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:42:16.363 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:42:16.367 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:42:16.366416+08:00, lastProcessTime=2024-09-11T17:42:06.353562+08:00 2024-09-11 17:42:16.377 INFO 2981023 --- [host-check-host-status2,ec8233cf1da2e91f,3a97898f5c00d011] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:42:16.384 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:42:16.392 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.393 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.393 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:42:16.395 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:42:16.398 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.402 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.403 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:16.404 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.406 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:42:16.407 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:42:16.420 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:42:16.421 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:16.421 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:42:16.431 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:16.436 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:42:16.438 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.441 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:42:16.441 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:42:16.442 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:42:16.454 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.455 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.455 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:16.456 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:42:16.457 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:42:16.461 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:42:16.465 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:16.465 INFO 2981023 --- [host-check-host-status3,ec8233cf1da2e91f,47c39a509bf99061] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:42:16.466 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:16.468 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:16.476 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.476 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.477 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:16.477 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:42:16.482 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:42:16.486 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:16.486 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:42:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:16.493 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.494 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.494 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:42:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:42:16.514 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.515 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.515 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:16.515 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:42:16.516 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.519 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:42:16.519 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:42:16.526 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:16.526 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:42:16.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:16.547 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.547 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.548 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:16.549 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:42:16.553 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:42:16.555 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:42:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:16.565 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:42:16.569 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:42:16.575 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:42:16.575 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:42:16.577 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:42:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.579 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:42:16.584 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:42:16.584 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:42:16.591 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:42:16.595 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:42:16.366416+08:00 2024-09-11 17:42:16.607 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:42:16.612 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:16.612 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:16.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:42:16.618 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:42:16.618 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:42:16.618 INFO 2981023 --- [ocp-schedules-3,448ea173e288e888,8e658b1d15380d75] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:42:16.977 INFO 2981023 --- [host-check-clockdiff2,5a1aef54051ce7cc,d2a3f680abf15f7e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047736 0 0 , error= 2024-09-11 17:42:17.260 INFO 2981023 --- [ocp-async-9,47a7109b2c2c189b,64ad39794a9ba8e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,ca62e6de0b00e6f9,4cd9786f261b67d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,ca62e6de0b00e6f9,4cd9786f261b67d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:17.348 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:17.349 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:17.757 INFO 2981023 --- [ocp-async-8,0a75891618e592be,b1ac35e3747efc20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:42:18.571 INFO 2981023 --- [host-check-clockdiff1,5a1aef54051ce7cc,6a23bffee9e38dca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047738 0 0 , error= 2024-09-11 17:42:19.261 INFO 2981023 --- [ocp-async-4,250ccda09b74e8d1,1c1d6d07ae190a5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,43a3173b517664e6,1e1ee6c207b70437] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,43a3173b517664e6,1e1ee6c207b70437] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:20.349 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:20.350 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:21.261 INFO 2981023 --- [ocp-async-8,5eda7567c1ef06ac,8a6d66bb64938300] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:22.076 INFO 2981023 --- [ocp-schedules-24,c976e2bc89ca8105,f8868fca776160b3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:42:22.686 INFO 2981023 --- [ocp-schedules-27,b7bf0588e2ecc592,7706f58dbc28171e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:42:22.730 WARN 2981023 --- [scanner-13532583-1,b7bf0588e2ecc592,191a64bef0843370] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:42:22.793 INFO 2981023 --- [alarm-detect-14,b7bf0588e2ecc592,b3cccaa44b987e9d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047705 ~ 1726047732 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726047720, value=1.826)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.826, values={1726047720=1.826}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:42:22.793 INFO 2981023 --- [alarm-detect-14,b7bf0588e2ecc592,b3cccaa44b987e9d] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Pending 2024-09-11 17:42:22.986 INFO 2981023 --- [ocp-schedules-28,aafd8621d237dd83,3ec22a51fa406efc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:42:22.986 INFO 2981023 --- [ocp-schedules-28,aafd8621d237dd83,3ec22a51fa406efc] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:42:22.986 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:42:22.986 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:42:22.989 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:22.990 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:22.990 INFO 2981023 --- [ocp-schedules-28,aafd8621d237dd83,3ec22a51fa406efc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:22.991 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:22.991 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:22.991 INFO 2981023 --- [ocp-schedules-28,aafd8621d237dd83,3ec22a51fa406efc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:23.006 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:42:23.006 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:42:23.007 INFO 2981023 --- [ocp-schedules-28,aafd8621d237dd83,3ec22a51fa406efc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:42:23.010 INFO 2981023 --- [ocp-schedules-28,aafd8621d237dd83,3ec22a51fa406efc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:42:23.261 INFO 2981023 --- [ocp-async-3,ad1c50f8dad5d7c9,e2d0246987761603] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,e2597d5f9cdfd2b7,a5b0426cba01c8d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,e2597d5f9cdfd2b7,a5b0426cba01c8d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:23.350 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:23.350 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:24.259 INFO 2981023 --- [ocp-schedules-3,68c831b62a0aaaa1,f8164d7154654266] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:42:25.261 INFO 2981023 --- [ocp-async-1,381519aa09440e16,e01db8ae9150d7dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,96051409ff2ae814,bb321ba912ab901b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,96051409ff2ae814,bb321ba912ab901b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:42:26.336294+08:00, lastProcessTime=2024-09-11T17:42:16.366416+08:00 2024-09-11 17:42:26.350 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:26.351 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:26.473 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:42:26.336294+08:00 2024-09-11 17:42:27.261 INFO 2981023 --- [ocp-async-10,5a7a2babd1d938a1,ec3eff7e6cbf89b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:27.758 INFO 2981023 --- [ocp-async-1,6d7b8abada159d13,51fa83fe7e076b55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:42:29.261 INFO 2981023 --- [ocp-async-3,b39f6f253f0d7ec7,54d2ed6ef92fdcdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,9abd93ecd9dc79d8,5760e6c93bf93597] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,9abd93ecd9dc79d8,5760e6c93bf93597] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:29.351 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:29.351 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:30.601 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:42:30.601 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:42:30.604 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:42:30.604 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:42:30.604 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:42:30.604 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:42:30.606 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:42:30.606 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:42:30.606 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:42:30.606 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:42:30.629 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:42:30.633 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:42:30.633 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:42:30.633 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:42:30.633 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:42:30.633 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:42:30.637 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:42:30.879 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:42:31.200 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.593857682S 2024-09-11 17:42:31.200 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:42:31.253 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:42:31.253 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:42:31.261 INFO 2981023 --- [ocp-async-1,e267a8acc6830e8a,d206a23594423055] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:31.270 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:42:31.322 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:42:31.322 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:42:31.322 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:42:31.322 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:42:31.322 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:42:31.322 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:42:31.328 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:42:31.408 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] o.s.boot.SpringApplication : Started application in 0.804 seconds (JVM running for 83551.281) 2024-09-11 17:42:31.408 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.804593912S 2024-09-11 17:42:31.619 INFO 2981023 --- [ocp-schedules-1,f9f9464065adf1ed,47b954c724a3660f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:42:32.078 INFO 2981023 --- [ocp-schedules-12,f2dd4a81a36fe699,dadd04ef7bd18e2a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:42:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,e7352ae2686e1fe7,cc6f1264a4b7ee10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:32.317 INFO 2981023 --- [prometheus-minute-schedule-1,e7352ae2686e1fe7,cc6f1264a4b7ee10] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:32.351 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:32.352 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:32.875 INFO 2981023 --- [ocp-schedules-8,c0f1c5b5816b6abb,bee7db03ccc950fa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:42:32.924 WARN 2981023 --- [scanner-13532816-1,c0f1c5b5816b6abb,2a12465afa099655] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:42:32.942 INFO 2981023 --- [alarm-detect-14,c0f1c5b5816b6abb,9d5911d0e87152ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644463, data is empty. 2024-09-11 17:42:32.947 INFO 2981023 --- [alarm-detect-14,c0f1c5b5816b6abb,9d5911d0e87152ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644463, data is empty. 2024-09-11 17:42:32.953 INFO 2981023 --- [alarm-detect-14,c0f1c5b5816b6abb,9d5911d0e87152ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644463, data is empty. 2024-09-11 17:42:32.972 INFO 2981023 --- [alarm-detect-5,c0f1c5b5816b6abb,60cf9555ce44a06f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644463, data is empty. 2024-09-11 17:42:32.977 INFO 2981023 --- [alarm-detect-5,c0f1c5b5816b6abb,60cf9555ce44a06f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644463, data is empty. 2024-09-11 17:42:32.982 INFO 2981023 --- [alarm-detect-5,c0f1c5b5816b6abb,60cf9555ce44a06f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644463, data is empty. 2024-09-11 17:42:32.985 INFO 2981023 --- [alarm-detect-18,c0f1c5b5816b6abb,45ca3242c1b622ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644463, data is empty. 2024-09-11 17:42:32.989 INFO 2981023 --- [alarm-detect-18,c0f1c5b5816b6abb,45ca3242c1b622ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644463, data is empty. 2024-09-11 17:42:32.992 INFO 2981023 --- [alarm-detect-18,c0f1c5b5816b6abb,45ca3242c1b622ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644463, data is empty. 2024-09-11 17:42:33.084 INFO 2981023 --- [alarm-detect-1,c0f1c5b5816b6abb,4a868ed01f7009c2] c.o.o.alarm.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1726047720 ~ 1726047743 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={monitor_exporter_avaliable=[SeriesMetricValues(metric=Metric{name='monitor_exporter_avaliable', labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://10.31.35.7:62888/metrics/stat, exporter_type=HOST_MONITOR, host=10.31.35.7, host_name=vm-eas-prd-rtd-mid05, scrape_interval=60, svr_ip=10.31.35.7}}, data=[ValueData(timestamp=1726047735, value=0.0)])]}, metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1726047735=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://10.31.35.7:62888/metrics/stat, exporter_type=HOST_MONITOR, host=10.31.35.7, host_name=vm-eas-prd-rtd-mid05, scrape_interval=60, svr_ip=10.31.35.7})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:42:33.084 INFO 2981023 --- [alarm-detect-1,c0f1c5b5816b6abb,4a868ed01f7009c2] c.o.o.alarm.core.detect.MetricDetector : alarmType monitor_exporter_unavaliable detect time range 1726047720 ~ 1726047743 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={monitor_exporter_avaliable=[SeriesMetricValues(metric=Metric{name='monitor_exporter_avaliable', labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://10.31.35.7:62889/metrics/stat, exporter_type=HOST_MONITOR, host=10.31.35.7, host_name=vm-eas-prd-rtd-mid05, scrape_interval=60, svr_ip=10.31.35.7}}, data=[ValueData(timestamp=1726047735, value=0.0)])]}, metricEvaluateResults={monitor_exporter_avaliable == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=monitor_exporter_avaliable, match=true, value=0.0, values={1726047735=0.0}, valueCount=1, labels={app=OCP, exporter=/metrics/stat, exporter_addr=http://10.31.35.7:62889/metrics/stat, exporter_type=HOST_MONITOR, host=10.31.35.7, host_name=vm-eas-prd-rtd-mid05, scrape_interval=60, svr_ip=10.31.35.7})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:42:33.085 INFO 2981023 --- [alarm-detect-1,c0f1c5b5816b6abb,4a868ed01f7009c2] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=monitor_exporter_unavaliable, target=alarm_template_id=0:host=10.31.35.7:exporter=/metrics/stat-3, currentState=Pending 2024-09-11 17:42:33.100 INFO 2981023 --- [alarm-detect-11,c0f1c5b5816b6abb,23d290fad915024f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644463, data is empty. 2024-09-11 17:42:33.106 INFO 2981023 --- [alarm-detect-11,c0f1c5b5816b6abb,23d290fad915024f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644463, data is empty. 2024-09-11 17:42:33.110 INFO 2981023 --- [alarm-detect-11,c0f1c5b5816b6abb,23d290fad915024f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644463, data is empty. 2024-09-11 17:42:33.261 INFO 2981023 --- [ocp-async-5,ebdd0501cf3dbcd8,03977efcd83eabe3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:33.485 INFO 2981023 --- [ocp-schedules-4,0accca95014aeb5d,c3c8929aa24abffe] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY,C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkprdocp-1719994437/H_10.31.35.10,C_linkrtd-1719994443/H_10.31.35.6,C_rtdob-1718689631/H_10.31.35.5,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7, current targets ,,,,,C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY,C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkprdocp-1719994437/H_10.31.35.10,C_linkrtd-1719994443/H_10.31.35.6,C_rtdob-1718689631/H_10.31.35.5,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:34.176 INFO 2981023 --- [ocp-schedules-3,3590ea0f95059a5a,6835a263a0cea5e0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:42:34.182 INFO 2981023 --- [ocp-schedules-3,3590ea0f95059a5a,6835a263a0cea5e0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:42:34.182 INFO 2981023 --- [ocp-schedules-3,3590ea0f95059a5a,6835a263a0cea5e0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:42:35.261 INFO 2981023 --- [ocp-async-2,6dbae4dfdb356435,30b1f45ce0eeb566] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,51ad4dfa7562303e,08237d827810f432] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,51ad4dfa7562303e,08237d827810f432] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:35.352 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:35.353 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:42:36.337285+08:00, lastProcessTime=2024-09-11T17:42:26.336294+08:00 2024-09-11 17:42:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:42:36.337285+08:00 2024-09-11 17:42:37.262 INFO 2981023 --- [ocp-async-7,f2a3d09b97dfe55c,33c5d7ee0030b879] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:37.314 INFO 2981023 --- [prometheus-cache-monitor1,ba6f4b140147a233,2e9ce4da0901889c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1875,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:42:37.315 INFO 2981023 --- [prometheus-cache-monitor1,ba6f4b140147a233,2e9ce4da0901889c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:42:37.758 INFO 2981023 --- [ocp-async-2,6140c4be93fa32b8,399c2c036f36d8db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:42:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,0911efa0f16b5997,2dd87fb12511deec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:38.317 INFO 2981023 --- [prometheus-minute-schedule-1,0911efa0f16b5997,2dd87fb12511deec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:38.353 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:38.355 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:39.262 INFO 2981023 --- [ocp-async-10,18653b20a23b2584,170cbe9c0a018c83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:40.004 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:42:40.004 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:42:40.593 INFO 2981023 --- [ocp-schedules-13,e53195427a4c26d7,af3305b3c80a2c1a] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:42:40.612 WARN 2981023 --- [ocp-schedules-13,e53195427a4c26d7,af3305b3c80a2c1a] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:42:41.262 INFO 2981023 --- [ocp-async-2,6963acfeffaa4c21,65095ca9eceb8c2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,d7655fdd9220223d,00de6b0d528157c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,d7655fdd9220223d,00de6b0d528157c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:41.355 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:41.355 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:42.081 INFO 2981023 --- [ocp-schedules-22,b460c52a9f1e5c61,0933438eef5395d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:42:43.167 INFO 2981023 --- [ocp-schedules-14,149fb6053b45880f,e4b0b50559fc0796] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:42:43.253 WARN 2981023 --- [scanner-13533145-1,149fb6053b45880f,55b1af74ca488c75] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:42:43.262 INFO 2981023 --- [ocp-async-9,388bd257bcee7687,0439bbdeeb8049a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,90557be6c4ce19c9,502ce426614f5ef4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:44.317 INFO 2981023 --- [prometheus-minute-schedule-1,90557be6c4ce19c9,502ce426614f5ef4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:44.356 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:44.357 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:45.089 INFO 2981023 --- [ocp-schedules-22,8ae1597febfb4e02,ac9d09fc65a77de4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-09-11 17:42:45.093 INFO 2981023 --- [ocp-schedules-22,8ae1597febfb4e02,ac9d09fc65a77de4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:45.101 INFO 2981023 --- [prometheus-second-schedule-1,f0e56a963a15fef8,4cb54f477b9bf31e] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:42:45.167 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:45.168 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:45.169 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:45.171 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 02:03:06.281634, 2024-09-11 17:42:45.119, 1000] 2024-09-11 17:42:45.171 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:21:42.083265, 2024-09-11 17:42:45.12, 1000] 2024-09-11 17:42:45.171 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-09-11 02:03:06.281634, endTime 2024-09-11 17:42:45.119 2024-09-11 17:42:45.172 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000006, startTime 2024-09-11 17:21:42.083265, endTime 2024-09-11 17:42:45.12 2024-09-11 17:42:45.173 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 10:40:18.658505, 2024-09-11 17:42:45.119, 1000] 2024-09-11 17:42:45.175 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2024-09-11 10:40:18.658505, endTime 2024-09-11 17:42:45.119 2024-09-11 17:42:45.186 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:45.190 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:45.211 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:21:42.083371, 2024-09-11 17:42:45.178, 1000] 2024-09-11 17:42:45.212 INFO 2981023 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:45.212 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000006, startTime 2024-09-11 17:21:42.083371, endTime 2024-09-11 17:42:45.178 2024-09-11 17:42:45.212 INFO 2981023 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:45.213 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:45.213 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:45.242 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 08:00:10.321716, 2024-09-11 17:42:45.206, 1000] 2024-09-11 17:42:45.252 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-09-11 08:00:10.321716, endTime 2024-09-11 17:42:45.206 2024-09-11 17:42:45.254 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 16:48:30.801707, 2024-09-11 17:42:45.206, 1000] 2024-09-11 17:42:45.255 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2024-09-11 16:48:30.801707, endTime 2024-09-11 17:42:45.206 2024-09-11 17:42:45.256 INFO 2981023 --- [ocp-schedules-22,8ae1597febfb4e02,ac9d09fc65a77de4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, tryCount=3, successCount=3 2024-09-11 17:42:45.262 INFO 2981023 --- [ocp-async-4,0d046c46dc294487,82144a24e1f4d33a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:46.019 INFO 2981023 --- [ocp-schedules-29,be141b27802c4824,8dbfd5555e0143ab] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:42:46.019 INFO 2981023 --- [host-check-clockdiff1,be141b27802c4824,52e90d5e336856b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:42:46.020 INFO 2981023 --- [host-check-clockdiff2,be141b27802c4824,554e57c0a6687dcb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:42:46.020 INFO 2981023 --- [host-check-clockdiff3,be141b27802c4824,030422d71baa05d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:42:46.026 INFO 2981023 --- [ocp-schedules-7,0f2eac7ee6b0eabe,ea432a929999f9ea] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:42:46.029 INFO 2981023 --- [ocp-schedules-17,1df9a5966867b802,cf7c94635924f8ff] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:42:46.029 INFO 2981023 --- [host-check-agent-status1,1df9a5966867b802,ebdffadd6924cf06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:42:46.029 INFO 2981023 --- [host-check-agent-status2,1df9a5966867b802,6aba994367db1c9b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:42:46.030 INFO 2981023 --- [host-check-agent-status3,1df9a5966867b802,801bcc9a496ac22e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:42:46.031 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:42:46.031 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:42:46.037 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:42:46.040 INFO 2981023 --- [host-check-clockdiff3,be141b27802c4824,030422d71baa05d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047766 0 0 , error= 2024-09-11 17:42:46.060 INFO 2981023 --- [host-check-agent-status3,1df9a5966867b802,801bcc9a496ac22e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:46.065 INFO 2981023 --- [host-check-agent-status3,1df9a5966867b802,801bcc9a496ac22e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:46.069 INFO 2981023 --- [host-check-agent-status1,1df9a5966867b802,ebdffadd6924cf06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:46.071 INFO 2981023 --- [host-check-agent-status1,1df9a5966867b802,ebdffadd6924cf06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:46.071 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:46.072 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:46.073 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:46.073 INFO 2981023 --- [host-check-agent-status2,1df9a5966867b802,6aba994367db1c9b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:46.074 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:46.076 INFO 2981023 --- [host-check-agent-status2,1df9a5966867b802,6aba994367db1c9b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:46.169 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:42:46.182 INFO 2981023 --- [ocp-schedules-8,a6ec86a62108ac75,2f895a2dcf79d450] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:42:46.187 INFO 2981023 --- [ocp-schedules-14,af39eb1b9d0754b4,2e0fef44845286e6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:42:46.187 INFO 2981023 --- [ocp-schedules-9,0f02ac22a5784038,0d497a4c586564bc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:42:46.188 INFO 2981023 --- [ocp-schedules-2,74785c28b6a442e0,f7b25e7ac0e36777] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:42:46.188 INFO 2981023 --- [ocp-schedules-1,6d171679217c0f2d,b8715ebe3d6aea85] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:42:46.197 INFO 2981023 --- [ocp-schedules-2,74785c28b6a442e0,f7b25e7ac0e36777] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:46.197 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:46.255 INFO 2981023 --- [ocp-schedules-9,0f02ac22a5784038,0d497a4c586564bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:46.263 INFO 2981023 --- [ocp-schedules-14,af39eb1b9d0754b4,2e0fef44845286e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:46.268 INFO 2981023 --- [ocp-schedules-9,0f02ac22a5784038,0d497a4c586564bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:42:46.291 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:46.303 INFO 2981023 --- [ocp-schedules-8,a6ec86a62108ac75,2f895a2dcf79d450] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:42:46.307 INFO 2981023 --- [host-check-agent-status1,1df9a5966867b802,ebdffadd6924cf06] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:42:46.311 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.311 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.312 INFO 2981023 --- [ocp-schedules-11,a99d471ac14f7d62,75e2cc9b97ffd38d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:42:46.312 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:46.312 INFO 2981023 --- [host-check-agent-status1,1df9a5966867b802,ebdffadd6924cf06] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:42:46.312 INFO 2981023 --- [host-check-agent-status1,1df9a5966867b802,ebdffadd6924cf06] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:42:46.312 INFO 2981023 --- [host-check-agent-status1,1df9a5966867b802,ebdffadd6924cf06] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:42:46.312 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:46.312 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:42:46.313 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:42:46.375 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.378 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.378 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:46.379 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:46.391 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:42:46.454 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:42:46.452493+08:00, lastProcessTime=2024-09-11T17:42:36.337285+08:00 2024-09-11 17:42:46.472 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.472 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:46.473 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:42:46.473 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:42:46.475 INFO 2981023 --- [ocp-schedules-14,af39eb1b9d0754b4,2e0fef44845286e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:42:46.475 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:42:46.474851849+08:00 2024-09-11 17:42:46.478 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:42:46.475+08:00 2024-09-11 17:42:46.480 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:42:46.481 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:42:46.480867541+08:00 2024-09-11 17:42:46.481 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:42:46.481+08:00 2024-09-11 17:42:46.485 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:42:46.475+08:00 and status AVAILABLE 2024-09-11 17:42:46.488 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:42:46.481+08:00 and status AVAILABLE 2024-09-11 17:42:46.489 INFO 2981023 --- [host-check-agent-status2,1df9a5966867b802,6aba994367db1c9b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:42:46.492 INFO 2981023 --- [host-check-agent-status2,1df9a5966867b802,6aba994367db1c9b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:42:46.492 INFO 2981023 --- [host-check-agent-status2,1df9a5966867b802,6aba994367db1c9b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:42:46.492 INFO 2981023 --- [host-check-agent-status2,1df9a5966867b802,6aba994367db1c9b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:42:46.493 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.499 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:42:46.499 INFO 2981023 --- [host-check-agent-status3,1df9a5966867b802,801bcc9a496ac22e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:42:46.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:42:46.501 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:42:46.505 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:42:46.505 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:42:46.505470360+08:00 2024-09-11 17:42:46.506 INFO 2981023 --- [host-check-agent-status3,1df9a5966867b802,801bcc9a496ac22e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:42:46.506 INFO 2981023 --- [host-check-agent-status3,1df9a5966867b802,801bcc9a496ac22e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:42:46.506 INFO 2981023 --- [host-check-agent-status3,1df9a5966867b802,801bcc9a496ac22e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:42:46.512 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:42:46.505+08:00 2024-09-11 17:42:46.557 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:42:46.505+08:00 and status AVAILABLE 2024-09-11 17:42:46.559 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.559 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.559 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:46.560 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.585 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:42:46.592 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:42:46.592 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:46.594 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:46.596 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.597 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:42:46.598 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:42:46.598 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.600 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:46.601 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:46.603 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:46.605 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:46.613 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:42:46.630 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:42:46.643 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.643 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:42:46.647 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.647 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.648 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:46.648 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:42:46.649 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.649 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:46.652 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.654 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.655 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:42:46.656 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:42:46.658 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:42:46.659 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:42:46.667 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:42:46.669 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:46.669 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:42:46.675 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:42:46.687 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:42:46.687 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:46.687 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:42:46.693 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.694 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:42:46.695 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.696 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:42:46.696 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:42:46.696 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:46.702 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:42:46.704 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:46.709 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.711 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.712 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:46.712 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:42:46.713 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:42:46.713 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.714 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.715 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:46.716 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:42:46.717 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:42:46.720 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:42:46.721 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:42:46.722 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:42:46.723 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:42:46.723 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:42:46.724 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:42:46.726 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:42:46.734 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:42:46.738 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:42:46.759 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:42:46.760 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:42:46.763 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:42:46.768 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:42:46.769 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.777 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.778 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.778 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:42:46.778 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:42:46.779 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:42:46.782 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.782 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:42:46.783 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:42:46.783 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.786 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:42:46.789 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:42:46.791 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:42:46.791 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:42:46.826 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.826 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.827 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:42:46.827 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:42:46.831 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:42:46.834 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:42:46.835 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:42:46.839 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:42:46.847 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.851 INFO 2981023 --- [host-check-host-status3,0f2eac7ee6b0eabe,6d50640bcb385e0e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:42:46.852 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.852 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:42:46.856 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:42:46.859 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.859 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.860 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:42:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:42:46.861 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:42:46.861 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.861 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:42:46.861 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.866 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:42:46.866 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:42:46.867 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:42:46.868 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:42:46.876 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:46.876 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:42:46.876 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:42:46.890 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:42:46.893 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:42:46.894 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:42:46.899 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.900 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.901 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:42:46.902 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:42:46.902 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:42:46.902 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.903 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.903 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:42:46.909 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:42:46.909 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:42:46.914 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:42:46.915 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:42:46.916 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:42:46.916 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:42:46.922 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.923 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:42:46.923 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.923 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:42:46.924 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:42:46.924 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.928 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:42:46.452493+08:00 2024-09-11 17:42:46.929 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:42:46.930 INFO 2981023 --- [host-check-host-status1,0f2eac7ee6b0eabe,b674edf05debbe8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:42:46.942 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:42:46.947 INFO 2981023 --- [host-check-host-status2,0f2eac7ee6b0eabe,ca4e3c45b6b28d22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:42:46.955 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:42:46.962 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:46.962 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:46.963 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:42:46.967 INFO 2981023 --- [host-check-clockdiff2,be141b27802c4824,554e57c0a6687dcb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047766 0 0 , error= 2024-09-11 17:42:46.970 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:42:46.970 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:42:46.970 INFO 2981023 --- [ocp-schedules-2,74785c28b6a442e0,f7b25e7ac0e36777] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:42:46.972 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:42:46.987 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:42:46.994 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:42:46.998 INFO 2981023 --- [ocp-schedules-25,70b4bf8c38291eb6,668664701b048dd9] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:42:47.262 INFO 2981023 --- [ocp-async-3,31f2fbb94d4f8b0f,ccceb090eac8edfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,144cbb388f7a2b85,6ff3e511f5fe2d58] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,144cbb388f7a2b85,6ff3e511f5fe2d58] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:47.357 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:47.358 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:47.455 INFO 2981023 --- [ocp-schedules-8,11083c50efd6df09,be75ad416a48308a] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 1 2024-09-11 17:42:47.457 INFO 2981023 --- [ocp-schedules-8,11083c50efd6df09,be75ad416a48308a] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-09-11 17:42:47.457 INFO 2981023 --- [ocp-schedules-8,11083c50efd6df09,be75ad416a48308a] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 1 2024-09-11 17:42:47.758 INFO 2981023 --- [ocp-async-8,132f71f30da0cd7c,d183ebdb9962b8cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:42:48.555 INFO 2981023 --- [host-check-clockdiff1,be141b27802c4824,52e90d5e336856b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047768 0 0 , error= 2024-09-11 17:42:49.263 INFO 2981023 --- [ocp-async-1,457520f46304cb81,d38a01a2b111808a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:42:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:42:50.153 INFO 2981023 --- [ocp-schedules-29,4ecaafd0c4fa7f9b,50e2ab5a8d0242d0] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:42:50.242 INFO 2981023 --- [ocp-schedules-30,22d2aba2bdb6763b,0ea387a4d96eb15a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:42:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,96d5bfe7064d5805,f717971e76e08ce0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,96d5bfe7064d5805,f717971e76e08ce0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:50.358 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:50.359 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:51.263 INFO 2981023 --- [ocp-async-6,d16f2758cc0c8f43,31a7ad04331daf18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:51.553 INFO 2981023 --- [ocp-schedules-20,31d61cce955991e7,7d3533526611fd33] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:42:51.555 INFO 2981023 --- [ocp-schedules-20,31d61cce955991e7,7d3533526611fd33] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:51.555 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:42:51.555 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:42:51.555 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:42:51.555 INFO 2981023 --- [ocp-schedules-20,31d61cce955991e7,7d3533526611fd33] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:42:52.083 INFO 2981023 --- [ocp-schedules-14,83070e084b1e4ce5,5f8535c2ac0e9cc1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:42:53.044 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:42:53.062 WARN 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:42:53.077 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2024-09-11 17:42:53.077 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2024-09-11 17:42:53.078 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2024-09-11 17:42:53.079 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2024-09-11 17:42:53.080 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2024-09-11 17:42:53.081 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2024-09-11 17:42:53.082 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2024-09-11 17:42:53.083 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2024-09-11 17:42:53.084 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2024-09-11 17:42:53.085 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2024-09-11 17:42:53.086 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 3 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2024-09-11 17:42:53.087 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2024-09-11 17:42:53.088 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2024-09-11 17:42:53.089 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2024-09-11 17:42:53.090 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2024-09-11 17:42:53.091 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2024-09-11 17:42:53.092 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2024-09-11 17:42:53.093 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2024-09-11 17:42:53.094 INFO 2981023 --- [ocp-schedules-22,e745c96f17eaf968,827df80d31713bde] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=126, updatedCount=126, failedToUpdateCount=0 2024-09-11 17:42:53.117 INFO 2981023 --- [ocp-schedules-17,089c71ae3e4b3a42,408a2c62d4371511] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:42:53.117 INFO 2981023 --- [ocp-schedules-17,089c71ae3e4b3a42,408a2c62d4371511] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:42:53.263 INFO 2981023 --- [ocp-async-3,11d85c0071855f3d,6b75981c3c477db1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,a02b0b168607b76e,129ac3f5dd869943] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,a02b0b168607b76e,129ac3f5dd869943] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:53.360 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:53.360 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:53.360 INFO 2981023 --- [ocp-schedules-9,f1c9b845aa123ddb,3115138e1e296a6b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:42:53.427 WARN 2981023 --- [scanner-13533372-1,f1c9b845aa123ddb,c32ea99f8085f9dc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:42:53.439 INFO 2981023 --- [alarm-detect-9,f1c9b845aa123ddb,f06131b566e88ecd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644463, data is empty. 2024-09-11 17:42:53.442 INFO 2981023 --- [alarm-detect-9,f1c9b845aa123ddb,f06131b566e88ecd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644463, data is empty. 2024-09-11 17:42:53.445 INFO 2981023 --- [alarm-detect-9,f1c9b845aa123ddb,f06131b566e88ecd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644463, data is empty. 2024-09-11 17:42:53.482 INFO 2981023 --- [alarm-detect-3,f1c9b845aa123ddb,dc52a53016f324a5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644463, data is empty. 2024-09-11 17:42:53.486 INFO 2981023 --- [alarm-detect-14,f1c9b845aa123ddb,bbf46a248bac5658] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644463, data is empty. 2024-09-11 17:42:53.486 INFO 2981023 --- [alarm-detect-3,f1c9b845aa123ddb,dc52a53016f324a5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644463, data is empty. 2024-09-11 17:42:53.490 INFO 2981023 --- [alarm-detect-14,f1c9b845aa123ddb,bbf46a248bac5658] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644463, data is empty. 2024-09-11 17:42:53.497 INFO 2981023 --- [alarm-detect-3,f1c9b845aa123ddb,dc52a53016f324a5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644463, data is empty. 2024-09-11 17:42:53.501 INFO 2981023 --- [alarm-detect-14,f1c9b845aa123ddb,bbf46a248bac5658] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644463, data is empty. 2024-09-11 17:42:53.512 INFO 2981023 --- [ocp-schedules-27,fd57f8886ba39572,bd2a119afcb95b8a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:42:53.512+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:42:53.529 INFO 2981023 --- [ocp-schedules-27,fd57f8886ba39572,bd2a119afcb95b8a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:42:53.618 INFO 2981023 --- [alarm-detect-5,f1c9b845aa123ddb,f8981d195e125f2f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644463, data is empty. 2024-09-11 17:42:53.623 INFO 2981023 --- [alarm-detect-5,f1c9b845aa123ddb,f8981d195e125f2f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644463, data is empty. 2024-09-11 17:42:53.627 INFO 2981023 --- [alarm-detect-5,f1c9b845aa123ddb,f8981d195e125f2f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644463, data is empty. 2024-09-11 17:42:53.716 INFO 2981023 --- [ocp-schedules-27,fd57f8886ba39572,bd2a119afcb95b8a] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 38 to SqlAggregateData 2024-09-11 17:42:53.718 INFO 2981023 --- [ocp-schedules-27,fd57f8886ba39572,bd2a119afcb95b8a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:42:54.185 INFO 2981023 --- [ocp-schedules-18,e94ff9e91d896bfb,39215a5588fa5a6e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:42:54.196 INFO 2981023 --- [ocp-schedules-18,e94ff9e91d896bfb,39215a5588fa5a6e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:42:54.196 INFO 2981023 --- [ocp-schedules-18,e94ff9e91d896bfb,39215a5588fa5a6e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:42:54.262 INFO 2981023 --- [ocp-schedules-26,9ef38dc9af675fb9,0c0ffe7e9484d95f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:42:54.292 INFO 2981023 --- [ocp-schedules-3,c2b0ca709e59cdde,17211c79e6a7f4fe] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:42:54.294 INFO 2981023 --- [ocp-schedules-3,c2b0ca709e59cdde,17211c79e6a7f4fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:42:54.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:42:54.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.339 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:42:54.339 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.340 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:42:54.340 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.341 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:42:54.343 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:42:54.344 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:42:54.349 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.349 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.352 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.352 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.353 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.353 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.355 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.355 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.360 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.360 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:42:54.365 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.365 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.366 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.366 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.368 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.368 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:42:54.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.380 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.380 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:42:54.380 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:42:54.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3011D6E7-0-0] 2024-09-11 17:42:54.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:42:54.381 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.381 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:42:54.381 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.381 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.387 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.387 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:42:54.393 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.394 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:42:54.394 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.394 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:42:54.395 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.395 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:42:54.395 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.396 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:42:54.397 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:42:54.398 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.399 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:42:54.399 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.400 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:42:54.401 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.401 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:42:54.402 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.402 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:42:54.403 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.404 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:42:54.404 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:42:54.405 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.405 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:42:54.406 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.406 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:42:54.407 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.408 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:42:54.408 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.408 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:42:54.408 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3011D6F1-0-0] 2024-09-11 17:42:54.408 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:42:54.408 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:42:54.409 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.409 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:42:54.409 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:42:54.411 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:42:54.415 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.417 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:42:54.417 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.418 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:42:54.418 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.419 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:54.419 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:42:54.420 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.421 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:42:54.421 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.421 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.421 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.422 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:42:54.422 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.423 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:42:54.423 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:42:54.425 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:54.425 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:42:54.426 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.427 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:42:54.434 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:42:54.438 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:42:54.438 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:42:54.439 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.439 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:42:54.443 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:42:54.445 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.445 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:42:54.446 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:42:54.446 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:42:54.446 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:42:54.447 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:42:54.449 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:42:54.449 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.449 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.454 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:42:54.455 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:42:54.455 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:42:54.455 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:42:54.455 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:42:54.456 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:42:54.457 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:54.457 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:42:54.465 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:42:54.466 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:42:54.466 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:42:54.466 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:42:54.467 INFO 2981023 --- [ocp-schedules-3,c2b0ca709e59cdde,17211c79e6a7f4fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:42:54.791 INFO 2981023 --- [ocp-schedules-17,ed7c0cca75fc3dd2,368f6cbacbee45cb] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:42:54.791 INFO 2981023 --- [ocp-schedules-17,ed7c0cca75fc3dd2,368f6cbacbee45cb] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:42:54.795 INFO 2981023 --- [ocp-schedules-17,ed7c0cca75fc3dd2,368f6cbacbee45cb] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:42:54.795 INFO 2981023 --- [ocp-schedules-17,ed7c0cca75fc3dd2,368f6cbacbee45cb] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:42:54.795 INFO 2981023 --- [ocp-schedules-17,ed7c0cca75fc3dd2,368f6cbacbee45cb] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:42:54.805 INFO 2981023 --- [ocp-schedules-17,ed7c0cca75fc3dd2,368f6cbacbee45cb] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:42:54.805 INFO 2981023 --- [ocp-schedules-17,ed7c0cca75fc3dd2,368f6cbacbee45cb] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:42:54.805 INFO 2981023 --- [ocp-schedules-17,ed7c0cca75fc3dd2,368f6cbacbee45cb] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:42:54.815 INFO 2981023 --- [ocp-schedules-17,ed7c0cca75fc3dd2,368f6cbacbee45cb] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:42:55.263 INFO 2981023 --- [ocp-async-9,d10c3c3b1df31b5a,d3dbe1fcc9c9832a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:55.706 INFO 2981023 --- [ocp-schedules-25,18a07206232ed1bd,0c1b9ef1bccc1592] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:42:55.706 INFO 2981023 --- [ocp-schedules-25,18a07206232ed1bd,0c1b9ef1bccc1592] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:42:55.710 INFO 2981023 --- [ocp-schedules-25,18a07206232ed1bd,0c1b9ef1bccc1592] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:42:55.710 INFO 2981023 --- [ocp-schedules-25,18a07206232ed1bd,0c1b9ef1bccc1592] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:42:55.712 INFO 2981023 --- [ocp-schedules-25,18a07206232ed1bd,0c1b9ef1bccc1592] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:42:55.712 INFO 2981023 --- [ocp-schedules-25,18a07206232ed1bd,0c1b9ef1bccc1592] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:42:55.811 INFO 2981023 --- [ocp-schedules-18,718c0143560e07d2,9a8f21d786dfbbdd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:42:55.817 INFO 2981023 --- [ocp-schedules-18,718c0143560e07d2,9a8f21d786dfbbdd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:42:55.817 INFO 2981023 --- [ocp-schedules-18,718c0143560e07d2,9a8f21d786dfbbdd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:42:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,4c0598f65ce7af10,4db852e8ac185b9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,4c0598f65ce7af10,4db852e8ac185b9d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:42:56.352070+08:00, lastProcessTime=2024-09-11T17:42:46.452493+08:00 2024-09-11 17:42:56.360 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:56.361 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:42:56.453 INFO 2981023 --- [ocp-schedules-23,1251392fb3ffd12f,d0986a221adaf494] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:42:56.453 INFO 2981023 --- [ocp-schedules-23,1251392fb3ffd12f,d0986a221adaf494] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:42:56.467 INFO 2981023 --- [Collect all tenants major compactions2,1251392fb3ffd12f,5e4ad167469531ef] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:42:56.469 INFO 2981023 --- [Collect all tenants major compactions1,1251392fb3ffd12f,5026cd71764c8546] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:42:56.475 INFO 2981023 --- [Collect all tenants major compactions3,1251392fb3ffd12f,e9e2031d50780f7f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:42:56.475 INFO 2981023 --- [Collect all tenants major compactions4,1251392fb3ffd12f,30c8b3b3ed1a2a69] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:42:56.479 INFO 2981023 --- [Collect all tenants major compactions5,1251392fb3ffd12f,ee87409f9ad9df57] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:42:56.482 INFO 2981023 --- [Collect all tenants major compactions1,1251392fb3ffd12f,5026cd71764c8546] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:56.483 INFO 2981023 --- [Collect all tenants major compactions6,1251392fb3ffd12f,3eadfe60800d9aca] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:42:56.483 INFO 2981023 --- [Collect all tenants major compactions1,1251392fb3ffd12f,5026cd71764c8546] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:42:56.483 INFO 2981023 --- [Collect all tenants major compactions2,1251392fb3ffd12f,5e4ad167469531ef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:56.489 INFO 2981023 --- [Collect all tenants major compactions2,1251392fb3ffd12f,5e4ad167469531ef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:42:56.491 INFO 2981023 --- [Collect all tenants major compactions1,1251392fb3ffd12f,5026cd71764c8546] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:42:56.493 INFO 2981023 --- [Collect all tenants major compactions2,1251392fb3ffd12f,5e4ad167469531ef] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:42:56.494 INFO 2981023 --- [Collect all tenants major compactions1,1251392fb3ffd12f,5026cd71764c8546] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:42:56.496 INFO 2981023 --- [Collect all tenants major compactions2,1251392fb3ffd12f,5e4ad167469531ef] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:42:56.497 INFO 2981023 --- [Collect all tenants major compactions5,1251392fb3ffd12f,ee87409f9ad9df57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:56.498 INFO 2981023 --- [Collect all tenants major compactions5,1251392fb3ffd12f,ee87409f9ad9df57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:42:56.500 INFO 2981023 --- [Collect all tenants major compactions5,1251392fb3ffd12f,ee87409f9ad9df57] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:42:56.500 INFO 2981023 --- [Collect all tenants major compactions5,1251392fb3ffd12f,ee87409f9ad9df57] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:42:56.504 INFO 2981023 --- [Collect all tenants major compactions4,1251392fb3ffd12f,30c8b3b3ed1a2a69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:56.505 INFO 2981023 --- [Collect all tenants major compactions4,1251392fb3ffd12f,30c8b3b3ed1a2a69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:42:56.507 INFO 2981023 --- [Collect all tenants major compactions3,1251392fb3ffd12f,e9e2031d50780f7f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:56.507 INFO 2981023 --- [Collect all tenants major compactions4,1251392fb3ffd12f,30c8b3b3ed1a2a69] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:42:56.507 INFO 2981023 --- [Collect all tenants major compactions4,1251392fb3ffd12f,30c8b3b3ed1a2a69] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:42:56.507 INFO 2981023 --- [Collect all tenants major compactions6,1251392fb3ffd12f,3eadfe60800d9aca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:42:56.508 INFO 2981023 --- [Collect all tenants major compactions3,1251392fb3ffd12f,e9e2031d50780f7f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:42:56.508 INFO 2981023 --- [Collect all tenants major compactions6,1251392fb3ffd12f,3eadfe60800d9aca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:42:56.509 INFO 2981023 --- [Collect all tenants major compactions3,1251392fb3ffd12f,e9e2031d50780f7f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:42:56.509 INFO 2981023 --- [Collect all tenants major compactions3,1251392fb3ffd12f,e9e2031d50780f7f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:42:56.510 INFO 2981023 --- [Collect all tenants major compactions6,1251392fb3ffd12f,3eadfe60800d9aca] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:42:56.510 INFO 2981023 --- [Collect all tenants major compactions6,1251392fb3ffd12f,3eadfe60800d9aca] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:42:56.510 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:42:56.352070+08:00 2024-09-11 17:42:56.510 INFO 2981023 --- [ocp-schedules-23,1251392fb3ffd12f,d0986a221adaf494] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:42:57.263 INFO 2981023 --- [ocp-async-8,3ba1f4d61652e14d,50f9ee6bf171894c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:57.758 INFO 2981023 --- [ocp-async-1,94a73a52f080869c,318ead930d4067a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:42:58.451 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.455 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=link 2024-09-11 17:42:58.462 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.468 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.468 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.474 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.474 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.475 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.476 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.477 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.477 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.478 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.479 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.479 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.483 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.485 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.489 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.493 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.496 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.499 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.502 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.505 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.509 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.512 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.515 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.519 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:36:58.352, end=2024-09-11T17:41:58.447} 2024-09-11 17:42:58.519 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:42:59.263 INFO 2981023 --- [ocp-async-3,e7af246270763eeb,2c2064cc077a40d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:42:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,60f3d61995d440ea,9dc4dd73a10c2d9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047780, interval=60 2024-09-11 17:42:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,60f3d61995d440ea,9dc4dd73a10c2d9a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:42:59.361 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:42:59.361 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:01.263 INFO 2981023 --- [ocp-async-9,94d6090bb1dca5e4,77b2aab313a50501] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:02.086 INFO 2981023 --- [ocp-schedules-25,0ba3b5799eca1342,193651c5efe7906e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:43:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,658722c5749fb33a,8a5d4a26fd217238] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,658722c5749fb33a,8a5d4a26fd217238] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:02.361 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:02.362 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:03.263 INFO 2981023 --- [ocp-async-8,c647ff2dc2aca001,76c9fa979e38fbc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:03.644 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:39:03.22, end=2024-09-11T17:41:03.639} 2024-09-11 17:43:03.647 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClu 2024-09-11 17:43:03.647 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:43:03.648 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:39:03.22, end=2024-09-11T17:41:03.639},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:43:03.648 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:43:03.648 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:43:03.649 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:43:03.650 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:39:03.22, end=2024-09-11T17:41:03.639},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:43:03.650 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:43:03.650 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:43:03.658 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:43:03.666 INFO 2981023 --- [ocp-schedules-3,3cdf639ecf8e9fec,d2586db68e58023d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:43:03.694 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961383694000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:03.694Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:03.703 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:43:03.703 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:03.715 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:03.715 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:03.716 WARN 2981023 --- [scanner-13533681-1,3cdf639ecf8e9fec,f4c29763a16cb130] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:43:03.722 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:03.722 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:03.732 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:03.732 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:03.737 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:43:03.737 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:03.694, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:03.737 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:43:03.746 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 4 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:39:03.22, end=2024-09-11T17:41:03.639},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:43:03.762 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:43:03.762 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:43:03.992 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961383992000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:03.992Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.008 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:43:04.008 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.008 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384008000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.008Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.013 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 62 2024-09-11 17:43:04.013 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.083 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.083 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.085 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.085 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.085 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.085 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.088 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.088 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.090 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384090000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.090Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.102 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.102 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.103 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.103 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.106 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.129 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.129 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.129 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.129 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.130 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384130000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.130Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.133 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.133 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.135 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:03.992, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.141 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.153 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384153000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.153Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.157 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.157 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.160 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384160000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.160Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.167 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.167 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.169 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.169 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.175 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.175 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.176 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384176000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.176Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.177 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.178 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384178000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.178Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.182 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.182 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.186 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.008, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.188 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.188 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.191 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:43:04.191 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.191 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.09, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.200 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.200 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.201 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:43:04.201 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.204 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.204 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.204 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.204 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.207 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.207 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.209 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.212 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.212 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.216 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384216000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.216Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.220 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.13, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.224 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.224 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.225 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.225 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.225 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.225 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.228 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.228 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:43:04.228 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.229 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.230 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.230 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.233 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.153, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.234 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.234 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.236 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.236 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.237 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.16, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.240 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384240000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.240Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.247 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.247 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.248 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384248000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.248Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.251 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.251 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.254 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.254 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.256 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.256 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.258 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.260 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384260000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.260Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.262 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.265 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.265 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.266 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:43:04.266 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.268 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.178, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.269 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.176, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.273 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.273 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.275 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384275000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.275Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.277 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.277 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.279 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.279 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.282 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.282 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.283 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.283 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.284 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384284000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.284Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.286 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.286 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.287 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.288 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.288 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.289 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.289 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.291 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.291 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.293 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.293 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.293 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.216, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.293 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.293 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.294 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.297 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.297 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.298 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.24, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.299 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.299 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.302 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.302 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.303 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.303 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.304 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.306 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.306 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.307 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.307 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.308 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.311 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.311 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.312 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.26, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.313 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.314 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.248, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.317 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384317000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.317Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.317 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961384317000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:43:04.317Z, endTime=2024-09-11T09:41:03.639Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:43:04.318 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.318 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.320 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.275, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.321 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.324 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.284, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.326 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.326 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.326 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:43:04.326 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:43:04.332 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.332 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.332 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.332 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.334 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.334 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:43:04.334 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.334 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:43:04.340 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.340 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.340 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:43:04.340 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047663639000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:43:04.341 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.341 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:43:04.344 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.317, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.344 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:43:04.317, end=2024-09-11T17:41:03.639}, clusterId=2, tenantId=6, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:43:04.344 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:43:04.346 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:39:03.22, end=2024-09-11T17:41:03.639},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:43:04.346 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:43:04.346 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:43:04.355 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:43:04.355 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:43:04.355 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:39:03.22, end=2024-09-11T17:41:03.639},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:43:04.355 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:43:04.355 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:43:04.356 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:43:04.356 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:39:03.22, end=2024-09-11T17:41:03.639},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:43:04.356 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:43:04.356 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:43:04.357 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:43:05.263 INFO 2981023 --- [ocp-async-7,32eaeb656ca29e24,53b228e763398f81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,650e2cc8034eb9be,d5f0dad97ccec11d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,650e2cc8034eb9be,d5f0dad97ccec11d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:05.362 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:05.362 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:43:06.337016+08:00, lastProcessTime=2024-09-11T17:42:56.352070+08:00 2024-09-11 17:43:06.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:43:06.337016+08:00 2024-09-11 17:43:07.264 INFO 2981023 --- [ocp-async-5,c2a14da294a7e5e1,e7adcc5894eaa9eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:07.440 INFO 2981023 --- [ocp-schedules-16,56ee517101aea838,e59206cdd2b580ac] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:43:07.456 INFO 2981023 --- [ocp-schedules-17,f2aafec99e126306,e37bf67b091219db] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:43:07.456+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:43:07.457 INFO 2981023 --- [ocp-schedules-17,f2aafec99e126306,e37bf67b091219db] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:43:07.458 INFO 2981023 --- [ocp-schedules-17,f2aafec99e126306,e37bf67b091219db] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:43:07.459 INFO 2981023 --- [ocp-schedules-17,f2aafec99e126306,e37bf67b091219db] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:43:07.758 INFO 2981023 --- [ocp-async-10,17798d1de47d553b,519a043c54ded10d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:43:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,9135c874d61f846c,09ee2aca02153007] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,9135c874d61f846c,09ee2aca02153007] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:08.363 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:08.363 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:09.264 INFO 2981023 --- [ocp-async-6,990832f4f8ce4782,4f225cb982f143ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:10.002 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:43:10.002 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:43:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:43:10.919 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:43:11.264 INFO 2981023 --- [ocp-async-7,9680eb0d347c9ae4,8f7b98673ce8f50e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,8cde6bd0efaaf217,02e66ecb3ca47834] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,8cde6bd0efaaf217,02e66ecb3ca47834] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:11.363 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:11.363 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:12.089 INFO 2981023 --- [ocp-schedules-8,1c566d2d74739be3,e647f846888ebe45] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:43:12.547 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,faa790b18f829863,016ac04200cedb3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=faa790b18f829863, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:43:12.548 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-7,faa790b18f829863,016ac04200cedb3c] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:43:12.577 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,faa790b18f829863,016ac04200cedb3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=faa790b18f829863, duration=35 ms] 2024-09-11 17:43:13.236 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047720, nodeCount=1000, affectRows=1000 2024-09-11 17:43:13.238 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047720, nodeCount=875, affectRows=875 2024-09-11 17:43:13.238 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047720, nodeCount=999, validCount=999 2024-09-11 17:43:13.240 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047720, nodeCount=875, validCount=875 2024-09-11 17:43:13.264 INFO 2981023 --- [ocp-async-1,dcefbc85f60e02f0,c3cba447da7a6f96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:13.985 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047720, nodeCount=463, affectRows=463 2024-09-11 17:43:14.197 INFO 2981023 --- [ocp-schedules-12,3fcebd28b5df225e,ab6b4224cb73d7b6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:43:14.204 INFO 2981023 --- [ocp-schedules-12,3fcebd28b5df225e,ab6b4224cb73d7b6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:43:14.204 INFO 2981023 --- [ocp-schedules-12,3fcebd28b5df225e,ab6b4224cb73d7b6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:43:14.224 INFO 2981023 --- [ocp-schedules-2,d948c8d383a0531b,294c87d5480e6c8c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:43:14.271 WARN 2981023 --- [scanner-13535389-1,d948c8d383a0531b,4e49dc1f5e914a1b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:43:14.296 INFO 2981023 --- [alarm-detect-17,d948c8d383a0531b,a00cd41f159e9b96] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644463, data is empty. 2024-09-11 17:43:14.302 INFO 2981023 --- [alarm-detect-17,d948c8d383a0531b,a00cd41f159e9b96] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644501, data is empty. 2024-09-11 17:43:14.308 INFO 2981023 --- [alarm-detect-17,d948c8d383a0531b,a00cd41f159e9b96] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644501, data is empty. 2024-09-11 17:43:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,09ed6c94a9dbca44,5347e10eeb1c7af0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,0a7bf89c0f2c1711,7d8ab57d4f5bb9c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:43:14.318 INFO 2981023 --- [prometheus-minute-schedule-1,09ed6c94a9dbca44,5347e10eeb1c7af0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:14.343 INFO 2981023 --- [alarm-detect-11,d948c8d383a0531b,6d784509cb92ae75] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644463, data is empty. 2024-09-11 17:43:14.345 INFO 2981023 --- [alarm-detect-6,d948c8d383a0531b,318101158736c118] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644463, data is empty. 2024-09-11 17:43:14.349 INFO 2981023 --- [alarm-detect-6,d948c8d383a0531b,318101158736c118] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644501, data is empty. 2024-09-11 17:43:14.351 INFO 2981023 --- [alarm-detect-11,d948c8d383a0531b,6d784509cb92ae75] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644501, data is empty. 2024-09-11 17:43:14.356 INFO 2981023 --- [alarm-detect-6,d948c8d383a0531b,318101158736c118] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644501, data is empty. 2024-09-11 17:43:14.359 INFO 2981023 --- [alarm-detect-11,d948c8d383a0531b,6d784509cb92ae75] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644501, data is empty. 2024-09-11 17:43:14.364 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:14.366 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:14.451 INFO 2981023 --- [alarm-detect-17,d948c8d383a0531b,21b12a01532841df] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644463, data is empty. 2024-09-11 17:43:14.457 INFO 2981023 --- [alarm-detect-17,d948c8d383a0531b,21b12a01532841df] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644501, data is empty. 2024-09-11 17:43:14.462 INFO 2981023 --- [alarm-detect-17,d948c8d383a0531b,21b12a01532841df] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644501, data is empty. 2024-09-11 17:43:15.264 INFO 2981023 --- [ocp-async-8,c5f1c46b97c50dc2,4279d79fc3fafbac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:15.740 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047720, nodeCount=463, validCount=463 2024-09-11 17:43:16.014 INFO 2981023 --- [ocp-schedules-12,b7d60ebb6c5782eb,eba6204cecf789c6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:43:16.015 INFO 2981023 --- [host-check-clockdiff1,b7d60ebb6c5782eb,7f7302d22acb251f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:43:16.015 INFO 2981023 --- [host-check-clockdiff2,b7d60ebb6c5782eb,30ebae84495c3b30] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:43:16.015 INFO 2981023 --- [host-check-clockdiff3,b7d60ebb6c5782eb,c0ec8664a7060cf4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:43:16.032 INFO 2981023 --- [host-check-clockdiff3,b7d60ebb6c5782eb,c0ec8664a7060cf4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047796 0 0 , error= 2024-09-11 17:43:16.032 INFO 2981023 --- [ocp-schedules-16,2543d36ba6db0e1a,cee2306e65ae5cad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:43:16.033 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:43:16.033 INFO 2981023 --- [ocp-schedules-2,93b6b83d2b41357a,e26477ae7bda4a0b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:43:16.033 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:43:16.033 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:43:16.034 INFO 2981023 --- [host-check-agent-status1,93b6b83d2b41357a,46a998cbaa1fcd4c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:43:16.034 INFO 2981023 --- [host-check-agent-status2,93b6b83d2b41357a,9207403ffbb43124] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:43:16.034 INFO 2981023 --- [host-check-agent-status3,93b6b83d2b41357a,0ace1fc2bab470c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:43:16.063 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:43:16.064 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:43:16.067 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:43:16.068 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:43:16.068 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:43:16.069 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:43:16.072 INFO 2981023 --- [host-check-agent-status1,93b6b83d2b41357a,46a998cbaa1fcd4c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:43:16.073 INFO 2981023 --- [host-check-agent-status2,93b6b83d2b41357a,9207403ffbb43124] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:43:16.073 INFO 2981023 --- [host-check-agent-status1,93b6b83d2b41357a,46a998cbaa1fcd4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:43:16.076 INFO 2981023 --- [host-check-agent-status2,93b6b83d2b41357a,9207403ffbb43124] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:43:16.187 INFO 2981023 --- [ocp-schedules-25,18843253b15c1860,784e5f8809641c59] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:43:16.193 INFO 2981023 --- [ocp-schedules-20,5779e8d1903fb83e,99ddd45d1336e14b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:43:16.193 INFO 2981023 --- [ocp-schedules-28,e21cb6edaaf3b224,a4b6ed6fd02477b8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:43:16.194 INFO 2981023 --- [ocp-schedules-27,4fb8e2e39a66ce98,02ac1be33c0aa2e5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:43:16.272 INFO 2981023 --- [host-check-agent-status3,93b6b83d2b41357a,0ace1fc2bab470c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:43:16.274 INFO 2981023 --- [ocp-schedules-28,e21cb6edaaf3b224,a4b6ed6fd02477b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:43:16.278 INFO 2981023 --- [host-check-agent-status3,93b6b83d2b41357a,0ace1fc2bab470c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:43:16.299 INFO 2981023 --- [ocp-schedules-25,18843253b15c1860,784e5f8809641c59] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:43:16.299 INFO 2981023 --- [ocp-schedules-20,5779e8d1903fb83e,99ddd45d1336e14b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:43:16.319 INFO 2981023 --- [ocp-schedules-22,44c108875e79b396,6b49dd592e6d8955] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:43:16.329 INFO 2981023 --- [ocp-schedules-20,5779e8d1903fb83e,99ddd45d1336e14b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:43:16.357 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:43:16.358 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:43:16.360 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:43:16.359663356+08:00 2024-09-11 17:43:16.360 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:43:16.359429435+08:00 2024-09-11 17:43:16.361 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:43:16.360+08:00 2024-09-11 17:43:16.362 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:43:16.360+08:00 2024-09-11 17:43:16.364 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:43:16.360+08:00 and status AVAILABLE 2024-09-11 17:43:16.364 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:43:16.360+08:00 and status AVAILABLE 2024-09-11 17:43:16.368 INFO 2981023 --- [host-check-agent-status2,93b6b83d2b41357a,9207403ffbb43124] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:43:16.370 INFO 2981023 --- [host-check-agent-status2,93b6b83d2b41357a,9207403ffbb43124] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:43:16.371 INFO 2981023 --- [host-check-agent-status2,93b6b83d2b41357a,9207403ffbb43124] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:43:16.371 INFO 2981023 --- [host-check-agent-status2,93b6b83d2b41357a,9207403ffbb43124] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:43:16.372 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:43:16.374 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:43:16.372545+08:00, lastProcessTime=2024-09-11T17:43:06.337016+08:00 2024-09-11 17:43:16.374 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:43:16.374046921+08:00 2024-09-11 17:43:16.375 INFO 2981023 --- [host-check-agent-status3,93b6b83d2b41357a,0ace1fc2bab470c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:43:16.375 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:43:16.374+08:00 2024-09-11 17:43:16.375 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:43:16.378 INFO 2981023 --- [host-check-agent-status3,93b6b83d2b41357a,0ace1fc2bab470c7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:43:16.378 INFO 2981023 --- [host-check-agent-status3,93b6b83d2b41357a,0ace1fc2bab470c7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:43:16.378 INFO 2981023 --- [host-check-agent-status3,93b6b83d2b41357a,0ace1fc2bab470c7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:43:16.379 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:43:16.374+08:00 and status AVAILABLE 2024-09-11 17:43:16.382 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:43:16.383 INFO 2981023 --- [host-check-agent-status1,93b6b83d2b41357a,46a998cbaa1fcd4c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:43:16.384 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.384 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:16.385 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:43:16.386 INFO 2981023 --- [host-check-agent-status1,93b6b83d2b41357a,46a998cbaa1fcd4c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:43:16.386 INFO 2981023 --- [host-check-agent-status1,93b6b83d2b41357a,46a998cbaa1fcd4c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:43:16.386 INFO 2981023 --- [host-check-agent-status1,93b6b83d2b41357a,46a998cbaa1fcd4c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:43:16.395 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.395 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:16.395 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:43:16.398 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.398 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:16.409 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.409 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:16.409 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:16.410 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:43:16.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:43:16.413 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:43:16.454 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:43:16.455 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:43:16.458 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.459 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:16.459 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:43:16.460 INFO 2981023 --- [ocp-schedules-25,18843253b15c1860,784e5f8809641c59] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:43:16.465 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:43:16.467 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:43:16.478 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:43:16.478 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:43:16.498 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:43:16.501 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.502 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.502 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:43:16.504 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:43:16.506 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:43:16.515 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:43:16.520 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:43:16.525 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:43:16.527 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:43:16.537 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:43:16.553 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.557 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.558 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.559 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:16.560 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.563 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:43:16.565 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:43:16.567 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:43:16.568 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:43:16.571 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.573 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:43:16.576 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:16.576 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:43:16.580 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:16.588 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:43:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.592 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:16.593 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:43:16.594 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:43:16.596 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:43:16.598 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.599 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.600 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:43:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:43:16.601 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:43:16.603 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.604 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.605 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:16.605 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:43:16.606 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:43:16.612 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:43:16.614 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:16.614 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:43:16.618 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:43:16.619 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:43:16.626 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:43:16.633 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:16.633 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.633 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:43:16.635 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.635 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:16.636 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:43:16.637 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.639 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:43:16.640 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:43:16.641 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:43:16.644 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.645 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.646 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:16.647 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:43:16.650 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:16.650 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:43:16.650 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:43:16.652 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:43:16.652 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:43:16.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:43:16.659 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:43:16.660 INFO 2981023 --- [host-check-host-status1,2543d36ba6db0e1a,3a98a9a613ee5c4f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:43:16.661 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:43:16.668 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:43:16.672 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:16.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:43:16.675 INFO 2981023 --- [host-check-host-status2,2543d36ba6db0e1a,a44498db1614211d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:43:16.682 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:43:16.688 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.689 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.689 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:16.689 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:43:16.694 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:43:16.699 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:43:16.702 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:43:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:43:16.719 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:43:16.726 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:43:16.728 INFO 2981023 --- [host-check-host-status3,2543d36ba6db0e1a,7985feca18358c81] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:43:16.738 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.739 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.740 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:43:16.747 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:43:16.748 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:43:16.748 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:43:16.752 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:43:16.372545+08:00 2024-09-11 17:43:16.754 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.755 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.755 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:43:16.758 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:43:16.762 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:43:16.762 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:43:16.764 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:16.764 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:16.764 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:43:16.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:43:16.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:43:16.769 INFO 2981023 --- [ocp-schedules-28,e21cb6edaaf3b224,a4b6ed6fd02477b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:43:16.960 INFO 2981023 --- [host-check-clockdiff2,b7d60ebb6c5782eb,30ebae84495c3b30] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047796 0 0 , error= 2024-09-11 17:43:17.264 INFO 2981023 --- [ocp-async-10,b2129aa88022d9a0,2e6936d0ff26d753] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,ea7b4b272472989c,5fdb1e149a093dda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,ea7b4b272472989c,5fdb1e149a093dda] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:17.366 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:17.367 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:17.758 INFO 2981023 --- [ocp-async-9,263db0ea18110167,32ef89d41b09528a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:43:19.264 INFO 2981023 --- [ocp-async-3,e1929088c84cd2d1,1d1f95abc3e2b5ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,876b5c65504d3f0e,1ee474e793105992] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,876b5c65504d3f0e,1ee474e793105992] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:20.367 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:20.369 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:21.073 INFO 2981023 --- [host-check-clockdiff1,b7d60ebb6c5782eb,7f7302d22acb251f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047801 0 0 , error= 2024-09-11 17:43:21.264 INFO 2981023 --- [ocp-async-4,6335e6358da4b19b,65e1df211ad7e7ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:22.093 INFO 2981023 --- [ocp-schedules-23,64b1cf02e9de49d0,9722a8e54024264e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:43:22.247 INFO 2981023 --- [ocp-schedules-10,7a922092d75871ab,c21ca2b09edaa379] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 17:43:22.248 INFO 2981023 --- [ocp-schedules-10,7a922092d75871ab,c21ca2b09edaa379] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 17:43:22.270 INFO 2981023 --- [pool-551-thread-1,7a922092d75871ab,78352fbe63949873] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:43:22.270 INFO 2981023 --- [pool-551-thread-3,7a922092d75871ab,2217af685e732bf7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:43:22.271 INFO 2981023 --- [pool-551-thread-2,7a922092d75871ab,309d3cb1a7a6ae5c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:43:22.293 INFO 2981023 --- [pool-551-thread-1,7a922092d75871ab,78352fbe63949873] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:22.294 INFO 2981023 --- [pool-551-thread-2,7a922092d75871ab,309d3cb1a7a6ae5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:22.295 INFO 2981023 --- [pool-551-thread-1,7a922092d75871ab,78352fbe63949873] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726047502000000, 1726047802000000] 2024-09-11 17:43:22.296 INFO 2981023 --- [pool-551-thread-3,7a922092d75871ab,2217af685e732bf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:22.297 INFO 2981023 --- [pool-551-thread-2,7a922092d75871ab,309d3cb1a7a6ae5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726047502000000, 1726047802000000] 2024-09-11 17:43:22.299 INFO 2981023 --- [pool-551-thread-3,7a922092d75871ab,2217af685e732bf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726047502000000, 1726047802000000] 2024-09-11 17:43:22.303 INFO 2981023 --- [pool-551-thread-3,7a922092d75871ab,2217af685e732bf7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:43:22.368 INFO 2981023 --- [pool-551-thread-1,7a922092d75871ab,78352fbe63949873] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:43:22.448 INFO 2981023 --- [pool-551-thread-2,7a922092d75871ab,309d3cb1a7a6ae5c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:43:22.448 INFO 2981023 --- [ocp-schedules-10,7a922092d75871ab,c21ca2b09edaa379] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 17:43:23.014 INFO 2981023 --- [ocp-schedules-1,0869f86a9442d73e,bf0ad88c3440b732] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:43:23.014 INFO 2981023 --- [ocp-schedules-1,0869f86a9442d73e,bf0ad88c3440b732] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:43:23.014 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:43:23.014 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:43:23.018 INFO 2981023 --- [ocp-schedules-1,0869f86a9442d73e,bf0ad88c3440b732] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:43:23.018 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:43:23.022 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:43:23.022 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:43:23.022 INFO 2981023 --- [ocp-schedules-1,0869f86a9442d73e,bf0ad88c3440b732] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:43:23.023 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:43:23.044 INFO 2981023 --- [ocp-schedules-1,0869f86a9442d73e,bf0ad88c3440b732] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:43:23.044 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:43:23.046 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:43:23.049 INFO 2981023 --- [ocp-schedules-1,0869f86a9442d73e,bf0ad88c3440b732] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:43:23.265 INFO 2981023 --- [ocp-async-10,c6949be8f1568b0f,e4d6258053dda0e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,48d61f7f746919b3,20246e0efd1705e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:23.318 INFO 2981023 --- [prometheus-minute-schedule-1,48d61f7f746919b3,20246e0efd1705e1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:23.369 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:23.370 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:24.265 INFO 2981023 --- [ocp-schedules-7,8a711d92e5a85d4a,c1824d2c4d4ea821] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:43:24.509 INFO 2981023 --- [ocp-schedules-16,85115f689316773d,3145aca58f7f24ff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:43:24.550 WARN 2981023 --- [scanner-13535671-1,85115f689316773d,294afc14b1c54892] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:43:24.617 INFO 2981023 --- [alarm-detect-8,85115f689316773d,585b2e9b6a542064] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047765 ~ 1726047794 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726047780, value=1.553)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.553, values={1726047780=1.553}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:43:24.618 INFO 2981023 --- [alarm-detect-8,85115f689316773d,585b2e9b6a542064] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Pending 2024-09-11 17:43:25.265 INFO 2981023 --- [ocp-async-5,29bd1c8bc901337a,e29b7e26ac8862a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,8388a3772d69d137,710b89cb4ae26e02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,8388a3772d69d137,710b89cb4ae26e02] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:43:26.352531+08:00, lastProcessTime=2024-09-11T17:43:16.372545+08:00 2024-09-11 17:43:26.370 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:26.371 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:26.475 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:43:26.352531+08:00 2024-09-11 17:43:27.086 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.092 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkr 2024-09-11 17:43:27.099 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.099 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:43:27.104 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.109 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.114 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.114 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.120 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.120 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.121 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.122 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.122 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.123 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.124 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.125 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.125 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.129 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.129 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:43:27.131 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.135 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.146 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.150 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.154 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 35 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.205 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 8 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.220 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 164 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.259 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.265 INFO 2981023 --- [ocp-async-1,f073abd72da88ec5,b51ea1bcbc5fec0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:27.265 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.265 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:43:27.268 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.272 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.276 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:37:26.914, end=2024-09-11T17:42:27.075} 2024-09-11 17:43:27.276 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 2 dubious sqls 2024-09-11 17:43:27.758 INFO 2981023 --- [ocp-async-5,28ac5f2a258caec9,464d3e9e78228fc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:43:29.265 INFO 2981023 --- [ocp-async-10,b6f63a67e46e14b6,5eabd7d10b41805f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,6d28ade8214889bb,65452354595468f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,6d28ade8214889bb,65452354595468f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:29.371 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:29.372 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:31.265 INFO 2981023 --- [ocp-async-7,72e70e74e1d648b9,aac4f3916cb4f4d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:32.095 INFO 2981023 --- [ocp-schedules-17,f53f5dcf1dcccada,7b27cbb319a43182] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:43:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,cee8db767065a03f,527f7f1a79a21334] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,cee8db767065a03f,527f7f1a79a21334] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:32.372 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:32.373 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:33.265 INFO 2981023 --- [ocp-async-1,985cdddf5ae717e1,e70482dbb4b3f96c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:34.207 INFO 2981023 --- [ocp-schedules-22,51617ab1038e3132,ca2a0ca178ce8a40] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:43:34.215 INFO 2981023 --- [ocp-schedules-22,51617ab1038e3132,ca2a0ca178ce8a40] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:43:34.215 INFO 2981023 --- [ocp-schedules-22,51617ab1038e3132,ca2a0ca178ce8a40] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:43:34.700 INFO 2981023 --- [ocp-schedules-20,91677df70e50b38f,2a42ed1d3aa77a2e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:43:34.749 WARN 2981023 --- [scanner-13535884-1,91677df70e50b38f,babdb27789b9941c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:43:34.764 INFO 2981023 --- [alarm-detect-14,91677df70e50b38f,a9a5a1598db7bfc2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644501, data is empty. 2024-09-11 17:43:34.769 INFO 2981023 --- [alarm-detect-14,91677df70e50b38f,a9a5a1598db7bfc2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644501, data is empty. 2024-09-11 17:43:34.778 INFO 2981023 --- [alarm-detect-14,91677df70e50b38f,a9a5a1598db7bfc2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644501, data is empty. 2024-09-11 17:43:34.794 INFO 2981023 --- [alarm-detect-14,91677df70e50b38f,d68574ea6a78d6a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644501, data is empty. 2024-09-11 17:43:34.800 INFO 2981023 --- [alarm-detect-6,91677df70e50b38f,83b5b2e13d677f52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644501, data is empty. 2024-09-11 17:43:34.801 INFO 2981023 --- [alarm-detect-14,91677df70e50b38f,d68574ea6a78d6a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644501, data is empty. 2024-09-11 17:43:34.808 INFO 2981023 --- [alarm-detect-14,91677df70e50b38f,d68574ea6a78d6a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644501, data is empty. 2024-09-11 17:43:34.808 INFO 2981023 --- [alarm-detect-6,91677df70e50b38f,83b5b2e13d677f52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644501, data is empty. 2024-09-11 17:43:34.814 INFO 2981023 --- [alarm-detect-6,91677df70e50b38f,83b5b2e13d677f52] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644501, data is empty. 2024-09-11 17:43:34.912 INFO 2981023 --- [alarm-detect-17,91677df70e50b38f,7f13cc04a7155e65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644501, data is empty. 2024-09-11 17:43:34.916 INFO 2981023 --- [alarm-detect-17,91677df70e50b38f,7f13cc04a7155e65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644501, data is empty. 2024-09-11 17:43:34.919 INFO 2981023 --- [alarm-detect-17,91677df70e50b38f,7f13cc04a7155e65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644501, data is empty. 2024-09-11 17:43:35.265 INFO 2981023 --- [ocp-async-6,cdd81bc5c42427a0,b855ace89603fa62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,8e9a15d376f5500a,48b41bf20a3ff610] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,8e9a15d376f5500a,48b41bf20a3ff610] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:35.373 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:35.373 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:43:36.352285+08:00, lastProcessTime=2024-09-11T17:43:26.352531+08:00 2024-09-11 17:43:36.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:43:36.352285+08:00 2024-09-11 17:43:37.265 INFO 2981023 --- [ocp-async-2,8fdd3b9c5db2085c,10677386e679de04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:37.314 INFO 2981023 --- [prometheus-cache-monitor1,dd130028944ab4bf,ef0379d07df57caa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8021,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:43:37.315 INFO 2981023 --- [prometheus-cache-monitor1,dd130028944ab4bf,ef0379d07df57caa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:43:37.758 INFO 2981023 --- [ocp-async-6,7fe8ae9904d58188,501a8269a1aba3fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:43:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,acbfed699420d420,2290182f7356f105] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:38.318 INFO 2981023 --- [prometheus-minute-schedule-1,acbfed699420d420,2290182f7356f105] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:38.374 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:38.375 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:39.265 INFO 2981023 --- [ocp-async-9,be2bbebb547b0461,3dfa11b73f846b8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:40.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:43:40.003 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:43:40.762 INFO 2981023 --- [ocp-schedules-24,157472b1580e0f4a,8314c4ad214c75db] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:43:40.784 WARN 2981023 --- [ocp-schedules-24,157472b1580e0f4a,8314c4ad214c75db] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:43:41.266 INFO 2981023 --- [ocp-async-8,27e0373dbf562a91,91f829f8daf93c29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,10ef2120d37c8781,961011cf6919789e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,10ef2120d37c8781,961011cf6919789e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:41.375 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:41.376 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:42.098 INFO 2981023 --- [ocp-schedules-30,2554e6450d8e68e6,dc6554aa68c2b524] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:43:43.266 INFO 2981023 --- [ocp-async-5,38d91aeff3c764e8,be4e7fda4c10d731] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,ebdbd972bdb8df85,abaae2e9ac456cf1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,ebdbd972bdb8df85,abaae2e9ac456cf1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:44.376 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:44.376 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:44.987 INFO 2981023 --- [ocp-schedules-20,dcdf9b2f9d2ad1c8,b5d9ac0d4a85be7c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:43:45.077 WARN 2981023 --- [scanner-13536237-1,dcdf9b2f9d2ad1c8,9bff3117e5a07e5e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:43:45.101 INFO 2981023 --- [prometheus-second-schedule-1,ac6f7edb66016b4a,133bae67c6a7dd57] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:43:45.266 INFO 2981023 --- [ocp-async-3,2c79860286344fd0,0feec8bb974e1e21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:46.018 INFO 2981023 --- [ocp-schedules-25,5e397fd18addfa8b,a69cf1e9ef59b29d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:43:46.019 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:43:46.019 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:43:46.019 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:43:46.027 INFO 2981023 --- [ocp-schedules-3,bf3b663317967800,bf495e4ad90955f1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:43:46.027 INFO 2981023 --- [host-check-agent-status1,bf3b663317967800,38ffaf9c963f71e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:43:46.027 INFO 2981023 --- [host-check-agent-status3,bf3b663317967800,5ce7688f21d43e97] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:43:46.028 INFO 2981023 --- [host-check-agent-status2,bf3b663317967800,be7a51f021b2b00f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:43:46.039 INFO 2981023 --- [ocp-schedules-29,562e0a351084b333,02ce6aa543f8b3e9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:43:46.039 INFO 2981023 --- [host-check-clockdiff1,562e0a351084b333,98ca6121d29b7aa7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:43:46.039 INFO 2981023 --- [host-check-clockdiff2,562e0a351084b333,b43fa12527a59cd7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:43:46.039 INFO 2981023 --- [host-check-clockdiff3,562e0a351084b333,db9eecebec0211a4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:43:46.048 INFO 2981023 --- [host-check-clockdiff3,562e0a351084b333,db9eecebec0211a4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047826 0 0 , error= 2024-09-11 17:43:46.049 INFO 2981023 --- [host-check-clockdiff2,562e0a351084b333,b43fa12527a59cd7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047826 0 0 , error= 2024-09-11 17:43:46.059 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:43:46.063 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:43:46.066 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:43:46.068 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:43:46.071 INFO 2981023 --- [host-check-agent-status1,bf3b663317967800,38ffaf9c963f71e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:43:46.071 INFO 2981023 --- [host-check-agent-status1,bf3b663317967800,38ffaf9c963f71e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:43:46.107 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:43:46.107 INFO 2981023 --- [host-check-agent-status3,bf3b663317967800,5ce7688f21d43e97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:43:46.179 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:43:46.183 INFO 2981023 --- [host-check-agent-status3,bf3b663317967800,5ce7688f21d43e97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:43:46.183 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:43:46.187 INFO 2981023 --- [ocp-schedules-23,b466697394dbaf62,3c2f15e6e9bce94b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:43:46.187 INFO 2981023 --- [ocp-schedules-11,90e8002afe83faaa,5615bb9f1e5b6c73] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:43:46.187 INFO 2981023 --- [ocp-schedules-13,74749cc3f555c0b5,70f92090aac8d898] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:43:46.187 INFO 2981023 --- [ocp-schedules-22,2a0330931082e5dc,a3dbc0c35e52dcac] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:43:46.187 INFO 2981023 --- [ocp-schedules-16,f081c5a9b5872f65,7b6413e22b98b82a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:43:46.188 INFO 2981023 --- [host-check-agent-status2,bf3b663317967800,be7a51f021b2b00f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:43:46.191 INFO 2981023 --- [ocp-schedules-11,90e8002afe83faaa,5615bb9f1e5b6c73] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:43:46.192 INFO 2981023 --- [ocp-schedules-13,74749cc3f555c0b5,70f92090aac8d898] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:43:46.194 INFO 2981023 --- [host-check-agent-status2,bf3b663317967800,be7a51f021b2b00f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:43:46.197 INFO 2981023 --- [ocp-schedules-22,2a0330931082e5dc,a3dbc0c35e52dcac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:43:46.197 INFO 2981023 --- [ocp-schedules-13,74749cc3f555c0b5,70f92090aac8d898] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:43:46.204 INFO 2981023 --- [ocp-schedules-23,b466697394dbaf62,3c2f15e6e9bce94b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:43:46.279 INFO 2981023 --- [ocp-schedules-20,a02d74415fbb7de1,f1bb6019aa4ec13f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:43:46.312 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:43:46.365 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:43:46.370 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:43:46.372 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:43:46.370635288+08:00 2024-09-11 17:43:46.372 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:43:46.372052763+08:00 2024-09-11 17:43:46.374 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:43:46.372+08:00 2024-09-11 17:43:46.375 INFO 2981023 --- [host-check-agent-status2,bf3b663317967800,be7a51f021b2b00f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:43:46.375 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:43:46.372+08:00 2024-09-11 17:43:46.376 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:43:46.372+08:00 and status AVAILABLE 2024-09-11 17:43:46.377 INFO 2981023 --- [host-check-agent-status2,bf3b663317967800,be7a51f021b2b00f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:43:46.377 INFO 2981023 --- [host-check-agent-status2,bf3b663317967800,be7a51f021b2b00f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:43:46.377 INFO 2981023 --- [host-check-agent-status2,bf3b663317967800,be7a51f021b2b00f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:43:46.381 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:43:46.372+08:00 and status AVAILABLE 2024-09-11 17:43:46.392 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.393 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:46.394 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:43:46.395 INFO 2981023 --- [host-check-agent-status1,bf3b663317967800,38ffaf9c963f71e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:43:46.395 INFO 2981023 --- [host-check-agent-status3,bf3b663317967800,5ce7688f21d43e97] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:43:46.399 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:43:46.400 INFO 2981023 --- [host-check-agent-status1,bf3b663317967800,38ffaf9c963f71e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:43:46.400 INFO 2981023 --- [host-check-agent-status1,bf3b663317967800,38ffaf9c963f71e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:43:46.400 INFO 2981023 --- [host-check-agent-status1,bf3b663317967800,38ffaf9c963f71e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:43:46.400 INFO 2981023 --- [host-check-agent-status3,bf3b663317967800,5ce7688f21d43e97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:43:46.400 INFO 2981023 --- [host-check-agent-status3,bf3b663317967800,5ce7688f21d43e97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:43:46.400 INFO 2981023 --- [host-check-agent-status3,bf3b663317967800,5ce7688f21d43e97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:43:46.401 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.401 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:46.402 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:43:46.405 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:43:46.404019+08:00, lastProcessTime=2024-09-11T17:43:36.352285+08:00 2024-09-11 17:43:46.406 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.406 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.406 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:46.407 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.407 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:46.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:46.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:43:46.410 INFO 2981023 --- [ocp-schedules-11,90e8002afe83faaa,5615bb9f1e5b6c73] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:43:46.461 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.461 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:43:46.461 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:43:46.461 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:43:46.462 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:43:46.461810032+08:00 2024-09-11 17:43:46.463 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:43:46.463 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:43:46.462+08:00 2024-09-11 17:43:46.463 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.472 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:43:46.462+08:00 and status AVAILABLE 2024-09-11 17:43:46.472 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.473 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:46.475 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:43:46.477 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:43:46.478 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:43:46.480 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:43:46.481 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:43:46.482 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:43:46.501 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:43:46.504 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:43:46.506 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:43:46.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:43:46.522 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:43:46.531 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:43:46.551 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:43:46.556 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.557 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.559 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:43:46.565 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.567 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.567 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:43:46.567 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:43:46.568 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:43:46.568 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.569 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:43:46.582 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.583 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.584 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:46.584 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:43:46.584 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:43:46.585 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.588 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:43:46.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:43:46.593 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:43:46.596 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.597 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.597 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:46.599 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:43:46.603 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:43:46.605 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:43:46.613 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:46.613 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:43:46.613 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:43:46.615 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:43:46.619 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:46.619 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:43:46.640 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:43:46.665 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:43:46.665 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:43:46.666 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.667 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:43:46.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:46.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:46.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:43:46.682 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.683 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:43:46.685 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.685 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:43:46.686 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:43:46.686 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.686 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:43:46.687 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:46.687 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:43:46.687 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.687 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:43:46.689 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:43:46.694 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:43:46.698 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:43:46.707 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.708 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.708 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:46.709 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:43:46.718 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:43:46.721 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:43:46.721 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:43:46.748 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:43:46.759 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:43:46.759 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:43:46.761 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:43:46.762 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:43:46.764 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:43:46.770 INFO 2981023 --- [host-check-host-status1,5e397fd18addfa8b,fb9f5b7e808f3067] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:43:46.800 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.801 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.801 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:43:46.802 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.802 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.802 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:43:46.803 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:43:46.805 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:43:46.805 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.807 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:43:46.822 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:43:46.823 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:43:46.832 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.832 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.833 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:43:46.834 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:43:46.840 INFO 2981023 --- [ocp-schedules-8,df86c005cc1a5534,b5960b9b77a882ca] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8021,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:43:46.840 INFO 2981023 --- [ocp-schedules-8,df86c005cc1a5534,b5960b9b77a882ca] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:43:46.846 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:43:46.848 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:43:46.848 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:43:46.850 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:43:46.857 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.857 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.858 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:43:46.859 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:43:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.861 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:43:46.862 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:43:46.871 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.872 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.872 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:43:46.873 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:43:46.873 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.874 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:46.874 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:43:46.876 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:43:46.877 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:43:46.886 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.887 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.888 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:43:46.893 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:43:46.895 INFO 2981023 --- [host-check-host-status2,5e397fd18addfa8b,59134625ce3ca434] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:43:46.896 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:43:46.896 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:43:46.897 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:43:46.906 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.906 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.907 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:43:46.907 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:43:46.922 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:43:46.927 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:43:46.928 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:43:46.929 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:43:46.932 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:43:46.946 INFO 2981023 --- [host-check-host-status3,5e397fd18addfa8b,f62e096b15c40638] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:43:46.949 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:43:46.951 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:43:46.959 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:43:46.961 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:43:46.404019+08:00 2024-09-11 17:43:46.963 INFO 2981023 --- [ocp-schedules-4,521068f02674c565,edd4fb2ca3396377] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:43:46.978 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:43:46.984 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:46.985 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:46.985 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:43:46.991 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:43:46.991 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:43:46.994 INFO 2981023 --- [ocp-schedules-22,2a0330931082e5dc,a3dbc0c35e52dcac] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:43:47.266 INFO 2981023 --- [ocp-async-10,b636dbf4e4e73979,0176b059fceacb00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,5d5b5f3f334dc324,8bf24bd8215e4998] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,5d5b5f3f334dc324,8bf24bd8215e4998] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:47.376 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:47.377 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:47.758 INFO 2981023 --- [ocp-async-4,c3165fed698faecb,13b76c6f5e53ea30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:43:49.266 INFO 2981023 --- [ocp-async-1,c9204722c6d51328,49995de6e51a796b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:43:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:43:50.245 INFO 2981023 --- [ocp-schedules-11,52152f3598b13395,04774a2c66989043] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:43:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,04c6210d87a58ea4,cddc9c3aee6c338c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,04c6210d87a58ea4,cddc9c3aee6c338c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:50.377 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:50.378 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:50.609 INFO 2981023 --- [ocp-schedules-19,e72fba27a4d8cfd1,91678bea1da564e7] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:43:51.266 INFO 2981023 --- [ocp-async-6,441d3c8d90303779,be2ace87f6571fe0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:51.301 INFO 2981023 --- [host-check-clockdiff1,562e0a351084b333,98ca6121d29b7aa7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047831 0 0 , error= 2024-09-11 17:43:51.555 INFO 2981023 --- [ocp-schedules-17,99d83ae66177358c,adcffa796197bfb2] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:43:51.556 INFO 2981023 --- [ocp-schedules-17,99d83ae66177358c,adcffa796197bfb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:43:51.557 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:43:51.557 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:43:51.557 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:43:51.557 INFO 2981023 --- [ocp-schedules-17,99d83ae66177358c,adcffa796197bfb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:43:52.101 INFO 2981023 --- [ocp-schedules-9,12f3d37a8bb23d3e,f93dcb22ab19772c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:43:53.106 INFO 2981023 --- [ocp-schedules-1,9fbcfdae35d0919d,c8dad946482b00bf] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:43:53.124 INFO 2981023 --- [ocp-schedules-6,ac11e9f93df79140,76ba1d72633b1953] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:43:53.124 INFO 2981023 --- [ocp-schedules-6,ac11e9f93df79140,76ba1d72633b1953] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:43:53.125 WARN 2981023 --- [ocp-schedules-1,9fbcfdae35d0919d,c8dad946482b00bf] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:43:53.266 INFO 2981023 --- [ocp-async-9,c3f91ccfe96bf9ab,42cc97ae9ff74bd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,899bd167006f9b3c,24ce3f7fb74d92ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,899bd167006f9b3c,24ce3f7fb74d92ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:53.378 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:53.379 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:53.747 INFO 2981023 --- [ocp-schedules-14,066711bff15f7d80,a5aeb6fd439a5078] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:43:53.747+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:43:53.751 INFO 2981023 --- [ocp-schedules-14,066711bff15f7d80,a5aeb6fd439a5078] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:43:53.906 INFO 2981023 --- [ocp-schedules-14,066711bff15f7d80,a5aeb6fd439a5078] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 56 to SqlAggregateData 2024-09-11 17:43:53.907 INFO 2981023 --- [ocp-schedules-14,066711bff15f7d80,a5aeb6fd439a5078] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:43:54.218 INFO 2981023 --- [ocp-schedules-7,2089326ae057bb5d,af66b09fecc7a4b5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:43:54.226 INFO 2981023 --- [ocp-schedules-7,2089326ae057bb5d,af66b09fecc7a4b5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:43:54.226 INFO 2981023 --- [ocp-schedules-7,2089326ae057bb5d,af66b09fecc7a4b5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:43:54.268 INFO 2981023 --- [ocp-schedules-29,edbc682a0e6de354,27ab2048d42d3cb0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:43:54.467 INFO 2981023 --- [ocp-schedules-23,97db9fca4b415dea,7a52435835c944ba] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:43:54.468 INFO 2981023 --- [ocp-schedules-23,97db9fca4b415dea,7a52435835c944ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:43:54.506 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:43:54.506 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.507 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:43:54.507 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.508 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:43:54.508 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.510 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:43:54.511 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:43:54.511 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:43:54.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.515 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.515 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.520 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.520 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.526 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.526 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.527 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.527 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:43:54.528 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.528 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.535 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.535 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:43:54.537 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.538 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.541 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.541 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:43:54.541 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3101D6DA-0-0] 2024-09-11 17:43:54.541 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:43:54.542 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.542 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.542 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:43:54.542 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:43:54.545 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.546 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:43:54.547 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.547 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:43:54.548 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.548 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:43:54.549 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:43:54.553 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.553 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:43:54.553 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:43:54.556 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.556 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:43:54.568 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:54.570 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.570 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.571 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.571 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:43:54.572 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:43:54.572 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3101D6E4-0-0] 2024-09-11 17:43:54.572 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.572 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:43:54.573 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:43:54.574 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.575 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:43:54.576 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.577 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:43:54.578 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.578 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:43:54.579 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.579 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.580 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:43:54.580 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:43:54.580 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.580 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.581 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:43:54.581 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:43:54.581 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.581 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:43:54.582 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:43:54.583 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.583 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.583 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:43:54.583 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:43:54.584 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.584 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.584 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:43:54.584 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:43:54.585 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:43:54.585 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:43:54.585 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.586 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:43:54.587 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:43:54.587 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:43:54.588 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:43:54.588 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:43:54.588 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:43:54.588 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:43:54.600 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:54.601 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:43:54.601 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.602 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:43:54.602 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.602 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:43:54.608 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:43:54.610 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:43:54.610 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.611 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:43:54.612 INFO 2981023 --- [ocp-schedules-1,168eb14ec291f937,c87c0a6771b4673b] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T17:43:54.612+08:00, tenantSize=6 2024-09-11 17:43:54.613 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:43:54.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:43:54.618 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:43:54.618 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.618 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.619 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:43:54.619 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:54.620 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:43:54.626 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:43:54.627 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:43:54.627 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:43:54.627 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:43:54.630 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:43:54.632 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:43:54.632 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:43:54.632 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:43:54.632 INFO 2981023 --- [ocp-schedules-1,168eb14ec291f937,c87c0a6771b4673b] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726047834612000 2024-09-11 17:43:54.632 INFO 2981023 --- [ocp-schedules-1,168eb14ec291f937,c87c0a6771b4673b] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726047534586000 2024-09-11 17:43:54.632 INFO 2981023 --- [ocp-schedules-23,97db9fca4b415dea,7a52435835c944ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:43:54.653 INFO 2981023 --- [ocp-schedules-1,168eb14ec291f937,c87c0a6771b4673b] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 17:43:55.193 INFO 2981023 --- [ocp-schedules-16,c88f8ec7e405bf4b,166d090f8320c4f1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:43:55.243 WARN 2981023 --- [scanner-13536453-1,c88f8ec7e405bf4b,c9f6d3c1fa956a77] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:43:55.267 INFO 2981023 --- [ocp-async-5,4aeb31e83b59ff0b,af3c2f5262412bcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:55.271 INFO 2981023 --- [alarm-detect-8,c88f8ec7e405bf4b,4642fcf6a6238363] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644501, data is empty. 2024-09-11 17:43:55.277 INFO 2981023 --- [alarm-detect-8,c88f8ec7e405bf4b,4642fcf6a6238363] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644557, data is empty. 2024-09-11 17:43:55.282 INFO 2981023 --- [alarm-detect-8,c88f8ec7e405bf4b,4642fcf6a6238363] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644557, data is empty. 2024-09-11 17:43:55.296 INFO 2981023 --- [alarm-detect-14,c88f8ec7e405bf4b,3f0dbfae2004b284] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644501, data is empty. 2024-09-11 17:43:55.300 INFO 2981023 --- [alarm-detect-10,c88f8ec7e405bf4b,d1be810ee62c3b53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644501, data is empty. 2024-09-11 17:43:55.302 INFO 2981023 --- [alarm-detect-14,c88f8ec7e405bf4b,3f0dbfae2004b284] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644557, data is empty. 2024-09-11 17:43:55.304 INFO 2981023 --- [alarm-detect-10,c88f8ec7e405bf4b,d1be810ee62c3b53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644557, data is empty. 2024-09-11 17:43:55.306 INFO 2981023 --- [alarm-detect-14,c88f8ec7e405bf4b,3f0dbfae2004b284] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644557, data is empty. 2024-09-11 17:43:55.308 INFO 2981023 --- [alarm-detect-10,c88f8ec7e405bf4b,d1be810ee62c3b53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644557, data is empty. 2024-09-11 17:43:55.414 INFO 2981023 --- [alarm-detect-16,c88f8ec7e405bf4b,d20e0e069ca901bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644501, data is empty. 2024-09-11 17:43:55.420 INFO 2981023 --- [alarm-detect-16,c88f8ec7e405bf4b,d20e0e069ca901bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644557, data is empty. 2024-09-11 17:43:55.434 INFO 2981023 --- [alarm-detect-16,c88f8ec7e405bf4b,d20e0e069ca901bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644557, data is empty. 2024-09-11 17:43:55.714 INFO 2981023 --- [ocp-schedules-7,dcd6d1ea163a0d69,581962c69216933d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:43:55.714 INFO 2981023 --- [ocp-schedules-7,dcd6d1ea163a0d69,581962c69216933d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:43:55.715 INFO 2981023 --- [ocp-schedules-7,dcd6d1ea163a0d69,581962c69216933d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:43:55.715 INFO 2981023 --- [ocp-schedules-7,dcd6d1ea163a0d69,581962c69216933d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:43:55.716 INFO 2981023 --- [ocp-schedules-7,dcd6d1ea163a0d69,581962c69216933d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:43:55.716 INFO 2981023 --- [ocp-schedules-7,dcd6d1ea163a0d69,581962c69216933d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:43:55.817 INFO 2981023 --- [ocp-schedules-29,ebd11a89c5515d4d,621f3b2d071da93d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:43:55.824 INFO 2981023 --- [ocp-schedules-29,ebd11a89c5515d4d,621f3b2d071da93d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:43:55.824 INFO 2981023 --- [ocp-schedules-29,ebd11a89c5515d4d,621f3b2d071da93d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:43:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,804990f8c92f45e1,2fcc8fe4276ed718] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,804990f8c92f45e1,2fcc8fe4276ed718] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:43:56.352897+08:00, lastProcessTime=2024-09-11T17:43:46.404019+08:00 2024-09-11 17:43:56.379 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:56.380 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:43:56.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:43:56.352897+08:00 2024-09-11 17:43:56.511 INFO 2981023 --- [ocp-schedules-27,e29a60dc0d71bef2,2b5216eb751ebb36] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:43:56.511 INFO 2981023 --- [ocp-schedules-27,e29a60dc0d71bef2,2b5216eb751ebb36] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:43:56.523 INFO 2981023 --- [Collect all tenants major compactions1,e29a60dc0d71bef2,584bc364494ff258] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:43:56.523 INFO 2981023 --- [Collect all tenants major compactions2,e29a60dc0d71bef2,64db24d2fb4e8d4f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:43:56.524 INFO 2981023 --- [Collect all tenants major compactions3,e29a60dc0d71bef2,377ce312fe3b20da] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:43:56.524 INFO 2981023 --- [Collect all tenants major compactions4,e29a60dc0d71bef2,5a141dc5559425b5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:43:56.525 INFO 2981023 --- [Collect all tenants major compactions5,e29a60dc0d71bef2,5806b525c2624062] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:43:56.525 INFO 2981023 --- [Collect all tenants major compactions6,e29a60dc0d71bef2,e7b3b1379d2769d5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:43:56.535 INFO 2981023 --- [Collect all tenants major compactions6,e29a60dc0d71bef2,e7b3b1379d2769d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:56.535 INFO 2981023 --- [Collect all tenants major compactions1,e29a60dc0d71bef2,584bc364494ff258] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:56.535 INFO 2981023 --- [Collect all tenants major compactions2,e29a60dc0d71bef2,64db24d2fb4e8d4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:56.536 INFO 2981023 --- [Collect all tenants major compactions6,e29a60dc0d71bef2,e7b3b1379d2769d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:43:56.536 INFO 2981023 --- [Collect all tenants major compactions1,e29a60dc0d71bef2,584bc364494ff258] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:43:56.537 INFO 2981023 --- [Collect all tenants major compactions3,e29a60dc0d71bef2,377ce312fe3b20da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:56.537 INFO 2981023 --- [Collect all tenants major compactions2,e29a60dc0d71bef2,64db24d2fb4e8d4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:43:56.538 INFO 2981023 --- [Collect all tenants major compactions4,e29a60dc0d71bef2,5a141dc5559425b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:56.539 INFO 2981023 --- [Collect all tenants major compactions3,e29a60dc0d71bef2,377ce312fe3b20da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:43:56.539 INFO 2981023 --- [Collect all tenants major compactions4,e29a60dc0d71bef2,5a141dc5559425b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:43:56.539 INFO 2981023 --- [Collect all tenants major compactions1,e29a60dc0d71bef2,584bc364494ff258] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:43:56.539 INFO 2981023 --- [Collect all tenants major compactions1,e29a60dc0d71bef2,584bc364494ff258] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:43:56.540 INFO 2981023 --- [Collect all tenants major compactions2,e29a60dc0d71bef2,64db24d2fb4e8d4f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:43:56.540 INFO 2981023 --- [Collect all tenants major compactions2,e29a60dc0d71bef2,64db24d2fb4e8d4f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:43:56.540 INFO 2981023 --- [Collect all tenants major compactions6,e29a60dc0d71bef2,e7b3b1379d2769d5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:43:56.540 INFO 2981023 --- [Collect all tenants major compactions6,e29a60dc0d71bef2,e7b3b1379d2769d5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:43:56.540 INFO 2981023 --- [Collect all tenants major compactions5,e29a60dc0d71bef2,5806b525c2624062] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:43:56.541 INFO 2981023 --- [Collect all tenants major compactions3,e29a60dc0d71bef2,377ce312fe3b20da] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:43:56.541 INFO 2981023 --- [Collect all tenants major compactions3,e29a60dc0d71bef2,377ce312fe3b20da] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:43:56.541 INFO 2981023 --- [Collect all tenants major compactions4,e29a60dc0d71bef2,5a141dc5559425b5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:43:56.541 INFO 2981023 --- [Collect all tenants major compactions4,e29a60dc0d71bef2,5a141dc5559425b5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:43:56.542 INFO 2981023 --- [Collect all tenants major compactions5,e29a60dc0d71bef2,5806b525c2624062] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:43:56.543 INFO 2981023 --- [Collect all tenants major compactions5,e29a60dc0d71bef2,5806b525c2624062] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:43:56.544 INFO 2981023 --- [Collect all tenants major compactions5,e29a60dc0d71bef2,5806b525c2624062] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:43:56.544 INFO 2981023 --- [ocp-schedules-27,e29a60dc0d71bef2,2b5216eb751ebb36] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:43:57.267 INFO 2981023 --- [ocp-async-6,4ee282c634c37866,c413441bcb1ad13b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:57.759 INFO 2981023 --- [ocp-async-1,7e83b5253a52f883,6f7c4404d2df47ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:43:59.267 INFO 2981023 --- [ocp-async-9,7621e3d81a3e0244,e6afc346ce7b41bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:43:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,a5c17ddc91b1988c,10b2de3eac2484f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047840, interval=60 2024-09-11 17:43:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,a5c17ddc91b1988c,10b2de3eac2484f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:43:59.380 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:43:59.381 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:01.270 INFO 2981023 --- [ocp-async-5,4c84ab050bd96f02,b4b816a8026116bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:02.103 INFO 2981023 --- [ocp-schedules-10,fdaf4eca900c6ac6,4fca17e5f885da93] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:44:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,8d62a75214287790,66556c51b096ad01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,8d62a75214287790,66556c51b096ad01] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:02.382 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:02.382 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:02.486 INFO 2981023 --- [ocp-schedules-19,2dc5aec555cb7455,f548642adc6a31db] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:44:02.487 INFO 2981023 --- [ocp-schedules-19,2dc5aec555cb7455,f548642adc6a31db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:44:02.502 INFO 2981023 --- [ocp-schedules-19,2dc5aec555cb7455,f548642adc6a31db] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:44:03.267 INFO 2981023 --- [ocp-async-4,12feaad61c4a9484,142e8b8de9babe13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:05.267 INFO 2981023 --- [ocp-async-10,7c07351a6967e61b,830ffc62a29f232e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,cc40dcc7a7a25a13,1b331c92e5c55b1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,cc40dcc7a7a25a13,1b331c92e5c55b1d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:05.382 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:05.384 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:05.470 INFO 2981023 --- [ocp-schedules-18,ca0be0f9e70ae364,3ee8f2dac8358547] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:44:05.494 WARN 2981023 --- [scanner-13536751-1,ca0be0f9e70ae364,9e9c6e55d3adbcb2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:44:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:44:05.971 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:05.971 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:44:05.971 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:44:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:44:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:44:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:44:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:44:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:44:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:44:06.008 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:44:06.009 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:44:06.010 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:06.010 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:06.010 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:06.010 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:06.077 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:41:40, start=2024-09-11T17:37:40, end=2024-09-11T17:40:10, updates=108, granule=120000ms, delay=240000ms 2024-09-11 17:44:06.087 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:41:41, start=2024-09-11T17:37:41, end=2024-09-11T17:40:11, updates=314, granule=120000ms, delay=240000ms 2024-09-11 17:44:06.104 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:41:41, target=2024-09-11T17:40:11, granule=120000ms, delay=90000ms) 2024-09-11 17:44:06.104 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:40:11, target=2024-09-11T17:30:51, granule=600000ms, delay=560000ms) 2024-09-11 17:44:06.106 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:41:40, target=2024-09-11T17:40:10, granule=120000ms, delay=90000ms) 2024-09-11 17:44:06.106 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:40:10, target=2024-09-11T17:29:31, granule=600000ms, delay=639000ms) 2024-09-11 17:44:06.106 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=135ms, period=120000ms 2024-09-11 17:44:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:44:06.352399+08:00, lastProcessTime=2024-09-11T17:43:56.352897+08:00 2024-09-11 17:44:06.502 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:44:06.352399+08:00 2024-09-11 17:44:07.267 INFO 2981023 --- [ocp-async-9,6bb0d6a0b8c36c0a,08288c63cd0d4123] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:07.452 INFO 2981023 --- [ocp-schedules-13,ff7207f86090455e,b042cfe567cec132] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:44:07.470 INFO 2981023 --- [ocp-schedules-2,115460c9e711992f,3e1108f8d6accf9c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:44:07.470+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:44:07.471 INFO 2981023 --- [ocp-schedules-2,115460c9e711992f,3e1108f8d6accf9c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:44:07.473 INFO 2981023 --- [ocp-schedules-2,115460c9e711992f,3e1108f8d6accf9c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:44:07.473 INFO 2981023 --- [ocp-schedules-2,115460c9e711992f,3e1108f8d6accf9c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:44:07.759 INFO 2981023 --- [ocp-async-2,a854c3c5c72f2725,8f8ed88e9327a6fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:44:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,f753f02d767797c3,bc17e201eef4e7a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,f753f02d767797c3,bc17e201eef4e7a9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:08.384 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:08.384 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:09.267 INFO 2981023 --- [ocp-async-6,a3b4335d2aeff24d,2fb9ac26801bdc05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:10.003 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:44:10.003 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:44:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:44:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:44:11.268 INFO 2981023 --- [ocp-async-10,e5b583b0e4ddabda,32b86e5e99c202ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,4061dfc16c6fb722,2e5ca40398c3c152] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,4061dfc16c6fb722,2e5ca40398c3c152] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:11.385 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:11.385 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:12.106 INFO 2981023 --- [ocp-schedules-25,8f42177db8c8c9d1,1804f86057247ee7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:44:13.268 INFO 2981023 --- [ocp-async-1,53a382218cba1dfa,fd5a69adf5b072a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:13.405 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047780, nodeCount=892, affectRows=892 2024-09-11 17:44:13.407 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047780, nodeCount=914, affectRows=914 2024-09-11 17:44:13.408 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047780, nodeCount=891, validCount=891 2024-09-11 17:44:13.410 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047780, nodeCount=914, validCount=914 2024-09-11 17:44:14.076 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047780, nodeCount=463, affectRows=463 2024-09-11 17:44:14.230 INFO 2981023 --- [ocp-schedules-17,a5494195876e1a23,a171704bb6a27448] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:44:14.243 INFO 2981023 --- [ocp-schedules-17,a5494195876e1a23,a171704bb6a27448] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:44:14.243 INFO 2981023 --- [ocp-schedules-17,a5494195876e1a23,a171704bb6a27448] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:44:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,3d91115a3cc3ceef,5c46fd8f31520c50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,cb4e242e15499c7e,803e60d66d910d02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:44:14.318 INFO 2981023 --- [prometheus-minute-schedule-1,3d91115a3cc3ceef,5c46fd8f31520c50] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:14.386 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:14.390 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:15.268 INFO 2981023 --- [ocp-async-5,c966aacae83dfc40,6fd00d39b39ade16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:15.664 INFO 2981023 --- [ocp-schedules-3,1a94920ff0265167,cf9e8a6c9701bb9d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:44:15.720 WARN 2981023 --- [scanner-13538473-1,1a94920ff0265167,2081d08d523dc08b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:44:15.737 INFO 2981023 --- [alarm-detect-12,1a94920ff0265167,4fdacb1f09ad2822] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644557, data is empty. 2024-09-11 17:44:15.742 INFO 2981023 --- [alarm-detect-12,1a94920ff0265167,4fdacb1f09ad2822] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644557, data is empty. 2024-09-11 17:44:15.746 INFO 2981023 --- [alarm-detect-12,1a94920ff0265167,4fdacb1f09ad2822] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644557, data is empty. 2024-09-11 17:44:15.757 INFO 2981023 --- [alarm-detect-12,1a94920ff0265167,81ae3c6529309e7b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644557, data is empty. 2024-09-11 17:44:15.760 INFO 2981023 --- [alarm-detect-6,1a94920ff0265167,7d2d18454d846cd3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644557, data is empty. 2024-09-11 17:44:15.763 INFO 2981023 --- [alarm-detect-12,1a94920ff0265167,81ae3c6529309e7b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644557, data is empty. 2024-09-11 17:44:15.768 INFO 2981023 --- [alarm-detect-6,1a94920ff0265167,7d2d18454d846cd3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644557, data is empty. 2024-09-11 17:44:15.772 INFO 2981023 --- [alarm-detect-12,1a94920ff0265167,81ae3c6529309e7b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644557, data is empty. 2024-09-11 17:44:15.775 INFO 2981023 --- [alarm-detect-6,1a94920ff0265167,7d2d18454d846cd3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644557, data is empty. 2024-09-11 17:44:15.797 INFO 2981023 --- [alarm-detect-8,1a94920ff0265167,9c7a580c2dd5cfa9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047825 ~ 1726047845 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726047840, value=2.091)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.091, values={1726047840=2.091}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:44:15.812 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047780, nodeCount=463, validCount=463 2024-09-11 17:44:15.932 INFO 2981023 --- [alarm-detect-1,1a94920ff0265167,8757aded2931f3a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644557, data is empty. 2024-09-11 17:44:15.939 INFO 2981023 --- [alarm-detect-1,1a94920ff0265167,8757aded2931f3a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644557, data is empty. 2024-09-11 17:44:15.944 INFO 2981023 --- [alarm-detect-1,1a94920ff0265167,8757aded2931f3a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644557, data is empty. 2024-09-11 17:44:16.017 INFO 2981023 --- [ocp-schedules-18,c8128ac396eb411b,674d5e2a804347cb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:44:16.017 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:44:16.018 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:44:16.018 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:44:16.019 INFO 2981023 --- [ocp-schedules-15,128e916f1d2f5935,81674e595a26b6e9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:44:16.019 INFO 2981023 --- [host-check-clockdiff1,128e916f1d2f5935,f4b0097964ce5125] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:44:16.021 INFO 2981023 --- [host-check-clockdiff2,128e916f1d2f5935,8958f3206f6d6e6c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:44:16.021 INFO 2981023 --- [host-check-clockdiff3,128e916f1d2f5935,4ddaaebde75de284] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:44:16.039 INFO 2981023 --- [host-check-clockdiff3,128e916f1d2f5935,4ddaaebde75de284] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047856 0 0 , error= 2024-09-11 17:44:16.060 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:16.061 INFO 2981023 --- [ocp-schedules-4,db8e2d882da81fc6,9b5285da6492258f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:44:16.061 INFO 2981023 --- [host-check-agent-status1,db8e2d882da81fc6,d5418d002b94e607] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:44:16.061 INFO 2981023 --- [host-check-agent-status2,db8e2d882da81fc6,d10421a0269e1f14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:44:16.062 INFO 2981023 --- [host-check-agent-status3,db8e2d882da81fc6,948f35be4dfc142d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:44:16.064 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:16.064 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:16.065 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:16.094 INFO 2981023 --- [host-check-agent-status3,db8e2d882da81fc6,948f35be4dfc142d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:16.116 INFO 2981023 --- [host-check-agent-status3,db8e2d882da81fc6,948f35be4dfc142d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:16.186 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:16.194 INFO 2981023 --- [host-check-agent-status2,db8e2d882da81fc6,d10421a0269e1f14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:16.195 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:16.195 INFO 2981023 --- [ocp-schedules-22,a153d2d5e5660120,f6c140a79d9839f2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:44:16.196 INFO 2981023 --- [ocp-schedules-30,ab9c2cdccbf83729,3ea091cacd4baea7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:44:16.196 INFO 2981023 --- [ocp-schedules-6,2fa21ccf2fa18b5b,03fabacd7ee9c004] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:44:16.196 INFO 2981023 --- [host-check-agent-status1,db8e2d882da81fc6,d5418d002b94e607] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:16.196 INFO 2981023 --- [ocp-schedules-20,627521daf033a0df,2ce962b7d71bfa52] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:44:16.200 INFO 2981023 --- [host-check-agent-status2,db8e2d882da81fc6,d10421a0269e1f14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:16.202 INFO 2981023 --- [host-check-agent-status1,db8e2d882da81fc6,d5418d002b94e607] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:16.205 INFO 2981023 --- [ocp-schedules-6,2fa21ccf2fa18b5b,03fabacd7ee9c004] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:44:16.205 INFO 2981023 --- [ocp-schedules-22,a153d2d5e5660120,f6c140a79d9839f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:44:16.206 INFO 2981023 --- [ocp-schedules-30,ab9c2cdccbf83729,3ea091cacd4baea7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:44:16.220 INFO 2981023 --- [ocp-schedules-14,56eb3d4cd63f8582,fbbb4ce6b127f087] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:44:16.253 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:44:16.257 INFO 2981023 --- [ocp-schedules-30,ab9c2cdccbf83729,3ea091cacd4baea7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:44:16.259 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:44:16.255068403+08:00 2024-09-11 17:44:16.267 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:44:16.259+08:00 2024-09-11 17:44:16.270 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:44:16.273 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:44:16.259+08:00 and status AVAILABLE 2024-09-11 17:44:16.273 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:44:16.273375811+08:00 2024-09-11 17:44:16.279 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:44:16.273+08:00 2024-09-11 17:44:16.292 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:44:16.273+08:00 and status AVAILABLE 2024-09-11 17:44:16.301 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:44:16.305 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:44:16.306 INFO 2981023 --- [host-check-agent-status1,db8e2d882da81fc6,d5418d002b94e607] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:44:16.307 INFO 2981023 --- [host-check-agent-status2,db8e2d882da81fc6,d10421a0269e1f14] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:44:16.307 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:44:16.307657929+08:00 2024-09-11 17:44:16.309 INFO 2981023 --- [host-check-agent-status1,db8e2d882da81fc6,d5418d002b94e607] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:44:16.309 INFO 2981023 --- [host-check-agent-status1,db8e2d882da81fc6,d5418d002b94e607] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:44:16.309 INFO 2981023 --- [host-check-agent-status1,db8e2d882da81fc6,d5418d002b94e607] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:44:16.311 INFO 2981023 --- [host-check-agent-status2,db8e2d882da81fc6,d10421a0269e1f14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:44:16.311 INFO 2981023 --- [host-check-agent-status2,db8e2d882da81fc6,d10421a0269e1f14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:44:16.311 INFO 2981023 --- [host-check-agent-status2,db8e2d882da81fc6,d10421a0269e1f14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:44:16.311 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:44:16.307+08:00 2024-09-11 17:44:16.318 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:44:16.307+08:00 and status AVAILABLE 2024-09-11 17:44:16.324 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:44:16.327 INFO 2981023 --- [host-check-agent-status3,db8e2d882da81fc6,948f35be4dfc142d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:44:16.330 INFO 2981023 --- [host-check-agent-status3,db8e2d882da81fc6,948f35be4dfc142d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:44:16.330 INFO 2981023 --- [host-check-agent-status3,db8e2d882da81fc6,948f35be4dfc142d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:44:16.330 INFO 2981023 --- [host-check-agent-status3,db8e2d882da81fc6,948f35be4dfc142d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:44:16.337 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:16.340 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:16.341 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:44:16.344 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.344 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:16.345 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:44:16.350 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:16.356 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.357 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:16.358 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:44:16.360 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:16.361 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:16.363 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:16.367 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.368 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:16.374 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:44:16.373403+08:00, lastProcessTime=2024-09-11T17:44:06.352399+08:00 2024-09-11 17:44:16.380 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.381 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:16.381 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:44:16.382 INFO 2981023 --- [ocp-schedules-22,a153d2d5e5660120,f6c140a79d9839f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:44:16.389 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.390 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:16.390 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:44:16.395 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.395 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:16.399 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:44:16.402 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:44:16.421 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:44:16.423 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:44:16.432 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:44:16.438 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:44:16.442 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:44:16.443 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:44:16.459 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.461 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.461 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:44:16.465 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.466 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.466 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:44:16.479 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:44:16.484 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:44:16.496 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:44:16.505 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.505 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.505 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:16.506 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:44:16.506 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:44:16.506 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.509 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:44:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:44:16.515 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:44:16.517 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:44:16.526 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:44:16.526 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:16.526 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:44:16.528 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.528 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.529 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:16.530 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.531 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:44:16.533 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:44:16.533 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:44:16.555 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:44:16.557 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:16.557 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:44:16.558 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:44:16.558 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:44:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:16.569 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:44:16.575 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:44:16.576 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.576 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.577 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:16.578 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:44:16.582 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:44:16.584 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:44:16.584 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.585 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.585 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:16.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:16.589 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.592 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:44:16.592 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.592 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.593 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:16.593 INFO 2981023 --- [host-check-host-status1,c8128ac396eb411b,d0cd2cad87cb4f3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:44:16.595 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:44:16.596 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:44:16.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:44:16.598 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:44:16.603 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:44:16.603 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:44:16.606 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:16.606 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:44:16.608 INFO 2981023 --- [host-check-host-status3,c8128ac396eb411b,e474b451d81b9829] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:44:16.610 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:16.611 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:16.613 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:44:16.614 INFO 2981023 --- [host-check-host-status2,c8128ac396eb411b,d7d35dce6378326b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:44:16.624 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:16.632 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.633 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.634 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:16.634 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:44:16.639 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:44:16.640 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:44:16.644 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:16.645 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:16.667 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:44:16.686 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:44:16.686 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:44:16.373403+08:00 2024-09-11 17:44:16.691 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.692 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.692 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:44:16.695 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:44:16.697 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:44:16.697 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:44:16.700 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.701 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.701 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:44:16.702 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:44:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:44:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:44:16.707 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:16.708 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:16.708 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:44:16.713 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:44:16.713 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:44:16.713 INFO 2981023 --- [ocp-schedules-6,2fa21ccf2fa18b5b,03fabacd7ee9c004] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:44:16.970 INFO 2981023 --- [host-check-clockdiff2,128e916f1d2f5935,8958f3206f6d6e6c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047856 0 0 , error= 2024-09-11 17:44:16.973 INFO 2981023 --- [host-check-clockdiff1,128e916f1d2f5935,f4b0097964ce5125] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047856 0 0 , error= 2024-09-11 17:44:17.268 INFO 2981023 --- [ocp-async-2,20ee59c125fbaf97,1e98ba6346088a5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,e28f7db76cc527ca,3927215bfb938ff3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,e28f7db76cc527ca,3927215bfb938ff3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:17.390 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:17.390 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:17.759 INFO 2981023 --- [ocp-async-3,44d56d5adfd8ca28,5549def1ce47058a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:44:19.268 INFO 2981023 --- [ocp-async-9,6800a6d6d47584da,92240788f0e86b49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:19.527 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:44:19.527 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000007 2024-09-11 17:44:19.528 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000009 2024-09-11 17:44:19.528 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2024-09-11 17:44:19.581 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:44:19.581 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:44:19.588 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:44:19.607 WARN 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:44:19.607 WARN 2981023 --- [ForkJoinPool.commonPool-worker-3,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:44:19.610 WARN 2981023 --- [ForkJoinPool.commonPool-worker-1,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:44:19.626 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:44:19.626 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:44:19.626 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:44:19.626 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000009 agent sql collect property 2024-09-11 17:44:19.627 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:44:19.627 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:44:19.627 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:44:19.627 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2024-09-11 17:44:19.628 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:44:19.628 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:44:19.629 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:44:19.629 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000007 agent sql collect property 2024-09-11 17:44:19.637 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2000009 to set sql collect property inactive 2024-09-11 17:44:19.639 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:19.639 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:19.641 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:19.643 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:19.643 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:19.645 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:19.671 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.6), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4 2024-09-11 17:44:19.672 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994437), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.10), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkprdocp), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"include 2024-09-11 17:44:19.675 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1718689631), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.5), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=rtdob), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"in 2024-09-11 17:44:19.686 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000007, do nothing 2024-09-11 17:44:19.691 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2024-09-11 17:44:19.694 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000009, do nothing 2024-09-11 17:44:19.695 INFO 2981023 --- [ocp-schedules-6,3c752edcbb435ac1,492395673cd99b53] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-09-11 17:44:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,558653ef61b6507a,d4b3eaa0308cab75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,558653ef61b6507a,d4b3eaa0308cab75] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:20.390 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:20.392 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:21.268 INFO 2981023 --- [ocp-async-4,803958d36952cf6d,85356eef82815e98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:22.109 INFO 2981023 --- [ocp-schedules-12,df916aec9378ff4a,9ed6a3201c9ca7e1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:44:23.053 INFO 2981023 --- [ocp-schedules-18,55d02983fc449091,a51c172e830082dc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:44:23.053 INFO 2981023 --- [ocp-schedules-18,55d02983fc449091,a51c172e830082dc] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:44:23.053 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:44:23.053 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:44:23.057 INFO 2981023 --- [ocp-schedules-18,55d02983fc449091,a51c172e830082dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:23.057 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:23.057 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:23.058 INFO 2981023 --- [ocp-schedules-18,55d02983fc449091,a51c172e830082dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:23.058 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:23.059 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:23.076 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:44:23.076 INFO 2981023 --- [ocp-schedules-18,55d02983fc449091,a51c172e830082dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:44:23.076 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:44:23.080 INFO 2981023 --- [ocp-schedules-18,55d02983fc449091,a51c172e830082dc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:44:23.268 INFO 2981023 --- [ocp-async-2,24a82f3579fb512c,e67b560fcaf9abde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,2457cdb0f9c7ae4b,23194c7f27d43ad8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,2457cdb0f9c7ae4b,23194c7f27d43ad8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:23.392 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:23.394 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:24.271 INFO 2981023 --- [ocp-schedules-25,57c1ad40b70bd7b7,db0063f111c8b34d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:44:25.268 INFO 2981023 --- [ocp-async-10,0bb9baf4ea52a794,bef570fa672d95ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:25.976 INFO 2981023 --- [ocp-schedules-20,b1698c3f680be326,aad63acb5dfebac4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:44:26.031 WARN 2981023 --- [scanner-13538767-1,b1698c3f680be326,adc169d493122535] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:44:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,65550300478dc13f,04b7260a4f71b5cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:26.315 INFO 2981023 --- [prometheus-minute-schedule-1,65550300478dc13f,04b7260a4f71b5cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:26.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:44:26.351280+08:00, lastProcessTime=2024-09-11T17:44:16.373403+08:00 2024-09-11 17:44:26.394 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:26.394 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:26.477 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:44:26.351280+08:00 2024-09-11 17:44:27.118 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 17:44:27.118 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 17:44:27.118 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.119 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:44:27.119 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.120 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.120 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.120 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 17:44:27.132 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.132 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.133 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.133 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 17:44:27.142 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.142 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.142 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.143 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 17:44:27.152 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.152 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.152 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.153 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 17:44:27.162 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.162 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.162 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.163 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 17:44:27.166 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.167 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.167 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.167 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 17:44:27.170 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.170 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.171 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.171 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 17:44:27.174 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.174 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.174 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.175 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 17:44:27.178 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.179 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.179 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.179 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 17:44:27.183 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.183 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.184 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.184 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 17:44:27.188 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.188 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.188 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.189 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 17:44:27.195 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.195 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.195 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.196 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 17:44:27.206 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.206 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.207 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.207 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 17:44:27.211 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.211 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.212 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.212 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 17:44:27.216 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.217 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.217 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.217 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 17:44:27.222 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.222 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.222 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.223 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 17:44:27.226 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.226 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.227 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.227 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 17:44:27.230 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.231 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.231 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.232 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 17:44:27.235 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.235 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.236 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.236 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 17:44:27.240 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.240 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.241 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.241 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 17:44:27.245 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.245 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.246 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.246 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 17:44:27.249 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.250 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.250 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.251 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 17:44:27.254 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.255 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.255 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.255 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 17:44:27.259 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.259 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.260 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.260 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 17:44:27.263 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.264 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.264 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.264 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 17:44:27.268 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.268 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.269 INFO 2981023 --- [ocp-async-3,4f3138c5e5378295,8eda6df2bdde8523] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:27.269 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.269 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 17:44:27.273 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.273 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.274 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.274 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 17:44:27.277 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.277 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.278 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.278 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 17:44:27.281 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.281 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.282 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.282 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 17:44:27.285 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.285 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.286 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.286 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 17:44:27.289 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.289 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.289 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.290 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 17:44:27.292 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.293 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.293 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.293 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 17:44:27.296 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.296 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.297 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.297 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 17:44:27.300 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.300 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.300 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.301 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 17:44:27.305 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.306 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.306 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.306 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 17:44:27.310 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.310 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.311 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.311 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 17:44:27.314 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.314 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.315 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.315 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 17:44:27.318 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.318 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.319 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.319 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 17:44:27.321 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.321 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.322 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.322 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 17:44:27.324 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.325 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.325 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.325 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 17:44:27.328 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.328 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.328 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.329 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 17:44:27.331 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.331 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.331 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.332 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 17:44:27.338 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.339 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.339 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.339 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 17:44:27.354 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.355 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.355 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.355 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 17:44:27.359 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.360 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.360 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.360 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 17:44:27.364 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.365 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.365 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.365 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 17:44:27.370 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.370 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.370 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.371 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 17:44:27.373 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.373 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.373 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.374 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 17:44:27.376 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.376 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.377 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.377 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 17:44:27.380 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 17:44:27.380 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 17:44:27.380 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.381 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:44:27.382 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.383 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.383 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.384 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 17:44:27.387 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.388 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:44:27.389 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:27.389 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 17:44:27.391 INFO 2981023 --- [ocp-schedules-22,5139865c7f4ab7f7,4da400d9982d1e9b] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 17:44:27.759 INFO 2981023 --- [ocp-async-10,45552fe631993ba4,120958f67f6e5830] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:44:29.269 INFO 2981023 --- [ocp-async-6,c1bed966f3a2af1e,034ea232bacda330] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,2d50323a39ed7897,0d11bc18677e445d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,2d50323a39ed7897,0d11bc18677e445d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:29.394 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:29.395 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:31.269 INFO 2981023 --- [ocp-async-7,5896b26786e9f132,71ad45334b023f57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:32.112 INFO 2981023 --- [ocp-schedules-13,05f78e0c548b20ee,93e556f5b23ee019] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:44:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,cb3d1338cae16ab8,5add7ac5fd32a7f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,cb3d1338cae16ab8,5add7ac5fd32a7f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:32.395 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:32.395 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:33.269 INFO 2981023 --- [ocp-async-3,483f2054d2f844c1,470736466506b421] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:34.246 INFO 2981023 --- [ocp-schedules-3,81c50e93e6befbda,0e78f0bac65e1db8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:44:34.254 INFO 2981023 --- [ocp-schedules-3,81c50e93e6befbda,0e78f0bac65e1db8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:44:34.254 INFO 2981023 --- [ocp-schedules-3,81c50e93e6befbda,0e78f0bac65e1db8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:44:35.269 INFO 2981023 --- [ocp-async-2,d544ec6524894aa6,421b74dd0e244bd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,9ecf1634ef0ea1f5,2bbc0bb51c41f207] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,9ecf1634ef0ea1f5,2bbc0bb51c41f207] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:35.395 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:35.396 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:36.203 INFO 2981023 --- [ocp-schedules-5,5c3348434ef9fa58,5c5657abb3a30aa7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:44:36.247 WARN 2981023 --- [scanner-13538993-1,5c3348434ef9fa58,be405d0d33fe9508] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:44:36.266 INFO 2981023 --- [alarm-detect-6,5c3348434ef9fa58,85933eabac5ea055] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644557, data is empty. 2024-09-11 17:44:36.270 INFO 2981023 --- [alarm-detect-6,5c3348434ef9fa58,85933eabac5ea055] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644557, data is empty. 2024-09-11 17:44:36.274 INFO 2981023 --- [alarm-detect-6,5c3348434ef9fa58,85933eabac5ea055] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644557, data is empty. 2024-09-11 17:44:36.332 INFO 2981023 --- [alarm-detect-14,5c3348434ef9fa58,3f23ceb709ae6ee5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644557, data is empty. 2024-09-11 17:44:36.332 INFO 2981023 --- [alarm-detect-10,5c3348434ef9fa58,564659fd3199ebad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644557, data is empty. 2024-09-11 17:44:36.339 INFO 2981023 --- [alarm-detect-14,5c3348434ef9fa58,3f23ceb709ae6ee5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644557, data is empty. 2024-09-11 17:44:36.342 INFO 2981023 --- [alarm-detect-10,5c3348434ef9fa58,564659fd3199ebad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644557, data is empty. 2024-09-11 17:44:36.347 INFO 2981023 --- [alarm-detect-14,5c3348434ef9fa58,3f23ceb709ae6ee5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644557, data is empty. 2024-09-11 17:44:36.348 INFO 2981023 --- [alarm-detect-10,5c3348434ef9fa58,564659fd3199ebad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644557, data is empty. 2024-09-11 17:44:36.357 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:44:36.356411+08:00, lastProcessTime=2024-09-11T17:44:26.351280+08:00 2024-09-11 17:44:36.379 INFO 2981023 --- [alarm-detect-19,5c3348434ef9fa58,b81da4dedd30b573] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047840 ~ 1726047866 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726047855, value=1.647)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.647, values={1726047855=1.647}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:44:36.380 INFO 2981023 --- [alarm-detect-19,5c3348434ef9fa58,b81da4dedd30b573] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Firing 2024-09-11 17:44:36.380 INFO 2981023 --- [alarm-detect-19,5c3348434ef9fa58,b81da4dedd30b573] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-3] 2024-09-11 17:44:36.428 INFO 2981023 --- [alarm-detect-19,5c3348434ef9fa58,b81da4dedd30b573] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007103 2024-09-11 17:44:36.459 INFO 2981023 --- [alarm-detect-11,5c3348434ef9fa58,b0dfc0d22b1d7553] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644557, data is empty. 2024-09-11 17:44:36.465 INFO 2981023 --- [alarm-detect-11,5c3348434ef9fa58,b0dfc0d22b1d7553] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644557, data is empty. 2024-09-11 17:44:36.468 INFO 2981023 --- [alarm-detect-11,5c3348434ef9fa58,b0dfc0d22b1d7553] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644557, data is empty. 2024-09-11 17:44:36.572 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:44:36.356411+08:00 2024-09-11 17:44:37.269 INFO 2981023 --- [ocp-async-8,820b1f49885345a6,d4198a12b098486e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:37.314 INFO 2981023 --- [prometheus-cache-monitor1,a296dc6ac4c6ea62,aa87ca653ccc13b8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7779,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1807,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:44:37.315 INFO 2981023 --- [prometheus-cache-monitor1,a296dc6ac4c6ea62,aa87ca653ccc13b8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:44:37.759 INFO 2981023 --- [ocp-async-2,b1e3c4e48ab6f0e3,acfaa90a7d03c194] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:44:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,4ab625e15b8297a2,2cb43b1e678cb0b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,4ab625e15b8297a2,2cb43b1e678cb0b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:38.396 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:38.396 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:39.269 INFO 2981023 --- [ocp-async-1,8efb1646f6bd8da2,2e60da065da7b38d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:40.003 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:44:40.004 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:44:40.918 INFO 2981023 --- [ocp-schedules-1,cbb406d56e6086c5,d6abb955d68d4639] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:44:40.938 WARN 2981023 --- [ocp-schedules-1,cbb406d56e6086c5,d6abb955d68d4639] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:44:41.269 INFO 2981023 --- [ocp-async-2,4743e94864eb36dc,3a437bd302613f94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,da9544138dffe3fb,d4137f2c150485e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,da9544138dffe3fb,d4137f2c150485e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:41.396 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:41.397 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:42.114 INFO 2981023 --- [ocp-schedules-19,34d486fc1ebcfde3,6af53d99b36fca98] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:44:43.270 INFO 2981023 --- [ocp-async-10,c28a9e8bb0e126e8,12a37504270400df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,e5e5f23c11001f5f,ac875f9edf595f8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:44.317 INFO 2981023 --- [prometheus-minute-schedule-1,e5e5f23c11001f5f,ac875f9edf595f8a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:44.397 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:44.399 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:45.101 INFO 2981023 --- [prometheus-second-schedule-1,e97317e61afcca5a,13c75ac923efa8ff] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:44:45.270 INFO 2981023 --- [ocp-async-3,7a7fc90409d43940,551bd8fe70d21a38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:46.019 INFO 2981023 --- [ocp-schedules-27,ef79db93a8a1e133,842bd61f3d4c9587] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:44:46.020 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:44:46.021 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:44:46.021 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:44:46.023 INFO 2981023 --- [ocp-schedules-24,5de295c5af06f19f,03315ea40a555e1c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:44:46.031 INFO 2981023 --- [host-check-clockdiff1,5de295c5af06f19f,9e32f409f0918181] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:44:46.032 INFO 2981023 --- [host-check-clockdiff2,5de295c5af06f19f,973b80bc5f7e4731] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:44:46.036 INFO 2981023 --- [host-check-clockdiff3,5de295c5af06f19f,99f253c777e26e7a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:44:46.047 INFO 2981023 --- [host-check-clockdiff2,5de295c5af06f19f,973b80bc5f7e4731] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047886 0 0 , error= 2024-09-11 17:44:46.049 INFO 2981023 --- [host-check-clockdiff3,5de295c5af06f19f,99f253c777e26e7a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047886 0 0 , error= 2024-09-11 17:44:46.065 INFO 2981023 --- [ocp-schedules-29,64c99e30a8ae56fa,dce183d02df3a854] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:44:46.065 INFO 2981023 --- [host-check-agent-status1,64c99e30a8ae56fa,a7e9756cf7b899be] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:44:46.065 INFO 2981023 --- [host-check-agent-status2,64c99e30a8ae56fa,ee3610934b2d1519] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:44:46.066 INFO 2981023 --- [host-check-agent-status3,64c99e30a8ae56fa,893637edbc3691e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:44:46.070 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:46.074 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:46.078 INFO 2981023 --- [host-check-agent-status1,64c99e30a8ae56fa,a7e9756cf7b899be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:46.078 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:46.079 INFO 2981023 --- [host-check-agent-status1,64c99e30a8ae56fa,a7e9756cf7b899be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:46.079 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:46.079 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:46.091 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:46.115 INFO 2981023 --- [host-check-agent-status2,64c99e30a8ae56fa,ee3610934b2d1519] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:46.121 INFO 2981023 --- [host-check-agent-status3,64c99e30a8ae56fa,893637edbc3691e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:46.157 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:44:46.162 INFO 2981023 --- [host-check-agent-status3,64c99e30a8ae56fa,893637edbc3691e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:46.173 INFO 2981023 --- [host-check-agent-status2,64c99e30a8ae56fa,ee3610934b2d1519] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:46.186 INFO 2981023 --- [ocp-schedules-9,cbd45182a19cbd04,388a562c547b5fb7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:44:46.188 INFO 2981023 --- [ocp-schedules-2,c26b4a18afb0b66a,219e444ef160b06d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:44:46.188 INFO 2981023 --- [ocp-schedules-22,4dea8f8f255fd5d1,f0edc3fa1d645d0e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:44:46.189 INFO 2981023 --- [ocp-schedules-13,2656561cc3be3bd4,04ccc376d0168b35] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:44:46.189 INFO 2981023 --- [ocp-schedules-17,004f5624e83dd511,732ea9e39d9ad3dc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:44:46.264 INFO 2981023 --- [ocp-schedules-9,cbd45182a19cbd04,388a562c547b5fb7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:44:46.270 INFO 2981023 --- [ocp-schedules-22,4dea8f8f255fd5d1,f0edc3fa1d645d0e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:44:46.270 INFO 2981023 --- [ocp-schedules-13,2656561cc3be3bd4,04ccc376d0168b35] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:44:46.270 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:44:46.289 INFO 2981023 --- [ocp-schedules-2,c26b4a18afb0b66a,219e444ef160b06d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:44:46.348 INFO 2981023 --- [ocp-schedules-22,4dea8f8f255fd5d1,f0edc3fa1d645d0e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:44:46.358 INFO 2981023 --- [ocp-schedules-25,efae04f5700c480c,7133004db5bdd78a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:44:46.373 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:44:46.374 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:44:46.375 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:44:46.374828792+08:00 2024-09-11 17:44:46.376 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:44:46.376175141+08:00 2024-09-11 17:44:46.378 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:44:46.375+08:00 2024-09-11 17:44:46.380 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:44:46.375+08:00 and status AVAILABLE 2024-09-11 17:44:46.382 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:44:46.376+08:00 2024-09-11 17:44:46.393 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:44:46.376+08:00 and status AVAILABLE 2024-09-11 17:44:46.456 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:44:46.456 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:44:46.452655+08:00, lastProcessTime=2024-09-11T17:44:36.356411+08:00 2024-09-11 17:44:46.457 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:44:46.464 INFO 2981023 --- [host-check-agent-status3,64c99e30a8ae56fa,893637edbc3691e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:44:46.468 INFO 2981023 --- [host-check-agent-status3,64c99e30a8ae56fa,893637edbc3691e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:44:46.468 INFO 2981023 --- [host-check-agent-status3,64c99e30a8ae56fa,893637edbc3691e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:44:46.468 INFO 2981023 --- [host-check-agent-status3,64c99e30a8ae56fa,893637edbc3691e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:44:46.480 INFO 2981023 --- [host-check-agent-status1,64c99e30a8ae56fa,a7e9756cf7b899be] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:44:46.481 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:44:46.483 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:44:46.482760148+08:00 2024-09-11 17:44:46.483 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:44:46.483+08:00 2024-09-11 17:44:46.484 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.485 INFO 2981023 --- [host-check-agent-status1,64c99e30a8ae56fa,a7e9756cf7b899be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:44:46.485 INFO 2981023 --- [host-check-agent-status1,64c99e30a8ae56fa,a7e9756cf7b899be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:44:46.485 INFO 2981023 --- [host-check-agent-status1,64c99e30a8ae56fa,a7e9756cf7b899be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:44:46.485 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:46.485 INFO 2981023 --- [host-check-agent-status2,64c99e30a8ae56fa,ee3610934b2d1519] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:44:46.486 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:44:46.486 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:44:46.483+08:00 and status AVAILABLE 2024-09-11 17:44:46.488 INFO 2981023 --- [host-check-agent-status2,64c99e30a8ae56fa,ee3610934b2d1519] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:44:46.488 INFO 2981023 --- [host-check-agent-status2,64c99e30a8ae56fa,ee3610934b2d1519] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:44:46.488 INFO 2981023 --- [host-check-agent-status2,64c99e30a8ae56fa,ee3610934b2d1519] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:44:46.492 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:46.493 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:46.493 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:44:46.497 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.498 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:46.499 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:46.499 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:44:46.500 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:46.500 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.500 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.501 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:46.502 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:46.503 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:44:46.504 INFO 2981023 --- [ocp-schedules-2,c26b4a18afb0b66a,219e444ef160b06d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:44:46.515 INFO 2981023 --- [ocp-schedules-21,0de2fe3e9600aa87,d20da7ecb453629f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:44:46.571 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.571 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:46.571 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:44:46.572 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:44:46.572 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.580 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:46.588 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:44:46.593 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:44:46.594 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.594 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:46.620 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:44:46.645 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:44:46.647 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:44:46.650 WARN 2981023 --- [scanner-13539303-1,0de2fe3e9600aa87,cc01b7b019636a4b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:44:46.676 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:44:46.678 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:44:46.683 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:44:46.695 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.695 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.696 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:44:46.702 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:44:46.724 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.725 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.726 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:44:46.726 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:44:46.726 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.730 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.732 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.733 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:44:46.743 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:44:46.751 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:44:46.763 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:44:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.791 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:46.793 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:44:46.794 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:44:46.796 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:44:46.797 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.800 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:44:46.802 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:44:46.814 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:44:46.820 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.821 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.821 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:46.822 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.825 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:46.825 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:44:46.825 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:44:46.826 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:44:46.826 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.833 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:44:46.839 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:46.839 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:44:46.840 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:44:46.844 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.844 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:44:46.845 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.845 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:44:46.846 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:44:46.847 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.847 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:44:46.851 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:44:46.852 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.859 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:44:46.865 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:44:46.870 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:44:46.872 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:44:46.875 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.875 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.876 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:44:46.880 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:46.885 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:46.891 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:44:46.897 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.898 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.898 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:46.899 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:44:46.900 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.901 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.902 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:46.902 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:44:46.904 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:44:46.909 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:44:46.909 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:44:46.910 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:44:46.911 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:44:46.914 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:44:46.914 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:44:46.915 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:44:46.916 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:44:46.920 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.920 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.921 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:44:46.922 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:44:46.922 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.925 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:44:46.926 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:44:46.928 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.929 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.930 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:44:46.930 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:44:46.931 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.939 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:44:46.939 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:44:46.940 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:44:46.944 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:46.944 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:44:46.947 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:44:46.947 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:44:46.949 INFO 2981023 --- [host-check-host-status1,ef79db93a8a1e133,587adb3a973d294c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:44:46.956 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:44:46.964 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:44:46.965 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.966 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.967 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:44:46.967 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:44:46.967 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:44:46.969 INFO 2981023 --- [host-check-host-status2,ef79db93a8a1e133,57c9ce245768a6bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:44:46.974 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:44:46.975 INFO 2981023 --- [host-check-host-status3,ef79db93a8a1e133,141773c8df4d4259] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:44:46.980 INFO 2981023 --- [host-check-clockdiff1,5de295c5af06f19f,9e32f409f0918181] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047886 0 0 , error= 2024-09-11 17:44:46.983 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:44:46.984 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:44:46.994 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:46.995 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:46.995 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:44:46.996 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:44:46.996 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:47.001 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:44:47.003 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:44:47.008 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:47.009 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:47.011 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:44:47.018 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:44:47.018 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:44:47.034 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:44:47.038 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:44:47.042 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:47.043 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:47.044 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:44:47.046 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:44:47.047 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:47.048 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:47.048 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:44:47.051 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:44:47.052 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:44:47.054 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:44:47.054 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:44:47.055 INFO 2981023 --- [ocp-schedules-13,2656561cc3be3bd4,04ccc376d0168b35] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:44:47.062 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:44:47.063 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:44:47.067 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:44:46.452655+08:00 2024-09-11 17:44:47.069 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:44:47.071 INFO 2981023 --- [ocp-schedules-30,1da1cf28ef49f951,60c48a0689a0453b] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:44:47.270 INFO 2981023 --- [ocp-async-4,f1a2f52d16a4923d,82d7829ac8828921] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,0b1ad9f05d8518f2,4a77c5e224e7c984] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,0b1ad9f05d8518f2,4a77c5e224e7c984] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:47.399 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:47.399 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:47.759 INFO 2981023 --- [ocp-async-1,ae7ef7c29d4611f0,afc21359187e5ec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:44:49.270 INFO 2981023 --- [ocp-async-7,a08bac8355b5398a,a412995f2a4707a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:44:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:44:50.242 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-19,36967b7825e0fcb0,6a69b54bf3529062] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.248 INFO 2981023 --- [ocp-schedules-9,a49f67adfefe8479,496a8d0baef5ea87] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:44:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,cb627ed0649731dd,1aeed7baf718ad5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,cb627ed0649731dd,1aeed7baf718ad5c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:50.399 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:50.400 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:51.053 INFO 2981023 --- [ocp-schedules-16,90a19d58214da3f5,67927ab8027029c3] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:44:51.270 INFO 2981023 --- [ocp-async-1,ab4fc7c2f4b7a199,eb4bc728535f4485] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:51.557 INFO 2981023 --- [ocp-schedules-3,fec07cd1faec3e66,9affdba808bfe007] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:44:51.558 INFO 2981023 --- [ocp-schedules-3,fec07cd1faec3e66,9affdba808bfe007] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:44:51.559 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:44:51.559 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:44:51.559 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:44:51.559 INFO 2981023 --- [ocp-schedules-3,fec07cd1faec3e66,9affdba808bfe007] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:44:52.117 INFO 2981023 --- [ocp-schedules-10,906e999884004873,8509209d422c91c2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:44:53.130 INFO 2981023 --- [ocp-schedules-1,019271f9c589ee66,523daadac04e3f87] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:44:53.130 INFO 2981023 --- [ocp-schedules-1,019271f9c589ee66,523daadac04e3f87] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:44:53.156 INFO 2981023 --- [ocp-schedules-3,cdd6ca5faf3cf96e,141d005063858cb8] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:44:53.173 WARN 2981023 --- [ocp-schedules-3,cdd6ca5faf3cf96e,141d005063858cb8] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:44:53.270 INFO 2981023 --- [ocp-async-6,c892b582de26b92f,cd526bb2838275c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,e8c5cd0cb3d6ca63,96e66588a05886dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,e8c5cd0cb3d6ca63,96e66588a05886dc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:53.400 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:53.400 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:53.923 INFO 2981023 --- [ocp-schedules-2,fe79573699491740,52f756dd6f926cbf] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:44:53.923+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:44:53.926 INFO 2981023 --- [ocp-schedules-2,fe79573699491740,52f756dd6f926cbf] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:44:54.067 INFO 2981023 --- [ocp-schedules-2,fe79573699491740,52f756dd6f926cbf] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 42 to SqlAggregateData 2024-09-11 17:44:54.068 INFO 2981023 --- [ocp-schedules-2,fe79573699491740,52f756dd6f926cbf] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:44:54.257 INFO 2981023 --- [ocp-schedules-16,7e2ecb7951bb9d39,e7356a6c1454cd34] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:44:54.265 INFO 2981023 --- [ocp-schedules-16,7e2ecb7951bb9d39,e7356a6c1454cd34] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:44:54.265 INFO 2981023 --- [ocp-schedules-16,7e2ecb7951bb9d39,e7356a6c1454cd34] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:44:54.272 INFO 2981023 --- [ocp-schedules-27,3cbdd08290ad8f21,b70955bf7f322ede] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:44:54.633 INFO 2981023 --- [ocp-schedules-26,4d1c101ddbd19f92,bd41a284657ff4b0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:44:54.634 INFO 2981023 --- [ocp-schedules-26,4d1c101ddbd19f92,bd41a284657ff4b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:44:54.661 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:44:54.661 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:44:54.661 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.661 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.662 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:44:54.662 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.665 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:44:54.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:44:54.665 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:44:54.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.669 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.670 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.674 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.674 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.679 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.680 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:44:54.681 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.682 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.683 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.683 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:44:54.691 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.692 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:44:54.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.694 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:44:54.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3221D6E7-0-0] 2024-09-11 17:44:54.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:44:54.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:44:54.697 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.698 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.704 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.705 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.705 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:44:54.705 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.705 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:44:54.705 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:44:54.705 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.706 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:44:54.707 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.707 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:44:54.709 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:44:54.712 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:44:54.719 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.720 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:44:54.720 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:44:54.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.722 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:44:54.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:44:54.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:44:54.724 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.724 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:44:54.724 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3221D6F1-0-0] 2024-09-11 17:44:54.724 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:44:54.725 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:44:54.725 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.726 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:44:54.727 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.727 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:44:54.728 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.728 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:54.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:44:54.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:44:54.730 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:44:54.730 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.732 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:44:54.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:44:54.734 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.734 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:44:54.735 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.736 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:44:54.737 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.738 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:44:54.738 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.739 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:44:54.739 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.740 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:44:54.740 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:44:54.743 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:44:54.744 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:44:54.748 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:54.749 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:44:54.749 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:44:54.749 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:44:54.750 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.750 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:44:54.757 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:44:54.757 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:44:54.758 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:44:54.758 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.759 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:44:54.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:44:54.765 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:44:54.767 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:44:54.767 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.768 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.770 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:44:54.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:44:54.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:54.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:44:54.773 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:44:54.775 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:44:54.775 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:44:54.775 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:44:54.782 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:44:54.784 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:44:54.784 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:44:54.784 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:44:54.784 INFO 2981023 --- [ocp-schedules-26,4d1c101ddbd19f92,bd41a284657ff4b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:44:55.270 INFO 2981023 --- [ocp-async-9,9cfe99921c66aa61,8a2fc53930dcda86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:55.716 INFO 2981023 --- [ocp-schedules-2,14a475dc6a93ad0a,93f7ac478a0c1c7c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:44:55.716 INFO 2981023 --- [ocp-schedules-2,14a475dc6a93ad0a,93f7ac478a0c1c7c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:44:55.717 INFO 2981023 --- [ocp-schedules-2,14a475dc6a93ad0a,93f7ac478a0c1c7c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:44:55.717 INFO 2981023 --- [ocp-schedules-2,14a475dc6a93ad0a,93f7ac478a0c1c7c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:44:55.717 INFO 2981023 --- [ocp-schedules-2,14a475dc6a93ad0a,93f7ac478a0c1c7c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:44:55.717 INFO 2981023 --- [ocp-schedules-2,14a475dc6a93ad0a,93f7ac478a0c1c7c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:44:55.824 INFO 2981023 --- [ocp-schedules-20,cbccb62ead83749b,70a03e7d402bf457] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:44:55.830 INFO 2981023 --- [ocp-schedules-20,cbccb62ead83749b,70a03e7d402bf457] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:44:55.830 INFO 2981023 --- [ocp-schedules-20,cbccb62ead83749b,70a03e7d402bf457] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:44:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,39566433111ceaa5,de34094c963059d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,39566433111ceaa5,de34094c963059d4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:56.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:44:56.336074+08:00, lastProcessTime=2024-09-11T17:44:46.452655+08:00 2024-09-11 17:44:56.400 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:56.401 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:44:56.466 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:44:56.336074+08:00 2024-09-11 17:44:56.544 INFO 2981023 --- [ocp-schedules-3,4f2f34310b884b31,030ea8c2f8828cf4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:44:56.544 INFO 2981023 --- [ocp-schedules-3,4f2f34310b884b31,030ea8c2f8828cf4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:44:56.558 INFO 2981023 --- [Collect all tenants major compactions4,4f2f34310b884b31,5859aa187bd73c6a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:44:56.558 INFO 2981023 --- [Collect all tenants major compactions5,4f2f34310b884b31,d285a89496d7fd3c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:44:56.560 INFO 2981023 --- [Collect all tenants major compactions1,4f2f34310b884b31,b4a1cfba4af9682b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:44:56.560 INFO 2981023 --- [Collect all tenants major compactions6,4f2f34310b884b31,7b6047a77d9b6a22] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:44:56.561 INFO 2981023 --- [Collect all tenants major compactions3,4f2f34310b884b31,d0d18355fbc182a2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:44:56.562 INFO 2981023 --- [Collect all tenants major compactions2,4f2f34310b884b31,87d1c19b139f3917] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:44:56.569 INFO 2981023 --- [Collect all tenants major compactions4,4f2f34310b884b31,5859aa187bd73c6a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:56.570 INFO 2981023 --- [Collect all tenants major compactions5,4f2f34310b884b31,d285a89496d7fd3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:56.571 INFO 2981023 --- [Collect all tenants major compactions4,4f2f34310b884b31,5859aa187bd73c6a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:44:56.571 INFO 2981023 --- [Collect all tenants major compactions5,4f2f34310b884b31,d285a89496d7fd3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:44:56.572 INFO 2981023 --- [Collect all tenants major compactions1,4f2f34310b884b31,b4a1cfba4af9682b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:56.572 INFO 2981023 --- [Collect all tenants major compactions3,4f2f34310b884b31,d0d18355fbc182a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:56.572 INFO 2981023 --- [Collect all tenants major compactions6,4f2f34310b884b31,7b6047a77d9b6a22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:56.573 INFO 2981023 --- [Collect all tenants major compactions1,4f2f34310b884b31,b4a1cfba4af9682b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:44:56.573 INFO 2981023 --- [Collect all tenants major compactions6,4f2f34310b884b31,7b6047a77d9b6a22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:44:56.574 INFO 2981023 --- [Collect all tenants major compactions3,4f2f34310b884b31,d0d18355fbc182a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:44:56.574 INFO 2981023 --- [Collect all tenants major compactions4,4f2f34310b884b31,5859aa187bd73c6a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:44:56.575 INFO 2981023 --- [Collect all tenants major compactions4,4f2f34310b884b31,5859aa187bd73c6a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:44:56.575 INFO 2981023 --- [Collect all tenants major compactions1,4f2f34310b884b31,b4a1cfba4af9682b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:44:56.575 INFO 2981023 --- [Collect all tenants major compactions5,4f2f34310b884b31,d285a89496d7fd3c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:44:56.575 INFO 2981023 --- [Collect all tenants major compactions1,4f2f34310b884b31,b4a1cfba4af9682b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:44:56.575 INFO 2981023 --- [Collect all tenants major compactions5,4f2f34310b884b31,d285a89496d7fd3c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:44:56.576 INFO 2981023 --- [Collect all tenants major compactions2,4f2f34310b884b31,87d1c19b139f3917] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:44:56.576 INFO 2981023 --- [Collect all tenants major compactions6,4f2f34310b884b31,7b6047a77d9b6a22] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:44:56.576 INFO 2981023 --- [Collect all tenants major compactions6,4f2f34310b884b31,7b6047a77d9b6a22] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:44:56.576 INFO 2981023 --- [Collect all tenants major compactions3,4f2f34310b884b31,d0d18355fbc182a2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:44:56.577 INFO 2981023 --- [Collect all tenants major compactions3,4f2f34310b884b31,d0d18355fbc182a2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:44:56.577 INFO 2981023 --- [Collect all tenants major compactions2,4f2f34310b884b31,87d1c19b139f3917] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:44:56.578 INFO 2981023 --- [Collect all tenants major compactions2,4f2f34310b884b31,87d1c19b139f3917] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:44:56.578 INFO 2981023 --- [Collect all tenants major compactions2,4f2f34310b884b31,87d1c19b139f3917] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:44:56.579 INFO 2981023 --- [ocp-schedules-3,4f2f34310b884b31,030ea8c2f8828cf4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:44:56.906 INFO 2981023 --- [ocp-schedules-19,e651f2014554b44a,c9f1cde074735f06] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:44:56.951 WARN 2981023 --- [scanner-13539526-1,e651f2014554b44a,57be461613299f62] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:44:56.985 INFO 2981023 --- [alarm-detect-3,e651f2014554b44a,3d7a0f3a12985de4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644557, data is empty. 2024-09-11 17:44:56.990 INFO 2981023 --- [alarm-detect-3,e651f2014554b44a,3d7a0f3a12985de4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644599, data is empty. 2024-09-11 17:44:56.994 INFO 2981023 --- [alarm-detect-3,e651f2014554b44a,3d7a0f3a12985de4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644599, data is empty. 2024-09-11 17:44:57.008 INFO 2981023 --- [alarm-detect-7,e651f2014554b44a,6e9977331862f9c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644557, data is empty. 2024-09-11 17:44:57.010 INFO 2981023 --- [alarm-detect-1,e651f2014554b44a,5e97a8a91df758b1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644557, data is empty. 2024-09-11 17:44:57.012 INFO 2981023 --- [alarm-detect-7,e651f2014554b44a,6e9977331862f9c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644599, data is empty. 2024-09-11 17:44:57.013 INFO 2981023 --- [alarm-detect-1,e651f2014554b44a,5e97a8a91df758b1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644599, data is empty. 2024-09-11 17:44:57.014 INFO 2981023 --- [alarm-detect-7,e651f2014554b44a,6e9977331862f9c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644599, data is empty. 2024-09-11 17:44:57.017 INFO 2981023 --- [alarm-detect-1,e651f2014554b44a,5e97a8a91df758b1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644599, data is empty. 2024-09-11 17:44:57.142 INFO 2981023 --- [alarm-detect-11,e651f2014554b44a,e9decf103ab120a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644557, data is empty. 2024-09-11 17:44:57.146 INFO 2981023 --- [alarm-detect-11,e651f2014554b44a,e9decf103ab120a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644599, data is empty. 2024-09-11 17:44:57.149 INFO 2981023 --- [alarm-detect-11,e651f2014554b44a,e9decf103ab120a9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644599, data is empty. 2024-09-11 17:44:57.270 INFO 2981023 --- [ocp-async-5,922a626b1af478a2,e6cfc42835a95e8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:57.759 INFO 2981023 --- [ocp-async-7,f3b81bbcfa5da3e3,7e2f4963b862b18d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:44:59.270 INFO 2981023 --- [ocp-async-6,ee91c6c27f89652a,d3937fb66882b355] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:44:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,6df5dec510c36db6,3e36a8a7f216bd40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047900, interval=60 2024-09-11 17:44:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,6df5dec510c36db6,3e36a8a7f216bd40] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:44:59.401 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:44:59.402 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:00.280 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:40:01.0, nextRun=2024-09-11 17:45:00.0, currentDate=Wed Sep 11 17:45:00 HKT 2024 2024-09-11 17:45:00.280 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:40:01.0, nextRunTime=2024-09-11 17:45:00.0} 2024-09-11 17:45:00.286 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:45:00.286 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:45:00.286 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:45:00.286+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:45:00.298 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:40:01.0, nextRun=2024-09-11 17:45:00.0, currentDate=Wed Sep 11 17:45:00 HKT 2024 2024-09-11 17:45:00.298 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:40:01.0, nextRunTime=2024-09-11 17:45:00.0} 2024-09-11 17:45:00.310 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2, 1000006] 2024-09-11 17:45:00.316 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:45:00.317 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:45:00.317 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:45:00.317 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,4a61bbda227aa42e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2,1000006]]}}, createTime=null, startTime=2024-09-11T17:45:00.317+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:45:00.538 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,bc97c1df977eda55] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048174, operation:EXECUTE, state: RUNNING 2024-09-11 17:45:00.549 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,bc97c1df977eda55] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048175, operation:EXECUTE, state: RUNNING 2024-09-11 17:45:00.570 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048174, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048191, task_operation=execute, latest_execution_start_time=2024-09-11T17:45:00.532+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048174}, listMap={}}, executor=10.31.35.10 2024-09-11 17:45:00.582 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048175, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048192, task_operation=execute, latest_execution_start_time=2024-09-11T17:45:00.547+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048175}, listMap={cluster_ids=[[1,2,1000006]]}}, executor=10.31.35.10 2024-09-11 17:45:00.583 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:45:00.586 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:45:00.603 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:45:00.607 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:45:00.619 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:45:00.620 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:45:00.620 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:45:00.628 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:45:00.638 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:45:00.647 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.perf.sql.internal.SqlParseService : parse 9 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:45:00.700 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:45:00.715 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:45:00.743 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:45:00.758 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:45:00.758 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:45:00.763 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:45:00.848 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1000006 2024-09-11 17:45:00.880 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,0cf242746e7a9820] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048175, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:45:00.882 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:45:00.942 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:45:00.972 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:45:00.996 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:45:01.032 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:45:01.045 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:45:01.060 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:45:01.077 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:45:01.077 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:45:01.094 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:45:01.274 INFO 2981023 --- [ocp-async-7,bdc403e5c90ad984,4937e0e9bb5f04c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:01.476 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:45:01.521 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048192, status:SUCCESS 2024-09-11 17:45:01.525 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048192, status:SUCCESS 2024-09-11 17:45:01.527 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:45:01.541 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:45:01.548 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:45:01.558 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:45:01.565 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:45:01.572 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:45:01.574 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:45:01.574 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:45:01.609 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:45:01.626 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:45:01.638 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,9f65a6cd41debb14] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048174, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:45:02.119 INFO 2981023 --- [ocp-schedules-24,e9f8aaad2fc26f8e,6f1f0714810241da] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:45:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,726567dcfa0b78e3,c2b90fa6c0366804] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,726567dcfa0b78e3,c2b90fa6c0366804] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:02.402 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:02.402 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:02.538 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048191, status:SUCCESS 2024-09-11 17:45:02.538 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048191, status:SUCCESS 2024-09-11 17:45:03.273 INFO 2981023 --- [ocp-async-1,9356837ea110bbfb,4b0d405745a7d39b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:04.374 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:41:03.639, end=2024-09-11T17:43:04.371} 2024-09-11 17:45:04.380 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId 2024-09-11 17:45:04.381 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:45:04.381 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:03.639, end=2024-09-11T17:43:04.371},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:45:04.381 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:45:04.381 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:45:04.382 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:45:04.384 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:03.639, end=2024-09-11T17:43:04.371},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:45:04.384 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:45:04.384 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:45:04.400 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:45:04.419 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504419000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.419Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.422 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:45:04.422 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.427 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.427 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.430 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.430 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.435 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.435 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.436 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:45:04.436 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.419, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.437 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:45:04.439 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 4 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:03.639, end=2024-09-11T17:43:04.371},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:45:04.452 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:45:04.452 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:45:04.498 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504498000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.498Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.502 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504501000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.501Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.502 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504502000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.502Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.508 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504508000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.508Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.508 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 62 2024-09-11 17:45:04.508 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.517 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504517000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.517Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.519 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504519000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.519Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.521 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.521 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.525 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.525 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.526 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.526 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.532 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:45:04.532 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.533 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.533 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.535 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.535 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.537 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.537 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.538 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.539 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.539 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.539 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.540 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504540000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.540Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.542 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.542 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.545 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504545000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.545Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.548 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.548 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.548 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.548 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.549 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.549 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.549 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.549 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.551 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.551 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.557 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:45:04.557 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.559 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.559 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.561 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.561 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.562 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.562 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.563 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.563 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.563 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.565 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.565 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.567 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.568 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.568 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.569 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.569 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.569 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.569 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.569 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.569 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.571 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.498, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.572 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.572 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.573 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.574 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.574 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.574 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.575 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.517, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.576 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.576 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.577 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.577 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.580 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.582 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.502, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.582 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.585 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.585 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.588 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.588 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.588 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.501, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.589 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.590 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.519, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.591 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.591 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.54, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.594 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.508, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.595 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.545, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.606 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504606000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.606Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.616 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:45:04.616 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.618 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504618000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.618Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.623 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504623000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.623Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.627 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.627 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.629 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504629000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.629Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.629 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:45:04.629 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.635 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504635000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.635Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.636 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504636000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.636Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.636 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504636000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.636Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.640 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.640 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.642 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504642000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.642Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.643 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.643 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.649 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:45:04.649 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.649 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.649 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.658 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.658 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.660 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.660 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.664 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.664 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.664 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.664 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.665 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.665 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.666 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.666 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.666 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.666 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.666 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.666 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.667 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.667 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.668 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.670 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.670 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.672 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.606, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.675 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.675 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.677 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.677 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.679 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.679 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.680 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.680 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.682 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.682 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.683 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.683 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.684 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.685 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.685 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.685 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.685 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.685 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.688 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.688 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.688 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.688 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.689 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.689 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.693 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.618, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.693 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.693 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.629, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.694 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.694 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.694 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.694 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.698 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.699 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.701 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.701 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.702 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.623, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.702 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.702 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.703 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.642, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.703 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.636, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.703 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.704 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.704 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961504704000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:45:04.704Z, endTime=2024-09-11T09:43:04.371Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:45:04.707 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.636, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.708 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.635, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.712 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:45:04.712 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:45:04.717 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.717 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.718 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:45:04.718 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:45:04.723 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:45:04.723 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047784371000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:45:04.725 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:45:04.727 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:45:04.704, end=2024-09-11T17:43:04.371}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:45:04.728 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:45:04.728 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:03.639, end=2024-09-11T17:43:04.371},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:45:04.728 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:45:04.728 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:45:04.729 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:45:04.730 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:03.639, end=2024-09-11T17:43:04.371},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:45:04.730 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:45:04.730 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:45:04.731 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:45:04.733 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:03.639, end=2024-09-11T17:43:04.371},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:45:04.733 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:45:04.733 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:45:04.743 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:45:04.743 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:45:05.273 INFO 2981023 --- [ocp-async-4,7013ccdd8c640053,562651ca3ad6855e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,dc63801f394be4f6,c19e7a1023dd520a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,dc63801f394be4f6,c19e7a1023dd520a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:05.402 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:05.404 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:45:06.353022+08:00, lastProcessTime=2024-09-11T17:44:56.336074+08:00 2024-09-11 17:45:06.488 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:45:06.353022+08:00 2024-09-11 17:45:07.191 INFO 2981023 --- [ocp-schedules-3,1a06cbcaaf09dd96,2a565c2b7e21f184] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:45:07.247 WARN 2981023 --- [scanner-13540008-1,1a06cbcaaf09dd96,f01ca30f496425ab] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:45:07.273 INFO 2981023 --- [ocp-async-3,c36bcd6138d562d6,dac7da6b13a336f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:07.467 INFO 2981023 --- [ocp-schedules-13,d79a47a4cf59aced,05ca69862d632dbb] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:45:07.489 INFO 2981023 --- [ocp-schedules-29,c68a62b82b77f73c,18b3029d2af2b124] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:45:07.489+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:45:07.490 INFO 2981023 --- [ocp-schedules-29,c68a62b82b77f73c,18b3029d2af2b124] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:45:07.491 INFO 2981023 --- [ocp-schedules-29,c68a62b82b77f73c,18b3029d2af2b124] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:45:07.492 INFO 2981023 --- [ocp-schedules-29,c68a62b82b77f73c,18b3029d2af2b124] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:45:07.759 INFO 2981023 --- [ocp-async-8,03a3e6ff881a601e,f3b6a7c8be2e5957] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:45:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,aeaa34b6525392f4,6e27e506521471ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,aeaa34b6525392f4,6e27e506521471ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:08.404 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:08.406 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:09.273 INFO 2981023 --- [ocp-async-1,3a57f594af194ae6,1a8d491880e3febf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:10.003 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:45:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:45:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:45:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:45:11.274 INFO 2981023 --- [ocp-async-4,c29a2013cc4bba27,da1e1e156b3e328d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,82ae535e1110c5f5,57aea22d3649e8d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,82ae535e1110c5f5,57aea22d3649e8d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:11.406 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:11.407 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:12.122 INFO 2981023 --- [ocp-schedules-23,a58a00157ab2376f,cbdb89a5c7b8eafb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:45:13.274 INFO 2981023 --- [ocp-async-10,4a80a700f10e7593,42c1a840a586151a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:13.549 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047840, nodeCount=1000, affectRows=1000 2024-09-11 17:45:13.551 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047840, nodeCount=1000, validCount=1000 2024-09-11 17:45:13.552 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047840, nodeCount=807, affectRows=807 2024-09-11 17:45:13.554 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047840, nodeCount=806, validCount=806 2024-09-11 17:45:14.268 INFO 2981023 --- [ocp-schedules-25,f25dcbbafdcc5048,750ee554c76e9598] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:45:14.277 INFO 2981023 --- [ocp-schedules-25,f25dcbbafdcc5048,750ee554c76e9598] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:45:14.277 INFO 2981023 --- [ocp-schedules-25,f25dcbbafdcc5048,750ee554c76e9598] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:45:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,f3139b58334c6be2,b392c3f2f4fb3fb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,6f3d18d83fad3403,11e0947b319ed150] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:45:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,f3139b58334c6be2,b392c3f2f4fb3fb2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:14.407 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:14.407 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:15.274 INFO 2981023 --- [ocp-async-9,8913554d617885ee,1f8d05e8be60d877] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:15.827 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047840, nodeCount=463, affectRows=463 2024-09-11 17:45:16.027 INFO 2981023 --- [ocp-schedules-20,9830d209755febd1,34729aded03559ee] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:45:16.028 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:45:16.028 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:45:16.028 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:45:16.028 INFO 2981023 --- [ocp-schedules-17,f701bc0aa81bdbf7,4c0ff8791263ab80] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:45:16.029 INFO 2981023 --- [ocp-schedules-16,9f190ebfdfff6a99,54113f187b0f1ac1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:45:16.030 INFO 2981023 --- [host-check-clockdiff1,f701bc0aa81bdbf7,5f746bac719827d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:45:16.031 INFO 2981023 --- [host-check-clockdiff2,f701bc0aa81bdbf7,d0989207364f1db8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:45:16.034 INFO 2981023 --- [host-check-clockdiff3,f701bc0aa81bdbf7,e47fece2244e66e8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:45:16.044 INFO 2981023 --- [host-check-agent-status1,9f190ebfdfff6a99,b7d4aa4a7154d8e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:45:16.045 INFO 2981023 --- [host-check-agent-status2,9f190ebfdfff6a99,05830282b0edd8bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:45:16.048 INFO 2981023 --- [host-check-agent-status3,9f190ebfdfff6a99,ccda546c3c075716] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:45:16.060 INFO 2981023 --- [host-check-clockdiff3,f701bc0aa81bdbf7,e47fece2244e66e8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047916 0 0 , error= 2024-09-11 17:45:16.063 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:45:16.065 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:45:16.068 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:45:16.070 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:45:16.077 INFO 2981023 --- [host-check-agent-status2,9f190ebfdfff6a99,05830282b0edd8bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:45:16.079 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:45:16.080 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:45:16.096 INFO 2981023 --- [host-check-agent-status1,9f190ebfdfff6a99,b7d4aa4a7154d8e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:45:16.179 INFO 2981023 --- [host-check-agent-status2,9f190ebfdfff6a99,05830282b0edd8bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:45:16.186 INFO 2981023 --- [host-check-agent-status1,9f190ebfdfff6a99,b7d4aa4a7154d8e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:45:16.191 INFO 2981023 --- [ocp-schedules-28,a55ec306800be08f,1f8048ab426b5bf0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:45:16.193 INFO 2981023 --- [ocp-schedules-19,a93c2afeb8603c67,55521b0ce6a1fd08] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:45:16.193 INFO 2981023 --- [ocp-schedules-22,ace370ba64795e99,c051f5b060f4e133] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:45:16.193 INFO 2981023 --- [ocp-schedules-9,6d47a34d1d0de0b5,55c3a3c407d9e176] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:45:16.195 INFO 2981023 --- [ocp-schedules-22,ace370ba64795e99,c051f5b060f4e133] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:45:16.196 INFO 2981023 --- [host-check-agent-status3,9f190ebfdfff6a99,ccda546c3c075716] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:45:16.262 INFO 2981023 --- [host-check-agent-status3,9f190ebfdfff6a99,ccda546c3c075716] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:45:16.265 INFO 2981023 --- [ocp-schedules-19,a93c2afeb8603c67,55521b0ce6a1fd08] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:45:16.266 INFO 2981023 --- [ocp-schedules-28,a55ec306800be08f,1f8048ab426b5bf0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:45:16.273 INFO 2981023 --- [ocp-schedules-19,a93c2afeb8603c67,55521b0ce6a1fd08] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:45:16.307 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:45:16.307 INFO 2981023 --- [ocp-schedules-18,ad74ec37bb988411,dedc3bd10a35ea9b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:45:16.307 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:45:16.309 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:45:16.308804453+08:00 2024-09-11 17:45:16.309 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:45:16.309+08:00 2024-09-11 17:45:16.311 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:45:16.309344193+08:00 2024-09-11 17:45:16.312 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:45:16.309+08:00 and status AVAILABLE 2024-09-11 17:45:16.313 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:45:16.311+08:00 2024-09-11 17:45:16.314 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:45:16.315 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:45:16.311+08:00 and status AVAILABLE 2024-09-11 17:45:16.316 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:45:16.316011712+08:00 2024-09-11 17:45:16.317 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:45:16.316+08:00 2024-09-11 17:45:16.320 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:45:16.316+08:00 and status AVAILABLE 2024-09-11 17:45:16.327 INFO 2981023 --- [host-check-agent-status1,9f190ebfdfff6a99,b7d4aa4a7154d8e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:45:16.329 INFO 2981023 --- [host-check-agent-status1,9f190ebfdfff6a99,b7d4aa4a7154d8e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:45:16.329 INFO 2981023 --- [host-check-agent-status1,9f190ebfdfff6a99,b7d4aa4a7154d8e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:45:16.329 INFO 2981023 --- [host-check-agent-status1,9f190ebfdfff6a99,b7d4aa4a7154d8e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:45:16.331 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.331 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:16.332 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:45:16.354 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:45:16.354 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:45:16.355 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:45:16.355 INFO 2981023 --- [host-check-agent-status3,9f190ebfdfff6a99,ccda546c3c075716] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:45:16.366 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.366 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:16.367 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:45:16.371 INFO 2981023 --- [host-check-agent-status3,9f190ebfdfff6a99,ccda546c3c075716] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:45:16.371 INFO 2981023 --- [host-check-agent-status3,9f190ebfdfff6a99,ccda546c3c075716] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:45:16.371 INFO 2981023 --- [host-check-agent-status3,9f190ebfdfff6a99,ccda546c3c075716] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:45:16.380 INFO 2981023 --- [host-check-agent-status2,9f190ebfdfff6a99,05830282b0edd8bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:45:16.384 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:45:16.386 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:45:16.387 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:45:16.388 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:45:16.391 INFO 2981023 --- [host-check-agent-status2,9f190ebfdfff6a99,05830282b0edd8bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:45:16.391 INFO 2981023 --- [host-check-agent-status2,9f190ebfdfff6a99,05830282b0edd8bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:45:16.391 INFO 2981023 --- [host-check-agent-status2,9f190ebfdfff6a99,05830282b0edd8bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:45:16.392 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.394 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:16.401 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:45:16.402 INFO 2981023 --- [ocp-schedules-28,a55ec306800be08f,1f8048ab426b5bf0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:45:16.404 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:45:16.404 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:45:16.404344+08:00, lastProcessTime=2024-09-11T17:45:06.353022+08:00 2024-09-11 17:45:16.406 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.406 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:45:16.406 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:16.407 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.407 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:16.410 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.411 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:16.460 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:45:16.463 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:45:16.473 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:45:16.482 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.482 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:45:16.492 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:45:16.493 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:45:16.502 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:45:16.506 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:45:16.511 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:45:16.512 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:45:16.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.526 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.528 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:16.528 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:45:16.529 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.531 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:45:16.532 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:45:16.532 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.533 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.533 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:45:16.535 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:45:16.548 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:16.548 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:45:16.555 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:45:16.555 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:45:16.564 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.565 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.565 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.569 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:45:16.570 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:45:16.572 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:16.576 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:45:16.579 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:16.579 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:45:16.580 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.580 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.581 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:45:16.581 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:16.581 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:45:16.584 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:45:16.588 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:45:16.590 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:45:16.594 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:45:16.597 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:45:16.597 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:45:16.598 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:45:16.602 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:16.609 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.610 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.611 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.611 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:45:16.611 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.612 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:16.613 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:45:16.615 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:45:16.617 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:45:16.622 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:45:16.626 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:45:16.629 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:45:16.630 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:45:16.631 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:45:16.637 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.638 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.638 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:16.639 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:45:16.639 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.642 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:45:16.642 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:45:16.651 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:16.651 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:45:16.656 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:45:16.656 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:45:16.659 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:45:16.668 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:45:16.669 INFO 2981023 --- [host-check-host-status1,9830d209755febd1,312142bcb28c2707] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:45:16.669 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:16.670 INFO 2981023 --- [host-check-host-status3,9830d209755febd1,4adb1f8de882bcc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:45:16.671 INFO 2981023 --- [host-check-host-status2,9830d209755febd1,308650971e642456] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:45:16.678 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.679 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.681 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:16.683 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:45:16.687 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:45:16.691 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:45:16.692 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:45:16.712 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:45:16.720 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:45:16.726 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.726 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.727 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:45:16.733 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:45:16.733 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:45:16.733 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:45:16.739 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.739 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.740 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:45:16.740 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:45:16.404344+08:00 2024-09-11 17:45:16.742 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:45:16.746 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:45:16.746 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:45:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:16.749 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:16.749 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:45:16.755 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:45:16.755 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:45:16.755 INFO 2981023 --- [ocp-schedules-22,ace370ba64795e99,c051f5b060f4e133] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:45:16.990 INFO 2981023 --- [host-check-clockdiff2,f701bc0aa81bdbf7,d0989207364f1db8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047916 0 0 , error= 2024-09-11 17:45:17.274 INFO 2981023 --- [ocp-async-8,17df02f7127bb859,e6b1cff84cd9d1e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,bee5ac590ddac573,878514845f245ea6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,bee5ac590ddac573,878514845f245ea6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:17.398 INFO 2981023 --- [ocp-schedules-21,b21a61326bd13b15,b310dc32a3396f27] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:45:17.408 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:17.409 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:17.409 WARN 2981023 --- [scanner-13541689-1,b21a61326bd13b15,e1ef4a921b4cd536] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:45:17.470 INFO 2981023 --- [alarm-detect-2,b21a61326bd13b15,8942308c0dc7bd35] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644599, data is empty. 2024-09-11 17:45:17.475 INFO 2981023 --- [alarm-detect-2,b21a61326bd13b15,8942308c0dc7bd35] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644599, data is empty. 2024-09-11 17:45:17.484 INFO 2981023 --- [alarm-detect-2,b21a61326bd13b15,8942308c0dc7bd35] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644599, data is empty. 2024-09-11 17:45:17.513 INFO 2981023 --- [alarm-detect-1,b21a61326bd13b15,a9fa25aa155e7615] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644599, data is empty. 2024-09-11 17:45:17.515 INFO 2981023 --- [alarm-detect-13,b21a61326bd13b15,7369bf3ee615ee5d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644599, data is empty. 2024-09-11 17:45:17.531 INFO 2981023 --- [alarm-detect-13,b21a61326bd13b15,7369bf3ee615ee5d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644599, data is empty. 2024-09-11 17:45:17.534 INFO 2981023 --- [alarm-detect-1,b21a61326bd13b15,a9fa25aa155e7615] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644599, data is empty. 2024-09-11 17:45:17.537 INFO 2981023 --- [alarm-detect-13,b21a61326bd13b15,7369bf3ee615ee5d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644599, data is empty. 2024-09-11 17:45:17.539 INFO 2981023 --- [alarm-detect-1,b21a61326bd13b15,a9fa25aa155e7615] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644599, data is empty. 2024-09-11 17:45:17.616 INFO 2981023 --- [alarm-detect-1,b21a61326bd13b15,bdf3bb23fb25ba8f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047885 ~ 1726047907 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726047900, value=1.968)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.968, values={1726047900=1.968}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:45:17.692 INFO 2981023 --- [alarm-detect-9,b21a61326bd13b15,3b381c3ae4c3c560] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644599, data is empty. 2024-09-11 17:45:17.698 INFO 2981023 --- [alarm-detect-9,b21a61326bd13b15,3b381c3ae4c3c560] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644599, data is empty. 2024-09-11 17:45:17.702 INFO 2981023 --- [alarm-detect-9,b21a61326bd13b15,3b381c3ae4c3c560] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644599, data is empty. 2024-09-11 17:45:17.760 INFO 2981023 --- [ocp-async-7,715353e63f40ac49,2cdd4949b8ee1091] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:45:18.029 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047840, nodeCount=463, validCount=463 2024-09-11 17:45:19.274 INFO 2981023 --- [ocp-async-3,d78cb7ce01ebc915,7b33472cb65d85b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,03134e8603cbfcc3,3700cb5313a867e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,03134e8603cbfcc3,3700cb5313a867e2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:20.409 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:20.409 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:21.106 INFO 2981023 --- [host-check-clockdiff1,f701bc0aa81bdbf7,5f746bac719827d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047921 0 0 , error= 2024-09-11 17:45:21.274 INFO 2981023 --- [ocp-async-7,730db6b007d0be4e,782da8fe353c0c86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:22.125 INFO 2981023 --- [ocp-schedules-1,f0827ea9d9c0d1fb,1896e497254fcefc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:45:23.083 INFO 2981023 --- [ocp-schedules-23,07d93383dbe3a0e4,eb7f92d0eb3157a0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:45:23.083 INFO 2981023 --- [ocp-schedules-23,07d93383dbe3a0e4,eb7f92d0eb3157a0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:45:23.083 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:45:23.084 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:45:23.087 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:45:23.088 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:45:23.088 INFO 2981023 --- [ocp-schedules-23,07d93383dbe3a0e4,eb7f92d0eb3157a0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:45:23.088 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:45:23.088 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:45:23.089 INFO 2981023 --- [ocp-schedules-23,07d93383dbe3a0e4,eb7f92d0eb3157a0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:45:23.103 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:45:23.103 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:45:23.103 INFO 2981023 --- [ocp-schedules-23,07d93383dbe3a0e4,eb7f92d0eb3157a0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:45:23.105 INFO 2981023 --- [ocp-schedules-23,07d93383dbe3a0e4,eb7f92d0eb3157a0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:45:23.274 INFO 2981023 --- [ocp-async-8,cdad89518205a014,fca84ff2760717de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,87f24be3712ba101,957c7a59022c41d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,87f24be3712ba101,957c7a59022c41d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:23.409 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:23.410 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:24.275 INFO 2981023 --- [ocp-schedules-24,0459dcf542d02ff9,6a1b419673e67a6c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:45:25.274 INFO 2981023 --- [ocp-async-6,64d21e996c392709,695734a518367d4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,2d720cd118284195,6cf38f48e68cfd98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,2d720cd118284195,6cf38f48e68cfd98] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:26.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:45:26.355508+08:00, lastProcessTime=2024-09-11T17:45:16.404344+08:00 2024-09-11 17:45:26.410 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:26.410 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:26.500 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:45:26.355508+08:00 2024-09-11 17:45:27.275 INFO 2981023 --- [ocp-async-10,232245504f6df14f,bebb94d9dca40a3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:27.756 INFO 2981023 --- [ocp-schedules-9,614e1073525d1be3,e5f345191cb99213] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:45:27.760 INFO 2981023 --- [ocp-async-6,7d6ee04910c617c3,911fdc1f080db60e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:45:27.805 WARN 2981023 --- [scanner-13542011-1,614e1073525d1be3,3cbbe769522b14ea] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:45:29.275 INFO 2981023 --- [ocp-async-1,666ffd631a9fde8c,21d20eb10144d6af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,d707115024f83e06,80c0c33f697c9d17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,d707115024f83e06,80c0c33f697c9d17] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:29.410 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:29.411 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:31.275 INFO 2981023 --- [ocp-async-6,4560c7e4d07351d3,2ac794cd642b89be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:31.619 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:45:31.619 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:45:31.623 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:45:31.623 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:45:31.623 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:45:31.623 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:45:31.626 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:45:31.626 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:45:31.626 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:45:31.626 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:45:31.673 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:45:31.677 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:45:31.677 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:45:31.677 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:45:31.677 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:45:31.677 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:45:31.681 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:45:31.935 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:45:32.128 INFO 2981023 --- [ocp-schedules-9,14840c96b65400e0,8924a62ee0c54512] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:45:32.197 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.571163907S 2024-09-11 17:45:32.197 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:45:32.246 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:45:32.246 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:45:32.264 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:45:32.297 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:45:32.297 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:45:32.297 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:45:32.297 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:45:32.297 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:45:32.297 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:45:32.303 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:45:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,b91b181a48b5e153,e43e8a7b483994b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,b91b181a48b5e153,e43e8a7b483994b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:32.388 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] o.s.boot.SpringApplication : Started application in 0.764 seconds (JVM running for 83732.26) 2024-09-11 17:45:32.388 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.765043727S 2024-09-11 17:45:32.411 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:32.411 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:32.608 INFO 2981023 --- [ocp-schedules-20,cd7b2687ffcd09a0,8040a561ad999ff0] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:45:33.275 INFO 2981023 --- [ocp-async-10,bf4e431ede62a38c,d2af154fc90a1fa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:34.280 INFO 2981023 --- [ocp-schedules-27,2cef9288d17f7be9,d0f85a15967124e4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:45:34.291 INFO 2981023 --- [ocp-schedules-27,2cef9288d17f7be9,d0f85a15967124e4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:45:34.291 INFO 2981023 --- [ocp-schedules-27,2cef9288d17f7be9,d0f85a15967124e4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:45:35.275 INFO 2981023 --- [ocp-async-8,98cdadf86eaf85a0,a4ecc973eba2e8ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,fae3b623a3460045,f2bb0b5d58d70594] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,fae3b623a3460045,f2bb0b5d58d70594] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:35.411 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:35.412 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:45:36.337809+08:00, lastProcessTime=2024-09-11T17:45:26.355508+08:00 2024-09-11 17:45:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:45:36.337809+08:00 2024-09-11 17:45:37.275 INFO 2981023 --- [ocp-async-2,f3088bb1c9f0d138,239fb4ae47caab05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:37.314 INFO 2981023 --- [prometheus-cache-monitor1,d0508da134a1508f,e125b4960edb6d71] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7536,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:45:37.314 INFO 2981023 --- [prometheus-cache-monitor1,d0508da134a1508f,e125b4960edb6d71] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:45:37.760 INFO 2981023 --- [ocp-async-8,6c8c2a4f550112f1,133599b0973ce239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:45:37.948 INFO 2981023 --- [ocp-schedules-29,d7a9e2eaca142412,51bb0b5f18499249] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:45:37.997 WARN 2981023 --- [scanner-13542240-1,d7a9e2eaca142412,30da1244dfc8db97] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:45:38.025 INFO 2981023 --- [alarm-detect-3,d7a9e2eaca142412,a72b24b1054e5c38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644599, data is empty. 2024-09-11 17:45:38.031 INFO 2981023 --- [alarm-detect-3,d7a9e2eaca142412,a72b24b1054e5c38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644599, data is empty. 2024-09-11 17:45:38.037 INFO 2981023 --- [alarm-detect-3,d7a9e2eaca142412,a72b24b1054e5c38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644599, data is empty. 2024-09-11 17:45:38.038 INFO 2981023 --- [alarm-detect-1,d7a9e2eaca142412,3e80833e7709e5dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644599, data is empty. 2024-09-11 17:45:38.040 INFO 2981023 --- [alarm-detect-14,d7a9e2eaca142412,2c2fa744b6b17ff8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644599, data is empty. 2024-09-11 17:45:38.043 INFO 2981023 --- [alarm-detect-1,d7a9e2eaca142412,3e80833e7709e5dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644599, data is empty. 2024-09-11 17:45:38.045 INFO 2981023 --- [alarm-detect-14,d7a9e2eaca142412,2c2fa744b6b17ff8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644599, data is empty. 2024-09-11 17:45:38.047 INFO 2981023 --- [alarm-detect-1,d7a9e2eaca142412,3e80833e7709e5dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644599, data is empty. 2024-09-11 17:45:38.048 INFO 2981023 --- [alarm-detect-14,d7a9e2eaca142412,2c2fa744b6b17ff8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644599, data is empty. 2024-09-11 17:45:38.071 INFO 2981023 --- [alarm-detect-4,d7a9e2eaca142412,85c88b043be05612] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726047900 ~ 1726047928 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726047915, value=1.542)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.542, values={1726047915=1.542}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:45:38.150 INFO 2981023 --- [alarm-detect-14,d7a9e2eaca142412,4d1037b0b8f1409c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644599, data is empty. 2024-09-11 17:45:38.154 INFO 2981023 --- [alarm-detect-14,d7a9e2eaca142412,4d1037b0b8f1409c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644599, data is empty. 2024-09-11 17:45:38.159 INFO 2981023 --- [alarm-detect-14,d7a9e2eaca142412,4d1037b0b8f1409c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644599, data is empty. 2024-09-11 17:45:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,0cd9f74f2ff127ce,5541356fbd11caab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,0cd9f74f2ff127ce,5541356fbd11caab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:38.412 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:38.412 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:39.275 INFO 2981023 --- [ocp-async-5,8157cfc82959d3ba,12fdbfb65c0ddc6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:40.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:45:40.003 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:45:41.075 INFO 2981023 --- [ocp-schedules-8,7a31cab7768393d0,9b5e9a3fe58ac7da] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:45:41.093 WARN 2981023 --- [ocp-schedules-8,7a31cab7768393d0,9b5e9a3fe58ac7da] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:45:41.275 INFO 2981023 --- [ocp-async-8,4f7d2af96a3b2510,31d1c5494e414b0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,5a9369a10af535a2,9201c06fecc46828] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,5a9369a10af535a2,9201c06fecc46828] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:41.413 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:41.413 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:42.132 INFO 2981023 --- [ocp-schedules-4,166fcd92288caaa9,4dc99df6f7aab5ff] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:45:43.275 INFO 2981023 --- [ocp-async-4,145c601e95c81367,b6a29ac9b4eb301d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,0df06e3b4e2e67cd,3180a509be1f20d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,0df06e3b4e2e67cd,3180a509be1f20d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:44.413 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:44.413 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:45.101 INFO 2981023 --- [prometheus-second-schedule-1,4338f645a2df78c1,0e2eee872f530e58] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:45:45.276 INFO 2981023 --- [ocp-async-10,9de1de82537ff32c,c7c5ecc5de28e54e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:46.014 INFO 2981023 --- [ocp-schedules-15,edf38208f021955d,fcf6c5fe609ad9d6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:45:46.015 INFO 2981023 --- [host-check-clockdiff1,edf38208f021955d,db10c366ca9d488b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:45:46.015 INFO 2981023 --- [host-check-clockdiff2,edf38208f021955d,238379afee3029b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:45:46.017 INFO 2981023 --- [host-check-clockdiff3,edf38208f021955d,e934ae49c18ec40d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:45:46.022 INFO 2981023 --- [ocp-schedules-22,6af690e7e774f0cd,9ce02ec6ec891438] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:45:46.022 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:45:46.022 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:45:46.028 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:45:46.029 INFO 2981023 --- [host-check-clockdiff3,edf38208f021955d,e934ae49c18ec40d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047946 0 0 , error= 2024-09-11 17:45:46.030 INFO 2981023 --- [ocp-schedules-1,c62a644ad8e51b56,0b0b3386375cf7f2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:45:46.031 INFO 2981023 --- [host-check-agent-status1,c62a644ad8e51b56,3fc6240e79a97f6c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:45:46.031 INFO 2981023 --- [host-check-agent-status2,c62a644ad8e51b56,fa89659a685d0348] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:45:46.034 INFO 2981023 --- [host-check-agent-status3,c62a644ad8e51b56,3535d971a843ca0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:45:46.034 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:45:46.035 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:45:46.036 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:45:46.037 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:45:46.038 INFO 2981023 --- [host-check-agent-status1,c62a644ad8e51b56,3fc6240e79a97f6c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:45:46.038 INFO 2981023 --- [host-check-agent-status2,c62a644ad8e51b56,fa89659a685d0348] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:45:46.038 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:45:46.039 INFO 2981023 --- [host-check-agent-status1,c62a644ad8e51b56,3fc6240e79a97f6c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:45:46.039 INFO 2981023 --- [host-check-agent-status2,c62a644ad8e51b56,fa89659a685d0348] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:45:46.039 INFO 2981023 --- [host-check-agent-status3,c62a644ad8e51b56,3535d971a843ca0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:45:46.040 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:45:46.041 INFO 2981023 --- [host-check-agent-status3,c62a644ad8e51b56,3535d971a843ca0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:45:46.059 INFO 2981023 --- [host-check-agent-status1,c62a644ad8e51b56,3fc6240e79a97f6c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:45:46.062 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:45:46.062 INFO 2981023 --- [host-check-agent-status2,c62a644ad8e51b56,fa89659a685d0348] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:45:46.063 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:45:46.063 INFO 2981023 --- [host-check-agent-status1,c62a644ad8e51b56,3fc6240e79a97f6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:45:46.063 INFO 2981023 --- [host-check-agent-status1,c62a644ad8e51b56,3fc6240e79a97f6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:45:46.063 INFO 2981023 --- [host-check-agent-status1,c62a644ad8e51b56,3fc6240e79a97f6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:45:46.064 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:45:46.064075474+08:00 2024-09-11 17:45:46.064 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:45:46.064+08:00 2024-09-11 17:45:46.064 INFO 2981023 --- [host-check-agent-status3,c62a644ad8e51b56,3535d971a843ca0c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:45:46.065 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:45:46.064662594+08:00 2024-09-11 17:45:46.066 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:45:46.065+08:00 2024-09-11 17:45:46.066 INFO 2981023 --- [host-check-agent-status2,c62a644ad8e51b56,fa89659a685d0348] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:45:46.066 INFO 2981023 --- [host-check-agent-status2,c62a644ad8e51b56,fa89659a685d0348] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:45:46.066 INFO 2981023 --- [host-check-agent-status2,c62a644ad8e51b56,fa89659a685d0348] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:45:46.067 INFO 2981023 --- [host-check-agent-status3,c62a644ad8e51b56,3535d971a843ca0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:45:46.067 INFO 2981023 --- [host-check-agent-status3,c62a644ad8e51b56,3535d971a843ca0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:45:46.067 INFO 2981023 --- [host-check-agent-status3,c62a644ad8e51b56,3535d971a843ca0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:45:46.067 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:45:46.069 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:45:46.068956506+08:00 2024-09-11 17:45:46.069 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:45:46.069+08:00 2024-09-11 17:45:46.072 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:45:46.064+08:00 and status AVAILABLE 2024-09-11 17:45:46.073 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:45:46.065+08:00 and status AVAILABLE 2024-09-11 17:45:46.081 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:45:46.069+08:00 and status AVAILABLE 2024-09-11 17:45:46.093 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:45:46.099 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:45:46.100 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:45:46.107 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:45:46.108 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:45:46.108 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:45:46.109 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:45:46.110 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:45:46.111 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:45:46.132 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:45:46.132 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:45:46.138 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:45:46.157 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:45:46.163 INFO 2981023 --- [ocp-schedules-26,1dd599e28375bd4c,fc99fd94a054bfd1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:45:46.170 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:45:46.172 INFO 2981023 --- [ocp-schedules-26,1dd599e28375bd4c,fc99fd94a054bfd1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:45:46.176 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:45:46.176 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:45:46.180 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:45:46.189 INFO 2981023 --- [ocp-schedules-10,5621467ac9da4f36,0bd9bcb300814219] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:45:46.190 INFO 2981023 --- [ocp-schedules-20,e3ca6c56b4acabdf,0908ff8c5b7e5d76] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:45:46.190 INFO 2981023 --- [ocp-schedules-25,299cd035a1a76ba6,8cee219b0cdd9b4f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:45:46.190 INFO 2981023 --- [ocp-schedules-18,9a12580b551d57d7,00500a5d0a852333] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:45:46.191 INFO 2981023 --- [ocp-schedules-25,299cd035a1a76ba6,8cee219b0cdd9b4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:45:46.196 INFO 2981023 --- [ocp-schedules-20,e3ca6c56b4acabdf,0908ff8c5b7e5d76] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:45:46.196 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:45:46.197 INFO 2981023 --- [ocp-schedules-10,5621467ac9da4f36,0bd9bcb300814219] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:45:46.203 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:45:46.217 INFO 2981023 --- [ocp-schedules-7,0302c393493ee47e,05185c8a9571f1cc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:45:46.220 INFO 2981023 --- [ocp-schedules-20,e3ca6c56b4acabdf,0908ff8c5b7e5d76] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:45:46.220 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:45:46.258 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:45:46.260 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.261 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:46.261 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:45:46.266 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:45:46.267 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.268 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:46.270 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.270 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:46.271 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.271 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:46.272 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.272 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:46.274 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:45:46.274 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:45:46.281 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.282 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:45:46.284 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:45:46.285 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.289 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:45:46.303 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:45:46.306 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:45:46.313 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.313 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:46.314 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:45:46.314 INFO 2981023 --- [ocp-schedules-10,5621467ac9da4f36,0bd9bcb300814219] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:45:46.324 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:45:46.330 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:45:46.337 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.343 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.344 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:45:46.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:45:46.350765+08:00, lastProcessTime=2024-09-11T17:45:36.337809+08:00 2024-09-11 17:45:46.363 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:45:46.372 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:45:46.376 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:45:46.380 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:45:46.386 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.387 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.388 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:45:46.389 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.390 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.390 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:45:46.391 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:45:46.391 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.392 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.394 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.394 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:46.395 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:45:46.396 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:45:46.400 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.403 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:45:46.408 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:45:46.410 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:45:46.413 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:45:46.415 INFO 2981023 --- [host-check-host-status1,6af690e7e774f0cd,eaa21dd6b453303d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:45:46.422 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:45:46.424 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:45:46.426 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:46.426 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:45:46.430 INFO 2981023 --- [host-check-host-status2,6af690e7e774f0cd,3b0983042a92c2cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:45:46.445 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.446 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.446 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:45:46.452 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.453 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.453 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:46.454 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.456 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:45:46.456 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.457 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:46.457 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:45:46.459 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:45:46.469 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.469 INFO 2981023 --- [host-check-host-status3,6af690e7e774f0cd,b65a6b2358336fc6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:45:46.470 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.470 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:45:46.470 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.471 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:45:46.471 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.471 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:46.471 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:45:46.471 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:46.472 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:45:46.475 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.475 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:45:46.476 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:45:46.476 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.476 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:45:46.480 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:45:46.480 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.481 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.481 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:45:46.481 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:45:46.482 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:45:46.482 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.482 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:45:46.485 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:45:46.485 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:45:46.491 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:46.494 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:46.494 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:45:46.501 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.501 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.502 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:46.503 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:45:46.503 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:45:46.507 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:45:46.510 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:45:46.510 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:45:46.513 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:45:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:45:46.529 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:45:46.529 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:45:46.531 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:45:46.536 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.537 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.537 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:45:46.537 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:45:46.538 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:45:46.538 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.542 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:45:46.543 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:45:46.553 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:45:46.558 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:45:46.570 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:45:46.571 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:45:46.576 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.580 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.581 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:45:46.586 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.587 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:45:46.587 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:45:46.587 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:45:46.587 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.588 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:45:46.588 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:45:46.588 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.593 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.593 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.594 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:45:46.595 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:45:46.601 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:45:46.601 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:45:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:46.603 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:46.603 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:45:46.609 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:45:46.609 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:45:46.609 INFO 2981023 --- [ocp-schedules-25,299cd035a1a76ba6,8cee219b0cdd9b4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:45:46.627 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:45:46.350765+08:00 2024-09-11 17:45:46.641 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:45:46.657 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:45:46.663 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:45:46.666 INFO 2981023 --- [ocp-schedules-21,a45c1aea827044a9,77a6c4802ec165ab] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:45:46.840 INFO 2981023 --- [ocp-schedules-3,69f29fb165534c12,760d4fe380a2a636] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7536,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:45:46.840 INFO 2981023 --- [ocp-schedules-3,69f29fb165534c12,760d4fe380a2a636] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:45:46.960 INFO 2981023 --- [host-check-clockdiff2,edf38208f021955d,238379afee3029b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047946 0 0 , error= 2024-09-11 17:45:46.965 INFO 2981023 --- [host-check-clockdiff1,edf38208f021955d,db10c366ca9d488b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047946 0 0 , error= 2024-09-11 17:45:47.276 INFO 2981023 --- [ocp-async-8,a04cae0dca9a0e9b,d4b033fde945a9b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,324d538d203495c1,b4e7e244a5518183] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,324d538d203495c1,b4e7e244a5518183] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:47.414 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:47.414 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:47.760 INFO 2981023 --- [ocp-async-7,121a59e4856aa87b,812ce8eaae025a06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:45:48.196 INFO 2981023 --- [ocp-schedules-4,e46916a1f25e5198,917bf8b87200e01c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:45:48.243 WARN 2981023 --- [scanner-13542579-1,e46916a1f25e5198,81ad80faa748bf44] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:45:49.276 INFO 2981023 --- [ocp-async-6,abe606538eb1761b,6ca38d96e0487b85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:45:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:45:50.251 INFO 2981023 --- [ocp-schedules-20,d8e10ba55317165e,8407995d3fde8c7e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:45:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,af0316483cd29819,9e077ab9fee51e85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,af0316483cd29819,9e077ab9fee51e85] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:50.414 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:50.416 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:51.276 INFO 2981023 --- [ocp-async-5,1eb7533b4436a1b3,a542c4fee02a0751] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:51.501 INFO 2981023 --- [ocp-schedules-25,e4ed238b225c5662,73e0c023fd95774b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:45:51.559 INFO 2981023 --- [ocp-schedules-8,2df8709783af92a8,022e8e542cd8008d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:45:51.560 INFO 2981023 --- [ocp-schedules-8,2df8709783af92a8,022e8e542cd8008d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:45:51.561 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:45:51.561 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:45:51.561 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:45:51.561 INFO 2981023 --- [ocp-schedules-8,2df8709783af92a8,022e8e542cd8008d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:45:52.135 INFO 2981023 --- [ocp-schedules-2,253a31d31f9c7a9e,d392b1ef67a4f1f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:45:53.133 INFO 2981023 --- [ocp-schedules-1,c765ba619b931122,4d56ba795af3e870] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:45:53.133 INFO 2981023 --- [ocp-schedules-1,c765ba619b931122,4d56ba795af3e870] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:45:53.201 INFO 2981023 --- [ocp-schedules-20,e5b3360bd50de044,2d2671a6f0d28437] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:45:53.220 WARN 2981023 --- [ocp-schedules-20,e5b3360bd50de044,2d2671a6f0d28437] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:45:53.276 INFO 2981023 --- [ocp-async-8,dfdd24b563cfe5ef,8424f65e0900d79f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,2a1840cf568346c2,b84abb53317370af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,2a1840cf568346c2,b84abb53317370af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:53.416 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:53.417 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:54.089 INFO 2981023 --- [ocp-schedules-21,24054aa2fd4d3cdf,68e8884dc4e88d59] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:45:54.089+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:45:54.093 INFO 2981023 --- [ocp-schedules-21,24054aa2fd4d3cdf,68e8884dc4e88d59] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:45:54.242 INFO 2981023 --- [ocp-schedules-21,24054aa2fd4d3cdf,68e8884dc4e88d59] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 28 to SqlAggregateData 2024-09-11 17:45:54.244 INFO 2981023 --- [ocp-schedules-21,24054aa2fd4d3cdf,68e8884dc4e88d59] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:45:54.277 INFO 2981023 --- [ocp-schedules-8,3e8bd2e295736bea,e5b186036c6e6afc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:45:54.294 INFO 2981023 --- [ocp-schedules-4,63271e5b911191de,a4ef61f497e57a85] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:45:54.302 INFO 2981023 --- [ocp-schedules-4,63271e5b911191de,a4ef61f497e57a85] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:45:54.302 INFO 2981023 --- [ocp-schedules-4,63271e5b911191de,a4ef61f497e57a85] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:45:54.784 INFO 2981023 --- [ocp-schedules-20,44eccaf92f1c4446,55b6121975df465d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:45:54.785 INFO 2981023 --- [ocp-schedules-20,44eccaf92f1c4446,55b6121975df465d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:45:54.815 INFO 2981023 --- [ocp-schedules-10,a9474dd31a865613,856aabef9cd46cc1] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:45:54.816 INFO 2981023 --- [ocp-schedules-10,a9474dd31a865613,856aabef9cd46cc1] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:45:54.823 INFO 2981023 --- [ocp-schedules-10,a9474dd31a865613,856aabef9cd46cc1] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:45:54.823 INFO 2981023 --- [ocp-schedules-10,a9474dd31a865613,856aabef9cd46cc1] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:45:54.823 INFO 2981023 --- [ocp-schedules-10,a9474dd31a865613,856aabef9cd46cc1] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:45:54.823 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:45:54.824 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.824 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:45:54.824 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.825 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:45:54.826 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.828 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:45:54.828 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:45:54.829 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:45:54.832 INFO 2981023 --- [ocp-schedules-10,a9474dd31a865613,856aabef9cd46cc1] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:45:54.832 INFO 2981023 --- [ocp-schedules-10,a9474dd31a865613,856aabef9cd46cc1] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:45:54.832 INFO 2981023 --- [ocp-schedules-10,a9474dd31a865613,856aabef9cd46cc1] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:45:54.837 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.837 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.838 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.838 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.839 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.839 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.846 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.846 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.849 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.849 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.850 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.850 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:45:54.859 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.860 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:45:54.862 INFO 2981023 --- [ocp-schedules-10,a9474dd31a865613,856aabef9cd46cc1] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:45:54.865 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.865 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.866 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:45:54.866 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3311D73C-0-0] 2024-09-11 17:45:54.866 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:45:54.866 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.867 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:45:54.867 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.867 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.867 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:45:54.867 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.872 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.873 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:45:54.880 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.880 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:45:54.881 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.882 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:45:54.884 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.885 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:45:54.885 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.886 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:45:54.887 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:45:54.888 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.889 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:45:54.890 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.890 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:45:54.891 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.892 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:45:54.892 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.893 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.893 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:45:54.893 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:45:54.894 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:45:54.894 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3311D746-0-0] 2024-09-11 17:45:54.894 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.894 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:45:54.895 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:45:54.895 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:45:54.896 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.896 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:45:54.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:45:54.898 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.899 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:45:54.899 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.900 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:45:54.900 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:45:54.901 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.902 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:45:54.903 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:45:54.903 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.903 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:45:54.904 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.905 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:45:54.905 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.906 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:45:54.907 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.908 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:45:54.908 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:54.908 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.909 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:45:54.909 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:45:54.911 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.911 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.911 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:45:54.916 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:54.919 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.919 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:45:54.925 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:45:54.925 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:45:54.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:45:54.931 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:45:54.931 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:45:54.931 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:45:54.931 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:45:54.931 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.932 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:45:54.932 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:45:54.937 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:45:54.939 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:45:54.939 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.940 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.945 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:45:54.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:45:54.947 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:45:54.947 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:54.947 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:45:54.947 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:45:54.947 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:45:54.947 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:45:54.958 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:45:54.959 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:45:54.959 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:45:54.959 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:45:54.959 INFO 2981023 --- [ocp-schedules-20,44eccaf92f1c4446,55b6121975df465d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:45:55.276 INFO 2981023 --- [ocp-async-2,4755d7dfb0269e94,c7f4c84873116a65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:55.718 INFO 2981023 --- [ocp-schedules-4,285076bd3cda6c75,3561e89caa77512d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:45:55.718 INFO 2981023 --- [ocp-schedules-4,285076bd3cda6c75,3561e89caa77512d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:45:55.721 INFO 2981023 --- [ocp-schedules-4,285076bd3cda6c75,3561e89caa77512d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:45:55.721 INFO 2981023 --- [ocp-schedules-4,285076bd3cda6c75,3561e89caa77512d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:45:55.745 INFO 2981023 --- [ocp-schedules-4,285076bd3cda6c75,3561e89caa77512d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:45:55.745 INFO 2981023 --- [ocp-schedules-4,285076bd3cda6c75,3561e89caa77512d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:45:55.830 INFO 2981023 --- [ocp-schedules-30,a9373183b1fc729f,bb031825aa6bc364] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:45:55.841 INFO 2981023 --- [ocp-schedules-30,a9373183b1fc729f,bb031825aa6bc364] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:45:55.841 INFO 2981023 --- [ocp-schedules-30,a9373183b1fc729f,bb031825aa6bc364] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:45:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,f892db515a8fc96e,99a8101015122569] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,f892db515a8fc96e,99a8101015122569] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:45:56.351528+08:00, lastProcessTime=2024-09-11T17:45:46.350765+08:00 2024-09-11 17:45:56.417 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:56.417 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:45:56.483 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:45:56.351528+08:00 2024-09-11 17:45:56.579 INFO 2981023 --- [ocp-schedules-11,a857f6004b072bb0,3ae51e001a0de1f6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:45:56.579 INFO 2981023 --- [ocp-schedules-11,a857f6004b072bb0,3ae51e001a0de1f6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:45:56.593 INFO 2981023 --- [Collect all tenants major compactions1,a857f6004b072bb0,af2043735a426581] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:45:56.593 INFO 2981023 --- [Collect all tenants major compactions2,a857f6004b072bb0,62b6c7e2681f9071] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:45:56.594 INFO 2981023 --- [Collect all tenants major compactions4,a857f6004b072bb0,1dd1b3451383eced] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:45:56.594 INFO 2981023 --- [Collect all tenants major compactions6,a857f6004b072bb0,2564186014380820] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:45:56.594 INFO 2981023 --- [Collect all tenants major compactions3,a857f6004b072bb0,5d9e0f3d55bef241] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:45:56.596 INFO 2981023 --- [Collect all tenants major compactions5,a857f6004b072bb0,2a54cfa4190e3812] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:45:56.607 INFO 2981023 --- [Collect all tenants major compactions6,a857f6004b072bb0,2564186014380820] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:56.607 INFO 2981023 --- [Collect all tenants major compactions2,a857f6004b072bb0,62b6c7e2681f9071] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:56.608 INFO 2981023 --- [Collect all tenants major compactions6,a857f6004b072bb0,2564186014380820] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:45:56.609 INFO 2981023 --- [Collect all tenants major compactions2,a857f6004b072bb0,62b6c7e2681f9071] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:45:56.611 INFO 2981023 --- [Collect all tenants major compactions1,a857f6004b072bb0,af2043735a426581] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:56.611 INFO 2981023 --- [Collect all tenants major compactions3,a857f6004b072bb0,5d9e0f3d55bef241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:56.611 INFO 2981023 --- [Collect all tenants major compactions4,a857f6004b072bb0,1dd1b3451383eced] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:56.611 INFO 2981023 --- [Collect all tenants major compactions5,a857f6004b072bb0,2a54cfa4190e3812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:45:56.612 INFO 2981023 --- [Collect all tenants major compactions3,a857f6004b072bb0,5d9e0f3d55bef241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:45:56.612 INFO 2981023 --- [Collect all tenants major compactions4,a857f6004b072bb0,1dd1b3451383eced] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:45:56.612 INFO 2981023 --- [Collect all tenants major compactions1,a857f6004b072bb0,af2043735a426581] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:45:56.612 INFO 2981023 --- [Collect all tenants major compactions6,a857f6004b072bb0,2564186014380820] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:45:56.613 INFO 2981023 --- [Collect all tenants major compactions5,a857f6004b072bb0,2a54cfa4190e3812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:45:56.614 INFO 2981023 --- [Collect all tenants major compactions2,a857f6004b072bb0,62b6c7e2681f9071] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:45:56.614 INFO 2981023 --- [Collect all tenants major compactions1,a857f6004b072bb0,af2043735a426581] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:45:56.615 INFO 2981023 --- [Collect all tenants major compactions4,a857f6004b072bb0,1dd1b3451383eced] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:45:56.615 INFO 2981023 --- [Collect all tenants major compactions5,a857f6004b072bb0,2a54cfa4190e3812] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:45:56.615 INFO 2981023 --- [Collect all tenants major compactions6,a857f6004b072bb0,2564186014380820] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:45:56.617 INFO 2981023 --- [Collect all tenants major compactions3,a857f6004b072bb0,5d9e0f3d55bef241] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:45:56.617 INFO 2981023 --- [Collect all tenants major compactions3,a857f6004b072bb0,5d9e0f3d55bef241] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:45:56.619 INFO 2981023 --- [Collect all tenants major compactions5,a857f6004b072bb0,2a54cfa4190e3812] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:45:56.620 INFO 2981023 --- [Collect all tenants major compactions2,a857f6004b072bb0,62b6c7e2681f9071] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:45:56.621 INFO 2981023 --- [Collect all tenants major compactions4,a857f6004b072bb0,1dd1b3451383eced] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:45:56.622 INFO 2981023 --- [Collect all tenants major compactions1,a857f6004b072bb0,af2043735a426581] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:45:56.623 INFO 2981023 --- [ocp-schedules-11,a857f6004b072bb0,3ae51e001a0de1f6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:45:57.276 INFO 2981023 --- [ocp-async-10,8a4841ffb4eaed81,598c8bc8cc4e5685] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:57.760 INFO 2981023 --- [ocp-async-2,3060eaec90483671,07d0ab745fadd5dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:45:58.386 INFO 2981023 --- [ocp-schedules-21,5fbc33a6263272e5,5194353716aac04f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:45:58.431 WARN 2981023 --- [scanner-13542760-1,5fbc33a6263272e5,aae89c62c467533c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:45:58.463 INFO 2981023 --- [alarm-detect-1,5fbc33a6263272e5,a37f9cc5d25423da] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644599, data is empty. 2024-09-11 17:45:58.467 INFO 2981023 --- [alarm-detect-1,5fbc33a6263272e5,a37f9cc5d25423da] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644627, data is empty. 2024-09-11 17:45:58.473 INFO 2981023 --- [alarm-detect-1,5fbc33a6263272e5,a37f9cc5d25423da] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644627, data is empty. 2024-09-11 17:45:58.491 INFO 2981023 --- [alarm-detect-3,5fbc33a6263272e5,4ff33de727ae584e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644599, data is empty. 2024-09-11 17:45:58.499 INFO 2981023 --- [alarm-detect-5,5fbc33a6263272e5,8497cab7a10c2a94] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644599, data is empty. 2024-09-11 17:45:58.500 INFO 2981023 --- [alarm-detect-3,5fbc33a6263272e5,4ff33de727ae584e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644627, data is empty. 2024-09-11 17:45:58.505 INFO 2981023 --- [alarm-detect-3,5fbc33a6263272e5,4ff33de727ae584e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644627, data is empty. 2024-09-11 17:45:58.507 INFO 2981023 --- [alarm-detect-5,5fbc33a6263272e5,8497cab7a10c2a94] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644627, data is empty. 2024-09-11 17:45:58.513 INFO 2981023 --- [alarm-detect-5,5fbc33a6263272e5,8497cab7a10c2a94] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644627, data is empty. 2024-09-11 17:45:58.612 INFO 2981023 --- [alarm-detect-1,5fbc33a6263272e5,e596c4a865108ece] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644599, data is empty. 2024-09-11 17:45:58.617 INFO 2981023 --- [alarm-detect-1,5fbc33a6263272e5,e596c4a865108ece] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644627, data is empty. 2024-09-11 17:45:58.621 INFO 2981023 --- [alarm-detect-1,5fbc33a6263272e5,e596c4a865108ece] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644627, data is empty. 2024-09-11 17:45:59.276 INFO 2981023 --- [ocp-async-1,19e1545a7783e01d,0efea6e01509a581] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:45:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,b02bb38b9b2b6b3e,26e8db7123403786] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726047960, interval=60 2024-09-11 17:45:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,b02bb38b9b2b6b3e,26e8db7123403786] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:45:59.417 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:45:59.418 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:01.285 INFO 2981023 --- [ocp-async-4,390fbdb6c1042979,32b9022404d3cdd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:02.073 INFO 2981023 --- [ocp-schedules-20,f68302997d4116d8,fc1324afb2a35e52] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:46:02.073 INFO 2981023 --- [ocp-schedules-20,f68302997d4116d8,fc1324afb2a35e52] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:46:02.138 INFO 2981023 --- [ocp-schedules-24,1f3ade4891af7a1f,4efa0492bf07bc7f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:46:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,82a8b70322d7c4c2,6b3ff43702d3f048] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,82a8b70322d7c4c2,6b3ff43702d3f048] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:02.418 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:02.418 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:02.502 INFO 2981023 --- [ocp-schedules-10,ffd04d18f5c921c8,eef7cca231ce884d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:46:02.503 INFO 2981023 --- [ocp-schedules-10,ffd04d18f5c921c8,eef7cca231ce884d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:02.514 INFO 2981023 --- [ocp-schedules-10,ffd04d18f5c921c8,eef7cca231ce884d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:46:03.276 INFO 2981023 --- [ocp-async-4,8c98e7d5474dedca,891481671745e4b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:05.277 INFO 2981023 --- [ocp-async-8,585a5c7631055dc0,2a3099c8d0fa7b1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,42392cd28232793c,756f529dad395e9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,42392cd28232793c,756f529dad395e9f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:05.418 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:05.419 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:05.984 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:05.984 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:46:05.985 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:05.985 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:46:05.986 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:46:05.997 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:46:06.002 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:46:06.007 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:46:06.012 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:46:06.012 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:46:06.012 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:46:06.014 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:06.014 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:06.015 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:46:06.015 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:46:06.015 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:06.016 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:06.016 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:46:06.016 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:46:06.053 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:43:40, start=2024-09-11T17:40:10, end=2024-09-11T17:42:40, updates=134, granule=120000ms, delay=210000ms 2024-09-11 17:46:06.059 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:43:40, target=2024-09-11T17:42:40, granule=120000ms, delay=60000ms) 2024-09-11 17:46:06.059 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:42:40, target=2024-09-11T17:29:31, granule=600000ms, delay=789000ms) 2024-09-11 17:46:06.152 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:43:41, start=2024-09-11T17:40:11, end=2024-09-11T17:42:41, updates=314, granule=120000ms, delay=210000ms 2024-09-11 17:46:06.157 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:43:41, target=2024-09-11T17:42:41, granule=120000ms, delay=60000ms) 2024-09-11 17:46:06.157 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:42:41, target=2024-09-11T17:30:51, granule=600000ms, delay=710000ms) 2024-09-11 17:46:06.157 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=171ms, period=120000ms 2024-09-11 17:46:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:46:06.337589+08:00, lastProcessTime=2024-09-11T17:45:56.351528+08:00 2024-09-11 17:46:06.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:46:06.337589+08:00 2024-09-11 17:46:07.277 INFO 2981023 --- [ocp-async-6,462583d4a7a76298,21e09233a0e2ea99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:07.480 INFO 2981023 --- [ocp-schedules-9,ac0805b168a6a163,1c39417174b56e29] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:46:07.503 INFO 2981023 --- [ocp-schedules-12,01db83a9073e09d9,f114106a983bbbdb] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:46:07.503+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:46:07.504 INFO 2981023 --- [ocp-schedules-12,01db83a9073e09d9,f114106a983bbbdb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:46:07.505 INFO 2981023 --- [ocp-schedules-12,01db83a9073e09d9,f114106a983bbbdb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:46:07.506 INFO 2981023 --- [ocp-schedules-12,01db83a9073e09d9,f114106a983bbbdb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:46:07.760 INFO 2981023 --- [ocp-async-3,d6e9a42cb0b5fab6,7bb50e8c97280a57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:46:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,11cffe772f9ae23c,ee007489bbda98dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,11cffe772f9ae23c,ee007489bbda98dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:08.419 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:08.419 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:08.659 INFO 2981023 --- [ocp-schedules-6,dcd53805b2240e96,47b2b4bad82fe37d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:46:08.703 WARN 2981023 --- [scanner-13543141-1,dcd53805b2240e96,6e5de70e978b58c1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:46:09.277 INFO 2981023 --- [ocp-async-7,5283a4caf1e7d04a,bfc43c8f136978d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:10.002 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:46:10.002 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:46:10.918 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:46:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:46:11.277 INFO 2981023 --- [ocp-async-3,c22b2cc64aad6e8b,ba91e2f2b0e54534] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,92529caa62206cd1,550ac2a70cfab5c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,92529caa62206cd1,550ac2a70cfab5c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:11.419 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:11.420 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:12.141 INFO 2981023 --- [ocp-schedules-28,befa0359b987d77d,c564a47f2b56c3f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:46:13.117 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047900, nodeCount=463, affectRows=463 2024-09-11 17:46:13.277 INFO 2981023 --- [ocp-async-1,a4abd115d29619ff,21c366cde7970018] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:13.714 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047900, nodeCount=848, affectRows=848 2024-09-11 17:46:13.716 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047900, nodeCount=847, validCount=847 2024-09-11 17:46:13.722 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047900, nodeCount=958, affectRows=958 2024-09-11 17:46:13.724 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047900, nodeCount=958, validCount=958 2024-09-11 17:46:14.303 INFO 2981023 --- [ocp-schedules-14,d55782e9e2981518,0d13620a2d0f4965] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:46:14.308 INFO 2981023 --- [ocp-schedules-14,d55782e9e2981518,0d13620a2d0f4965] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:46:14.308 INFO 2981023 --- [ocp-schedules-14,d55782e9e2981518,0d13620a2d0f4965] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:46:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,0f1f9ee7bcd04224,6af34a6af0a83bf3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,981d27bed8e50914,27af8f86a7d8fa06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:46:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,0f1f9ee7bcd04224,6af34a6af0a83bf3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:14.420 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:14.420 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:14.909 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047900, nodeCount=463, validCount=463 2024-09-11 17:46:15.277 INFO 2981023 --- [ocp-async-5,689dce01f0bd19ba,d613e62a60656e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:15.998 INFO 2981023 --- [ocp-schedules-20,ebf133cc770346b6,c3cad853e27d49e6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:46:15.998 INFO 2981023 --- [host-check-clockdiff1,ebf133cc770346b6,0bcd913217498bd0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:46:15.998 INFO 2981023 --- [host-check-clockdiff2,ebf133cc770346b6,ae74e814a2ef442b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:46:15.999 INFO 2981023 --- [host-check-clockdiff3,ebf133cc770346b6,85c9136d862d14f1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:46:16.005 INFO 2981023 --- [host-check-clockdiff3,ebf133cc770346b6,85c9136d862d14f1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047976 0 0 , error= 2024-09-11 17:46:16.009 INFO 2981023 --- [host-check-clockdiff2,ebf133cc770346b6,ae74e814a2ef442b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047976 0 0 , error= 2024-09-11 17:46:16.008 INFO 2981023 --- [ocp-schedules-22,2d9328ff9627e59f,4d17dc5e18aecc5c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:46:16.010 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:46:16.010 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:46:16.010 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:46:16.012 INFO 2981023 --- [ocp-schedules-8,ba53a12d7c650c12,ffe9ce7d36fb471f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:46:16.013 INFO 2981023 --- [host-check-agent-status1,ba53a12d7c650c12,6ae1a7040a894cfa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:46:16.013 INFO 2981023 --- [host-check-agent-status2,ba53a12d7c650c12,bfcc1edfad0378d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:46:16.015 INFO 2981023 --- [host-check-agent-status3,ba53a12d7c650c12,e6dbb89340803385] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:46:16.015 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:46:16.017 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:46:16.017 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:46:16.020 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:46:16.020 INFO 2981023 --- [host-check-agent-status1,ba53a12d7c650c12,6ae1a7040a894cfa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:46:16.021 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:46:16.021 INFO 2981023 --- [host-check-agent-status1,ba53a12d7c650c12,6ae1a7040a894cfa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:46:16.022 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:46:16.024 INFO 2981023 --- [host-check-agent-status2,ba53a12d7c650c12,bfcc1edfad0378d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:46:16.025 INFO 2981023 --- [host-check-agent-status3,ba53a12d7c650c12,e6dbb89340803385] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:46:16.026 INFO 2981023 --- [host-check-agent-status2,ba53a12d7c650c12,bfcc1edfad0378d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:46:16.026 INFO 2981023 --- [host-check-agent-status3,ba53a12d7c650c12,e6dbb89340803385] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:46:16.044 INFO 2981023 --- [host-check-agent-status1,ba53a12d7c650c12,6ae1a7040a894cfa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:46:16.047 INFO 2981023 --- [host-check-agent-status2,ba53a12d7c650c12,bfcc1edfad0378d7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:46:16.048 INFO 2981023 --- [host-check-agent-status1,ba53a12d7c650c12,6ae1a7040a894cfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:46:16.048 INFO 2981023 --- [host-check-agent-status1,ba53a12d7c650c12,6ae1a7040a894cfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:46:16.048 INFO 2981023 --- [host-check-agent-status1,ba53a12d7c650c12,6ae1a7040a894cfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:46:16.053 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:46:16.054 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:46:16.056 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:46:16.055915840+08:00 2024-09-11 17:46:16.056 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:46:16.055588924+08:00 2024-09-11 17:46:16.056 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:46:16.056+08:00 2024-09-11 17:46:16.056 INFO 2981023 --- [host-check-agent-status3,ba53a12d7c650c12,e6dbb89340803385] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:46:16.058 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:46:16.059 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:46:16.056+08:00 and status AVAILABLE 2024-09-11 17:46:16.059 INFO 2981023 --- [host-check-agent-status2,ba53a12d7c650c12,bfcc1edfad0378d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:46:16.059 INFO 2981023 --- [host-check-agent-status2,ba53a12d7c650c12,bfcc1edfad0378d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:46:16.059 INFO 2981023 --- [host-check-agent-status2,ba53a12d7c650c12,bfcc1edfad0378d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:46:16.059 INFO 2981023 --- [host-check-agent-status3,ba53a12d7c650c12,e6dbb89340803385] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:46:16.059 INFO 2981023 --- [host-check-agent-status3,ba53a12d7c650c12,e6dbb89340803385] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:46:16.059 INFO 2981023 --- [host-check-agent-status3,ba53a12d7c650c12,e6dbb89340803385] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:46:16.059 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:46:16.059106799+08:00 2024-09-11 17:46:16.060 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:46:16.056+08:00 2024-09-11 17:46:16.060 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:46:16.059+08:00 2024-09-11 17:46:16.061 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:46:16.056+08:00 and status AVAILABLE 2024-09-11 17:46:16.062 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:46:16.059+08:00 and status AVAILABLE 2024-09-11 17:46:16.074 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:46:16.077 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:46:16.078 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:46:16.102 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:46:16.103 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:46:16.103 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:46:16.104 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:46:16.119 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:46:16.120 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:46:16.140 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:46:16.141 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:46:16.155 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:46:16.159 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:46:16.170 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:46:16.187 INFO 2981023 --- [ocp-schedules-3,dc9509c58a5da220,3b78c887ec741165] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:46:16.189 INFO 2981023 --- [ocp-schedules-3,dc9509c58a5da220,3b78c887ec741165] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:16.189 INFO 2981023 --- [ocp-schedules-16,15820b48a45d1a2c,20e8ec470472de3d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:46:16.189 INFO 2981023 --- [ocp-schedules-30,5ad97c8aec3d1ca5,9c5684a5c1905445] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:46:16.190 INFO 2981023 --- [ocp-schedules-16,15820b48a45d1a2c,20e8ec470472de3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:16.191 INFO 2981023 --- [ocp-schedules-30,5ad97c8aec3d1ca5,9c5684a5c1905445] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:16.192 INFO 2981023 --- [ocp-schedules-12,4a1af8a5656258f4,e2a925436c81cffc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:46:16.195 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:46:16.211 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:46:16.213 INFO 2981023 --- [ocp-schedules-25,f722137d66ee4995,7e1b618a1ca895c5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:46:16.218 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.220 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:46:16.221 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:16.223 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:46:16.225 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:46:16.229 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:46:16.234 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:46:16.241 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.242 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:16.242 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.243 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:46:16.243 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:16.244 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:46:16.250 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:46:16.252 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:46:16.264 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:46:16.267 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:46:16.273 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:46:16.276 INFO 2981023 --- [ocp-schedules-3,dc9509c58a5da220,3b78c887ec741165] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:46:16.278 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.279 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:16.285 INFO 2981023 --- [ocp-schedules-16,15820b48a45d1a2c,20e8ec470472de3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:46:16.295 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.296 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:16.306 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:46:16.311 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.314 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.314 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:46:16.314 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:46:16.315 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.316 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:16.326 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:46:16.336 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:46:16.336 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:46:16.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:46:16.331430+08:00, lastProcessTime=2024-09-11T17:46:06.337589+08:00 2024-09-11 17:46:16.340 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:46:16.348 INFO 2981023 --- [host-check-host-status1,2d9328ff9627e59f,a08c08cbd2dfb40d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:46:16.358 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:46:16.358 INFO 2981023 --- [host-check-host-status2,2d9328ff9627e59f,1761234e5b54f4f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:46:16.360 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.361 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.361 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:16.362 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:46:16.362 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.365 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:46:16.368 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:46:16.380 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.380 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.381 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:46:16.382 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:16.382 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:46:16.387 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:46:16.394 INFO 2981023 --- [host-check-host-status3,2d9328ff9627e59f,75885358f655ae00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:46:16.403 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.404 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.404 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:16.405 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.408 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:46:16.409 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:46:16.412 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:16.420 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.420 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:16.420 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:46:16.421 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.421 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:16.422 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:46:16.426 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:46:16.434 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:46:16.437 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:46:16.438 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:16.447 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.448 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.448 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:16.449 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:46:16.452 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:46:16.453 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:46:16.453 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.454 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.454 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:46:16.457 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:46:16.458 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:46:16.481 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.481 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:46:16.481 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.482 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:16.482 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:46:16.482 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.485 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:46:16.485 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:46:16.496 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:16.497 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:46:16.509 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:16.516 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:46:16.517 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.518 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.518 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:16.519 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:46:16.522 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:46:16.522 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.523 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.523 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:46:16.525 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:46:16.527 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:46:16.530 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:46:16.530 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:46:16.541 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:46:16.548 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:46:16.555 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.555 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.556 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:46:16.562 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:46:16.562 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:46:16.569 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:46:16.331430+08:00 2024-09-11 17:46:16.571 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:46:16.576 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:16.576 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:16.577 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:46:16.583 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:46:16.583 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:46:16.583 INFO 2981023 --- [ocp-schedules-30,5ad97c8aec3d1ca5,9c5684a5c1905445] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:46:17.277 INFO 2981023 --- [ocp-async-4,2a5509619f600e2d,616875ab35624ecf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,7f7dec253a27fdc8,e9e1fab3fd34c1f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,7f7dec253a27fdc8,e9e1fab3fd34c1f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:17.420 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:17.421 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:17.760 INFO 2981023 --- [ocp-async-5,095936b48b83859d,cea8f428b569a514] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:46:18.527 INFO 2981023 --- [host-check-clockdiff1,ebf133cc770346b6,0bcd913217498bd0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726047978 0 0 , error= 2024-09-11 17:46:18.845 INFO 2981023 --- [ocp-schedules-13,6c4c7f01da656848,ddb8f554e9211cc2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:46:18.894 WARN 2981023 --- [scanner-13544827-1,6c4c7f01da656848,a824460d6fd4a510] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:46:18.915 INFO 2981023 --- [alarm-detect-8,6c4c7f01da656848,a53659ccd51e8102] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644627, data is empty. 2024-09-11 17:46:18.925 INFO 2981023 --- [alarm-detect-8,6c4c7f01da656848,a53659ccd51e8102] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644627, data is empty. 2024-09-11 17:46:18.929 INFO 2981023 --- [alarm-detect-8,6c4c7f01da656848,a53659ccd51e8102] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644627, data is empty. 2024-09-11 17:46:18.948 INFO 2981023 --- [alarm-detect-6,6c4c7f01da656848,27fc4d79dc8d084e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644627, data is empty. 2024-09-11 17:46:18.951 INFO 2981023 --- [alarm-detect-1,6c4c7f01da656848,5dc42b9fcb746ff4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644627, data is empty. 2024-09-11 17:46:18.956 INFO 2981023 --- [alarm-detect-1,6c4c7f01da656848,5dc42b9fcb746ff4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644627, data is empty. 2024-09-11 17:46:18.961 INFO 2981023 --- [alarm-detect-6,6c4c7f01da656848,27fc4d79dc8d084e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644627, data is empty. 2024-09-11 17:46:18.962 INFO 2981023 --- [alarm-detect-1,6c4c7f01da656848,5dc42b9fcb746ff4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644627, data is empty. 2024-09-11 17:46:18.970 INFO 2981023 --- [alarm-detect-6,6c4c7f01da656848,27fc4d79dc8d084e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644627, data is empty. 2024-09-11 17:46:19.059 INFO 2981023 --- [alarm-detect-14,6c4c7f01da656848,542f8257b1ca1751] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644627, data is empty. 2024-09-11 17:46:19.064 INFO 2981023 --- [alarm-detect-14,6c4c7f01da656848,542f8257b1ca1751] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644627, data is empty. 2024-09-11 17:46:19.073 INFO 2981023 --- [alarm-detect-14,6c4c7f01da656848,542f8257b1ca1751] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644627, data is empty. 2024-09-11 17:46:19.277 INFO 2981023 --- [ocp-async-1,96420a61758e8a48,aefbd0576aa6970c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,32565270ce18b156,ed99d26b14b058ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,32565270ce18b156,ed99d26b14b058ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:20.421 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:20.422 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:21.278 INFO 2981023 --- [ocp-async-5,eb76644c5844c2a7,e00d65efe2ac5e43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:22.146 INFO 2981023 --- [ocp-schedules-19,29c4161312147aa7,d77296277c39c012] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:46:23.116 INFO 2981023 --- [ocp-schedules-16,14bf14ba99b7af89,4aedde5457a66e09] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:46:23.116 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:46:23.116 INFO 2981023 --- [ocp-schedules-16,14bf14ba99b7af89,4aedde5457a66e09] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:46:23.117 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:46:23.120 INFO 2981023 --- [ocp-schedules-16,14bf14ba99b7af89,4aedde5457a66e09] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:46:23.120 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:46:23.121 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:46:23.124 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:46:23.124 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:46:23.124 INFO 2981023 --- [ocp-schedules-16,14bf14ba99b7af89,4aedde5457a66e09] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:46:23.149 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:46:23.149 INFO 2981023 --- [ocp-schedules-16,14bf14ba99b7af89,4aedde5457a66e09] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:46:23.149 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:46:23.151 INFO 2981023 --- [ocp-schedules-16,14bf14ba99b7af89,4aedde5457a66e09] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:46:23.278 INFO 2981023 --- [ocp-async-4,aa4cb7cde35d9afd,ae32a19e98aae423] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,fa8eaee9cc7208d8,940d0d1d5e487585] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,fa8eaee9cc7208d8,940d0d1d5e487585] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:23.422 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:23.424 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:24.280 INFO 2981023 --- [ocp-schedules-4,86fff9fd608e2146,b5c77adf63da053c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:46:25.278 INFO 2981023 --- [ocp-async-2,d08aed2056ddedae,8d7ffd523827f879] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,f4aeff573ffe5775,7330fb7ffe7a3cac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,f4aeff573ffe5775,7330fb7ffe7a3cac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:46:26.352015+08:00, lastProcessTime=2024-09-11T17:46:16.331430+08:00 2024-09-11 17:46:26.424 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:26.425 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:26.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:46:26.352015+08:00 2024-09-11 17:46:27.278 INFO 2981023 --- [ocp-async-6,c088018a9cbe8ff6,3cee9e22564990a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:27.760 INFO 2981023 --- [ocp-async-2,fd9dce33aad0ee93,44739c4f1f532eb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:46:29.108 INFO 2981023 --- [ocp-schedules-17,7bd28dc8d84911cc,6daafbf105cf8acd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:46:29.156 WARN 2981023 --- [scanner-13545116-1,7bd28dc8d84911cc,8b0f0eb7192f8aa6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:46:29.279 INFO 2981023 --- [ocp-async-4,ece71878dc168695,ec6a11fa69a2fe87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,97c04d76d207abab,741710cb2bbd45ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:29.315 INFO 2981023 --- [prometheus-minute-schedule-1,97c04d76d207abab,741710cb2bbd45ac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:29.425 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:29.425 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:31.279 INFO 2981023 --- [ocp-async-2,c4e3c79a351e7b10,a648fff46612b868] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:32.149 INFO 2981023 --- [ocp-schedules-17,aa0ce50842eaae3a,a4b408ee78246987] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:46:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,b1da2fc207357c52,0f51e4629af2ac7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,b1da2fc207357c52,0f51e4629af2ac7a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:32.426 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:32.426 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:33.279 INFO 2981023 --- [ocp-async-10,074229a67e52b3c1,f9ae2532d538ec36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:34.312 INFO 2981023 --- [ocp-schedules-8,28380dedd96c2fa9,dcbec319a26d32b1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:46:34.320 INFO 2981023 --- [ocp-schedules-8,28380dedd96c2fa9,dcbec319a26d32b1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:46:34.320 INFO 2981023 --- [ocp-schedules-8,28380dedd96c2fa9,dcbec319a26d32b1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:46:35.279 INFO 2981023 --- [ocp-async-9,f26421d79d6a0812,f90b66919a71dff0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,1c1317f632bd0495,fc2d44880d341642] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,1c1317f632bd0495,fc2d44880d341642] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:35.426 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:35.426 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:46:36.337774+08:00, lastProcessTime=2024-09-11T17:46:26.352015+08:00 2024-09-11 17:46:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:46:36.337774+08:00 2024-09-11 17:46:37.279 INFO 2981023 --- [ocp-async-3,45014325dd0d25ad,641c0dce6f995ca8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:37.314 INFO 2981023 --- [prometheus-cache-monitor1,821a916dca84eec8,bb40e4d562b58773] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7294,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:46:37.315 INFO 2981023 --- [prometheus-cache-monitor1,821a916dca84eec8,bb40e4d562b58773] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:46:37.761 INFO 2981023 --- [ocp-async-9,0936aa8da06ab028,c8a4478c4fa6c4eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:46:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,286994a59eee227b,96250895bad64248] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,286994a59eee227b,96250895bad64248] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:38.427 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:38.427 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:39.279 INFO 2981023 --- [ocp-async-10,996d394374685881,807f3f7b1ae8f4cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:39.292 INFO 2981023 --- [ocp-schedules-20,dd23666e11c4ecd0,188077f4431b80b8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:46:39.338 WARN 2981023 --- [scanner-13545324-1,dd23666e11c4ecd0,f4000c50eac1edb8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:46:39.347 INFO 2981023 --- [alarm-detect-15,dd23666e11c4ecd0,bc271e38bd7b22af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644627, data is empty. 2024-09-11 17:46:39.357 INFO 2981023 --- [alarm-detect-15,dd23666e11c4ecd0,bc271e38bd7b22af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644627, data is empty. 2024-09-11 17:46:39.361 INFO 2981023 --- [alarm-detect-15,dd23666e11c4ecd0,bc271e38bd7b22af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644627, data is empty. 2024-09-11 17:46:39.381 INFO 2981023 --- [alarm-detect-7,dd23666e11c4ecd0,fd87ba043c9874e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644627, data is empty. 2024-09-11 17:46:39.390 INFO 2981023 --- [alarm-detect-7,dd23666e11c4ecd0,fd87ba043c9874e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644627, data is empty. 2024-09-11 17:46:39.393 INFO 2981023 --- [alarm-detect-12,dd23666e11c4ecd0,48b76b451d8e6900] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644627, data is empty. 2024-09-11 17:46:39.395 INFO 2981023 --- [alarm-detect-7,dd23666e11c4ecd0,fd87ba043c9874e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644627, data is empty. 2024-09-11 17:46:39.397 INFO 2981023 --- [alarm-detect-12,dd23666e11c4ecd0,48b76b451d8e6900] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644627, data is empty. 2024-09-11 17:46:39.404 INFO 2981023 --- [alarm-detect-12,dd23666e11c4ecd0,48b76b451d8e6900] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644627, data is empty. 2024-09-11 17:46:39.496 INFO 2981023 --- [alarm-detect-13,dd23666e11c4ecd0,024c6fdcfc4f54f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644627, data is empty. 2024-09-11 17:46:39.500 INFO 2981023 --- [alarm-detect-13,dd23666e11c4ecd0,024c6fdcfc4f54f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644627, data is empty. 2024-09-11 17:46:39.504 INFO 2981023 --- [alarm-detect-13,dd23666e11c4ecd0,024c6fdcfc4f54f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644627, data is empty. 2024-09-11 17:46:40.005 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:46:40.005 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:46:41.223 INFO 2981023 --- [ocp-schedules-21,4ffabbf9334188d2,228a673daf144a01] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:46:41.241 WARN 2981023 --- [ocp-schedules-21,4ffabbf9334188d2,228a673daf144a01] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:46:41.279 INFO 2981023 --- [ocp-async-7,1002425ae2ca8c99,f06c62af55b11237] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,0facbc01558177b3,0d26e87898a7bedd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,0facbc01558177b3,0d26e87898a7bedd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:41.427 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:41.428 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:42.152 INFO 2981023 --- [ocp-schedules-7,cc2d74b07cc144e4,1f965bd2536a1cd3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:46:43.279 INFO 2981023 --- [ocp-async-8,b259be877c083afc,dfd2791ac1183408] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,ad3f40bf5957ce2a,f0be99b894f4bcd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,ad3f40bf5957ce2a,f0be99b894f4bcd8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:44.428 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:44.428 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:45.101 INFO 2981023 --- [prometheus-second-schedule-1,0274809a5e82a91a,f454b1b0bf620fcc] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:46:45.279 INFO 2981023 --- [ocp-async-6,72048e8753b5dfb1,733813e16f6dff25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:46.016 INFO 2981023 --- [ocp-schedules-3,c595315fdcdda84d,3bbbe9128c34b28e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:46:46.016 INFO 2981023 --- [host-check-clockdiff1,c595315fdcdda84d,294ab18945a7315d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:46:46.017 INFO 2981023 --- [host-check-clockdiff2,c595315fdcdda84d,f375f33a295b7cca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:46:46.017 INFO 2981023 --- [host-check-clockdiff3,c595315fdcdda84d,30f535fd36f555fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:46:46.023 INFO 2981023 --- [ocp-schedules-21,53c274b313f0c7c7,c5d74725fbbe4883] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:46:46.026 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:46:46.026 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:46:46.026 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:46:46.030 INFO 2981023 --- [ocp-schedules-2,12be500a4a8f76be,cce199ce719a7700] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:46:46.031 INFO 2981023 --- [host-check-agent-status1,12be500a4a8f76be,03871331815ce49c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:46:46.034 INFO 2981023 --- [host-check-agent-status2,12be500a4a8f76be,d17ed828fe8cf36b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:46:46.034 INFO 2981023 --- [host-check-clockdiff2,c595315fdcdda84d,f375f33a295b7cca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048006 0 0 , error= 2024-09-11 17:46:46.035 INFO 2981023 --- [host-check-agent-status3,12be500a4a8f76be,1aa023c7636067c6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:46:46.036 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:46:46.036 INFO 2981023 --- [host-check-clockdiff3,c595315fdcdda84d,30f535fd36f555fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048006 0 0 , error= 2024-09-11 17:46:46.038 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:46:46.038 INFO 2981023 --- [host-check-clockdiff1,c595315fdcdda84d,294ab18945a7315d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048006 0 0 , error= 2024-09-11 17:46:46.040 INFO 2981023 --- [host-check-agent-status1,12be500a4a8f76be,03871331815ce49c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:46:46.041 INFO 2981023 --- [host-check-agent-status3,12be500a4a8f76be,1aa023c7636067c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:46:46.041 INFO 2981023 --- [host-check-agent-status1,12be500a4a8f76be,03871331815ce49c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:46:46.041 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:46:46.042 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:46:46.042 INFO 2981023 --- [host-check-agent-status3,12be500a4a8f76be,1aa023c7636067c6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:46:46.043 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:46:46.044 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:46:46.044 INFO 2981023 --- [host-check-agent-status2,12be500a4a8f76be,d17ed828fe8cf36b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:46:46.046 INFO 2981023 --- [host-check-agent-status2,12be500a4a8f76be,d17ed828fe8cf36b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:46:46.073 INFO 2981023 --- [host-check-agent-status2,12be500a4a8f76be,d17ed828fe8cf36b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:46:46.074 INFO 2981023 --- [host-check-agent-status3,12be500a4a8f76be,1aa023c7636067c6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:46:46.074 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:46:46.076 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:46:46.076039108+08:00 2024-09-11 17:46:46.077 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:46:46.077 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:46:46.076+08:00 2024-09-11 17:46:46.078 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:46:46.077832526+08:00 2024-09-11 17:46:46.078 INFO 2981023 --- [host-check-agent-status2,12be500a4a8f76be,d17ed828fe8cf36b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:46:46.078 INFO 2981023 --- [host-check-agent-status2,12be500a4a8f76be,d17ed828fe8cf36b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:46:46.078 INFO 2981023 --- [host-check-agent-status2,12be500a4a8f76be,d17ed828fe8cf36b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:46:46.078 INFO 2981023 --- [host-check-agent-status3,12be500a4a8f76be,1aa023c7636067c6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:46:46.078 INFO 2981023 --- [host-check-agent-status3,12be500a4a8f76be,1aa023c7636067c6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:46:46.078 INFO 2981023 --- [host-check-agent-status3,12be500a4a8f76be,1aa023c7636067c6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:46:46.079 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:46:46.078+08:00 2024-09-11 17:46:46.079 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:46:46.076+08:00 and status AVAILABLE 2024-09-11 17:46:46.080 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:46:46.081 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:46:46.078+08:00 and status AVAILABLE 2024-09-11 17:46:46.082 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:46:46.081982362+08:00 2024-09-11 17:46:46.090 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:46:46.082+08:00 2024-09-11 17:46:46.092 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:46:46.082+08:00 and status AVAILABLE 2024-09-11 17:46:46.093 INFO 2981023 --- [host-check-agent-status1,12be500a4a8f76be,03871331815ce49c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:46:46.095 INFO 2981023 --- [host-check-agent-status1,12be500a4a8f76be,03871331815ce49c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:46:46.095 INFO 2981023 --- [host-check-agent-status1,12be500a4a8f76be,03871331815ce49c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:46:46.095 INFO 2981023 --- [host-check-agent-status1,12be500a4a8f76be,03871331815ce49c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:46:46.097 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:46:46.103 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:46:46.120 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:46:46.126 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:46:46.128 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:46:46.129 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:46:46.130 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:46:46.131 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:46:46.132 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:46:46.158 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:46:46.160 INFO 2981023 --- [ocp-schedules-22,be3efde60c93f2e6,4d524d4837453468] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:46:46.162 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:46:46.163 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:46:46.164 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:46:46.168 INFO 2981023 --- [ocp-schedules-22,be3efde60c93f2e6,4d524d4837453468] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:46:46.187 INFO 2981023 --- [ocp-schedules-5,a8edcc6621ab22fb,fbce56ab79f3dc75] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:46:46.187 INFO 2981023 --- [ocp-schedules-10,3852cc694a0f5532,b1665e0276934fed] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:46:46.188 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:46:46.188 INFO 2981023 --- [ocp-schedules-10,3852cc694a0f5532,b1665e0276934fed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:46.189 INFO 2981023 --- [ocp-schedules-5,a8edcc6621ab22fb,fbce56ab79f3dc75] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:46.196 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:46:46.201 INFO 2981023 --- [ocp-schedules-27,f9f82e42ea0c3937,a520d4cad5a61707] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:46:46.201 INFO 2981023 --- [ocp-schedules-6,be04b98990b4b09c,4fd13c005d161912] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:46:46.202 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:46:46.205 INFO 2981023 --- [ocp-schedules-27,f9f82e42ea0c3937,a520d4cad5a61707] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:46.208 INFO 2981023 --- [ocp-schedules-5,a8edcc6621ab22fb,fbce56ab79f3dc75] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:46:46.234 INFO 2981023 --- [ocp-schedules-15,fa2a430e63544832,bb6926807cd29b41] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:46:46.234 INFO 2981023 --- [ocp-schedules-15,fa2a430e63544832,bb6926807cd29b41] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:46:46.241 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:46:46.242 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:46:46.242 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:46:46.256 INFO 2981023 --- [ocp-schedules-19,1f3777027d8309ba,c40ed98e59b931e6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:46:46.268 INFO 2981023 --- [ocp-schedules-14,c5ed801dad04b46a,d4a799963c0cf2c4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:46:46.271 INFO 2981023 --- [ocp-schedules-14,c5ed801dad04b46a,d4a799963c0cf2c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:46.273 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:46:46.295 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.300 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:46.308 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.309 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:46.310 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:46:46.317 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.317 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:46.320 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:46:46.325 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.325 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:46:46.326 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:46.328 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:46:46.336 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.337 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:46.338 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:46:46.338 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.339 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:46:46.339 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.339 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.339 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:46:46.341 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:46.342 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:46:46.345 INFO 2981023 --- [ocp-schedules-10,3852cc694a0f5532,b1665e0276934fed] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:46:46.346 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:46:46.343330+08:00, lastProcessTime=2024-09-11T17:46:36.337774+08:00 2024-09-11 17:46:46.350 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.350 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:46:46.350 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,a647b02fcebc3a50,93541313eee9c798] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=a647b02fcebc3a50, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:46:46.350 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,a647b02fcebc3a50,93541313eee9c798] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=a647b02fcebc3a50, duration=1 ms] 2024-09-11 17:46:46.353 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.354 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:46:46.354 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:46:46.355 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.355 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:46:46.355 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:46:46.362 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,73ec9103526aa4e3,776041e15d2d84ad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=73ec9103526aa4e3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:46:46.362 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,73ec9103526aa4e3,776041e15d2d84ad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=73ec9103526aa4e3, duration=0 ms] 2024-09-11 17:46:46.362 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.365 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:46:46.366 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.366 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:46:46.379 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:46:46.395 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:46:46.405 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.406 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.408 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:46:46.412 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:46:46.418 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:46:46.424 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:46:46.430 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d3fd6b03891801c9,61768cf42a31a432] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&ObClusterId=1719994443&version=2, client=10.31.35.10, traceId=d3fd6b03891801c9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkrtd,1719994443,false,2,] 2024-09-11 17:46:46.431 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d3fd6b03891801c9,61768cf42a31a432] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=d3fd6b03891801c9, duration=1 ms] 2024-09-11 17:46:46.432 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.433 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:46:46.437 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.438 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.438 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:46:46.445 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.447 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:46:46.448 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:46:46.452 INFO 2981023 --- [ocp-schedules-14,c5ed801dad04b46a,d4a799963c0cf2c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:46:46.453 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:46:46.473 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:46.484 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:46:46.485 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.489 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:46:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:46:46.492 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.493 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.496 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.496 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:46:46.496 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:46:46.497 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.497 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:46:46.500 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.503 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:46:46.509 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:46.509 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:46:46.509 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:46:46.524 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:46:46.528 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:46:46.533 INFO 2981023 --- [pool-10-thread-15,703d7a8cc48ef8c3,269333456a5d12b0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:46:46.533 INFO 2981023 --- [pool-10-thread-15,703d7a8cc48ef8c3,269333456a5d12b0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:46:46.533 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:46:46.539 INFO 2981023 --- [host-check-host-status3,53c274b313f0c7c7,79e537fd2b6616bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:46:46.541 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:46:46.549 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:46.553 INFO 2981023 --- [pool-10-thread-14,703d7a8cc48ef8c3,04b21ea0a28422e4] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:46:46.553 INFO 2981023 --- [pool-10-thread-16,703d7a8cc48ef8c3,4b00492cd5d163e8] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:46:46.556 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:46:46.564 INFO 2981023 --- [host-check-host-status2,53c274b313f0c7c7,994f25794cd01bd5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:46:46.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.567 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.569 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:46:46.572 INFO 2981023 --- [host-check-host-status1,53c274b313f0c7c7,adeea920ba967d3c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:46:46.573 INFO 2981023 --- [pool-10-thread-16,703d7a8cc48ef8c3,4b00492cd5d163e8] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:46:46.573 INFO 2981023 --- [pool-10-thread-16,703d7a8cc48ef8c3,4b00492cd5d163e8] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:46:46.576 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.577 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.577 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:46.582 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:46:46.587 INFO 2981023 --- [pool-10-thread-13,703d7a8cc48ef8c3,5a9a18477b26c0a5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:46:46.598 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:46:46.608 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:46:46.609 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.610 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.610 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:46.611 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:46:46.612 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.614 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:46:46.615 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.615 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:46:46.618 INFO 2981023 --- [ForkJoinPool-6-worker-12,703d7a8cc48ef8c3,fa79d2ee9f49b87c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:46:46.618 INFO 2981023 --- [ForkJoinPool-6-worker-12,703d7a8cc48ef8c3,fa79d2ee9f49b87c] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:46:46.620 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:46:46.620 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:46:46.629 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.631 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.632 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:46:46.633 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:46:46.633 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.639 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:46:46.640 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:46:46.642 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:46:46.642 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:46:46.656 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.656 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:46.656 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:46:46.657 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.657 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:46:46.657 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:46.657 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:46:46.658 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:46:46.659 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.659 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:46:46.659 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.661 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:46:46.662 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:46:46.662 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.664 INFO 2981023 --- [ForkJoinPool-6-worker-10,703d7a8cc48ef8c3,2a3f687faf95eccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.666 INFO 2981023 --- [ForkJoinPool-6-worker-10,703d7a8cc48ef8c3,2a3f687faf95eccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:46:46.669 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:46:46.674 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.675 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:46:46.676 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:46:46.677 WARN 2981023 --- [ForkJoinPool-6-worker-12,703d7a8cc48ef8c3,fa79d2ee9f49b87c] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:46:46.681 INFO 2981023 --- [ForkJoinPool-6-worker-10,703d7a8cc48ef8c3,2a3f687faf95eccc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:46:46.682 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:46:46.682 INFO 2981023 --- [ForkJoinPool-6-worker-10,703d7a8cc48ef8c3,2a3f687faf95eccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.684 INFO 2981023 --- [ForkJoinPool-6-worker-10,703d7a8cc48ef8c3,2a3f687faf95eccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:46:46.684 INFO 2981023 --- [ForkJoinPool-6-worker-10,703d7a8cc48ef8c3,2a3f687faf95eccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.686 INFO 2981023 --- [ForkJoinPool-6-worker-10,703d7a8cc48ef8c3,2a3f687faf95eccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:46:46.687 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:46.703 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:46:46.708 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.709 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:46:46.711 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.712 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:46.712 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:46:46.713 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:46:46.714 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:46:46.715 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.716 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:46:46.717 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.717 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:46:46.718 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:46:46.719 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.719 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:46:46.719 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:46:46.721 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.722 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.723 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:46:46.723 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:46:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:46:46.731 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:46:46.734 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:46:46.736 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.736 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:46:46.737 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:46:46.737 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:46:46.738 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.738 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:46:46.742 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:46:46.747 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:46:46.759 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:46:46.767 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:46:46.767 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.768 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.769 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:46:46.770 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:46:46, 2024-09-11 17:46:46] 2024-09-11 17:46:46.771 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:46:46, 2024-09-11 17:46:46] 2024-09-11 17:46:46.772 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.772 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:46:46.772 INFO 2981023 --- [ForkJoinPool-6-worker-5,703d7a8cc48ef8c3,5deb796ec7b113b5] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:46:46.772 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.773 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:46:46.775 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:46:46.775 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.778 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.780 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:46:46.786 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:46:46.787 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.788 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:46:46, 2024-09-11 17:46:46] 2024-09-11 17:46:46.789 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:46:46.789 INFO 2981023 --- [ForkJoinPool-6-worker-3,703d7a8cc48ef8c3,136761277d50320c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:46:46.790 INFO 2981023 --- [pool-10-thread-13,703d7a8cc48ef8c3,5a9a18477b26c0a5] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:46:46.798 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:46:46.799 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:46:46.809 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:46:46.810 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.811 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.811 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.812 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:46:46.814 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:46:46.819 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.820 INFO 2981023 --- [ForkJoinPool-6-worker-8,703d7a8cc48ef8c3,355a90e70a0c6754] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:46:46.822 INFO 2981023 --- [pool-10-thread-14,703d7a8cc48ef8c3,04b21ea0a28422e4] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:46:46.824 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:46:46.824 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:46:46.825 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:46:46.825 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:46:46.829 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.835 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:46:46.835 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.840 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:46:46.841 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.842 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.842 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:46:46.845 INFO 2981023 --- [pool-10-thread-15,703d7a8cc48ef8c3,269333456a5d12b0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:46:46.849 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.850 INFO 2981023 --- [pool-10-thread-15,703d7a8cc48ef8c3,269333456a5d12b0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:46:46.850 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:46:46.850 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:46:46.850 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.851 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:46:46.851 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:46:46.851 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.885 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:46:46.894 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:46.895 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:46.895 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:46:46.902 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:46:46.903 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:46:46.903 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:46:46.903 INFO 2981023 --- [ocp-schedules-27,f9f82e42ea0c3937,a520d4cad5a61707] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:46:46.919 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:46:46.923 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:46:46.343330+08:00 2024-09-11 17:46:46.925 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:46:46.928 INFO 2981023 --- [ocp-schedules-29,0b99ef82b9c3f174,59e7b83aea7a5cef] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:46:47.279 INFO 2981023 --- [ocp-async-7,68e3615562db246a,3034db82dfe2b6f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,826283182c811fba,5a7ad7130759c044] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,826283182c811fba,5a7ad7130759c044] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:47.428 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:47.429 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:47.761 INFO 2981023 --- [ocp-async-10,9088c508d74e86a6,ed60ec57339d9fc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:46:49.279 INFO 2981023 --- [ocp-async-1,5cc6bf2f7dc0d789,f275a39475e2b405] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:49.544 INFO 2981023 --- [ocp-schedules-2,c098f41962570c35,2b068b1c778a94ef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:46:49.584 WARN 2981023 --- [scanner-13545624-1,c098f41962570c35,8e476485cc040c13] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:46:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:46:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:46:50.253 INFO 2981023 --- [ocp-schedules-25,de6ea54dbf395e80,37fc3076d069ec3e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:46:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,eac3e3ad55a01af1,077f6a7f0d4f4bd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,eac3e3ad55a01af1,077f6a7f0d4f4bd0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:50.429 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:50.430 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:51.280 INFO 2981023 --- [ocp-async-10,69f06da10905a8e2,7d0bf52560689ac6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:51.562 INFO 2981023 --- [ocp-schedules-24,a351cbc250abc55b,50ca515bed3e9163] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:46:51.563 INFO 2981023 --- [ocp-schedules-24,a351cbc250abc55b,50ca515bed3e9163] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:51.563 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:46:51.563 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:46:51.563 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:46:51.563 INFO 2981023 --- [ocp-schedules-24,a351cbc250abc55b,50ca515bed3e9163] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:46:51.963 INFO 2981023 --- [ocp-schedules-25,f7b1e1cf7c8e3a7b,12b214fca6ff0b55] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:46:52.154 INFO 2981023 --- [ocp-schedules-30,20bcef4a786a72cd,6370f38f87e1dc92] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:46:53.141 INFO 2981023 --- [ocp-schedules-26,637be3de10ef86f7,5f97c3821d8b181f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:46:53.141 INFO 2981023 --- [ocp-schedules-26,637be3de10ef86f7,5f97c3821d8b181f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:46:53.250 INFO 2981023 --- [ocp-schedules-17,d8ac250cb7247231,68636c61eccaa0d3] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:46:53.268 WARN 2981023 --- [ocp-schedules-17,d8ac250cb7247231,68636c61eccaa0d3] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:46:53.280 INFO 2981023 --- [ocp-async-7,9fb519d6abc8ca74,a4cbfd3bcbd779ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,a83c932a2389d691,9858509e0578a2ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,a83c932a2389d691,9858509e0578a2ea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:53.430 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:53.430 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:54.256 INFO 2981023 --- [ocp-schedules-1,4a011c8bd9ddfa19,6acb95a55a77f109] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:46:54.256+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:46:54.260 INFO 2981023 --- [ocp-schedules-1,4a011c8bd9ddfa19,6acb95a55a77f109] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:46:54.282 INFO 2981023 --- [ocp-schedules-27,7283a0ba47826cb1,3390104830324f6d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:46:54.323 INFO 2981023 --- [ocp-schedules-16,11aea2a46ae39cf5,471da300265eebb3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:46:54.362 INFO 2981023 --- [ocp-schedules-16,11aea2a46ae39cf5,471da300265eebb3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:46:54.362 INFO 2981023 --- [ocp-schedules-16,11aea2a46ae39cf5,471da300265eebb3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:46:54.403 INFO 2981023 --- [ocp-schedules-1,4a011c8bd9ddfa19,6acb95a55a77f109] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 44 to SqlAggregateData 2024-09-11 17:46:54.405 INFO 2981023 --- [ocp-schedules-1,4a011c8bd9ddfa19,6acb95a55a77f109] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:46:54.959 INFO 2981023 --- [ocp-schedules-10,ab9c6f3de36b9eb0,838a6042fab06215] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:46:54.961 INFO 2981023 --- [ocp-schedules-10,ab9c6f3de36b9eb0,838a6042fab06215] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:46:54.987 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:46:54.987 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:46:54.987 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:54.988 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:54.989 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:46:54.989 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:54.990 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:46:54.991 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:46:54.992 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:46:54.995 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:54.995 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:54.995 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:54.996 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:54.996 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:54.996 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:55.001 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.001 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:55.006 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.006 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:46:55.008 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.008 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:55.010 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.010 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:55.021 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.022 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:46:55.023 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.023 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:55.023 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.024 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:55.024 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:46:55.024 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3441D6E5-0-0] 2024-09-11 17:46:55.024 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:46:55.025 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.025 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:46:55.029 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.029 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:46:55.031 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.032 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:46:55.033 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.033 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:46:55.034 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.034 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:46:55.035 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:46:55.036 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.037 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:46:55.040 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.041 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:46:55.041 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:46:55.052 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.053 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:46:55.053 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.054 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:46:55.054 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.055 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.055 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:46:55.055 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:46:55.055 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:46:55.055 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:55.056 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3441D6EF-0-0] 2024-09-11 17:46:55.056 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:46:55.056 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.057 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:46:55.057 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.057 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.058 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:46:55.058 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:55.058 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.059 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:46:55.060 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.060 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:46:55.061 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.061 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:46:55.086 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.086 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.087 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:46:55.087 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:46:55.087 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:46:55.087 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.088 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:46:55.088 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.089 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:46:55.089 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:46:55.090 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.090 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:46:55.091 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.091 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:46:55.092 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:46:55.092 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.093 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:46:55.093 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:46:55.095 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:46:55.095 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:46:55.095 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:46:55.095 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:46:55.103 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:55.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:46:55.107 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:46:55.108 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.109 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:46:55.111 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:46:55.113 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:46:55.113 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.113 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:46:55.117 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:46:55.117 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:46:55.119 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:46:55.119 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.119 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:55.120 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:46:55.120 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:55.120 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:46:55.125 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:46:55.126 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:46:55.126 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:46:55.126 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:46:55.132 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:46:55.133 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:46:55.133 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:46:55.133 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:46:55.134 INFO 2981023 --- [ocp-schedules-10,ab9c6f3de36b9eb0,838a6042fab06215] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:46:55.280 INFO 2981023 --- [ocp-async-2,1e747d91a9dd2d67,bd0287b7a3d4b7aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:55.748 INFO 2981023 --- [ocp-schedules-14,e1f8748174dbd997,ec1380f84085ca0d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:46:55.748 INFO 2981023 --- [ocp-schedules-14,e1f8748174dbd997,ec1380f84085ca0d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:46:55.749 INFO 2981023 --- [ocp-schedules-14,e1f8748174dbd997,ec1380f84085ca0d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:46:55.749 INFO 2981023 --- [ocp-schedules-14,e1f8748174dbd997,ec1380f84085ca0d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:46:55.749 INFO 2981023 --- [ocp-schedules-14,e1f8748174dbd997,ec1380f84085ca0d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:46:55.749 INFO 2981023 --- [ocp-schedules-14,e1f8748174dbd997,ec1380f84085ca0d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:46:55.841 INFO 2981023 --- [ocp-schedules-19,ead4e9223b5bb595,600e87771c2a3598] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:46:55.848 INFO 2981023 --- [ocp-schedules-19,ead4e9223b5bb595,600e87771c2a3598] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:46:55.848 INFO 2981023 --- [ocp-schedules-19,ead4e9223b5bb595,600e87771c2a3598] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:46:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,dfdece54d43377e9,dd59936e5af9fdab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,dfdece54d43377e9,dd59936e5af9fdab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:56.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:46:56.337876+08:00, lastProcessTime=2024-09-11T17:46:46.343330+08:00 2024-09-11 17:46:56.430 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:56.431 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:56.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:46:56.337876+08:00 2024-09-11 17:46:56.623 INFO 2981023 --- [ocp-schedules-10,21ade4a4d18084ae,847e57725d3e4e92] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:46:56.623 INFO 2981023 --- [ocp-schedules-10,21ade4a4d18084ae,847e57725d3e4e92] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:46:56.639 INFO 2981023 --- [Collect all tenants major compactions5,21ade4a4d18084ae,446b06a62612a514] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:46:56.639 INFO 2981023 --- [Collect all tenants major compactions4,21ade4a4d18084ae,bf138d2b94b6cece] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:46:56.640 INFO 2981023 --- [Collect all tenants major compactions2,21ade4a4d18084ae,bf4be5250044a8d6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:46:56.641 INFO 2981023 --- [Collect all tenants major compactions3,21ade4a4d18084ae,69f400ab3394f89c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:46:56.641 INFO 2981023 --- [Collect all tenants major compactions1,21ade4a4d18084ae,62c8e4268f2cd0f8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:46:56.642 INFO 2981023 --- [Collect all tenants major compactions6,21ade4a4d18084ae,297ade544b171e60] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:46:56.652 INFO 2981023 --- [Collect all tenants major compactions5,21ade4a4d18084ae,446b06a62612a514] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:56.652 INFO 2981023 --- [Collect all tenants major compactions4,21ade4a4d18084ae,bf138d2b94b6cece] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:56.653 INFO 2981023 --- [Collect all tenants major compactions5,21ade4a4d18084ae,446b06a62612a514] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:46:56.653 INFO 2981023 --- [Collect all tenants major compactions6,21ade4a4d18084ae,297ade544b171e60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:56.654 INFO 2981023 --- [Collect all tenants major compactions6,21ade4a4d18084ae,297ade544b171e60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:46:56.655 INFO 2981023 --- [Collect all tenants major compactions2,21ade4a4d18084ae,bf4be5250044a8d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:56.655 INFO 2981023 --- [Collect all tenants major compactions1,21ade4a4d18084ae,62c8e4268f2cd0f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:56.655 INFO 2981023 --- [Collect all tenants major compactions3,21ade4a4d18084ae,69f400ab3394f89c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:46:56.656 INFO 2981023 --- [Collect all tenants major compactions2,21ade4a4d18084ae,bf4be5250044a8d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:46:56.656 INFO 2981023 --- [Collect all tenants major compactions4,21ade4a4d18084ae,bf138d2b94b6cece] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:46:56.657 INFO 2981023 --- [Collect all tenants major compactions3,21ade4a4d18084ae,69f400ab3394f89c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:46:56.657 INFO 2981023 --- [Collect all tenants major compactions5,21ade4a4d18084ae,446b06a62612a514] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:46:56.657 INFO 2981023 --- [Collect all tenants major compactions5,21ade4a4d18084ae,446b06a62612a514] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:46:56.658 INFO 2981023 --- [Collect all tenants major compactions6,21ade4a4d18084ae,297ade544b171e60] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:46:56.658 INFO 2981023 --- [Collect all tenants major compactions6,21ade4a4d18084ae,297ade544b171e60] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:46:56.658 INFO 2981023 --- [Collect all tenants major compactions2,21ade4a4d18084ae,bf4be5250044a8d6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:46:56.658 INFO 2981023 --- [Collect all tenants major compactions2,21ade4a4d18084ae,bf4be5250044a8d6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:46:56.658 INFO 2981023 --- [Collect all tenants major compactions1,21ade4a4d18084ae,62c8e4268f2cd0f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:46:56.659 INFO 2981023 --- [Collect all tenants major compactions4,21ade4a4d18084ae,bf138d2b94b6cece] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:46:56.659 INFO 2981023 --- [Collect all tenants major compactions4,21ade4a4d18084ae,bf138d2b94b6cece] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:46:56.660 INFO 2981023 --- [Collect all tenants major compactions1,21ade4a4d18084ae,62c8e4268f2cd0f8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:46:56.660 INFO 2981023 --- [Collect all tenants major compactions1,21ade4a4d18084ae,62c8e4268f2cd0f8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:46:56.660 INFO 2981023 --- [Collect all tenants major compactions3,21ade4a4d18084ae,69f400ab3394f89c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:46:56.660 INFO 2981023 --- [Collect all tenants major compactions3,21ade4a4d18084ae,69f400ab3394f89c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:46:56.662 INFO 2981023 --- [ocp-schedules-10,21ade4a4d18084ae,847e57725d3e4e92] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:46:57.280 INFO 2981023 --- [ocp-async-10,d4c59ce5085f92ba,75932f0aa22bdd70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:57.761 INFO 2981023 --- [ocp-async-2,96409624e0a6bec4,565b382e01f06c34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:46:59.280 INFO 2981023 --- [ocp-async-7,46042325a627f327,249d73ab73148196] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:46:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,9e9cbabb4a3a01da,ea5dad77eefb5f4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048020, interval=60 2024-09-11 17:46:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,9e9cbabb4a3a01da,ea5dad77eefb5f4c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:46:59.431 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:46:59.433 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:46:59.731 INFO 2981023 --- [ocp-schedules-25,ac22373054dee228,9b2f207c88b15487] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:46:59.783 WARN 2981023 --- [scanner-13545850-1,ac22373054dee228,be5edf95c7683d8c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:46:59.805 INFO 2981023 --- [alarm-detect-10,ac22373054dee228,29496baf8b3d19ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644627, data is empty. 2024-09-11 17:46:59.812 INFO 2981023 --- [alarm-detect-10,ac22373054dee228,29496baf8b3d19ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644671, data is empty. 2024-09-11 17:46:59.821 INFO 2981023 --- [alarm-detect-10,ac22373054dee228,29496baf8b3d19ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644671, data is empty. 2024-09-11 17:46:59.826 INFO 2981023 --- [alarm-detect-3,ac22373054dee228,6b473dfbeaa2629f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644627, data is empty. 2024-09-11 17:46:59.830 INFO 2981023 --- [alarm-detect-10,ac22373054dee228,6661f122401ee557] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644627, data is empty. 2024-09-11 17:46:59.831 INFO 2981023 --- [alarm-detect-3,ac22373054dee228,6b473dfbeaa2629f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644671, data is empty. 2024-09-11 17:46:59.833 INFO 2981023 --- [alarm-detect-10,ac22373054dee228,6661f122401ee557] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644671, data is empty. 2024-09-11 17:46:59.836 INFO 2981023 --- [alarm-detect-3,ac22373054dee228,6b473dfbeaa2629f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644671, data is empty. 2024-09-11 17:46:59.839 INFO 2981023 --- [alarm-detect-10,ac22373054dee228,6661f122401ee557] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644671, data is empty. 2024-09-11 17:46:59.941 INFO 2981023 --- [alarm-detect-10,ac22373054dee228,f2203cd433e4e33e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644627, data is empty. 2024-09-11 17:46:59.949 INFO 2981023 --- [alarm-detect-10,ac22373054dee228,f2203cd433e4e33e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644671, data is empty. 2024-09-11 17:46:59.956 INFO 2981023 --- [alarm-detect-10,ac22373054dee228,f2203cd433e4e33e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644671, data is empty. 2024-09-11 17:47:01.280 INFO 2981023 --- [ocp-async-1,d6d7d3b999715a60,31958e068a34855a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:02.157 INFO 2981023 --- [ocp-schedules-4,656a3d20d8d7a9b5,0b5d59a56d78c0b1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:47:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,12aee959e5b31f99,dda6a165652c6925] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,12aee959e5b31f99,dda6a165652c6925] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:02.433 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:02.433 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:03.280 INFO 2981023 --- [ocp-async-5,dbf44da34485107c,ca787ab5ef9d25f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:04.769 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:43:04.371, end=2024-09-11T17:45:04.766} 2024-09-11 17:47:04.775 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=l 2024-09-11 17:47:04.776 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:47:04.778 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:43:04.371, end=2024-09-11T17:45:04.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:47:04.778 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:47:04.778 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:47:04.835 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624835000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.835Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.835 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624835000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.835Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.836 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624836000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.836Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.837 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624837000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.837Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.841 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624841000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.841Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.843 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624843000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.843Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.843 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624843000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.843Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.846 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624846000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.846Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.852 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:47:04.853 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.853 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:47:04.853 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.859 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 62 2024-09-11 17:47:04.859 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.861 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.861 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.867 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:04.867 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.867 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.867 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.867 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:04.867 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.868 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.868 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.873 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:04.873 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.875 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.875 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.876 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.876 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.877 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.877 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.879 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:47:04.879 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.880 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.880 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.881 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.881 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.882 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.882 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.884 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:04.884 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.886 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.886 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.886 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.886 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.887 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.887 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.892 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.892 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.892 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.893 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.893 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.894 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.894 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.896 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.896 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.896 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.896 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.898 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.898 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.899 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.899 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.899 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.900 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.900 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.900 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.835, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:04.902 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.902 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.903 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.904 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.841, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:04.904 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.835, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:04.904 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.904 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.905 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.905 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.909 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.910 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.836, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:04.910 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.837, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:04.914 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.914 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.918 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.919 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.843, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:04.921 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.921 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.923 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.925 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.846, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:04.930 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.843, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:04.934 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624934000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.934Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.940 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624939000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.939Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.947 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624947000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.947Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.949 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:04.949 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.951 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:47:04.951 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.951 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624951000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.951Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.952 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624952000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.952Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.957 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:47:04.957 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.960 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 91 2024-09-11 17:47:04.960 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.961 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.961 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.962 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:04.962 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.966 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.966 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.970 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.970 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.971 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624971000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.971Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.971 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.971 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.972 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.972 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.974 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.974 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.977 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624977000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.977Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.977 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.977 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.979 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.979 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.980 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.980 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.980 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.980 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.981 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:04.981 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.982 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:04.982 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:04.986 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.990 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:04.990 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:04.990 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961624990000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:04.990Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:04.991 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.991 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.992 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.992 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.993 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.939, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:04.998 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.998 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:04.999 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:04.999 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:04.999 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.001 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:05.001 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:05.001 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:05.002 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:05.003 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.947, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.004 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.004 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.005 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:05.005 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:05.005 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.951, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.007 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:05.007 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:05.008 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.008 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.008 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.952, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.010 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:05.014 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.934, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.018 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.018 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.018 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.018 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.019 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:05.020 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961625020000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:05.020Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:05.020 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.020 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.021 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:05.021 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:05.022 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:05.024 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.971, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.025 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.977, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.030 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:47:05.030 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:05.030 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.030 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.031 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961625031000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=215632585E2CA5F0C915A1E0D150020F, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:05.031Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:05.032 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:05.036 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:04.99, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.036 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 91 2024-09-11 17:47:05.036 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:05.037 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.037 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.039 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:05.039 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:05.043 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.043 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=215632585E2CA5F0C915A1E0D150020F, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.044 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:05.044 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:05.045 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.045 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.047 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:05.049 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:05.02, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.050 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.050 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=215632585E2CA5F0C915A1E0D150020F, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.051 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:47:05.055 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:05.031, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=6, sqlId=215632585E2CA5F0C915A1E0D150020F, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.055 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:47:05.055 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:43:04.371, end=2024-09-11T17:45:04.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:47:05.055 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:47:05.055 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:47:05.056 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:47:05.057 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:43:04.371, end=2024-09-11T17:45:04.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:47:05.057 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:47:05.057 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:47:05.058 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:47:05.060 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:43:04.371, end=2024-09-11T17:45:04.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:47:05.060 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:47:05.060 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:47:05.069 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:47:05.070 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:47:05.070 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:43:04.371, end=2024-09-11T17:45:04.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:47:05.070 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:47:05.070 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:47:05.071 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:47:05.078 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:43:04.371, end=2024-09-11T17:45:04.766},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:47:05.078 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:47:05.078 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:47:05.088 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:47:05.103 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961625102000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:47:05.102Z, endTime=2024-09-11T09:45:04.766Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:47:05.104 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:47:05.104 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:47:05.110 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.110 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.111 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:47:05.111 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:47:05.116 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:47:05.116 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726047904766000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:47:05.117 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:47:05.117 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:47:05.102, end=2024-09-11T17:45:04.766}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:47:05.117 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:47:05.280 INFO 2981023 --- [ocp-async-4,ac030131a835a87a,61d6616a45c883c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,50a1080b2bfe263c,f4832b56d2a2a5ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:05.315 INFO 2981023 --- [prometheus-minute-schedule-1,50a1080b2bfe263c,f4832b56d2a2a5ec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:05.433 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:05.434 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:06.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:47:06.350815+08:00, lastProcessTime=2024-09-11T17:46:56.337876+08:00 2024-09-11 17:47:06.478 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:47:06.350815+08:00 2024-09-11 17:47:07.280 INFO 2981023 --- [ocp-async-8,e6705ded48cb7825,fa5be624fc22d067] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:07.492 INFO 2981023 --- [ocp-schedules-2,499f36b9f641a1b0,50b52ac1be90b140] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:47:07.519 INFO 2981023 --- [ocp-schedules-27,cfb5df4c0e14695d,aeea02d1f61b1849] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:47:07.519+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:47:07.519 INFO 2981023 --- [ocp-schedules-27,cfb5df4c0e14695d,aeea02d1f61b1849] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:47:07.522 INFO 2981023 --- [ocp-schedules-27,cfb5df4c0e14695d,aeea02d1f61b1849] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:47:07.523 INFO 2981023 --- [ocp-schedules-27,cfb5df4c0e14695d,aeea02d1f61b1849] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:47:07.761 INFO 2981023 --- [ocp-async-3,a6dcbc3370273ab0,295d55348d94d7dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:47:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,46034010423ab15c,61db83084fb165c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,46034010423ab15c,61db83084fb165c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:08.434 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:08.435 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:09.281 INFO 2981023 --- [ocp-async-10,b3465c01804136c5,b040f3de0c8e7a3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:09.993 INFO 2981023 --- [ocp-schedules-16,1e326015ce39e54d,6531be68974b1457] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:47:10.007 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:47:10.011 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:47:10.035 WARN 2981023 --- [scanner-13546220-1,1e326015ce39e54d,36029c45d65199b5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:47:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:47:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:47:11.281 INFO 2981023 --- [ocp-async-9,69c17ec5c07438bc,ef1b57e76e726f3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,7c4cbad54545642c,4520e607bb169d40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,7c4cbad54545642c,4520e607bb169d40] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:11.435 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:11.435 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:12.160 INFO 2981023 --- [ocp-schedules-30,00843c8981210ce3,5713a71150d9375d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:47:12.780 INFO 2981023 --- [ocp-schedules-11,43e05a35c201a9fa,b9f001a0a8d607b9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:47:12.793 INFO 2981023 --- [ocp-schedules-11,43e05a35c201a9fa,b9f001a0a8d607b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:12.796 INFO 2981023 --- [ocp-schedules-11,43e05a35c201a9fa,b9f001a0a8d607b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:12.855 INFO 2981023 --- [ocp-schedules-11,43e05a35c201a9fa,b9f001a0a8d607b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:12.856 INFO 2981023 --- [ocp-schedules-11,43e05a35c201a9fa,b9f001a0a8d607b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:12.867 INFO 2981023 --- [ocp-schedules-11,43e05a35c201a9fa,b9f001a0a8d607b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:12.868 INFO 2981023 --- [ocp-schedules-11,43e05a35c201a9fa,b9f001a0a8d607b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:12.879 INFO 2981023 --- [ocp-schedules-11,43e05a35c201a9fa,b9f001a0a8d607b9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=5 2024-09-11 17:47:13.281 INFO 2981023 --- [ocp-async-6,85ad3c9b572df7c8,38681832f3433991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:13.863 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047960, nodeCount=806, affectRows=806 2024-09-11 17:47:13.865 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047960, nodeCount=806, validCount=806 2024-09-11 17:47:13.890 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726047960, nodeCount=1000, affectRows=1000 2024-09-11 17:47:13.892 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726047960, nodeCount=999, validCount=999 2024-09-11 17:47:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,a52b91e026c33676,c0684b92475bcf89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,8cecdc604b538016,272f0071a9310532] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:47:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,a52b91e026c33676,c0684b92475bcf89] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:14.363 INFO 2981023 --- [ocp-schedules-10,2009cf57f2d512cb,2ce7a7ed32f76963] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:47:14.370 INFO 2981023 --- [ocp-schedules-10,2009cf57f2d512cb,2ce7a7ed32f76963] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:47:14.370 INFO 2981023 --- [ocp-schedules-10,2009cf57f2d512cb,2ce7a7ed32f76963] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:47:14.435 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:14.436 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:15.001 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726047960, nodeCount=463, affectRows=463 2024-09-11 17:47:15.281 INFO 2981023 --- [ocp-async-5,f9578181d3d89e72,bd904a8687abb337] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:16.015 INFO 2981023 --- [ocp-schedules-7,41b6b9095079b117,b1972284a58ccd02] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:47:16.015 INFO 2981023 --- [host-check-clockdiff1,41b6b9095079b117,920acc11f850c45b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:47:16.015 INFO 2981023 --- [host-check-clockdiff2,41b6b9095079b117,f586a702fa094f26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:47:16.015 INFO 2981023 --- [host-check-clockdiff3,41b6b9095079b117,d872967b98fbc0d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:47:16.025 INFO 2981023 --- [host-check-clockdiff3,41b6b9095079b117,d872967b98fbc0d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048036 0 0 , error= 2024-09-11 17:47:16.029 INFO 2981023 --- [host-check-clockdiff1,41b6b9095079b117,920acc11f850c45b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048036 0 0 , error= 2024-09-11 17:47:16.030 INFO 2981023 --- [host-check-clockdiff2,41b6b9095079b117,f586a702fa094f26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048036 0 0 , error= 2024-09-11 17:47:16.031 INFO 2981023 --- [ocp-schedules-18,25829703a4271ccf,9d77972469f071ad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:47:16.031 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:47:16.031 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:47:16.031 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:47:16.035 INFO 2981023 --- [ocp-schedules-13,7628ada285cb9509,82389b958b1ebf54] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:47:16.035 INFO 2981023 --- [host-check-agent-status1,7628ada285cb9509,ea3234898d4cfe9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:47:16.035 INFO 2981023 --- [host-check-agent-status2,7628ada285cb9509,a2730bae1dc1c7ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:47:16.035 INFO 2981023 --- [host-check-agent-status3,7628ada285cb9509,3d918357309e4f1c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:47:16.057 INFO 2981023 --- [host-check-agent-status2,7628ada285cb9509,a2730bae1dc1c7ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:16.058 INFO 2981023 --- [host-check-agent-status1,7628ada285cb9509,ea3234898d4cfe9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:16.059 INFO 2981023 --- [host-check-agent-status2,7628ada285cb9509,a2730bae1dc1c7ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:16.059 INFO 2981023 --- [host-check-agent-status3,7628ada285cb9509,3d918357309e4f1c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:16.059 INFO 2981023 --- [host-check-agent-status1,7628ada285cb9509,ea3234898d4cfe9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:16.060 INFO 2981023 --- [host-check-agent-status3,7628ada285cb9509,3d918357309e4f1c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:16.065 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:16.066 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:16.066 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:16.067 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:16.071 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:16.083 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:16.086 INFO 2981023 --- [host-check-agent-status1,7628ada285cb9509,ea3234898d4cfe9f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:47:16.090 INFO 2981023 --- [host-check-agent-status1,7628ada285cb9509,ea3234898d4cfe9f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:47:16.090 INFO 2981023 --- [host-check-agent-status1,7628ada285cb9509,ea3234898d4cfe9f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:47:16.090 INFO 2981023 --- [host-check-agent-status1,7628ada285cb9509,ea3234898d4cfe9f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:47:16.090 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:47:16.090 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:47:16.092 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:47:16.092168961+08:00 2024-09-11 17:47:16.093 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:47:16.092414316+08:00 2024-09-11 17:47:16.093 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:47:16.092+08:00 2024-09-11 17:47:16.094 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:47:16.093+08:00 2024-09-11 17:47:16.096 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:47:16.092+08:00 and status AVAILABLE 2024-09-11 17:47:16.097 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:47:16.093+08:00 and status AVAILABLE 2024-09-11 17:47:16.097 INFO 2981023 --- [host-check-agent-status2,7628ada285cb9509,a2730bae1dc1c7ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:47:16.099 INFO 2981023 --- [host-check-agent-status2,7628ada285cb9509,a2730bae1dc1c7ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:47:16.099 INFO 2981023 --- [host-check-agent-status2,7628ada285cb9509,a2730bae1dc1c7ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:47:16.099 INFO 2981023 --- [host-check-agent-status2,7628ada285cb9509,a2730bae1dc1c7ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:47:16.105 INFO 2981023 --- [host-check-agent-status3,7628ada285cb9509,3d918357309e4f1c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:47:16.109 INFO 2981023 --- [host-check-agent-status3,7628ada285cb9509,3d918357309e4f1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:47:16.109 INFO 2981023 --- [host-check-agent-status3,7628ada285cb9509,3d918357309e4f1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:47:16.109 INFO 2981023 --- [host-check-agent-status3,7628ada285cb9509,3d918357309e4f1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:47:16.121 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:47:16.121 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:47:16.177 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:16.178 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:16.178 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:16.180 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:16.195 INFO 2981023 --- [ocp-schedules-4,7799d651a96a38b5,6a21be19b0d918ce] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:47:16.195 INFO 2981023 --- [ocp-schedules-3,9ae7f28da345e908,9057454fbeacb2a5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:47:16.195 INFO 2981023 --- [ocp-schedules-8,cd6b1bd468763d70,a0cf40420575d388] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:47:16.195 INFO 2981023 --- [ocp-schedules-6,e493545aaf406f37,1c4439e6e4988fbd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:47:16.196 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:47:16.197 INFO 2981023 --- [ocp-schedules-8,cd6b1bd468763d70,a0cf40420575d388] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:47:16.202 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:47:16.203 INFO 2981023 --- [ocp-schedules-3,9ae7f28da345e908,9057454fbeacb2a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:47:16.204 INFO 2981023 --- [ocp-schedules-4,7799d651a96a38b5,6a21be19b0d918ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:47:16.207 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:47:16.196560358+08:00 2024-09-11 17:47:16.215 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:47:16.207+08:00 2024-09-11 17:47:16.221 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:47:16.207+08:00 and status AVAILABLE 2024-09-11 17:47:16.221 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:47:16.230 INFO 2981023 --- [ocp-schedules-19,19480e9e0eb6e629,cc0ccc2395cdffd3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:47:16.242 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:47:16.249 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:47:16.251 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.252 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:16.253 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:47:16.254 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.255 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:16.267 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:16.270 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:16.279 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.279 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:16.281 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:47:16.283 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:47:16.284 INFO 2981023 --- [ocp-schedules-3,9ae7f28da345e908,9057454fbeacb2a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:47:16.320 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:47:16.321 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.322 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:16.323 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:47:16.325 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.325 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:16.327 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:47:16.329 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:47:16.332 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.332 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.332 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:16.332 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.333 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:47:16.337 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:47:16.342 INFO 2981023 --- [ocp-schedules-4,7799d651a96a38b5,6a21be19b0d918ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:47:16.343 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007102] will be resolved. 2024-09-11 17:47:16.349 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:47:16.349 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:47:16.364 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:47:16.378 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:47:16.379 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:47:16.391 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 17:47:16.394 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:47:16.397 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.398 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.398 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:16.399 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:47:16.399 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.401 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.402 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.402 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:47:16.402 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:47:16.402 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:47:16.404 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : resolveByTimeout, timeoutCount=1 2024-09-11 17:47:16.404 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:47:16.405 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:47:16.404637+08:00, lastProcessTime=2024-09-11T17:47:06.350815+08:00 2024-09-11 17:47:16.407 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:47:16.411 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:47:16.413 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:16.413 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:47:16.426 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.426 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.427 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:16.428 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.430 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:47:16.431 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:47:16.435 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:16.435 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:47:16.440 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:47:16.442 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:47:16.444 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.447 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.448 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:16.448 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:16.448 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:47:16.448 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:47:16.454 INFO 2981023 --- [host-check-host-status2,25829703a4271ccf,6669b17cf49309a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:47:16.456 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:47:16.462 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:16.463 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:47:16.469 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:16.470 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:16.474 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:47:16.480 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.481 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.482 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:16.483 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:47:16.484 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.485 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.485 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:47:16.486 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:47:16.489 INFO 2981023 --- [host-check-host-status3,25829703a4271ccf,b5b070b752e2a724] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:47:16.490 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:47:16.491 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:16.492 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:16.506 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:47:16.512 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:47:16.514 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.515 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.515 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:16.516 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:47:16.516 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.519 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:47:16.519 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:47:16.526 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:16.526 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:47:16.532 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:47:16.537 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:16.540 INFO 2981023 --- [host-check-host-status1,25829703a4271ccf,fa5d5f6a775a8cb0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:47:16.546 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.546 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.547 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:16.547 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:47:16.551 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:47:16.554 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:16.555 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:16.560 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:47:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.569 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.569 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:47:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:47:16.576 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:47:16.576 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:47:16.581 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:47:16.587 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.588 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.588 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:47:16.594 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:47:16.594 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:47:16.601 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:47:16.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:16.608 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:16.609 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:47:16.614 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:47:16.614 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:47:16.614 INFO 2981023 --- [ocp-schedules-8,cd6b1bd468763d70,a0cf40420575d388] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:47:16.638 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:47:16.643 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:47:16.404637+08:00 2024-09-11 17:47:17.281 INFO 2981023 --- [ocp-async-4,3b4e6313dd826d25,45be63669e8ef4ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,88a2e53178139f86,b67a755642affee1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,88a2e53178139f86,b67a755642affee1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:17.436 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:17.437 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:17.761 INFO 2981023 --- [ocp-async-1,a9cbf9fbe5db2467,5f0b57f0d1b5b2dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:47:17.991 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726047960, nodeCount=463, validCount=463 2024-09-11 17:47:19.281 INFO 2981023 --- [ocp-async-8,758ce78b349be175,554a222950fdbb55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:20.180 INFO 2981023 --- [ocp-schedules-19,fa192213f03eb022,0b5c0b7c5c2d2ce0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:47:20.277 WARN 2981023 --- [scanner-13547906-1,fa192213f03eb022,4a853c1923892c85] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:47:20.299 INFO 2981023 --- [alarm-detect-10,fa192213f03eb022,f571dfd319d73074] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644671, data is empty. 2024-09-11 17:47:20.303 INFO 2981023 --- [alarm-detect-10,fa192213f03eb022,f571dfd319d73074] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644671, data is empty. 2024-09-11 17:47:20.307 INFO 2981023 --- [alarm-detect-10,fa192213f03eb022,f571dfd319d73074] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644671, data is empty. 2024-09-11 17:47:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,2fe0e04d9a663ebc,fb20deb5c8db1a08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:20.316 INFO 2981023 --- [alarm-detect-12,fa192213f03eb022,60626bc683c219fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644671, data is empty. 2024-09-11 17:47:20.317 INFO 2981023 --- [alarm-detect-14,fa192213f03eb022,73177ab1f8adac61] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644671, data is empty. 2024-09-11 17:47:20.322 INFO 2981023 --- [prometheus-minute-schedule-1,2fe0e04d9a663ebc,fb20deb5c8db1a08] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:20.323 INFO 2981023 --- [alarm-detect-14,fa192213f03eb022,73177ab1f8adac61] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644671, data is empty. 2024-09-11 17:47:20.325 INFO 2981023 --- [alarm-detect-12,fa192213f03eb022,60626bc683c219fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644671, data is empty. 2024-09-11 17:47:20.344 INFO 2981023 --- [alarm-detect-14,fa192213f03eb022,73177ab1f8adac61] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644671, data is empty. 2024-09-11 17:47:20.350 INFO 2981023 --- [alarm-detect-12,fa192213f03eb022,60626bc683c219fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644671, data is empty. 2024-09-11 17:47:20.437 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:20.439 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:20.439 INFO 2981023 --- [alarm-detect-18,fa192213f03eb022,6efab5b135bf1220] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644671, data is empty. 2024-09-11 17:47:20.444 INFO 2981023 --- [alarm-detect-18,fa192213f03eb022,6efab5b135bf1220] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644671, data is empty. 2024-09-11 17:47:20.447 INFO 2981023 --- [alarm-detect-18,fa192213f03eb022,6efab5b135bf1220] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644671, data is empty. 2024-09-11 17:47:21.281 INFO 2981023 --- [ocp-async-2,69d2bdb309b06a89,4356df84a0057725] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:22.162 INFO 2981023 --- [ocp-schedules-2,7e2545767bb81e23,722bfbb8fe8c605c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:47:23.154 INFO 2981023 --- [ocp-schedules-9,b3ee305744ba8dfd,62fa970a2af4b081] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:47:23.154 INFO 2981023 --- [ocp-schedules-9,b3ee305744ba8dfd,62fa970a2af4b081] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:47:23.154 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:47:23.155 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:47:23.158 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:23.158 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:23.159 INFO 2981023 --- [ocp-schedules-9,b3ee305744ba8dfd,62fa970a2af4b081] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:23.162 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:23.162 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:23.162 INFO 2981023 --- [ocp-schedules-9,b3ee305744ba8dfd,62fa970a2af4b081] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:23.187 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:47:23.187 INFO 2981023 --- [ocp-schedules-9,b3ee305744ba8dfd,62fa970a2af4b081] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:47:23.188 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:47:23.190 INFO 2981023 --- [ocp-schedules-9,b3ee305744ba8dfd,62fa970a2af4b081] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:47:23.281 INFO 2981023 --- [ocp-async-4,48aadac4169b0f1d,ccc4df975222aa1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,fdb6ad3608131c49,239494e964c4eabe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,fdb6ad3608131c49,239494e964c4eabe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:23.439 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:23.439 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:24.285 INFO 2981023 --- [ocp-schedules-13,2f1530a7eba7df94,ea4e5de1403aca2a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:47:25.282 INFO 2981023 --- [ocp-async-9,123bead4486c98d8,7d22f8c9f70d8246] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,330695256b40b860,0d9b9b808d8b3f51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,330695256b40b860,0d9b9b808d8b3f51] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:47:26.351392+08:00, lastProcessTime=2024-09-11T17:47:16.404637+08:00 2024-09-11 17:47:26.440 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:26.440 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:26.483 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:47:26.351392+08:00 2024-09-11 17:47:27.282 INFO 2981023 --- [ocp-async-8,6afabac8b15a499a,2071835a618c42b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:27.761 INFO 2981023 --- [ocp-async-9,c96584d605a8ed07,6493c2de882c2331] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:47:29.282 INFO 2981023 --- [ocp-async-10,c6eb7f7ca6c16e7c,b3239c4ed6e68c6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,3db72bb4b67a7904,d17baac6b828c1ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,3db72bb4b67a7904,d17baac6b828c1ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:29.440 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:29.441 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:30.490 INFO 2981023 --- [ocp-schedules-15,148746c81e750f36,8538c59ddddeb9ae] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:47:30.538 WARN 2981023 --- [scanner-13548203-1,148746c81e750f36,17bf274c56e2ddd4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:47:31.282 INFO 2981023 --- [ocp-async-7,b0befb94f1363df3,5f49b1c1f0495faf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:32.165 INFO 2981023 --- [ocp-schedules-23,8b140667496cc1c0,9586a26c19ba489d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:47:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,c30f5f463974f545,5934cf20756ac80b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,c30f5f463974f545,5934cf20756ac80b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:32.441 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:32.442 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:33.282 INFO 2981023 --- [ocp-async-1,e4d113a46987411f,ad75e9ad6a443548] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:34.374 INFO 2981023 --- [ocp-schedules-22,fee43f172b0989ef,ff9aa403944588f7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:47:34.382 INFO 2981023 --- [ocp-schedules-22,fee43f172b0989ef,ff9aa403944588f7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:47:34.382 INFO 2981023 --- [ocp-schedules-22,fee43f172b0989ef,ff9aa403944588f7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:47:35.282 INFO 2981023 --- [ocp-async-5,e6a468344d4a6eef,f6f41000575a567b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,082512c54d620dab,a46398c5c9f44a15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,082512c54d620dab,a46398c5c9f44a15] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:35.442 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:35.443 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:47:36.336964+08:00, lastProcessTime=2024-09-11T17:47:26.351392+08:00 2024-09-11 17:47:36.466 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:47:36.336964+08:00 2024-09-11 17:47:37.282 INFO 2981023 --- [ocp-async-9,3b9542ff9b10f479,172a622eaf3f370a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:37.314 INFO 2981023 --- [prometheus-cache-monitor1,b1d803b824504529,954c9665c67bd308] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:47:37.315 INFO 2981023 --- [prometheus-cache-monitor1,b1d803b824504529,954c9665c67bd308] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:47:37.761 INFO 2981023 --- [ocp-async-4,b4aaf51f131c2a52,91e9b81a49aad7a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:47:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,fa6d028e9ad50cd9,cf85e98ab9614c7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,fa6d028e9ad50cd9,cf85e98ab9614c7c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:38.443 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:38.444 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:39.282 INFO 2981023 --- [ocp-async-8,7deea4c917faee46,44640764c7d643f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:40.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:47:40.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:47:40.678 INFO 2981023 --- [ocp-schedules-18,4c59804db6a1100e,3b3b94497dd51eb6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:47:40.689 WARN 2981023 --- [scanner-13548389-1,4c59804db6a1100e,4c7a05984d300e99] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:47:40.739 INFO 2981023 --- [alarm-detect-11,4c59804db6a1100e,bc09bfe3c0463707] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644671, data is empty. 2024-09-11 17:47:40.743 INFO 2981023 --- [alarm-detect-11,4c59804db6a1100e,bc09bfe3c0463707] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644671, data is empty. 2024-09-11 17:47:40.746 INFO 2981023 --- [alarm-detect-11,4c59804db6a1100e,bc09bfe3c0463707] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644671, data is empty. 2024-09-11 17:47:40.765 INFO 2981023 --- [alarm-detect-13,4c59804db6a1100e,dd616f2fade83fba] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644671, data is empty. 2024-09-11 17:47:40.768 INFO 2981023 --- [alarm-detect-13,4c59804db6a1100e,dd616f2fade83fba] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644671, data is empty. 2024-09-11 17:47:40.775 INFO 2981023 --- [alarm-detect-13,4c59804db6a1100e,dd616f2fade83fba] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644671, data is empty. 2024-09-11 17:47:40.776 INFO 2981023 --- [alarm-detect-5,4c59804db6a1100e,d8b350cd837f0cc4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644671, data is empty. 2024-09-11 17:47:40.780 INFO 2981023 --- [alarm-detect-5,4c59804db6a1100e,d8b350cd837f0cc4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644671, data is empty. 2024-09-11 17:47:40.785 INFO 2981023 --- [alarm-detect-5,4c59804db6a1100e,d8b350cd837f0cc4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644671, data is empty. 2024-09-11 17:47:40.879 INFO 2981023 --- [alarm-detect-1,4c59804db6a1100e,c360ca27cba59719] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644671, data is empty. 2024-09-11 17:47:40.884 INFO 2981023 --- [alarm-detect-1,4c59804db6a1100e,c360ca27cba59719] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644671, data is empty. 2024-09-11 17:47:40.888 INFO 2981023 --- [alarm-detect-1,4c59804db6a1100e,c360ca27cba59719] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644671, data is empty. 2024-09-11 17:47:41.282 INFO 2981023 --- [ocp-async-5,4e2bcce8cdb17c8a,9e70d1f0a1bcfae0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,4486d494b071ec10,e7bd036202bc1f88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,4486d494b071ec10,e7bd036202bc1f88] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:41.395 INFO 2981023 --- [ocp-schedules-27,c34b41ecf14cb6f3,5eaa11d468f7a44c] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:47:41.414 WARN 2981023 --- [ocp-schedules-27,c34b41ecf14cb6f3,5eaa11d468f7a44c] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:47:41.444 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:41.444 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:42.168 INFO 2981023 --- [ocp-schedules-22,d4ee868e989fc381,3c77879e418fa050] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:47:43.282 INFO 2981023 --- [ocp-async-9,6fa0dbc33e15e507,005507cc6ba8c6ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,ca7131999f07d42f,8d4805ed0215217d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,ca7131999f07d42f,8d4805ed0215217d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:44.444 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:44.445 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:45.101 INFO 2981023 --- [prometheus-second-schedule-1,9027c092b4cc8d0f,e22cae091b773954] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:47:45.282 INFO 2981023 --- [ocp-async-1,3d6fee6c1a88563c,138cc4134435b15c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:46.019 INFO 2981023 --- [ocp-schedules-5,cdcd5420dbf810e9,e41efd42626e3601] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:47:46.020 INFO 2981023 --- [host-check-clockdiff2,cdcd5420dbf810e9,204e23e426300dc3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:47:46.020 INFO 2981023 --- [ocp-schedules-16,c9b82562ca909154,45f63407ba390330] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:47:46.020 INFO 2981023 --- [host-check-clockdiff1,cdcd5420dbf810e9,b473429fa3465ffc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:47:46.022 INFO 2981023 --- [host-check-clockdiff3,cdcd5420dbf810e9,d5b1d56badb66ebc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:47:46.025 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:47:46.026 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:47:46.026 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:47:46.033 INFO 2981023 --- [ocp-schedules-17,17ce014f1657c533,688405c68dcb2c27] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:47:46.034 INFO 2981023 --- [host-check-agent-status1,17ce014f1657c533,bf8a605eac95864d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:47:46.035 INFO 2981023 --- [host-check-agent-status2,17ce014f1657c533,13ecca4bde914fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:47:46.037 INFO 2981023 --- [host-check-agent-status3,17ce014f1657c533,88a224ef3d6405d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:47:46.037 INFO 2981023 --- [host-check-clockdiff2,cdcd5420dbf810e9,204e23e426300dc3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048066 0 0 , error= 2024-09-11 17:47:46.038 INFO 2981023 --- [host-check-clockdiff3,cdcd5420dbf810e9,d5b1d56badb66ebc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048066 0 0 , error= 2024-09-11 17:47:46.038 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:46.039 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:46.040 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:46.040 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:46.041 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:46.042 INFO 2981023 --- [host-check-agent-status1,17ce014f1657c533,bf8a605eac95864d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:46.043 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:46.043 INFO 2981023 --- [host-check-agent-status2,17ce014f1657c533,13ecca4bde914fb4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:46.044 INFO 2981023 --- [host-check-agent-status1,17ce014f1657c533,bf8a605eac95864d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:46.045 INFO 2981023 --- [host-check-agent-status2,17ce014f1657c533,13ecca4bde914fb4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:46.046 INFO 2981023 --- [host-check-agent-status3,17ce014f1657c533,88a224ef3d6405d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:46.047 INFO 2981023 --- [host-check-agent-status3,17ce014f1657c533,88a224ef3d6405d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:46.066 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:47:46.066 INFO 2981023 --- [host-check-agent-status1,17ce014f1657c533,bf8a605eac95864d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:47:46.067 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:47:46.068 INFO 2981023 --- [host-check-agent-status2,17ce014f1657c533,13ecca4bde914fb4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:47:46.068 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:47:46.068492868+08:00 2024-09-11 17:47:46.069 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:47:46.068+08:00 2024-09-11 17:47:46.069 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:47:46.070 INFO 2981023 --- [host-check-agent-status3,17ce014f1657c533,88a224ef3d6405d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:47:46.071 INFO 2981023 --- [host-check-agent-status1,17ce014f1657c533,bf8a605eac95864d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:47:46.071 INFO 2981023 --- [host-check-agent-status1,17ce014f1657c533,bf8a605eac95864d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:47:46.071 INFO 2981023 --- [host-check-agent-status1,17ce014f1657c533,bf8a605eac95864d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:47:46.071 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:47:46.070692808+08:00 2024-09-11 17:47:46.071 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:47:46.071399511+08:00 2024-09-11 17:47:46.071 INFO 2981023 --- [host-check-agent-status2,17ce014f1657c533,13ecca4bde914fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:47:46.071 INFO 2981023 --- [host-check-agent-status2,17ce014f1657c533,13ecca4bde914fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:47:46.071 INFO 2981023 --- [host-check-agent-status2,17ce014f1657c533,13ecca4bde914fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:47:46.072 INFO 2981023 --- [host-check-agent-status3,17ce014f1657c533,88a224ef3d6405d5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:47:46.072 INFO 2981023 --- [host-check-agent-status3,17ce014f1657c533,88a224ef3d6405d5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:47:46.072 INFO 2981023 --- [host-check-agent-status3,17ce014f1657c533,88a224ef3d6405d5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:47:46.077 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:47:46.071+08:00 2024-09-11 17:47:46.077 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:47:46.068+08:00 and status AVAILABLE 2024-09-11 17:47:46.078 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:47:46.071+08:00 2024-09-11 17:47:46.079 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:47:46.071+08:00 and status AVAILABLE 2024-09-11 17:47:46.080 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:47:46.071+08:00 and status AVAILABLE 2024-09-11 17:47:46.119 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:47:46.119 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:47:46.128 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:46.129 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:46.130 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:47:46.143 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:46.145 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:46.145 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:47:46.160 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:47:46.161 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:47:46.176 INFO 2981023 --- [ocp-schedules-22,515246174abe993c,e3394e95cb55bea8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:47:46.178 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:46.179 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:47:46.180 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:46.185 INFO 2981023 --- [ocp-schedules-22,515246174abe993c,e3394e95cb55bea8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:47:46.188 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:47:46.188 INFO 2981023 --- [ocp-schedules-12,30a2221b39cb4c30,02d8b76764e195fb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:47:46.188 INFO 2981023 --- [ocp-schedules-8,4d723215f7af426f,e29069605175df3d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:47:46.190 INFO 2981023 --- [ocp-schedules-12,30a2221b39cb4c30,02d8b76764e195fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:47:46.195 INFO 2981023 --- [ocp-schedules-29,64da44e858b60210,09673aa869eb1b32] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:47:46.195 INFO 2981023 --- [ocp-schedules-4,5a9c0b338cc09d86,5c0d01ec7a028925] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:47:46.203 INFO 2981023 --- [ocp-schedules-8,4d723215f7af426f,e29069605175df3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:47:46.208 INFO 2981023 --- [ocp-schedules-29,64da44e858b60210,09673aa869eb1b32] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:47:46.220 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:47:46.228 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:47:46.237 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:47:46.244 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:47:46.246 INFO 2981023 --- [ocp-schedules-6,1879bcb5d67e670c,c1afceea1bb30710] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:47:46.249 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.249 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:46.251 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:47:46.256 INFO 2981023 --- [ocp-schedules-8,4d723215f7af426f,e29069605175df3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:47:46.269 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.270 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:46.272 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:47:46.279 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.281 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:46.282 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:47:46.283 INFO 2981023 --- [ocp-schedules-12,30a2221b39cb4c30,02d8b76764e195fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:47:46.295 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.296 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:47:46.297 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:47:46.297 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.306 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:47:46.306 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.307 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:47:46.307 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:46.313 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.313 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:46.315 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:47:46.322 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.323 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:46.331 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:47:46.331 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:47:46.336 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:47:46.343 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:47:46.340303+08:00, lastProcessTime=2024-09-11T17:47:36.336964+08:00 2024-09-11 17:47:46.351 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:47:46.357 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:47:46.359 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:47:46.362 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.363 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.363 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:47:46.381 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:47:46.395 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.397 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:47:46.398 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.400 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.400 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:47:46.400 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.400 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:46.401 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:47:46.401 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.401 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:47:46.405 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.408 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:47:46.408 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:47:46.426 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:47:46.427 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:47:46.432 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.433 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.433 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:47:46.434 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:46.434 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:47:46.437 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:47:46.438 INFO 2981023 --- [host-check-host-status2,c9b82562ca909154,d3d6d018431f4ceb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:47:46.438 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:47:46.439 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:47:46.447 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.448 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.449 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:47:46.450 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:47:46.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:46.466 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.467 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.467 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.467 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.467 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.468 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:46.468 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:46.469 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:47:46.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.469 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:47:46.472 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.472 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:47:46.472 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.473 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:47:46.473 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:47:46.473 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:47:46.473 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:47:46.473 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.474 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:47:46.476 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:47:46.477 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:47:46.479 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.480 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.480 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:47:46.481 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:47:46.481 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.481 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:47:46.482 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:47:46.482 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.485 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:47:46.486 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:47:46.488 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:47:46.490 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:46.490 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:47:46.491 INFO 2981023 --- [host-check-host-status1,c9b82562ca909154,70c383fb4927b25e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:47:46.494 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:46.494 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:47:46.506 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:47:46.509 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:46.509 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:47:46.515 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:47:46.516 INFO 2981023 --- [host-check-host-status3,c9b82562ca909154,1124c7aedefff632] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:47:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:47:46.517 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:46.520 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:47:46.525 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:47:46.529 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:47:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:47:46.534 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.534 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.536 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:47:46.537 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:47:46.540 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.541 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.541 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:47:46.542 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:47:46.542 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.546 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:47:46.557 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:47:46.558 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:47:46.561 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:47:46.562 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:47:46.574 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:47:46.577 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:47:46.586 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.588 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.588 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:47:46.602 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:47:46.606 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:47:46.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:47:46.607 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.607 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:47:46.608 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:47:46.608 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.614 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:47:46.614 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.614 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.615 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:47:46.622 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:47:46.622 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:47:46.636 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:47:46.340303+08:00 2024-09-11 17:47:46.653 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:47:46.669 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:47:46.674 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:47:46.677 INFO 2981023 --- [ocp-schedules-18,daaa57daf3eb31bb,f12e6cd0f3354567] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:47:46.679 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:47:46.685 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:46.685 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:46.686 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:47:46.691 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:47:46.691 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:47:46.691 INFO 2981023 --- [ocp-schedules-29,64da44e858b60210,09673aa869eb1b32] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:47:46.841 INFO 2981023 --- [ocp-schedules-11,e6423d4f6e4e9909,24dee04f927fa420] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:47:46.841 INFO 2981023 --- [ocp-schedules-11,e6423d4f6e4e9909,24dee04f927fa420] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:47:47.283 INFO 2981023 --- [ocp-async-5,1ce5debcdde2ca61,e864c5f530d24139] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,0236366075a798ce,d7e0434042673993] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,0236366075a798ce,d7e0434042673993] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:47.445 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:47.445 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:47.761 INFO 2981023 --- [ocp-async-8,701611b64afd2532,ccebf98618415c9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:47:49.283 INFO 2981023 --- [ocp-async-7,be087fbd6af0471b,b2b22d29a5c414ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:47:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:47:50.251 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:47:50.255 INFO 2981023 --- [ocp-schedules-27,0c330f5de3a8cfbd,5cb8812c8e48ba51] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.256 INFO 2981023 --- [ocp-schedules-15,c2bde7c0c86ac363,1d2da195605d2f9c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:47:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,ed0c93050474e18b,4c6fe0bb42461947] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,ed0c93050474e18b,4c6fe0bb42461947] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:50.446 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:50.446 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:50.928 INFO 2981023 --- [ocp-schedules-24,4b60fd9c7f8b06d6,fe1cbcae4aa88e62] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:47:50.970 WARN 2981023 --- [scanner-13548732-1,4b60fd9c7f8b06d6,fcac1a7afecb68e6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:47:51.055 INFO 2981023 --- [alarm-detect-15,4b60fd9c7f8b06d6,69268972fad44f2c] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-2, alarm_template_id=0:host=10.31.35.10-3, alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.6-2, alarm_template_id=0:host=10.31.35.5-2, alarm_template_id=0:host=10.31.35.6-3] 2024-09-11 17:47:51.283 INFO 2981023 --- [ocp-async-2,bd1135b64757147f,7348df6ab7c2c954] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:51.563 INFO 2981023 --- [ocp-schedules-23,e7e001ce56433215,936882a34dd09803] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:47:51.565 INFO 2981023 --- [ocp-schedules-23,e7e001ce56433215,936882a34dd09803] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:47:51.565 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:47:51.565 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:47:51.565 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:47:51.565 INFO 2981023 --- [ocp-schedules-23,e7e001ce56433215,936882a34dd09803] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:47:51.752 INFO 2981023 --- [host-check-clockdiff1,cdcd5420dbf810e9,b473429fa3465ffc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048071 0 0 , error= 2024-09-11 17:47:52.171 INFO 2981023 --- [ocp-schedules-22,11fc1a6a36a2c2ee,71a8d55211b8fad0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:47:52.405 INFO 2981023 --- [ocp-schedules-12,4840b5e2489535ec,29b795b6f89146d2] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:47:53.144 INFO 2981023 --- [ocp-schedules-6,1a98e51420a81beb,77df61f98fed042b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:47:53.144 INFO 2981023 --- [ocp-schedules-6,1a98e51420a81beb,77df61f98fed042b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:47:53.283 INFO 2981023 --- [ocp-async-10,7e838f0e2f3c0e23,0239f3128631b3d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:53.327 INFO 2981023 --- [prometheus-minute-schedule-1,4a307f4f16f9ec65,46f9afa7e9d23484] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:53.328 INFO 2981023 --- [ocp-schedules-19,765fa85d24aa1beb,8cd4870dd9cb45da] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:47:53.338 INFO 2981023 --- [prometheus-minute-schedule-1,4a307f4f16f9ec65,46f9afa7e9d23484] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:53.353 WARN 2981023 --- [ocp-schedules-19,765fa85d24aa1beb,8cd4870dd9cb45da] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:47:53.446 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:53.448 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:54.288 INFO 2981023 --- [ocp-schedules-11,c46462748861c290,73ecb6d722b71169] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:47:54.385 INFO 2981023 --- [ocp-schedules-7,217eb4b8667041e1,c0566e6fa1a83950] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:47:54.393 INFO 2981023 --- [ocp-schedules-7,217eb4b8667041e1,c0566e6fa1a83950] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:47:54.393 INFO 2981023 --- [ocp-schedules-7,217eb4b8667041e1,c0566e6fa1a83950] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:47:54.425 INFO 2981023 --- [ocp-schedules-14,e5161434d15da4ea,7f62fe30ddee14c3] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:47:54.425+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:47:54.429 INFO 2981023 --- [ocp-schedules-14,e5161434d15da4ea,7f62fe30ddee14c3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:47:54.535 INFO 2981023 --- [ocp-schedules-14,e5161434d15da4ea,7f62fe30ddee14c3] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 21 to SqlAggregateData 2024-09-11 17:47:54.536 INFO 2981023 --- [ocp-schedules-14,e5161434d15da4ea,7f62fe30ddee14c3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:47:55.134 INFO 2981023 --- [ocp-schedules-15,3a44ce82545dd082,32910b2ce9edd03f] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:47:55.135 INFO 2981023 --- [ocp-schedules-15,3a44ce82545dd082,32910b2ce9edd03f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:47:55.171 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:47:55.171 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:47:55.171 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:47:55.172 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.172 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.174 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:47:55.176 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:47:55.176 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:47:55.179 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.179 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.179 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.180 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.180 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.180 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.186 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.186 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.191 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.191 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.192 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.193 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.193 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.193 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:47:55.200 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.200 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:47:55.204 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.204 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.205 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:47:55.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3531D6DA-0-0] 2024-09-11 17:47:55.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:47:55.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.206 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:47:55.207 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.207 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:47:55.216 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.216 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.218 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.219 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:47:55.229 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.229 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:47:55.231 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.231 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:47:55.232 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:47:55.232 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3531D6E4-0-0] 2024-09-11 17:47:55.232 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:47:55.232 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.233 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:47:55.234 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.235 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:47:55.235 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.236 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:47:55.236 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:47:55.241 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:47:55.242 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.242 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:47:55.244 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.244 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:47:55.245 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.246 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:47:55.246 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.246 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.246 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:47:55.247 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:47:55.247 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.247 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.248 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:47:55.248 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:47:55.248 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.248 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.249 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:47:55.249 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:47:55.250 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:47:55.250 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.251 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:47:55.251 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.252 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:47:55.252 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.252 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:47:55.253 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:47:55.254 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.254 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:47:55.255 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.255 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:55.255 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:47:55.256 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.256 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:47:55.257 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:47:55.257 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.258 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.259 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:47:55.266 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:55.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.268 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:47:55.272 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:47:55.274 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:47:55.275 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.275 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:47:55.275 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:47:55.275 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:47:55.275 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:47:55.279 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:47:55.280 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:47:55.281 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.281 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.282 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:47:55.283 INFO 2981023 --- [ocp-async-3,87a3e6ff0b3e7ad1,864c97a80bee7c30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:55.283 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:47:55.284 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.284 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:47:55.290 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:47:55.292 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:47:55.293 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:55.293 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:47:55.294 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:47:55.296 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:47:55.296 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:47:55.296 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:47:55.299 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:47:55.301 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:47:55.301 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:47:55.301 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:47:55.301 INFO 2981023 --- [ocp-schedules-15,3a44ce82545dd082,32910b2ce9edd03f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:47:55.749 INFO 2981023 --- [ocp-schedules-11,a71801983290e737,2043a9f1021e8176] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:47:55.749 INFO 2981023 --- [ocp-schedules-11,a71801983290e737,2043a9f1021e8176] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:47:55.750 INFO 2981023 --- [ocp-schedules-11,a71801983290e737,2043a9f1021e8176] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:47:55.750 INFO 2981023 --- [ocp-schedules-11,a71801983290e737,2043a9f1021e8176] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:47:55.751 INFO 2981023 --- [ocp-schedules-11,a71801983290e737,2043a9f1021e8176] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:47:55.751 INFO 2981023 --- [ocp-schedules-11,a71801983290e737,2043a9f1021e8176] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:47:55.848 INFO 2981023 --- [ocp-schedules-23,bfe3bc73bfc10d38,6dbf6291bfa974f3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:47:55.854 INFO 2981023 --- [ocp-schedules-23,bfe3bc73bfc10d38,6dbf6291bfa974f3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:47:55.854 INFO 2981023 --- [ocp-schedules-23,bfe3bc73bfc10d38,6dbf6291bfa974f3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:47:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,afd612b7c0474c20,b9278b4b190170f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,afd612b7c0474c20,b9278b4b190170f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:47:56.351784+08:00, lastProcessTime=2024-09-11T17:47:46.340303+08:00 2024-09-11 17:47:56.448 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:56.449 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:47:56.479 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:47:56.483 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:47:56.351784+08:00 2024-09-11 17:47:56.663 INFO 2981023 --- [ocp-schedules-6,6f0d2b0d85f1fbbd,5d6fa901a66bd99c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:47:56.663 INFO 2981023 --- [ocp-schedules-6,6f0d2b0d85f1fbbd,5d6fa901a66bd99c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:47:56.675 INFO 2981023 --- [Collect all tenants major compactions2,6f0d2b0d85f1fbbd,529c0d3b08536c21] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:47:56.675 INFO 2981023 --- [Collect all tenants major compactions1,6f0d2b0d85f1fbbd,71c04c921c4c5ae5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:47:56.677 INFO 2981023 --- [Collect all tenants major compactions5,6f0d2b0d85f1fbbd,16f09906bca8710a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:47:56.677 INFO 2981023 --- [Collect all tenants major compactions3,6f0d2b0d85f1fbbd,db1a4353b9d11f59] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:47:56.677 INFO 2981023 --- [Collect all tenants major compactions6,6f0d2b0d85f1fbbd,fad4bdbe326d8677] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:47:56.678 INFO 2981023 --- [Collect all tenants major compactions4,6f0d2b0d85f1fbbd,d9dea58004febefd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:47:56.689 INFO 2981023 --- [Collect all tenants major compactions5,6f0d2b0d85f1fbbd,16f09906bca8710a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:56.689 INFO 2981023 --- [Collect all tenants major compactions2,6f0d2b0d85f1fbbd,529c0d3b08536c21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:56.689 INFO 2981023 --- [Collect all tenants major compactions6,6f0d2b0d85f1fbbd,fad4bdbe326d8677] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:56.689 INFO 2981023 --- [Collect all tenants major compactions1,6f0d2b0d85f1fbbd,71c04c921c4c5ae5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:56.689 INFO 2981023 --- [Collect all tenants major compactions3,6f0d2b0d85f1fbbd,db1a4353b9d11f59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:56.690 INFO 2981023 --- [Collect all tenants major compactions4,6f0d2b0d85f1fbbd,d9dea58004febefd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:47:56.690 INFO 2981023 --- [Collect all tenants major compactions5,6f0d2b0d85f1fbbd,16f09906bca8710a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:47:56.690 INFO 2981023 --- [Collect all tenants major compactions2,6f0d2b0d85f1fbbd,529c0d3b08536c21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:47:56.690 INFO 2981023 --- [Collect all tenants major compactions6,6f0d2b0d85f1fbbd,fad4bdbe326d8677] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:47:56.691 INFO 2981023 --- [Collect all tenants major compactions1,6f0d2b0d85f1fbbd,71c04c921c4c5ae5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:47:56.691 INFO 2981023 --- [Collect all tenants major compactions3,6f0d2b0d85f1fbbd,db1a4353b9d11f59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:47:56.692 INFO 2981023 --- [Collect all tenants major compactions4,6f0d2b0d85f1fbbd,d9dea58004febefd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions5,6f0d2b0d85f1fbbd,16f09906bca8710a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions1,6f0d2b0d85f1fbbd,71c04c921c4c5ae5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions5,6f0d2b0d85f1fbbd,16f09906bca8710a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions1,6f0d2b0d85f1fbbd,71c04c921c4c5ae5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions3,6f0d2b0d85f1fbbd,db1a4353b9d11f59] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions3,6f0d2b0d85f1fbbd,db1a4353b9d11f59] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions2,6f0d2b0d85f1fbbd,529c0d3b08536c21] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions2,6f0d2b0d85f1fbbd,529c0d3b08536c21] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions6,6f0d2b0d85f1fbbd,fad4bdbe326d8677] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:47:56.694 INFO 2981023 --- [Collect all tenants major compactions6,6f0d2b0d85f1fbbd,fad4bdbe326d8677] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:47:56.695 INFO 2981023 --- [Collect all tenants major compactions4,6f0d2b0d85f1fbbd,d9dea58004febefd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:47:56.695 INFO 2981023 --- [Collect all tenants major compactions4,6f0d2b0d85f1fbbd,d9dea58004febefd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:47:56.695 INFO 2981023 --- [ocp-schedules-6,6f0d2b0d85f1fbbd,5d6fa901a66bd99c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:47:57.283 INFO 2981023 --- [ocp-async-8,9032862c68edc48b,b5022c001515ef00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:57.762 INFO 2981023 --- [ocp-async-3,6ca95e93b73bd8a7,c9bf9c29124ec358] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:47:58.547 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.549 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obCluster 2024-09-11 17:47:58.553 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.557 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.560 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.563 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.564 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.565 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.565 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.566 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.567 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.567 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.568 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.569 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.572 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.574 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.577 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.579 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.582 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.584 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.586 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.589 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.589 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.590 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.591 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.591 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:41:58.447, end=2024-09-11T17:46:58.542} 2024-09-11 17:47:58.591 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:47:59.283 INFO 2981023 --- [ocp-async-4,59ee6cc4283754c0,662ea2369ef99473] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:47:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,fe88d7efd0d9daf9,af2ff6de857a229f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048080, interval=60 2024-09-11 17:47:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,fe88d7efd0d9daf9,af2ff6de857a229f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:47:59.449 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:47:59.450 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:01.129 INFO 2981023 --- [ocp-schedules-4,e1e37f2e2014404b,b30c074977b20fce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:48:01.289 INFO 2981023 --- [ocp-async-9,401c89adc4656dc8,96240e947dd8f288] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:01.455 WARN 2981023 --- [scanner-13548931-1,e1e37f2e2014404b,a96fd6191d51cad8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:48:01.496 INFO 2981023 --- [alarm-detect-19,e1e37f2e2014404b,d3ca9e2bc663b864] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644671, data is empty. 2024-09-11 17:48:01.503 INFO 2981023 --- [alarm-detect-19,e1e37f2e2014404b,d3ca9e2bc663b864] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644692, data is empty. 2024-09-11 17:48:01.507 INFO 2981023 --- [alarm-detect-19,e1e37f2e2014404b,d3ca9e2bc663b864] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644692, data is empty. 2024-09-11 17:48:01.530 INFO 2981023 --- [alarm-detect-5,e1e37f2e2014404b,870f384b15678626] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644671, data is empty. 2024-09-11 17:48:01.530 INFO 2981023 --- [alarm-detect-17,e1e37f2e2014404b,5d0bb9bb30bc9891] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644671, data is empty. 2024-09-11 17:48:01.534 INFO 2981023 --- [alarm-detect-5,e1e37f2e2014404b,870f384b15678626] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644692, data is empty. 2024-09-11 17:48:01.535 INFO 2981023 --- [alarm-detect-17,e1e37f2e2014404b,5d0bb9bb30bc9891] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644692, data is empty. 2024-09-11 17:48:01.539 INFO 2981023 --- [alarm-detect-5,e1e37f2e2014404b,870f384b15678626] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644692, data is empty. 2024-09-11 17:48:01.540 INFO 2981023 --- [alarm-detect-17,e1e37f2e2014404b,5d0bb9bb30bc9891] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644692, data is empty. 2024-09-11 17:48:01.644 INFO 2981023 --- [alarm-detect-3,e1e37f2e2014404b,560ff976048f4d5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644671, data is empty. 2024-09-11 17:48:01.648 INFO 2981023 --- [alarm-detect-3,e1e37f2e2014404b,560ff976048f4d5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644692, data is empty. 2024-09-11 17:48:01.650 INFO 2981023 --- [alarm-detect-3,e1e37f2e2014404b,560ff976048f4d5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644692, data is empty. 2024-09-11 17:48:02.173 INFO 2981023 --- [ocp-schedules-17,0cae4b330bac7262,8fda80eee6ae7be6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:48:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,6dee3e2867a63b03,2175201f2f10433a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,6dee3e2867a63b03,2175201f2f10433a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:02.450 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:02.451 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:02.514 INFO 2981023 --- [ocp-schedules-9,c3ce6e91d9b73a40,1510294037af7bdd] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:48:02.515 INFO 2981023 --- [ocp-schedules-9,c3ce6e91d9b73a40,1510294037af7bdd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:48:02.524 INFO 2981023 --- [ocp-schedules-9,c3ce6e91d9b73a40,1510294037af7bdd] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:48:03.289 INFO 2981023 --- [ocp-async-9,7eeb88c9a506f941,000682f724dbfdf4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:05.289 INFO 2981023 --- [ocp-async-5,6758a2c2bfd09069,5e925a43a4066a41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,122e32014c921602,68a774424964a5bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:05.315 INFO 2981023 --- [prometheus-minute-schedule-1,122e32014c921602,68a774424964a5bd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:05.451 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:05.451 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:48:05.984 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:48:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:48:05.995 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:48:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:48:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:48:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:48:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:06.005 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:48:06.005 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:48:06.006 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:06.006 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:06.006 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:48:06.006 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:48:06.087 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:45:40, start=2024-09-11T17:42:40, end=2024-09-11T17:45:10, updates=109, granule=120000ms, delay=180000ms 2024-09-11 17:48:06.093 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:45:40, target=2024-09-11T17:45:10, granule=120000ms, delay=30000ms) 2024-09-11 17:48:06.187 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:45:41, start=2024-09-11T17:42:41, end=2024-09-11T17:45:11, updates=299, granule=120000ms, delay=180000ms 2024-09-11 17:48:06.194 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - source=2024-09-11T17:45:10, start=2024-09-11T17:29:31, end=2024-09-11T17:39:31, updates=140, granule=600000ms, delay=939000ms 2024-09-11 17:48:06.199 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:45:41, target=2024-09-11T17:45:11, granule=120000ms, delay=30000ms) 2024-09-11 17:48:06.200 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:45:10, target=2024-09-11T17:39:31, granule=600000ms, delay=339000ms) 2024-09-11 17:48:06.329 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T17:45:11, start=2024-09-11T17:30:51, end=2024-09-11T17:40:51, updates=396, granule=600000ms, delay=860000ms 2024-09-11 17:48:06.333 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:45:11, target=2024-09-11T17:40:51, granule=600000ms, delay=260000ms) 2024-09-11 17:48:06.333 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=366ms, period=120000ms 2024-09-11 17:48:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:48:06.351590+08:00, lastProcessTime=2024-09-11T17:47:56.351784+08:00 2024-09-11 17:48:06.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:48:06.351590+08:00 2024-09-11 17:48:07.289 INFO 2981023 --- [ocp-async-7,edb54154b47f241e,f34811bff40be75a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:07.504 INFO 2981023 --- [ocp-schedules-25,b100301686189973,7ca5cb05b46bbc91] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:48:07.535 INFO 2981023 --- [ocp-schedules-20,cd44591015566ec9,fea2d3f8d544499d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:48:07.535+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:48:07.536 INFO 2981023 --- [ocp-schedules-20,cd44591015566ec9,fea2d3f8d544499d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:48:07.537 INFO 2981023 --- [ocp-schedules-20,cd44591015566ec9,fea2d3f8d544499d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:48:07.538 INFO 2981023 --- [ocp-schedules-20,cd44591015566ec9,fea2d3f8d544499d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:48:07.762 INFO 2981023 --- [ocp-async-5,0847c56b65d81aa1,8b2afd7a06c57759] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:48:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,7129ab1e7dc6452a,31e1ed3effe28f93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,7129ab1e7dc6452a,31e1ed3effe28f93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:08.451 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:08.452 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:09.289 INFO 2981023 --- [ocp-async-2,a6bd64bb0c8303df,550717d2d214f4ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:10.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:48:10.003 INFO 2981023 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:48:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:48:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:48:11.289 INFO 2981023 --- [ocp-async-5,1a41d3dac8b273bb,1d5eba697eb35de9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,7bcbf7cf89c70ff5,74dfab642939e7ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,7bcbf7cf89c70ff5,74dfab642939e7ea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:11.452 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:11.453 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:11.700 INFO 2981023 --- [ocp-schedules-2,ab998fa369d2c02a,6d5ea80c280c530a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:48:11.758 WARN 2981023 --- [scanner-13549852-1,ab998fa369d2c02a,4b4b21c1cce41ccf] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:48:12.176 INFO 2981023 --- [ocp-schedules-4,f15b9bbe0a9895a2,2d5a9a2b4cde27ca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:48:13.101 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048020, nodeCount=463, affectRows=463 2024-09-11 17:48:13.289 INFO 2981023 --- [ocp-async-7,4a76662a9e470e4f,1d224bcf1548f92d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:14.112 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048020, nodeCount=1000, affectRows=1000 2024-09-11 17:48:14.112 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048020, nodeCount=806, affectRows=806 2024-09-11 17:48:14.114 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048020, nodeCount=806, validCount=806 2024-09-11 17:48:14.115 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048020, nodeCount=999, validCount=999 2024-09-11 17:48:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,2e682de9e976d639,b84f9cb70a72f755] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,026d6f503ca489d6,ec1101df8181920f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:48:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,2e682de9e976d639,b84f9cb70a72f755] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:14.396 INFO 2981023 --- [ocp-schedules-9,d73bd3c318b5020c,be317797eb1d405c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:48:14.406 INFO 2981023 --- [ocp-schedules-9,d73bd3c318b5020c,be317797eb1d405c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:48:14.406 INFO 2981023 --- [ocp-schedules-9,d73bd3c318b5020c,be317797eb1d405c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:48:14.453 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:14.453 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:14.915 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048020, nodeCount=463, validCount=463 2024-09-11 17:48:15.290 INFO 2981023 --- [ocp-async-9,900a69464e8089e9,4179bf6ec60f5857] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:16.020 INFO 2981023 --- [ocp-schedules-29,64002de465a256f1,07acd1c8f24b5f9f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:48:16.021 INFO 2981023 --- [ocp-schedules-23,f3554883fd3cd829,2c92d8edc1cc927b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:48:16.021 INFO 2981023 --- [host-check-clockdiff1,64002de465a256f1,bffab1295f1f80f4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:48:16.021 INFO 2981023 --- [host-check-clockdiff2,64002de465a256f1,4435aa24635c03f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:48:16.021 INFO 2981023 --- [host-check-clockdiff3,64002de465a256f1,b2566e15d68921cd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:48:16.021 INFO 2981023 --- [host-check-agent-status1,f3554883fd3cd829,463f9738519717cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:48:16.023 INFO 2981023 --- [host-check-agent-status2,f3554883fd3cd829,224f83ddea18d4a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:48:16.023 INFO 2981023 --- [host-check-agent-status3,f3554883fd3cd829,4d9c4197d3631bbf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:48:16.038 INFO 2981023 --- [host-check-clockdiff2,64002de465a256f1,4435aa24635c03f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048096 0 0 , error= 2024-09-11 17:48:16.042 INFO 2981023 --- [ocp-schedules-20,508279919c831dc3,516f6737e8bc3a6d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:48:16.042 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:48:16.043 INFO 2981023 --- [host-check-clockdiff3,64002de465a256f1,b2566e15d68921cd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048096 0 0 , error= 2024-09-11 17:48:16.045 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:48:16.045 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:48:16.061 INFO 2981023 --- [host-check-agent-status1,f3554883fd3cd829,463f9738519717cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:48:16.062 INFO 2981023 --- [host-check-agent-status3,f3554883fd3cd829,4d9c4197d3631bbf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:48:16.062 INFO 2981023 --- [host-check-agent-status2,f3554883fd3cd829,224f83ddea18d4a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:48:16.066 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:48:16.066 INFO 2981023 --- [host-check-agent-status1,f3554883fd3cd829,463f9738519717cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:48:16.066 INFO 2981023 --- [host-check-agent-status2,f3554883fd3cd829,224f83ddea18d4a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:48:16.066 INFO 2981023 --- [host-check-agent-status3,f3554883fd3cd829,4d9c4197d3631bbf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:48:16.066 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:48:16.067 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:48:16.070 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:48:16.070 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:48:16.071 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:48:16.189 INFO 2981023 --- [ocp-schedules-22,1893980c5591780e,e207c98a0d67cff5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:48:16.195 INFO 2981023 --- [ocp-schedules-21,1af95e034db74c9b,679dedd4353502fd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:48:16.195 INFO 2981023 --- [ocp-schedules-19,da2e83932fc8d8b8,a5b403e6c629a5c7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:48:16.195 INFO 2981023 --- [ocp-schedules-8,0597f2b84786f9ed,e9e99226d7f393c6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:48:16.282 INFO 2981023 --- [ocp-schedules-19,da2e83932fc8d8b8,a5b403e6c629a5c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:48:16.294 INFO 2981023 --- [ocp-schedules-21,1af95e034db74c9b,679dedd4353502fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:48:16.296 INFO 2981023 --- [ocp-schedules-22,1893980c5591780e,e207c98a0d67cff5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:48:16.307 INFO 2981023 --- [ocp-schedules-21,1af95e034db74c9b,679dedd4353502fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:48:16.380 INFO 2981023 --- [ocp-schedules-7,4283412805cc4948,ab71a4338069b81b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:48:16.382 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:48:16.379732+08:00, lastProcessTime=2024-09-11T17:48:06.351590+08:00 2024-09-11 17:48:16.393 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:48:16.395 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:48:16.395 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:48:16.395294072+08:00 2024-09-11 17:48:16.396 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:48:16.395+08:00 2024-09-11 17:48:16.397 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:48:16.396693729+08:00 2024-09-11 17:48:16.398 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:48:16.397+08:00 2024-09-11 17:48:16.399 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:48:16.395+08:00 and status AVAILABLE 2024-09-11 17:48:16.400 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:48:16.397+08:00 and status AVAILABLE 2024-09-11 17:48:16.400 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.400 INFO 2981023 --- [host-check-agent-status2,f3554883fd3cd829,224f83ddea18d4a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:48:16.401 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:16.401 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:48:16.402 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:48:16.403 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.403 INFO 2981023 --- [host-check-agent-status2,f3554883fd3cd829,224f83ddea18d4a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:48:16.403 INFO 2981023 --- [host-check-agent-status2,f3554883fd3cd829,224f83ddea18d4a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:48:16.403 INFO 2981023 --- [host-check-agent-status2,f3554883fd3cd829,224f83ddea18d4a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:48:16.403 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:16.404 INFO 2981023 --- [host-check-agent-status1,f3554883fd3cd829,463f9738519717cd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:48:16.404 INFO 2981023 --- [host-check-agent-status3,f3554883fd3cd829,4d9c4197d3631bbf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:48:16.407 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:48:16.404547156+08:00 2024-09-11 17:48:16.407 INFO 2981023 --- [host-check-agent-status1,f3554883fd3cd829,463f9738519717cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:48:16.407 INFO 2981023 --- [host-check-agent-status1,f3554883fd3cd829,463f9738519717cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:48:16.407 INFO 2981023 --- [host-check-agent-status1,f3554883fd3cd829,463f9738519717cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:48:16.408 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:48:16.407+08:00 2024-09-11 17:48:16.408 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:48:16.408 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:48:16.410 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:48:16.407+08:00 and status AVAILABLE 2024-09-11 17:48:16.411 INFO 2981023 --- [host-check-agent-status3,f3554883fd3cd829,4d9c4197d3631bbf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:48:16.411 INFO 2981023 --- [host-check-agent-status3,f3554883fd3cd829,4d9c4197d3631bbf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:48:16.411 INFO 2981023 --- [host-check-agent-status3,f3554883fd3cd829,4d9c4197d3631bbf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:48:16.413 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.413 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:16.414 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:48:16.414 INFO 2981023 --- [ocp-schedules-22,1893980c5591780e,e207c98a0d67cff5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:48:16.415 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:48:16.453 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:48:16.469 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:48:16.469 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.470 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:16.470 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:48:16.471 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:48:16.472 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:48:16.478 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.479 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:16.480 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:48:16.481 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:48:16.488 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:48:16.493 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.494 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:16.496 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:48:16.507 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.508 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.508 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:48:16.508 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:48:16.522 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:48:16.531 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:48:16.532 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:48:16.564 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:48:16.566 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:48:16.568 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:48:16.578 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.579 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.580 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:48:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:16.588 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:48:16.590 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:48:16.593 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:48:16.594 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:48:16.597 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:48:16.606 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:48:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:48:16.622 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:48:16.623 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.624 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.624 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:16.627 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.630 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:48:16.631 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:48:16.634 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:48:16.635 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:48:16.635 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:48:16.662 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:48:16.662 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:16.664 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:48:16.666 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:16.666 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:48:16.673 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.674 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.675 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:16.675 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:48:16.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:48:16.680 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:48:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.683 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.683 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:48:16.684 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:48:16.684 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:16.686 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:48:16.687 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:48:16.689 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:48:16.699 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.700 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.700 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:16.701 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:48:16.704 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:48:16.708 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:48:16.712 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:48:16.713 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:48:16.720 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:48:16.723 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:48:16.723 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.730 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:16.732 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:48:16.732 INFO 2981023 --- [host-check-host-status2,508279919c831dc3,0ab64036bda924a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:48:16.732 INFO 2981023 --- [host-check-host-status1,508279919c831dc3,b25cd332bc2774ec] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:48:16.734 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.737 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:48:16.739 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:48:16.741 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:48:16.748 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:48:16.756 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:16.756 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:48:16.759 INFO 2981023 --- [host-check-host-status3,508279919c831dc3,97122cc7f199dd96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:48:16.774 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:48:16.379732+08:00 2024-09-11 17:48:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:16.787 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.787 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:48:16.791 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:48:16.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:48:16.795 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:48:16.807 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:48:16.812 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.813 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.816 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:48:16.818 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:48:16.820 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:48:16.823 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:48:16.823 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:48:16.824 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.824 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.824 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:48:16.829 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:48:16.829 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:48:16.849 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:48:16.854 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:16.854 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:16.855 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:48:16.861 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:48:16.861 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:48:16.861 INFO 2981023 --- [ocp-schedules-19,da2e83932fc8d8b8,a5b403e6c629a5c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:48:17.290 INFO 2981023 --- [ocp-async-8,9b8d9ced6d28f55e,c80fd1dbcc3db35c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,2e707a40c14f91b3,cf2729a31b889aa6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:17.315 INFO 2981023 --- [prometheus-minute-schedule-1,2e707a40c14f91b3,cf2729a31b889aa6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:17.454 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:17.454 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:17.762 INFO 2981023 --- [ocp-async-9,6b54a03540df559a,1b8bc1f2b54052b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:48:19.290 INFO 2981023 --- [ocp-async-7,ff655aac04f75f71,f14b762ba89c2a24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:19.745 INFO 2981023 --- [host-check-clockdiff1,64002de465a256f1,bffab1295f1f80f4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048099 0 0 , error= 2024-09-11 17:48:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,28c5040af4c00d7d,ece2c44a7b4a2480] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,28c5040af4c00d7d,ece2c44a7b4a2480] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:20.454 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:20.456 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:21.290 INFO 2981023 --- [ocp-async-9,15b8027620bc07f8,32515fb088292eea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:21.932 INFO 2981023 --- [ocp-schedules-9,b0c4576179ffdcb5,145faec4d53de2a9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:48:22.003 WARN 2981023 --- [scanner-13550991-1,b0c4576179ffdcb5,d58dbd6c7434bc7b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:48:22.034 INFO 2981023 --- [alarm-detect-14,b0c4576179ffdcb5,9d4dc765c1d37798] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644692, data is empty. 2024-09-11 17:48:22.040 INFO 2981023 --- [alarm-detect-14,b0c4576179ffdcb5,9d4dc765c1d37798] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644692, data is empty. 2024-09-11 17:48:22.044 INFO 2981023 --- [alarm-detect-14,b0c4576179ffdcb5,9d4dc765c1d37798] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644692, data is empty. 2024-09-11 17:48:22.060 INFO 2981023 --- [alarm-detect-8,b0c4576179ffdcb5,9f4a8d333a3fc42f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644692, data is empty. 2024-09-11 17:48:22.062 INFO 2981023 --- [alarm-detect-5,b0c4576179ffdcb5,d73c685398cea069] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644692, data is empty. 2024-09-11 17:48:22.068 INFO 2981023 --- [alarm-detect-5,b0c4576179ffdcb5,d73c685398cea069] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644692, data is empty. 2024-09-11 17:48:22.069 INFO 2981023 --- [alarm-detect-8,b0c4576179ffdcb5,9f4a8d333a3fc42f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644692, data is empty. 2024-09-11 17:48:22.073 INFO 2981023 --- [alarm-detect-8,b0c4576179ffdcb5,9f4a8d333a3fc42f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644692, data is empty. 2024-09-11 17:48:22.074 INFO 2981023 --- [alarm-detect-5,b0c4576179ffdcb5,d73c685398cea069] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644692, data is empty. 2024-09-11 17:48:22.172 INFO 2981023 --- [alarm-detect-8,b0c4576179ffdcb5,9a91e080ac5252cc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644692, data is empty. 2024-09-11 17:48:22.177 INFO 2981023 --- [alarm-detect-8,b0c4576179ffdcb5,9a91e080ac5252cc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644692, data is empty. 2024-09-11 17:48:22.179 INFO 2981023 --- [ocp-schedules-26,3c48af5369036e34,fdb5b4609d39d13e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:48:22.179 INFO 2981023 --- [alarm-detect-8,b0c4576179ffdcb5,9a91e080ac5252cc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644692, data is empty. 2024-09-11 17:48:22.448 INFO 2981023 --- [ocp-schedules-25,745791b359661685,0cd0a70b84638bf2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 17:48:22.448 INFO 2981023 --- [ocp-schedules-25,745791b359661685,0cd0a70b84638bf2] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 17:48:22.470 INFO 2981023 --- [pool-552-thread-2,745791b359661685,bd54a0db34de4600] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:48:22.470 INFO 2981023 --- [pool-552-thread-3,745791b359661685,2b793a6e29d66bb5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:48:22.470 INFO 2981023 --- [pool-552-thread-1,745791b359661685,ad61f5943d0dbe3f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:48:22.494 INFO 2981023 --- [pool-552-thread-2,745791b359661685,bd54a0db34de4600] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:22.495 INFO 2981023 --- [pool-552-thread-3,745791b359661685,2b793a6e29d66bb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:22.496 INFO 2981023 --- [pool-552-thread-2,745791b359661685,bd54a0db34de4600] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726047802000000, 1726048102000000] 2024-09-11 17:48:22.497 INFO 2981023 --- [pool-552-thread-1,745791b359661685,ad61f5943d0dbe3f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:22.499 INFO 2981023 --- [pool-552-thread-3,745791b359661685,2b793a6e29d66bb5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726047802000000, 1726048102000000] 2024-09-11 17:48:22.500 INFO 2981023 --- [pool-552-thread-1,745791b359661685,ad61f5943d0dbe3f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726047802000000, 1726048102000000] 2024-09-11 17:48:22.502 INFO 2981023 --- [pool-552-thread-3,745791b359661685,2b793a6e29d66bb5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:48:22.574 WARN 2981023 --- [pool-552-thread-1,745791b359661685,ad61f5943d0dbe3f] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:linkprdocp, ob tenant id:1003 2024-09-11 17:48:22.581 INFO 2981023 --- [pool-552-thread-1,745791b359661685,ad61f5943d0dbe3f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:48:22.640 INFO 2981023 --- [pool-552-thread-2,745791b359661685,bd54a0db34de4600] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:48:22.641 INFO 2981023 --- [ocp-schedules-25,745791b359661685,0cd0a70b84638bf2] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 17:48:23.193 INFO 2981023 --- [ocp-schedules-18,91edad7248237228,9256138be0e151af] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:48:23.193 INFO 2981023 --- [ocp-schedules-18,91edad7248237228,9256138be0e151af] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:48:23.193 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:48:23.193 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:48:23.197 INFO 2981023 --- [ocp-schedules-18,91edad7248237228,9256138be0e151af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:48:23.197 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:48:23.197 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:48:23.201 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:48:23.201 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:48:23.201 INFO 2981023 --- [ocp-schedules-18,91edad7248237228,9256138be0e151af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:48:23.222 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:48:23.223 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:48:23.225 INFO 2981023 --- [ocp-schedules-18,91edad7248237228,9256138be0e151af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:48:23.227 INFO 2981023 --- [ocp-schedules-18,91edad7248237228,9256138be0e151af] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:48:23.290 INFO 2981023 --- [ocp-async-8,6177e7ee666bcfd3,505a5fd39ec68534] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,a03774d2c6eb5bf5,5155c29c6052de8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:23.315 INFO 2981023 --- [prometheus-minute-schedule-1,a03774d2c6eb5bf5,5155c29c6052de8c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:23.456 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:23.457 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:24.291 INFO 2981023 --- [ocp-schedules-29,7349a0d258ca534e,bd61a4c0af21813f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:48:25.290 INFO 2981023 --- [ocp-async-4,cffd87227ff32808,1ebef47be81dd0d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,bfe980232c8e8189,be355286dd6d6385] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,bfe980232c8e8189,be355286dd6d6385] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:26.358 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:48:26.357330+08:00, lastProcessTime=2024-09-11T17:48:16.379732+08:00 2024-09-11 17:48:26.457 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:26.458 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:26.493 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:48:26.357330+08:00 2024-09-11 17:48:27.290 INFO 2981023 --- [ocp-async-1,d1dfd6473e0d0541,05c9b4c38437fe8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:27.294 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.301 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obCl 2024-09-11 17:48:27.307 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.325 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 32 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.392 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 10 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.407 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 154 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.422 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.427 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.428 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:48:27.429 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.432 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.435 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.439 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.439 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:48:27.440 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.444 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 4 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.448 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.449 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.450 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.450 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.451 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.452 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.452 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.452 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:48:27.453 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.454 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.454 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.455 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.456 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.456 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.457 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.458 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:42:27.075, end=2024-09-11T17:47:27.291} 2024-09-11 17:48:27.458 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 4 dubious sqls 2024-09-11 17:48:27.762 INFO 2981023 --- [ocp-async-4,5569e0691fc2bf7d,219f8773586c43c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:48:29.290 INFO 2981023 --- [ocp-async-10,92937947be2dfe61,2e4c3c0dd8b0366a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,c9556f0cf57a2420,aae4aceea861c3f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,c9556f0cf57a2420,aae4aceea861c3f4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:29.458 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:29.459 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:31.290 INFO 2981023 --- [ocp-async-4,1149a856a3e2458b,e260bb07edbc3e83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:32.183 INFO 2981023 --- [ocp-schedules-24,a24d3d605264496a,413439b94d5795ed] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:48:32.215 INFO 2981023 --- [ocp-schedules-29,40bf10a033976ca5,d970d53da290daf1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:48:32.260 WARN 2981023 --- [scanner-13551249-1,40bf10a033976ca5,bfc8aceaeefdacf6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:48:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,9af60951bfb1bf11,5b42cb4e78cfcb0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:32.321 INFO 2981023 --- [prometheus-minute-schedule-1,9af60951bfb1bf11,5b42cb4e78cfcb0c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:32.459 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:32.459 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:32.608 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:48:32.608 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:48:32.611 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:48:32.611 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:48:32.611 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:48:32.611 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:48:32.613 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:48:32.613 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:48:32.613 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:48:32.613 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:48:32.638 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:48:32.642 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:48:32.642 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:48:32.642 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:48:32.642 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:48:32.642 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:48:32.645 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:48:32.880 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:48:33.134 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.520879154S 2024-09-11 17:48:33.134 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:48:33.214 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:48:33.214 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:48:33.244 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:48:33.335 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:48:33.335 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:48:33.335 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:48:33.335 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:48:33.335 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:48:33.335 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:48:33.343 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:48:33.478 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] o.s.boot.SpringApplication : Started application in 0.867 seconds (JVM running for 83913.351) 2024-09-11 17:48:33.478 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.867687587S 2024-09-11 17:48:33.785 INFO 2981023 --- [ocp-schedules-20,f0bcaa75a37d7212,19b10a1323fa7a61] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:48:34.409 INFO 2981023 --- [ocp-schedules-28,acd4684fde83c21c,cfdf8985ca68360a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:48:34.417 INFO 2981023 --- [ocp-schedules-28,acd4684fde83c21c,cfdf8985ca68360a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:48:34.417 INFO 2981023 --- [ocp-schedules-28,acd4684fde83c21c,cfdf8985ca68360a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:48:35.291 INFO 2981023 --- [ocp-async-2,46dfe0074f239aed,dca9fa1e91380fbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,44263130484a38b4,10d9b37e211af6c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,44263130484a38b4,10d9b37e211af6c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:35.459 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:35.460 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:48:36.337875+08:00, lastProcessTime=2024-09-11T17:48:26.357330+08:00 2024-09-11 17:48:36.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:48:36.337875+08:00 2024-09-11 17:48:37.291 INFO 2981023 --- [ocp-async-5,6c15f35017b723ca,152e64a5e6085d69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:37.314 INFO 2981023 --- [prometheus-cache-monitor1,4ad69d07e523356c,2e69eac6866bce5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:48:37.314 INFO 2981023 --- [prometheus-cache-monitor1,4ad69d07e523356c,2e69eac6866bce5e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:48:37.762 INFO 2981023 --- [ocp-async-9,33384e98c1769fca,7c0d5f4a8a554ddd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:48:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,1be9b434d5c58430,1b133dd5a77a58da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,1be9b434d5c58430,1b133dd5a77a58da] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:38.460 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:38.461 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:39.291 INFO 2981023 --- [ocp-async-3,106dacee52bcd64b,0f09e713d4e90841] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:40.003 INFO 2981023 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:48:40.003 INFO 2981023 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:48:41.291 INFO 2981023 --- [ocp-async-8,bbe76581233e6a07,65f497c658a3d858] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,36d608dbba22fb87,9923da362ac64782] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:41.319 INFO 2981023 --- [prometheus-minute-schedule-1,36d608dbba22fb87,9923da362ac64782] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:41.461 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:41.462 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:41.564 INFO 2981023 --- [ocp-schedules-13,01b342c7b49c3e97,cc4680b397923981] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:48:41.583 WARN 2981023 --- [ocp-schedules-13,01b342c7b49c3e97,cc4680b397923981] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:48:42.186 INFO 2981023 --- [ocp-schedules-30,f907cab0ca56c48b,78a4ff6a7fc06b56] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:48:42.410 INFO 2981023 --- [ocp-schedules-26,581611a70222f07c,7e72f275035bffa6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:48:42.458 WARN 2981023 --- [scanner-13551511-1,581611a70222f07c,040e0b114bdd5caa] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:48:42.469 INFO 2981023 --- [alarm-detect-12,581611a70222f07c,61b2c1f293ecde54] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644692, data is empty. 2024-09-11 17:48:42.472 INFO 2981023 --- [alarm-detect-12,581611a70222f07c,61b2c1f293ecde54] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644692, data is empty. 2024-09-11 17:48:42.474 INFO 2981023 --- [alarm-detect-12,581611a70222f07c,61b2c1f293ecde54] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644692, data is empty. 2024-09-11 17:48:42.514 INFO 2981023 --- [alarm-detect-12,581611a70222f07c,01aa8ddc0b6795c6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644692, data is empty. 2024-09-11 17:48:42.519 INFO 2981023 --- [alarm-detect-6,581611a70222f07c,f4f9836f2d04d341] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644692, data is empty. 2024-09-11 17:48:42.522 INFO 2981023 --- [alarm-detect-12,581611a70222f07c,01aa8ddc0b6795c6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644692, data is empty. 2024-09-11 17:48:42.526 INFO 2981023 --- [alarm-detect-12,581611a70222f07c,01aa8ddc0b6795c6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644692, data is empty. 2024-09-11 17:48:42.528 INFO 2981023 --- [alarm-detect-6,581611a70222f07c,f4f9836f2d04d341] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644692, data is empty. 2024-09-11 17:48:42.533 INFO 2981023 --- [alarm-detect-6,581611a70222f07c,f4f9836f2d04d341] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644692, data is empty. 2024-09-11 17:48:42.610 INFO 2981023 --- [alarm-detect-11,581611a70222f07c,06afe4121cbd46f5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644692, data is empty. 2024-09-11 17:48:42.615 INFO 2981023 --- [alarm-detect-11,581611a70222f07c,06afe4121cbd46f5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644692, data is empty. 2024-09-11 17:48:42.618 INFO 2981023 --- [alarm-detect-11,581611a70222f07c,06afe4121cbd46f5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644692, data is empty. 2024-09-11 17:48:43.291 INFO 2981023 --- [ocp-async-10,8fc5e368b9cf01fb,c9449a42d9b37ec4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,e4f2970abaf66c39,d2d34704d0498932] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,e4f2970abaf66c39,d2d34704d0498932] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:44.462 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:44.463 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:45.101 INFO 2981023 --- [prometheus-second-schedule-1,93d3dd8eabbd219d,9b84e2b5d01454c6] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:48:45.291 INFO 2981023 --- [ocp-async-7,8a3a970646683042,4958d38931ed962f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:46.020 INFO 2981023 --- [ocp-schedules-6,a112665e1cf8d606,927a65e716faba50] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:48:46.020 INFO 2981023 --- [host-check-clockdiff1,a112665e1cf8d606,889eb88a9dde128e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:48:46.021 INFO 2981023 --- [host-check-clockdiff2,a112665e1cf8d606,6c80768531f85faf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:48:46.021 INFO 2981023 --- [host-check-clockdiff3,a112665e1cf8d606,a420d24a6e1ff4f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:48:46.024 INFO 2981023 --- [ocp-schedules-13,2d665fcfff32c451,dea3ed3bc90dfdad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:48:46.024 INFO 2981023 --- [ocp-schedules-25,41a332b7c4681b9c,676a7bb673195c85] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:48:46.024 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:48:46.024 INFO 2981023 --- [host-check-agent-status1,41a332b7c4681b9c,e330f621706043fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:48:46.030 INFO 2981023 --- [host-check-agent-status2,41a332b7c4681b9c,6f05fe00d4fd7b07] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:48:46.035 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:48:46.037 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:48:46.039 INFO 2981023 --- [host-check-clockdiff1,a112665e1cf8d606,889eb88a9dde128e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048126 0 0 , error= 2024-09-11 17:48:46.039 INFO 2981023 --- [host-check-clockdiff3,a112665e1cf8d606,a420d24a6e1ff4f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048126 0 0 , error= 2024-09-11 17:48:46.040 INFO 2981023 --- [host-check-clockdiff2,a112665e1cf8d606,6c80768531f85faf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048126 0 0 , error= 2024-09-11 17:48:46.039 INFO 2981023 --- [host-check-agent-status3,41a332b7c4681b9c,93cd002ccc134845] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:48:46.043 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:48:46.044 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:48:46.045 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:48:46.045 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:48:46.045 INFO 2981023 --- [host-check-agent-status1,41a332b7c4681b9c,e330f621706043fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:48:46.045 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:48:46.046 INFO 2981023 --- [host-check-agent-status2,41a332b7c4681b9c,6f05fe00d4fd7b07] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:48:46.046 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:48:46.046 INFO 2981023 --- [host-check-agent-status1,41a332b7c4681b9c,e330f621706043fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:48:46.047 INFO 2981023 --- [host-check-agent-status2,41a332b7c4681b9c,6f05fe00d4fd7b07] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:48:46.048 INFO 2981023 --- [host-check-agent-status3,41a332b7c4681b9c,93cd002ccc134845] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:48:46.048 INFO 2981023 --- [host-check-agent-status3,41a332b7c4681b9c,93cd002ccc134845] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:48:46.067 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:48:46.067 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:48:46.069 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:48:46.069 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:48:46.069145280+08:00 2024-09-11 17:48:46.070 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:48:46.069+08:00 2024-09-11 17:48:46.070 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:48:46.070487036+08:00 2024-09-11 17:48:46.071 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:48:46.070937504+08:00 2024-09-11 17:48:46.072 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:48:46.070+08:00 2024-09-11 17:48:46.072 INFO 2981023 --- [host-check-agent-status2,41a332b7c4681b9c,6f05fe00d4fd7b07] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:48:46.074 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:48:46.071+08:00 2024-09-11 17:48:46.077 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:48:46.070+08:00 and status AVAILABLE 2024-09-11 17:48:46.077 INFO 2981023 --- [host-check-agent-status2,41a332b7c4681b9c,6f05fe00d4fd7b07] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:48:46.077 INFO 2981023 --- [host-check-agent-status2,41a332b7c4681b9c,6f05fe00d4fd7b07] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:48:46.077 INFO 2981023 --- [host-check-agent-status2,41a332b7c4681b9c,6f05fe00d4fd7b07] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:48:46.078 INFO 2981023 --- [host-check-agent-status3,41a332b7c4681b9c,93cd002ccc134845] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:48:46.078 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:48:46.069+08:00 and status AVAILABLE 2024-09-11 17:48:46.081 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:48:46.071+08:00 and status AVAILABLE 2024-09-11 17:48:46.083 INFO 2981023 --- [host-check-agent-status3,41a332b7c4681b9c,93cd002ccc134845] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:48:46.083 INFO 2981023 --- [host-check-agent-status3,41a332b7c4681b9c,93cd002ccc134845] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:48:46.083 INFO 2981023 --- [host-check-agent-status3,41a332b7c4681b9c,93cd002ccc134845] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:48:46.102 INFO 2981023 --- [host-check-agent-status1,41a332b7c4681b9c,e330f621706043fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:48:46.103 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:48:46.103 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:48:46.103 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:48:46.109 INFO 2981023 --- [host-check-agent-status1,41a332b7c4681b9c,e330f621706043fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:48:46.109 INFO 2981023 --- [host-check-agent-status1,41a332b7c4681b9c,e330f621706043fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:48:46.109 INFO 2981023 --- [host-check-agent-status1,41a332b7c4681b9c,e330f621706043fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:48:46.118 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:48:46.118 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:48:46.118 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:48:46.119 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:48:46.119 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:48:46.120 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:48:46.138 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:48:46.140 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:48:46.142 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:48:46.158 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:48:46.162 INFO 2981023 --- [ocp-schedules-12,4e9271a562bb93ee,fe9c4c4dacd2bd9b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:48:46.170 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:48:46.170 INFO 2981023 --- [ocp-schedules-12,4e9271a562bb93ee,fe9c4c4dacd2bd9b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:48:46.175 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:48:46.188 INFO 2981023 --- [ocp-schedules-1,af748762f7687d82,ffe808dfba1e5627] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:48:46.188 INFO 2981023 --- [ocp-schedules-21,e9c61a3f4a5920e0,66efedb01a9ae798] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:48:46.191 INFO 2981023 --- [ocp-schedules-21,e9c61a3f4a5920e0,66efedb01a9ae798] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:48:46.192 INFO 2981023 --- [ocp-schedules-1,af748762f7687d82,ffe808dfba1e5627] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:48:46.195 INFO 2981023 --- [ocp-schedules-27,90b94bba06304be4,26ad97214b010482] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:48:46.195 INFO 2981023 --- [ocp-schedules-15,ba13942a52b89988,e2ee311e493d7a09] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:48:46.197 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:48:46.197 INFO 2981023 --- [ocp-schedules-27,90b94bba06304be4,26ad97214b010482] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:48:46.197 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:48:46.210 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:48:46.223 INFO 2981023 --- [ocp-schedules-21,e9c61a3f4a5920e0,66efedb01a9ae798] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:48:46.237 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:48:46.247 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:48:46.252 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.253 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:46.254 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:48:46.265 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:48:46.279 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.281 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:46.282 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:48:46.285 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.286 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:46.287 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:48:46.290 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:48:46.295 INFO 2981023 --- [ocp-schedules-29,96da3ee68ba26504,e248fef3804f4682] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:48:46.298 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.299 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:46.299 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:48:46.300 INFO 2981023 --- [ocp-schedules-1,af748762f7687d82,ffe808dfba1e5627] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:48:46.303 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.304 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:46.305 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.306 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:46.306 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:48:46.315 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:48:46.317 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:48:46.323 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.323 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:48:46.323 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:48:46.324 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.337 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:48:46.338 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:48:46.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:48:46.350468+08:00, lastProcessTime=2024-09-11T17:48:36.337875+08:00 2024-09-11 17:48:46.364 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.367 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:48:46.372 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:48:46.378 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.380 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:48:46.390 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:48:46.396 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:48:46.400 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.402 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.403 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:48:46.403 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:48:46.403 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.405 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:48:46.408 INFO 2981023 --- [host-check-host-status1,2d665fcfff32c451,bed3a124978c40ee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:48:46.419 INFO 2981023 --- [host-check-host-status2,2d665fcfff32c451,5bb6452fa7de7642] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:48:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.429 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:48:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:46.431 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:48:46.432 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:48:46.436 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.439 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:48:46.439 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:48:46.440 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:48:46.452 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.452 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.453 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:48:46.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:46.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:48:46.461 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.463 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:48:46.470 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.471 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.471 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:48:46.472 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.472 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.473 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:48:46.474 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:48:46.477 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.478 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:48:46.479 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:48:46.479 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:46.483 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.484 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.484 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:46.485 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.485 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:48:46.488 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:48:46.488 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:48:46.491 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:46.493 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:48:46.496 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:48:46.502 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:46.502 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:48:46.502 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:48:46.503 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.504 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.504 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:48:46.505 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:48:46.505 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.505 INFO 2981023 --- [host-check-host-status3,2d665fcfff32c451,99e8ffd990163250] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:48:46.508 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:48:46.508 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:48:46.509 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:48:46.510 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:48:46.511 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:48:46.518 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:48:46.527 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.528 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.528 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:48:46.529 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:48:46.529 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.531 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:48:46.534 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:48:46.540 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.541 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:46.541 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:48:46.542 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.543 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:48:46.544 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.544 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:48:46.545 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:48:46.545 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:48:46.547 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:48:46.549 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:48:46.552 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:48:46.553 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:48:46.555 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:48:46.563 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:48:46.572 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:48:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:48:46.580 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.580 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.581 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:48:46.581 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:48:46.581 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.599 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:48:46.605 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:48:46.608 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.612 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.612 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:48:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.614 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:48:46.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:48:46.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:48:46.621 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:48:46.621 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:48:46.631 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:48:46.640 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:48:46.646 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:46.647 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:46.647 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:48:46.648 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:48:46.649 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:48:46.350468+08:00 2024-09-11 17:48:46.653 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:48:46.653 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:48:46.654 INFO 2981023 --- [ocp-schedules-27,90b94bba06304be4,26ad97214b010482] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:48:46.655 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:48:46.658 INFO 2981023 --- [ocp-schedules-7,191af2f8d3530660,87f724c18ffb4ef2] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:48:47.291 INFO 2981023 --- [ocp-async-8,2bdcfa07c882075a,31194adeabc8e1dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,4cafcaefcefdd452,1b62a29532b13ac6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,4cafcaefcefdd452,1b62a29532b13ac6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:47.463 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:47.463 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:47.762 INFO 2981023 --- [ocp-async-2,890a685850431a87,a91a3c74bf36144c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:48:49.292 INFO 2981023 --- [ocp-async-5,84d2ae5dd559b659,43ab5afe6364fbcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:48:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:48:50.258 INFO 2981023 --- [ocp-schedules-23,7bb20adf42648ba5,5e3402fc5a919216] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:48:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,d17f98a5ab23e03d,1db829c9ffde4fdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:50.318 INFO 2981023 --- [prometheus-minute-schedule-1,d17f98a5ab23e03d,1db829c9ffde4fdb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:50.463 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:50.465 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:51.292 INFO 2981023 --- [ocp-async-2,b64232ed8b750536,64a30ba1eaa93366] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:51.565 INFO 2981023 --- [ocp-schedules-5,df4338a4b493932e,36f45e26532864c9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:48:51.566 INFO 2981023 --- [ocp-schedules-5,df4338a4b493932e,36f45e26532864c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:48:51.567 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:48:51.567 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:48:51.567 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:48:51.567 INFO 2981023 --- [ocp-schedules-5,df4338a4b493932e,36f45e26532864c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:48:52.189 INFO 2981023 --- [ocp-schedules-23,adbf2b002e9cfb67,e6c03a8c2aa9827f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:48:52.653 INFO 2981023 --- [ocp-schedules-6,8afba61801fa3608,39daa06eb70d78ec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:48:52.708 WARN 2981023 --- [scanner-13551846-1,8afba61801fa3608,94e6526789678c10] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:48:52.852 INFO 2981023 --- [ocp-schedules-24,f4c7c3afc529594c,4f07a9f81a9e8971] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:48:53.148 INFO 2981023 --- [ocp-schedules-5,e68e1b15d4728754,118c07deb0cda0ff] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:48:53.148 INFO 2981023 --- [ocp-schedules-5,e68e1b15d4728754,118c07deb0cda0ff] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:48:53.292 INFO 2981023 --- [ocp-async-8,f2ba1206364981de,b63a873155858328] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,4ed0186f54d5ab7c,046a6b9d008e27cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,4ed0186f54d5ab7c,046a6b9d008e27cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:53.393 INFO 2981023 --- [ocp-schedules-9,4d3e767603210010,0129e87f8d88ac70] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:48:53.412 WARN 2981023 --- [ocp-schedules-9,4d3e767603210010,0129e87f8d88ac70] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:48:53.465 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:53.467 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:54.294 INFO 2981023 --- [ocp-schedules-20,8d26660690ea4724,1232fdac7e502012] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:48:54.420 INFO 2981023 --- [ocp-schedules-6,ecab83b3c975fa8e,5178a4f6b2985541] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:48:54.429 INFO 2981023 --- [ocp-schedules-6,ecab83b3c975fa8e,5178a4f6b2985541] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:48:54.429 INFO 2981023 --- [ocp-schedules-6,ecab83b3c975fa8e,5178a4f6b2985541] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:48:54.592 INFO 2981023 --- [ocp-schedules-16,da7847c9ed9cc463,b4e9101c3ebb1444] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:48:54.592+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:48:54.597 INFO 2981023 --- [ocp-schedules-16,da7847c9ed9cc463,b4e9101c3ebb1444] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:48:54.668 INFO 2981023 --- [ocp-schedules-5,dd160761dc6f6e61,ee21c07652af051c] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T17:48:54.668+08:00, tenantSize=6 2024-09-11 17:48:54.712 INFO 2981023 --- [ocp-schedules-5,dd160761dc6f6e61,ee21c07652af051c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726048134668000 2024-09-11 17:48:54.712 INFO 2981023 --- [ocp-schedules-5,dd160761dc6f6e61,ee21c07652af051c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726047834612000 2024-09-11 17:48:54.729 INFO 2981023 --- [ocp-schedules-5,dd160761dc6f6e61,ee21c07652af051c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 17:48:54.759 INFO 2981023 --- [ocp-schedules-16,da7847c9ed9cc463,b4e9101c3ebb1444] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 47 to SqlAggregateData 2024-09-11 17:48:54.760 INFO 2981023 --- [ocp-schedules-16,da7847c9ed9cc463,b4e9101c3ebb1444] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:48:54.862 INFO 2981023 --- [ocp-schedules-25,9c7b81e4b77b3555,faa22bbceb13651a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:48:54.862 INFO 2981023 --- [ocp-schedules-25,9c7b81e4b77b3555,faa22bbceb13651a] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:48:54.867 INFO 2981023 --- [ocp-schedules-25,9c7b81e4b77b3555,faa22bbceb13651a] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:48:54.867 INFO 2981023 --- [ocp-schedules-25,9c7b81e4b77b3555,faa22bbceb13651a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:48:54.867 INFO 2981023 --- [ocp-schedules-25,9c7b81e4b77b3555,faa22bbceb13651a] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:48:54.876 INFO 2981023 --- [ocp-schedules-25,9c7b81e4b77b3555,faa22bbceb13651a] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:48:54.876 INFO 2981023 --- [ocp-schedules-25,9c7b81e4b77b3555,faa22bbceb13651a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:48:54.876 INFO 2981023 --- [ocp-schedules-25,9c7b81e4b77b3555,faa22bbceb13651a] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:48:54.887 INFO 2981023 --- [ocp-schedules-25,9c7b81e4b77b3555,faa22bbceb13651a] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:48:55.292 INFO 2981023 --- [ocp-async-10,63b81787e87f9005,d4289dc52390c12a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:55.301 INFO 2981023 --- [ocp-schedules-11,c56f900296632db5,43c7037ff2d7f27c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:48:55.302 INFO 2981023 --- [ocp-schedules-11,c56f900296632db5,43c7037ff2d7f27c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:48:55.347 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:48:55.347 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:48:55.347 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.348 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.348 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:48:55.348 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.351 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:48:55.351 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:48:55.352 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:48:55.357 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.358 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.359 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.360 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.360 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.360 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.364 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.365 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.370 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.370 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:48:55.370 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.371 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.372 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.372 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:48:55.386 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.387 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.387 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.388 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:48:55.388 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.388 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:48:55.388 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3611D6EB-0-0] 2024-09-11 17:48:55.388 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:48:55.388 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.388 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.388 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:48:55.400 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.400 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.400 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:48:55.400 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:48:55.400 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.401 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:48:55.401 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.402 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:48:55.402 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.403 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:48:55.404 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:48:55.411 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.412 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:48:55.412 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:48:55.413 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3611D6F5-0-0] 2024-09-11 17:48:55.413 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:48:55.413 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:48:55.417 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.417 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:48:55.418 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.419 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:48:55.419 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.420 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:48:55.420 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.421 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.421 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:48:55.421 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:48:55.422 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.422 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.423 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:48:55.423 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:48:55.423 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.424 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.424 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:48:55.424 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.425 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:48:55.425 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:48:55.425 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.425 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.426 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:48:55.426 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:48:55.426 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.426 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.427 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:48:55.427 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:48:55.427 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.428 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:48:55.428 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:48:55.428 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:55.428 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:48:55.430 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:48:55.430 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.430 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:48:55.431 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.443 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:55.444 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:48:55.444 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.444 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:48:55.445 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:48:55.446 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.446 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:48:55.448 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:48:55.448 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:48:55.448 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:48:55.451 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:48:55.453 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:48:55.453 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.454 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:48:55.456 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:48:55.458 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:48:55.458 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.459 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.460 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:48:55.461 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:48:55.461 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:55.462 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:48:55.468 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:48:55.469 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:48:55.469 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:48:55.469 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:48:55.469 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:48:55.471 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:48:55.471 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:48:55.471 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:48:55.471 INFO 2981023 --- [ocp-schedules-11,c56f900296632db5,43c7037ff2d7f27c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:48:55.751 INFO 2981023 --- [ocp-schedules-22,4ef2bb0a0d588307,3e7cb0306855439c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:48:55.751 INFO 2981023 --- [ocp-schedules-22,4ef2bb0a0d588307,3e7cb0306855439c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:48:55.755 INFO 2981023 --- [ocp-schedules-22,4ef2bb0a0d588307,3e7cb0306855439c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:48:55.755 INFO 2981023 --- [ocp-schedules-22,4ef2bb0a0d588307,3e7cb0306855439c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:48:55.757 INFO 2981023 --- [ocp-schedules-22,4ef2bb0a0d588307,3e7cb0306855439c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:48:55.757 INFO 2981023 --- [ocp-schedules-22,4ef2bb0a0d588307,3e7cb0306855439c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:48:55.854 INFO 2981023 --- [ocp-schedules-10,79becae104fb1f6e,f2ffdb76e85123ec] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:48:55.861 INFO 2981023 --- [ocp-schedules-10,79becae104fb1f6e,f2ffdb76e85123ec] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:48:55.861 INFO 2981023 --- [ocp-schedules-10,79becae104fb1f6e,f2ffdb76e85123ec] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:48:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,720a4b427b662626,4b3d715f09bea9a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,720a4b427b662626,4b3d715f09bea9a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:48:56.352842+08:00, lastProcessTime=2024-09-11T17:48:46.350468+08:00 2024-09-11 17:48:56.467 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:56.467 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:48:56.504 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:48:56.352842+08:00 2024-09-11 17:48:56.695 INFO 2981023 --- [ocp-schedules-27,6956dcd99845f33f,8a71c2dd31d77321] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:48:56.695 INFO 2981023 --- [ocp-schedules-27,6956dcd99845f33f,8a71c2dd31d77321] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:48:56.709 INFO 2981023 --- [Collect all tenants major compactions1,6956dcd99845f33f,f322c6c39ca5809d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:48:56.709 INFO 2981023 --- [Collect all tenants major compactions3,6956dcd99845f33f,2ffa471463494a32] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:48:56.709 INFO 2981023 --- [Collect all tenants major compactions6,6956dcd99845f33f,cef891f361f8b19e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:48:56.710 INFO 2981023 --- [Collect all tenants major compactions2,6956dcd99845f33f,8842e9054106bc9f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:48:56.711 INFO 2981023 --- [Collect all tenants major compactions4,6956dcd99845f33f,76fadebafb06cd6d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:48:56.712 INFO 2981023 --- [Collect all tenants major compactions5,6956dcd99845f33f,94f321677a08753e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:48:56.725 INFO 2981023 --- [Collect all tenants major compactions6,6956dcd99845f33f,cef891f361f8b19e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:56.726 INFO 2981023 --- [Collect all tenants major compactions6,6956dcd99845f33f,cef891f361f8b19e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:48:56.727 INFO 2981023 --- [Collect all tenants major compactions5,6956dcd99845f33f,94f321677a08753e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:56.728 INFO 2981023 --- [Collect all tenants major compactions2,6956dcd99845f33f,8842e9054106bc9f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:56.728 INFO 2981023 --- [Collect all tenants major compactions5,6956dcd99845f33f,94f321677a08753e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:48:56.729 INFO 2981023 --- [Collect all tenants major compactions2,6956dcd99845f33f,8842e9054106bc9f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:48:56.729 INFO 2981023 --- [Collect all tenants major compactions4,6956dcd99845f33f,76fadebafb06cd6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:56.729 INFO 2981023 --- [Collect all tenants major compactions6,6956dcd99845f33f,cef891f361f8b19e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:48:56.730 INFO 2981023 --- [Collect all tenants major compactions4,6956dcd99845f33f,76fadebafb06cd6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:48:56.731 INFO 2981023 --- [Collect all tenants major compactions1,6956dcd99845f33f,f322c6c39ca5809d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:56.731 INFO 2981023 --- [Collect all tenants major compactions3,6956dcd99845f33f,2ffa471463494a32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:48:56.732 INFO 2981023 --- [Collect all tenants major compactions1,6956dcd99845f33f,f322c6c39ca5809d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:48:56.732 INFO 2981023 --- [Collect all tenants major compactions6,6956dcd99845f33f,cef891f361f8b19e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:48:56.733 INFO 2981023 --- [Collect all tenants major compactions1,6956dcd99845f33f,f322c6c39ca5809d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:48:56.733 INFO 2981023 --- [Collect all tenants major compactions1,6956dcd99845f33f,f322c6c39ca5809d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:48:56.735 INFO 2981023 --- [Collect all tenants major compactions4,6956dcd99845f33f,76fadebafb06cd6d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:48:56.735 INFO 2981023 --- [Collect all tenants major compactions4,6956dcd99845f33f,76fadebafb06cd6d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:48:56.735 INFO 2981023 --- [Collect all tenants major compactions5,6956dcd99845f33f,94f321677a08753e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:48:56.735 INFO 2981023 --- [Collect all tenants major compactions5,6956dcd99845f33f,94f321677a08753e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:48:56.736 INFO 2981023 --- [Collect all tenants major compactions2,6956dcd99845f33f,8842e9054106bc9f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:48:56.736 INFO 2981023 --- [Collect all tenants major compactions2,6956dcd99845f33f,8842e9054106bc9f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:48:56.736 INFO 2981023 --- [Collect all tenants major compactions3,6956dcd99845f33f,2ffa471463494a32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:48:56.738 INFO 2981023 --- [Collect all tenants major compactions3,6956dcd99845f33f,2ffa471463494a32] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:48:56.738 INFO 2981023 --- [Collect all tenants major compactions3,6956dcd99845f33f,2ffa471463494a32] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:48:56.739 INFO 2981023 --- [ocp-schedules-27,6956dcd99845f33f,8a71c2dd31d77321] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:48:57.292 INFO 2981023 --- [ocp-async-2,f5c6b6efb9fc413b,00a58a3770033e63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:57.762 INFO 2981023 --- [ocp-async-5,6ecdf9d092e7939f,173aeda9f57b71bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:48:59.292 INFO 2981023 --- [ocp-async-8,cb710f4bc7a8c316,b7b2bacf0c3f8e6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:48:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,3d4acd77d1809c54,0a739ca94a866378] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048140, interval=60 2024-09-11 17:48:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,3d4acd77d1809c54,0a739ca94a866378] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:48:59.467 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:48:59.468 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:01.292 INFO 2981023 --- [ocp-async-5,90dd4254f45b9c1f,cc8de6a6b4b295a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:02.191 INFO 2981023 --- [ocp-schedules-5,1fb1c27f60f1825b,d521f0ce5cc6d432] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:49:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,46c4d40553c10110,a7bc3a49b9a409a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,46c4d40553c10110,a7bc3a49b9a409a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:02.468 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:02.468 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:02.902 INFO 2981023 --- [ocp-schedules-2,2c918f9688a483b6,45ff86d76f59c0be] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:49:02.949 WARN 2981023 --- [scanner-13552065-1,2c918f9688a483b6,62cd65778914da18] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:49:02.970 INFO 2981023 --- [alarm-detect-17,2c918f9688a483b6,0acea08f6927b157] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644692, data is empty. 2024-09-11 17:49:02.973 INFO 2981023 --- [alarm-detect-17,2c918f9688a483b6,0acea08f6927b157] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644739, data is empty. 2024-09-11 17:49:02.978 INFO 2981023 --- [alarm-detect-17,2c918f9688a483b6,0acea08f6927b157] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644739, data is empty. 2024-09-11 17:49:02.999 INFO 2981023 --- [alarm-detect-19,2c918f9688a483b6,09d13fd284ef85da] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644692, data is empty. 2024-09-11 17:49:02.999 INFO 2981023 --- [alarm-detect-14,2c918f9688a483b6,133d1345c4526feb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644692, data is empty. 2024-09-11 17:49:03.006 INFO 2981023 --- [alarm-detect-14,2c918f9688a483b6,133d1345c4526feb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644739, data is empty. 2024-09-11 17:49:03.007 INFO 2981023 --- [alarm-detect-19,2c918f9688a483b6,09d13fd284ef85da] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644739, data is empty. 2024-09-11 17:49:03.010 INFO 2981023 --- [alarm-detect-19,2c918f9688a483b6,09d13fd284ef85da] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644739, data is empty. 2024-09-11 17:49:03.012 INFO 2981023 --- [alarm-detect-14,2c918f9688a483b6,133d1345c4526feb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644739, data is empty. 2024-09-11 17:49:03.113 INFO 2981023 --- [alarm-detect-9,2c918f9688a483b6,f7d1fb2e428e5254] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644692, data is empty. 2024-09-11 17:49:03.116 INFO 2981023 --- [alarm-detect-9,2c918f9688a483b6,f7d1fb2e428e5254] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644739, data is empty. 2024-09-11 17:49:03.119 INFO 2981023 --- [alarm-detect-9,2c918f9688a483b6,f7d1fb2e428e5254] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644739, data is empty. 2024-09-11 17:49:03.292 INFO 2981023 --- [ocp-async-3,1c44fc87e267c034,438004d83e0a7766] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:05.131 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:45:04.766, end=2024-09-11T17:47:05.127} 2024-09-11 17:49:05.135 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClu 2024-09-11 17:49:05.135 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:49:05.136 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:45:04.766, end=2024-09-11T17:47:05.127},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:49:05.137 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:49:05.137 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:49:05.145 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:49:05.164 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745164000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.164Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.167 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:49:05.167 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.172 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.172 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.174 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.174 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.179 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.179 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.180 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:49:05.180 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.164, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.181 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:49:05.181 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:45:04.766, end=2024-09-11T17:47:05.127},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:49:05.181 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:49:05.181 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:49:05.182 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:49:05.183 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:45:04.766, end=2024-09-11T17:47:05.127},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:49:05.183 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:49:05.183 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:49:05.184 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:49:05.184 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:45:04.766, end=2024-09-11T17:47:05.127},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:49:05.184 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:49:05.184 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:49:05.185 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:49:05.187 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:45:04.766, end=2024-09-11T17:47:05.127},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:49:05.187 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:49:05.187 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:49:05.228 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745227000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.227Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.230 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745230000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.230Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.235 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745235000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.235Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.235 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745235000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.235Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.238 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745238000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.238Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.240 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745240000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.240Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.241 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745241000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.241Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.242 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745242000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.242Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.244 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 63 2024-09-11 17:49:05.244 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.254 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.254 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.256 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:49:05.256 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.257 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:49:05.257 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.264 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:49:05.264 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.267 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:49:05.267 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.269 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.269 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.273 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:49:05.273 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.274 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.274 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.276 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.276 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.278 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:49:05.278 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.279 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.279 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.281 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:49:05.281 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.281 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.281 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.284 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.284 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.286 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.286 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.287 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.287 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.287 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.287 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.290 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.290 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.290 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.290 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.291 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.291 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.291 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.291 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.292 INFO 2981023 --- [ocp-async-9,2d2c8c07f620026e,23788c11f9b4d312] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:05.293 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.293 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.293 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.294 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.294 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.295 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.296 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.296 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.297 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.297 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.299 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.300 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.300 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.303 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.303 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.305 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.305 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.308 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.308 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.308 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.309 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.309 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.310 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.23, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.311 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.312 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.235, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.312 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.313 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.313 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.313 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,06c155a3c8050b98,c52fe5abf5cf58fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:05.314 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.227, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.317 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.238, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.318 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.235, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.320 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.320 INFO 2981023 --- [prometheus-minute-schedule-1,06c155a3c8050b98,c52fe5abf5cf58fe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:05.321 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.24, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.326 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.241, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.341 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.242, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.351 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745351000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.351Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.352 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745352000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.352Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.355 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745355000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.355Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.362 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745362000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.362Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.366 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:49:05.366 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.368 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745368000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=C0995E24A01F869BEB311E8C790E6C86, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.368Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.370 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745370000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.370Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.371 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:49:05.371 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.371 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961745371000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:49:05.371Z, endTime=2024-09-11T09:47:05.127Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:49:05.373 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:49:05.373 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.377 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:49:05.377 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.379 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.379 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.383 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.383 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.384 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:49:05.384 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.387 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.387 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.387 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.387 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.387 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:49:05.387 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.388 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.388 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.390 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:49:05.390 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:49:05.392 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.392 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.398 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.398 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.400 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.400 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.400 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.400 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.401 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.401 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.401 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.401 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.401 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.401 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.403 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.403 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.404 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.404 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.404 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.405 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.405 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.405 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:49:05.405 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:49:05.405 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.406 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.406 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.409 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.410 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.351, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.412 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.352, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.414 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.355, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.415 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.415 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.417 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.417 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.417 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.418 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.418 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.418 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:49:05.418 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048025127000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:49:05.419 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.420 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.420 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:49:05.421 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.362, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.423 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.371, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.424 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.368, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=C0995E24A01F869BEB311E8C790E6C86, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.424 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:49:05.37, end=2024-09-11T17:47:05.127}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:49:05.424 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:49:05.426 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:45:04.766, end=2024-09-11T17:47:05.127},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:49:05.426 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:49:05.426 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:49:05.435 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:49:05.435 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:49:05.468 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:05.469 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:49:06.336692+08:00, lastProcessTime=2024-09-11T17:48:56.352842+08:00 2024-09-11 17:49:06.466 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:49:06.336692+08:00 2024-09-11 17:49:07.292 INFO 2981023 --- [ocp-async-4,38682cdfe68dff88,83eb6e02d6af9a77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:07.520 INFO 2981023 --- [ocp-schedules-17,88f849f14aaaed6a,8ab9e8eac16a5961] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:49:07.549 INFO 2981023 --- [ocp-schedules-6,e8108e226cf61326,ec1a0e4c7bac9877] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:49:07.549+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:49:07.550 INFO 2981023 --- [ocp-schedules-6,e8108e226cf61326,ec1a0e4c7bac9877] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:49:07.552 INFO 2981023 --- [ocp-schedules-6,e8108e226cf61326,ec1a0e4c7bac9877] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:49:07.552 INFO 2981023 --- [ocp-schedules-6,e8108e226cf61326,ec1a0e4c7bac9877] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:49:07.762 INFO 2981023 --- [ocp-async-9,5d22f58e6c4691eb,61218b5d38065eff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:49:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,fb398a9f6240c9b9,d6882b7dfaf9ed0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,fb398a9f6240c9b9,d6882b7dfaf9ed0e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:08.469 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:08.469 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:09.293 INFO 2981023 --- [ocp-async-2,78ed8f7ca741fe52,2430e02e116cbc5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:10.002 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:49:10.002 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:49:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:49:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:49:11.293 INFO 2981023 --- [ocp-async-9,f99fc4e726b877d7,e4562804ed5de41c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,2082525a9e2b0587,4d752564e9dfbdc4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:11.319 INFO 2981023 --- [prometheus-minute-schedule-1,2082525a9e2b0587,4d752564e9dfbdc4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:11.470 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:11.470 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:12.194 INFO 2981023 --- [ocp-schedules-13,edf6c5f923c4b47b,421212ffc06a6ada] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:49:13.165 INFO 2981023 --- [ocp-schedules-9,c26cd0ee0526b495,85b4b6a2eb28d200] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:49:13.251 WARN 2981023 --- [scanner-13553892-1,c26cd0ee0526b495,1931bb019c3474c5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:49:13.293 INFO 2981023 --- [ocp-async-4,931e55688c285d8e,acac33e4ee08fb17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:14.240 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048080, nodeCount=831, affectRows=831 2024-09-11 17:49:14.243 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048080, nodeCount=830, validCount=830 2024-09-11 17:49:14.245 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048080, nodeCount=975, affectRows=975 2024-09-11 17:49:14.248 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048080, nodeCount=975, validCount=975 2024-09-11 17:49:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,6de5ddc53c131a5e,39986e63ee13cde0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,b09574f330a4aa02,56c4a13ad120d35d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:49:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,6de5ddc53c131a5e,39986e63ee13cde0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:14.431 INFO 2981023 --- [ocp-schedules-25,905a082715e7e38a,19c8630ec188b3fa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:49:14.436 INFO 2981023 --- [ocp-schedules-25,905a082715e7e38a,19c8630ec188b3fa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:49:14.436 INFO 2981023 --- [ocp-schedules-25,905a082715e7e38a,19c8630ec188b3fa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:49:14.470 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:14.471 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:15.011 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048080, nodeCount=463, affectRows=463 2024-09-11 17:49:15.293 INFO 2981023 --- [ocp-async-3,882a02c82ff03f62,c2cddc60f7c9f4f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:16.020 INFO 2981023 --- [ocp-schedules-24,94e4877008f23a00,961fb169145a23f6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:49:16.020 INFO 2981023 --- [host-check-clockdiff1,94e4877008f23a00,71847af7eb17ce06] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:49:16.023 INFO 2981023 --- [ocp-schedules-25,054e3f113102ad6b,1b4dec1af745cd67] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:49:16.024 INFO 2981023 --- [host-check-clockdiff2,94e4877008f23a00,63e3d8f85d9ce37c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:49:16.028 INFO 2981023 --- [host-check-clockdiff3,94e4877008f23a00,2b13dec20117292a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:49:16.029 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:49:16.029 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:49:16.029 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:49:16.052 INFO 2981023 --- [host-check-clockdiff3,94e4877008f23a00,2b13dec20117292a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048156 0 0 , error= 2024-09-11 17:49:16.053 INFO 2981023 --- [ocp-schedules-30,5b146197051beb06,764a5010564c1319] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:49:16.063 INFO 2981023 --- [host-check-agent-status1,5b146197051beb06,a22d3f9981294df1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:49:16.064 INFO 2981023 --- [host-check-agent-status2,5b146197051beb06,0a196990cf64789e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:49:16.064 INFO 2981023 --- [host-check-agent-status3,5b146197051beb06,9d20a506261b5260] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:49:16.066 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:49:16.071 INFO 2981023 --- [host-check-agent-status3,5b146197051beb06,9d20a506261b5260] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:49:16.072 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:49:16.072 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:49:16.073 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:49:16.073 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:49:16.074 INFO 2981023 --- [host-check-agent-status3,5b146197051beb06,9d20a506261b5260] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:49:16.074 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:49:16.084 INFO 2981023 --- [host-check-agent-status1,5b146197051beb06,a22d3f9981294df1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:49:16.100 INFO 2981023 --- [host-check-agent-status1,5b146197051beb06,a22d3f9981294df1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:49:16.190 INFO 2981023 --- [ocp-schedules-9,a73fd23122f57205,eeb28462f2e5a0a7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:49:16.191 INFO 2981023 --- [ocp-schedules-22,0178eac992f067b2,0c57cdd88c7f08a4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:49:16.196 INFO 2981023 --- [host-check-agent-status2,5b146197051beb06,0a196990cf64789e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:49:16.196 INFO 2981023 --- [ocp-schedules-23,55049bf7a01d6de2,d65d918f1b2f85f4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:49:16.196 INFO 2981023 --- [ocp-schedules-12,dc5cab6a57177789,6c086011235f825f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:49:16.256 INFO 2981023 --- [ocp-schedules-9,a73fd23122f57205,eeb28462f2e5a0a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:49:16.278 INFO 2981023 --- [host-check-agent-status2,5b146197051beb06,0a196990cf64789e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:49:16.278 INFO 2981023 --- [ocp-schedules-23,55049bf7a01d6de2,d65d918f1b2f85f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:49:16.285 INFO 2981023 --- [ocp-schedules-22,0178eac992f067b2,0c57cdd88c7f08a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:49:16.299 INFO 2981023 --- [ocp-schedules-22,0178eac992f067b2,0c57cdd88c7f08a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:49:16.300 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:49:16.303 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:49:16.302662540+08:00 2024-09-11 17:49:16.307 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:49:16.304+08:00 2024-09-11 17:49:16.311 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:49:16.304+08:00 and status AVAILABLE 2024-09-11 17:49:16.362 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:49:16.364 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:49:16.364415996+08:00 2024-09-11 17:49:16.366 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:49:16.364+08:00 2024-09-11 17:49:16.368 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:49:16.364+08:00 and status AVAILABLE 2024-09-11 17:49:16.371 INFO 2981023 --- [host-check-agent-status2,5b146197051beb06,0a196990cf64789e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:49:16.373 INFO 2981023 --- [ocp-schedules-26,3acbab21b9c04809,e5a58080d8192311] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:49:16.374 INFO 2981023 --- [host-check-agent-status1,5b146197051beb06,a22d3f9981294df1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:49:16.375 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:49:16.374083+08:00, lastProcessTime=2024-09-11T17:49:06.336692+08:00 2024-09-11 17:49:16.376 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.376 INFO 2981023 --- [host-check-agent-status2,5b146197051beb06,0a196990cf64789e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:49:16.376 INFO 2981023 --- [host-check-agent-status2,5b146197051beb06,0a196990cf64789e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:49:16.376 INFO 2981023 --- [host-check-agent-status2,5b146197051beb06,0a196990cf64789e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:49:16.377 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:16.377 INFO 2981023 --- [host-check-agent-status1,5b146197051beb06,a22d3f9981294df1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:49:16.377 INFO 2981023 --- [host-check-agent-status1,5b146197051beb06,a22d3f9981294df1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:49:16.377 INFO 2981023 --- [host-check-agent-status1,5b146197051beb06,a22d3f9981294df1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:49:16.377 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.378 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:49:16.378 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:16.378 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:49:16.379 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:49:16.379 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:49:16.379 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:49:16.379264502+08:00 2024-09-11 17:49:16.380 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:49:16.381 INFO 2981023 --- [host-check-agent-status3,5b146197051beb06,9d20a506261b5260] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:49:16.386 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.386 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:49:16.379+08:00 2024-09-11 17:49:16.386 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:16.387 INFO 2981023 --- [host-check-agent-status3,5b146197051beb06,9d20a506261b5260] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:49:16.387 INFO 2981023 --- [host-check-agent-status3,5b146197051beb06,9d20a506261b5260] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:49:16.387 INFO 2981023 --- [host-check-agent-status3,5b146197051beb06,9d20a506261b5260] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:49:16.387 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:49:16.388 INFO 2981023 --- [ocp-schedules-9,a73fd23122f57205,eeb28462f2e5a0a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:49:16.401 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:49:16.379+08:00 and status AVAILABLE 2024-09-11 17:49:16.407 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.407 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:16.411 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:49:16.413 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:49:16.416 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.417 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:16.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:16.418 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:49:16.421 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:49:16.422 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:49:16.454 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:49:16.456 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:49:16.476 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:49:16.478 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:49:16.480 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:49:16.482 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.482 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:49:16.508 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:49:16.510 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:49:16.514 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.515 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.515 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:49:16.520 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:49:16.523 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.524 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.524 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:16.524 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:49:16.525 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.527 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:49:16.527 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:49:16.535 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:49:16.535 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:49:16.548 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:49:16.548 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:16.548 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:49:16.556 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.557 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.558 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:16.559 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.585 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:49:16.586 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:49:16.591 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:49:16.594 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:49:16.595 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.606 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:16.606 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:49:16.607 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:16.607 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:49:16.614 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:49:16.615 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:49:16.616 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.617 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.617 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:49:16.621 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:49:16.621 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:49:16.621 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:49:16.622 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:49:16.627 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:16.631 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:49:16.633 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:49:16.641 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.641 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:49:16.641 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.642 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:16.642 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:49:16.643 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.646 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:49:16.646 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:49:16.652 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.652 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:49:16.653 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.653 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:49:16.654 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:16.655 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:16.655 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:49:16.655 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:49:16.659 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:49:16.659 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:49:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:16.669 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:49:16.676 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:49:16.677 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.677 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.677 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:49:16.678 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:16.678 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:49:16.681 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:49:16.681 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:49:16.682 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:49:16.685 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:49:16.687 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:49:16.689 INFO 2981023 --- [host-check-host-status2,054e3f113102ad6b,a282220ac8fa29f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:49:16.710 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:49:16.712 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:49:16.713 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:49:16.726 INFO 2981023 --- [host-check-host-status1,054e3f113102ad6b,e4ea13cc07db634c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:49:16.727 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:49:16.737 INFO 2981023 --- [host-check-host-status3,054e3f113102ad6b,41db89a7e12d7250] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:49:16.743 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:49:16.749 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.750 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.751 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:49:16.752 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:49:16.756 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:49:16.374083+08:00 2024-09-11 17:49:16.758 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:49:16.758 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:49:16.758 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.758 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.759 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:49:16.765 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:49:16.765 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:49:16.779 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:49:16.785 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:16.786 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:16.787 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:49:16.793 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:49:16.793 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:49:16.795 INFO 2981023 --- [ocp-schedules-23,55049bf7a01d6de2,d65d918f1b2f85f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:49:16.969 INFO 2981023 --- [host-check-clockdiff2,94e4877008f23a00,63e3d8f85d9ce37c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048156 0 0 , error= 2024-09-11 17:49:17.101 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048080, nodeCount=463, validCount=463 2024-09-11 17:49:17.293 INFO 2981023 --- [ocp-async-1,5b5c7e26bcb90df6,381cecedaecc6716] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,77b44d241719d5fb,994b3946ac5b5425] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,77b44d241719d5fb,994b3946ac5b5425] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:17.471 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:17.471 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:17.762 INFO 2981023 --- [ocp-async-5,294c16fff57aee10,4ffaeb49a21385da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:49:19.293 INFO 2981023 --- [ocp-async-10,4e6d1b0a4361a6ca,bfe0c21dca57d19e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:20.179 INFO 2981023 --- [host-check-clockdiff1,94e4877008f23a00,71847af7eb17ce06] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048160 0 0 , error= 2024-09-11 17:49:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,e25b8f578d65da4d,37acdd989df3e495] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,e25b8f578d65da4d,37acdd989df3e495] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:20.471 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:20.472 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:21.293 INFO 2981023 --- [ocp-async-3,d10ae9da57f35b55,c38dc51e54a71ae6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:22.197 INFO 2981023 --- [ocp-schedules-13,48864f917ae13f34,85ea13a9632f0ac1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:49:23.231 INFO 2981023 --- [ocp-schedules-1,2ba1a5834c5b8efc,c48449b0f3a9981a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:49:23.231 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:49:23.231 INFO 2981023 --- [ocp-schedules-1,2ba1a5834c5b8efc,c48449b0f3a9981a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:49:23.231 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:49:23.235 INFO 2981023 --- [ocp-schedules-1,2ba1a5834c5b8efc,c48449b0f3a9981a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:49:23.235 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:49:23.236 INFO 2981023 --- [ocp-schedules-1,2ba1a5834c5b8efc,c48449b0f3a9981a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:49:23.236 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:49:23.237 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:49:23.237 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:49:23.251 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:49:23.252 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:49:23.254 INFO 2981023 --- [ocp-schedules-1,2ba1a5834c5b8efc,c48449b0f3a9981a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:49:23.256 INFO 2981023 --- [ocp-schedules-1,2ba1a5834c5b8efc,c48449b0f3a9981a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:49:23.293 INFO 2981023 --- [ocp-async-2,8e03e2a7dd219867,f916349ce50d4427] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,5fe1f3843fe474d4,cd8a75d24a9c8f05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,5fe1f3843fe474d4,cd8a75d24a9c8f05] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:23.356 INFO 2981023 --- [ocp-schedules-16,6ab2d2f428496c4e,e29c645fc5bb8bd4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:49:23.405 WARN 2981023 --- [scanner-13554061-1,6ab2d2f428496c4e,253b527f90639c12] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:49:23.433 INFO 2981023 --- [alarm-detect-14,6ab2d2f428496c4e,6dae8f080594db2c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644739, data is empty. 2024-09-11 17:49:23.440 INFO 2981023 --- [alarm-detect-14,6ab2d2f428496c4e,6dae8f080594db2c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644739, data is empty. 2024-09-11 17:49:23.444 INFO 2981023 --- [alarm-detect-14,6ab2d2f428496c4e,6dae8f080594db2c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644739, data is empty. 2024-09-11 17:49:23.452 INFO 2981023 --- [alarm-detect-1,6ab2d2f428496c4e,912b74d292eab8f1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644739, data is empty. 2024-09-11 17:49:23.452 INFO 2981023 --- [alarm-detect-19,6ab2d2f428496c4e,ff5f04a3b2e891e4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644739, data is empty. 2024-09-11 17:49:23.455 INFO 2981023 --- [alarm-detect-1,6ab2d2f428496c4e,912b74d292eab8f1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644739, data is empty. 2024-09-11 17:49:23.456 INFO 2981023 --- [alarm-detect-19,6ab2d2f428496c4e,ff5f04a3b2e891e4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644739, data is empty. 2024-09-11 17:49:23.459 INFO 2981023 --- [alarm-detect-19,6ab2d2f428496c4e,ff5f04a3b2e891e4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644739, data is empty. 2024-09-11 17:49:23.459 INFO 2981023 --- [alarm-detect-1,6ab2d2f428496c4e,912b74d292eab8f1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644739, data is empty. 2024-09-11 17:49:23.472 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:23.473 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:23.523 INFO 2981023 --- [alarm-detect-5,6ab2d2f428496c4e,a4cd41ad1415b052] c.o.o.alarm.core.detect.MetricDetector : alarmType agent_process_count_abnormal detect time range 1726048033 ~ 1726048093 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={agent_process_count=[SeriesMetricValues(metric=Metric{name='agent_process_count', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, name=ocp_mgragent, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726048080, value=2.0)])]}, metricEvaluateResults={agent_process_count != 1=[ExpressionEvaluateResult(matchValue=1.0, metric=agent_process_count, match=true, value=2.0, values={1726048080=2.0}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, name=ocp_mgragent, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:49:23.523 INFO 2981023 --- [alarm-detect-5,6ab2d2f428496c4e,a4cd41ad1415b052] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=agent_process_count_abnormal, target=alarm_template_id=0:host=10.31.35.10:name=ocp_mgragent-3, currentState=Pending 2024-09-11 17:49:23.564 INFO 2981023 --- [alarm-detect-6,6ab2d2f428496c4e,0a1065e4199a92fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644739, data is empty. 2024-09-11 17:49:23.568 INFO 2981023 --- [alarm-detect-6,6ab2d2f428496c4e,0a1065e4199a92fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644739, data is empty. 2024-09-11 17:49:23.573 INFO 2981023 --- [alarm-detect-6,6ab2d2f428496c4e,0a1065e4199a92fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644739, data is empty. 2024-09-11 17:49:24.295 INFO 2981023 --- [ocp-schedules-2,bc6d5a7a549e541d,65d581f07db93ecf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:49:25.293 INFO 2981023 --- [ocp-async-9,a80d19d86e114478,77c442fe020a0ad3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,c9f09eecec07ab4c,55261e8485b06372] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,c9f09eecec07ab4c,55261e8485b06372] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:49:26.336768+08:00, lastProcessTime=2024-09-11T17:49:16.374083+08:00 2024-09-11 17:49:26.467 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:49:26.336768+08:00 2024-09-11 17:49:26.473 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:26.473 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:27.294 INFO 2981023 --- [ocp-async-7,fd10b2a38dd25634,0b46626f2887e279] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:27.764 INFO 2981023 --- [ocp-async-8,8bf8b4d7b35a6c25,f6a9dab614782c32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:49:29.294 INFO 2981023 --- [ocp-async-1,45f6de00c20a31a4,2acaeffad8165b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,0b65c9bb2da89475,6bedcb4791a782e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,0b65c9bb2da89475,6bedcb4791a782e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:29.473 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:29.474 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:31.294 INFO 2981023 --- [ocp-async-8,586b7cc98fbd8b9e,bd1762b2a2e45f37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:32.200 INFO 2981023 --- [ocp-schedules-16,fa583684cf8e490e,6e99f6244d0547c9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:49:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,1c6b32ed155d1b21,d1dd6f30ee6b123b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,1c6b32ed155d1b21,d1dd6f30ee6b123b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:32.474 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:32.474 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:33.294 INFO 2981023 --- [ocp-async-10,017389345e57509f,dcb90ca852cb8865] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:33.609 INFO 2981023 --- [ocp-schedules-28,8d0a0e742fba6201,6cae16ed25971a49] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:49:33.649 WARN 2981023 --- [scanner-13554378-1,8d0a0e742fba6201,8a4a1ba4d76219de] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:49:34.438 INFO 2981023 --- [ocp-schedules-10,78ea877fed18b41c,c501398a74812a6f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:49:34.443 INFO 2981023 --- [ocp-schedules-10,78ea877fed18b41c,c501398a74812a6f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:49:34.443 INFO 2981023 --- [ocp-schedules-10,78ea877fed18b41c,c501398a74812a6f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:49:35.294 INFO 2981023 --- [ocp-async-3,5bf8dfd1b0b4b78c,c3e3e384a0d82bcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,52c967fde1bc9d87,e88ada469803748b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:35.315 INFO 2981023 --- [prometheus-minute-schedule-1,52c967fde1bc9d87,e88ada469803748b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:35.475 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:35.475 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:49:36.336148+08:00, lastProcessTime=2024-09-11T17:49:26.336768+08:00 2024-09-11 17:49:36.473 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:49:36.336148+08:00 2024-09-11 17:49:37.294 INFO 2981023 --- [ocp-async-1,c92253504abeb6ae,75a217972a536ecb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:37.314 INFO 2981023 --- [prometheus-cache-monitor1,c30a31b5082567a9,cde7f6a66462aac3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7224,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1807,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:49:37.315 INFO 2981023 --- [prometheus-cache-monitor1,c30a31b5082567a9,cde7f6a66462aac3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:49:37.763 INFO 2981023 --- [ocp-async-3,72719a4af2f14fba,42eed43a0f5e568f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:49:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,32e070c12a7ce395,a3f325758fd4e5a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,32e070c12a7ce395,a3f325758fd4e5a6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:38.475 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:38.476 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:39.294 INFO 2981023 --- [ocp-async-5,bf0bb8a0955b849f,47421548e6027466] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:40.004 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:49:40.004 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:49:41.294 INFO 2981023 --- [ocp-async-6,47d98ca3f88ca6e5,e3dd4f124cf1e920] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,60654f547d0d1665,5dc412998f372382] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:41.315 INFO 2981023 --- [prometheus-minute-schedule-1,60654f547d0d1665,5dc412998f372382] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:41.476 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:41.476 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:41.725 INFO 2981023 --- [ocp-schedules-27,e9bdcac078858661,68215af029b97cdc] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:49:41.754 WARN 2981023 --- [ocp-schedules-27,e9bdcac078858661,68215af029b97cdc] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:49:42.202 INFO 2981023 --- [ocp-schedules-13,e970e10fea77caf0,453dc53b9ada38b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:49:43.295 INFO 2981023 --- [ocp-async-9,9f8bd71412314dc8,90b66dc1cc3fda71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:43.782 INFO 2981023 --- [ocp-schedules-30,9d34109f0750fb1a,65b404f3f34cca67] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:49:43.832 WARN 2981023 --- [scanner-13554600-1,9d34109f0750fb1a,87811450dd2cbb1f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:49:43.844 INFO 2981023 --- [alarm-detect-2,9d34109f0750fb1a,96ed2476423effb1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644739, data is empty. 2024-09-11 17:49:43.847 INFO 2981023 --- [alarm-detect-2,9d34109f0750fb1a,96ed2476423effb1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644739, data is empty. 2024-09-11 17:49:43.850 INFO 2981023 --- [alarm-detect-2,9d34109f0750fb1a,96ed2476423effb1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644739, data is empty. 2024-09-11 17:49:43.874 INFO 2981023 --- [alarm-detect-6,9d34109f0750fb1a,3018c08bd422285b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644739, data is empty. 2024-09-11 17:49:43.877 INFO 2981023 --- [alarm-detect-19,9d34109f0750fb1a,77d2b701856e5d06] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644739, data is empty. 2024-09-11 17:49:43.882 INFO 2981023 --- [alarm-detect-6,9d34109f0750fb1a,3018c08bd422285b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644739, data is empty. 2024-09-11 17:49:43.885 INFO 2981023 --- [alarm-detect-19,9d34109f0750fb1a,77d2b701856e5d06] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644739, data is empty. 2024-09-11 17:49:43.890 INFO 2981023 --- [alarm-detect-6,9d34109f0750fb1a,3018c08bd422285b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644739, data is empty. 2024-09-11 17:49:43.891 INFO 2981023 --- [alarm-detect-19,9d34109f0750fb1a,77d2b701856e5d06] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644739, data is empty. 2024-09-11 17:49:43.993 INFO 2981023 --- [alarm-detect-1,9d34109f0750fb1a,3f47ef94aa19f478] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644739, data is empty. 2024-09-11 17:49:43.999 INFO 2981023 --- [alarm-detect-1,9d34109f0750fb1a,3f47ef94aa19f478] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644739, data is empty. 2024-09-11 17:49:44.002 INFO 2981023 --- [alarm-detect-1,9d34109f0750fb1a,3f47ef94aa19f478] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644739, data is empty. 2024-09-11 17:49:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,64bc7a981f3be0af,f07ea23b9170f285] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,64bc7a981f3be0af,f07ea23b9170f285] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:44.476 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:44.477 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:45.101 INFO 2981023 --- [prometheus-second-schedule-1,3c33b26a5f3cedab,da664c79d901aac0] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:49:45.295 INFO 2981023 --- [ocp-async-8,6183a29f29d5cbde,259287fa229f1de2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:46.014 INFO 2981023 --- [ocp-schedules-19,04d506864f6a26e2,166920266696b672] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:49:46.014 INFO 2981023 --- [host-check-clockdiff1,04d506864f6a26e2,1d6dec40252d59d7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:49:46.014 INFO 2981023 --- [host-check-clockdiff2,04d506864f6a26e2,0f9c15c88ff61e88] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:49:46.015 INFO 2981023 --- [host-check-clockdiff3,04d506864f6a26e2,076e2561c703d3cc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:49:46.027 INFO 2981023 --- [host-check-clockdiff3,04d506864f6a26e2,076e2561c703d3cc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048186 0 0 , error= 2024-09-11 17:49:46.027 INFO 2981023 --- [ocp-schedules-8,2b8b4cfc81c65b58,6df617f22058767b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:49:46.027 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:49:46.027 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:49:46.028 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:49:46.028 INFO 2981023 --- [ocp-schedules-25,b59490b277831031,9e858eccb5150e37] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:49:46.028 INFO 2981023 --- [host-check-agent-status1,b59490b277831031,16a1c5307fbea466] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:49:46.029 INFO 2981023 --- [host-check-agent-status2,b59490b277831031,af71b9389f6289ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:49:46.030 INFO 2981023 --- [host-check-agent-status3,b59490b277831031,aa14efec2ae71d8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:49:46.034 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:49:46.034 INFO 2981023 --- [host-check-agent-status1,b59490b277831031,16a1c5307fbea466] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:49:46.034 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:49:46.035 INFO 2981023 --- [host-check-agent-status2,b59490b277831031,af71b9389f6289ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:49:46.035 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:49:46.035 INFO 2981023 --- [host-check-agent-status1,b59490b277831031,16a1c5307fbea466] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:49:46.036 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:49:46.036 INFO 2981023 --- [host-check-agent-status2,b59490b277831031,af71b9389f6289ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:49:46.036 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:49:46.036 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:49:46.037 INFO 2981023 --- [host-check-agent-status3,b59490b277831031,aa14efec2ae71d8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:49:46.040 INFO 2981023 --- [host-check-agent-status3,b59490b277831031,aa14efec2ae71d8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:49:46.055 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:49:46.057 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:49:46.057 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:49:46.056648546+08:00 2024-09-11 17:49:46.057 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:49:46.057+08:00 2024-09-11 17:49:46.059 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:49:46.058694162+08:00 2024-09-11 17:49:46.059 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:49:46.057+08:00 and status AVAILABLE 2024-09-11 17:49:46.059 INFO 2981023 --- [host-check-agent-status1,b59490b277831031,16a1c5307fbea466] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:49:46.059 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:49:46.059+08:00 2024-09-11 17:49:46.060 INFO 2981023 --- [host-check-agent-status2,b59490b277831031,af71b9389f6289ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:49:46.061 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:49:46.059+08:00 and status AVAILABLE 2024-09-11 17:49:46.063 INFO 2981023 --- [host-check-agent-status2,b59490b277831031,af71b9389f6289ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:49:46.063 INFO 2981023 --- [host-check-agent-status2,b59490b277831031,af71b9389f6289ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:49:46.063 INFO 2981023 --- [host-check-agent-status2,b59490b277831031,af71b9389f6289ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:49:46.063 INFO 2981023 --- [host-check-agent-status1,b59490b277831031,16a1c5307fbea466] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:49:46.063 INFO 2981023 --- [host-check-agent-status1,b59490b277831031,16a1c5307fbea466] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:49:46.063 INFO 2981023 --- [host-check-agent-status1,b59490b277831031,16a1c5307fbea466] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:49:46.066 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:49:46.066 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:49:46.066 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:49:46.066395318+08:00 2024-09-11 17:49:46.069 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:49:46.066+08:00 2024-09-11 17:49:46.069 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:49:46.082 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:49:46.066+08:00 and status AVAILABLE 2024-09-11 17:49:46.087 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:49:46.088 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:49:46.090 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:49:46.090 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:49:46.103 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:49:46.103 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:49:46.104 INFO 2981023 --- [host-check-agent-status3,b59490b277831031,aa14efec2ae71d8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:49:46.104 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:49:46.115 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:49:46.116 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:49:46.136 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:49:46.139 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:49:46.145 INFO 2981023 --- [host-check-agent-status3,b59490b277831031,aa14efec2ae71d8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:49:46.145 INFO 2981023 --- [host-check-agent-status3,b59490b277831031,aa14efec2ae71d8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:49:46.145 INFO 2981023 --- [host-check-agent-status3,b59490b277831031,aa14efec2ae71d8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:49:46.152 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:49:46.161 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:49:46.164 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:49:46.168 INFO 2981023 --- [ocp-schedules-17,e28635bfba4f9aec,4c844b65d1127226] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:49:46.174 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:49:46.183 INFO 2981023 --- [ocp-schedules-17,e28635bfba4f9aec,4c844b65d1127226] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:49:46.187 INFO 2981023 --- [ocp-schedules-23,ba236e5ca842eb79,65cd079c5dfcd9ed] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:49:46.187 INFO 2981023 --- [ocp-schedules-12,7d7323dfa2726c5c,d772e400c9cec8f8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:49:46.188 INFO 2981023 --- [ocp-schedules-9,2d4825270ea2b4f3,755f5726b7086ef0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:49:46.188 INFO 2981023 --- [ocp-schedules-14,d1c13eec188608c7,53dc25f0b0f647bc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:49:46.192 INFO 2981023 --- [ocp-schedules-23,ba236e5ca842eb79,65cd079c5dfcd9ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:49:46.195 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:49:46.195 INFO 2981023 --- [ocp-schedules-12,7d7323dfa2726c5c,d772e400c9cec8f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:49:46.206 INFO 2981023 --- [ocp-schedules-9,2d4825270ea2b4f3,755f5726b7086ef0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:49:46.214 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:49:46.256 INFO 2981023 --- [ocp-schedules-10,80e08951ee5f0d1e,fb8de1cff4e96fe4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:49:46.263 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:49:46.268 INFO 2981023 --- [ocp-schedules-12,7d7323dfa2726c5c,d772e400c9cec8f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:49:46.275 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:49:46.276 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:49:46.288 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.288 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.289 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:46.290 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:46.290 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:49:46.290 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:49:46.291 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:49:46.291 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.292 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.293 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:49:46.293 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:49:46.294 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.299 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.300 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:49:46.300 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:46.301 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:49:46.302 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:49:46.307 INFO 2981023 --- [ocp-schedules-23,ba236e5ca842eb79,65cd079c5dfcd9ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:49:46.310 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:49:46.314 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:49:46.315 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.315 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:46.320 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.321 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:46.322 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:49:46.323 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.329 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:46.343 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:49:46.355 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:49:46.363 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:49:46.362570+08:00, lastProcessTime=2024-09-11T17:49:36.336148+08:00 2024-09-11 17:49:46.364 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:49:46.366 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:49:46.367 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.368 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.368 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:49:46.369 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:49:46.370 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:49:46.370 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.384 INFO 2981023 --- [host-check-host-status2,2b8b4cfc81c65b58,f351ed3984e161df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:49:46.394 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:49:46.399 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:49:46.401 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:49:46.406 INFO 2981023 --- [host-check-host-status1,2b8b4cfc81c65b58,ec4e06a622c63800] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:49:46.411 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.412 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.412 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:49:46.412 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:49:46.414 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.414 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.415 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:49:46.438 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.441 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:49:46.442 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.443 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.443 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:46.444 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.447 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:49:46.447 INFO 2981023 --- [host-check-host-status3,2b8b4cfc81c65b58,e75896cc1e7aaa8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:49:46.448 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:49:46.464 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.464 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.465 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:46.465 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:49:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.466 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.466 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.467 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:49:46.467 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:49:46.467 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.468 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:49:46.468 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:49:46.469 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:49:46.470 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:49:46.470 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.476 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:49:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:46.484 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:49:46.493 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:46.493 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:49:46.506 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:46.506 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:49:46.508 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:46.511 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.512 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.513 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:49:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.517 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.517 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.518 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:46.518 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:46.518 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:49:46.518 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:49:46.522 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:49:46.523 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:49:46.525 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.525 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.526 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:49:46.526 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:49:46.526 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:49:46.526 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.527 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:49:46.528 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:49:46.534 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:49:46.535 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.536 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.536 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:49:46.537 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:49:46.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.542 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:49:46.543 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:49:46.545 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:49:46.546 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:49:46.555 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:46.555 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:49:46.561 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:49:46.565 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:49:46.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:49:46.581 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.581 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.582 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:49:46.582 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:49:46.584 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.584 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.585 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:49:46.585 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:49:46.585 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.596 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:49:46.601 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:49:46.603 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:49:46.606 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:49:46.614 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.616 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.617 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:49:46.622 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:49:46.627 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:49:46.627 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:49:46.637 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:49:46.645 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:49:46.645 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.646 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.646 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:49:46.654 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:49:46.654 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:49:46.656 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:49:46.663 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:49:46.665 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:49:46.362570+08:00 2024-09-11 17:49:46.665 INFO 2981023 --- [ocp-schedules-7,2a005778f2575c47,ec0a702e9069ddb2] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:49:46.675 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:49:46.680 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:46.681 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:46.681 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:49:46.686 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:49:46.686 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047612666000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:49:46.686 INFO 2981023 --- [ocp-schedules-9,2d4825270ea2b4f3,755f5726b7086ef0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:49:46.841 INFO 2981023 --- [ocp-schedules-4,0c565988ec4524a6,6740f120fee56690] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7224,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1807,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:49:46.841 INFO 2981023 --- [ocp-schedules-4,0c565988ec4524a6,6740f120fee56690] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:49:47.295 INFO 2981023 --- [ocp-async-2,243ed52d2473bcf6,20822f7791fcf541] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,75825a8effc9b7ba,da8ec26b202c92b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:47.315 INFO 2981023 --- [prometheus-minute-schedule-1,75825a8effc9b7ba,da8ec26b202c92b2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:47.477 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:47.478 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:47.763 INFO 2981023 --- [ocp-async-3,cca392c5509cd916,53317180252e1184] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:49:47.803 INFO 2981023 --- [host-check-clockdiff2,04d506864f6a26e2,0f9c15c88ff61e88] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048187 0 0 , error= 2024-09-11 17:49:47.803 INFO 2981023 --- [host-check-clockdiff1,04d506864f6a26e2,1d6dec40252d59d7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048187 0 0 , error= 2024-09-11 17:49:49.295 INFO 2981023 --- [ocp-async-7,40bf162e03f8f05a,8a74e04a22e56c6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:49.295 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,5d20cd06631392ba,c5c42b3ef8eeece4] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&version=2&ObClusterId=1719994437, client=10.31.35.10, traceId=5d20cd06631392ba, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkprdocp,1719994437,2,{"ObRegion":"linkprdocp","ObCluster":"linkprdocp","ObRegionId":1719994437,"ObClusterId":1719994437,"Type":"PRIMARY","timestamp":1726048189294136,"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:49:49.306 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,5d20cd06631392ba,c5c42b3ef8eeece4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkprdocp, obClusterId=1719994437, rootServerJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, groupName=linkprdocp, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:49:49.306 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,5d20cd06631392ba,c5c42b3ef8eeece4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:49:49.306 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,5d20cd06631392ba,c5c42b3ef8eeece4] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkprdocp, clusterId=1719994437, affectRows=1 2024-09-11 17:49:49.307 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,5d20cd06631392ba,c5c42b3ef8eeece4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.10, traceId=5d20cd06631392ba, duration=13 ms] 2024-09-11 17:49:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:49:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:49:50.261 INFO 2981023 --- [ocp-schedules-20,cc99ae9de1b4dd0d,25990421f226cbeb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:49:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,8383abf3c5b3b7cb,b284f70cab0e6d9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,8383abf3c5b3b7cb,b284f70cab0e6d9a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:50.478 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:50.479 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:51.295 INFO 2981023 --- [ocp-async-3,585900e05473954a,252d7a03adb2f8a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:51.567 INFO 2981023 --- [ocp-schedules-18,97bd4d703bb1c488,05d50a9c41904722] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:49:51.568 INFO 2981023 --- [ocp-schedules-18,97bd4d703bb1c488,05d50a9c41904722] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:49:51.569 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:49:51.569 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:49:51.569 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:49:51.569 INFO 2981023 --- [ocp-schedules-18,97bd4d703bb1c488,05d50a9c41904722] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:49:52.205 INFO 2981023 --- [ocp-schedules-10,45f6f5cac6cf816c,695b1da7ec369fff] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:49:53.156 INFO 2981023 --- [ocp-schedules-2,d8133c883fec8ca9,a192aa57ccbf6010] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:49:53.156 INFO 2981023 --- [ocp-schedules-2,d8133c883fec8ca9,a192aa57ccbf6010] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:49:53.295 INFO 2981023 --- [ocp-async-6,31a97d8ce67f1a40,5937e204daa9f7ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:53.310 INFO 2981023 --- [ocp-schedules-29,0e3db54e52afdf69,5a0f487525451d2f] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:49:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,8741c64b3fbdcdcb,e8260521e0837cca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,8741c64b3fbdcdcb,e8260521e0837cca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:53.441 INFO 2981023 --- [ocp-schedules-1,ffd86841e5c85991,03f01e5308e755b9] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:49:53.463 WARN 2981023 --- [ocp-schedules-1,ffd86841e5c85991,03f01e5308e755b9] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:49:53.479 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:53.509 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:54.043 INFO 2981023 --- [ocp-schedules-4,f7a075036008717d,43f3b3df45839ac8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:49:54.121 WARN 2981023 --- [scanner-13554911-1,f7a075036008717d,7d60f6314f8c14f9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:49:54.297 INFO 2981023 --- [ocp-schedules-27,f0fcdebc53ed6993,dc2042bbb96c8ae4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:49:54.445 INFO 2981023 --- [ocp-schedules-3,1b7d3fb53a35864f,2e2f2d3045945248] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:49:54.453 INFO 2981023 --- [ocp-schedules-3,1b7d3fb53a35864f,2e2f2d3045945248] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:49:54.453 INFO 2981023 --- [ocp-schedules-3,1b7d3fb53a35864f,2e2f2d3045945248] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:49:54.781 INFO 2981023 --- [ocp-schedules-2,a03f7c32f9b5e0f4,6bd9098919451c43] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:49:54.781+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:49:54.785 INFO 2981023 --- [ocp-schedules-2,a03f7c32f9b5e0f4,6bd9098919451c43] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:49:54.902 INFO 2981023 --- [ocp-schedules-2,a03f7c32f9b5e0f4,6bd9098919451c43] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 48 to SqlAggregateData 2024-09-11 17:49:54.903 INFO 2981023 --- [ocp-schedules-2,a03f7c32f9b5e0f4,6bd9098919451c43] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:49:55.295 INFO 2981023 --- [ocp-async-1,3564daf1a8587f19,acdcef027dc86f79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:55.471 INFO 2981023 --- [ocp-schedules-28,4b0a9f8b066c7994,cade781d2a3ea0e7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:49:55.473 INFO 2981023 --- [ocp-schedules-28,4b0a9f8b066c7994,cade781d2a3ea0e7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:49:55.505 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:49:55.505 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:49:55.505 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:49:55.505 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.505 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.505 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.508 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:49:55.509 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:49:55.509 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:49:55.513 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.513 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.514 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.521 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.521 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.524 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.525 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.528 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:49:55.529 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.536 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.537 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:49:55.539 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.540 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:49:55.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3721D6DA-0-0] 2024-09-11 17:49:55.541 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:49:55.541 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.541 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:49:55.542 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.542 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.544 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.544 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:49:55.550 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.551 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:49:55.552 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.552 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:49:55.553 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.553 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:49:55.554 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:49:55.554 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.554 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:49:55.558 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.558 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:49:55.565 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.565 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:49:55.565 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:49:55.565 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:49:55.566 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3721D6E4-0-0] 2024-09-11 17:49:55.566 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:49:55.572 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.572 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.572 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:49:55.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:49:55.573 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.574 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:49:55.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:49:55.575 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.575 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.575 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:49:55.575 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:49:55.576 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.576 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.576 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:49:55.577 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.577 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:49:55.578 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.578 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:49:55.578 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.578 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:49:55.579 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:49:55.579 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.580 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.580 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:49:55.580 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:49:55.581 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:49:55.581 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.582 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:49:55.582 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:55.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:49:55.583 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.584 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:49:55.584 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.584 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:49:55.584 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.586 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:49:55.595 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:55.596 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.597 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:49:55.598 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:49:55.598 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:49:55.600 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.600 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:49:55.607 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:49:55.607 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:49:55.607 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:49:55.607 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:49:55.608 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:49:55.608 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:49:55.608 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.609 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:49:55.610 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:49:55.610 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.611 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.614 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:49:55.616 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:49:55.616 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:55.616 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:49:55.623 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:49:55.623 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:49:55.624 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:49:55.624 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:49:55.624 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:49:55.625 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:49:55.625 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:49:55.625 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:49:55.625 INFO 2981023 --- [ocp-schedules-28,4b0a9f8b066c7994,cade781d2a3ea0e7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:49:55.759 INFO 2981023 --- [ocp-schedules-14,1e1222f3ea6b35a9,03bcaa8768b9ad12] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:49:55.759 INFO 2981023 --- [ocp-schedules-14,1e1222f3ea6b35a9,03bcaa8768b9ad12] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:49:55.760 INFO 2981023 --- [ocp-schedules-14,1e1222f3ea6b35a9,03bcaa8768b9ad12] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:49:55.760 INFO 2981023 --- [ocp-schedules-14,1e1222f3ea6b35a9,03bcaa8768b9ad12] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:49:55.760 INFO 2981023 --- [ocp-schedules-14,1e1222f3ea6b35a9,03bcaa8768b9ad12] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:49:55.760 INFO 2981023 --- [ocp-schedules-14,1e1222f3ea6b35a9,03bcaa8768b9ad12] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:49:55.861 INFO 2981023 --- [ocp-schedules-18,10bf08a1758caa7c,3132fd429fd0d40a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:49:55.868 INFO 2981023 --- [ocp-schedules-18,10bf08a1758caa7c,3132fd429fd0d40a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:49:55.868 INFO 2981023 --- [ocp-schedules-18,10bf08a1758caa7c,3132fd429fd0d40a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:49:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,dfdc990ee6edfe90,32a2f608f9c93bb6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,dfdc990ee6edfe90,32a2f608f9c93bb6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:49:56.350490+08:00, lastProcessTime=2024-09-11T17:49:46.362570+08:00 2024-09-11 17:49:56.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:49:56.350490+08:00 2024-09-11 17:49:56.509 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:56.509 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:49:56.739 INFO 2981023 --- [ocp-schedules-30,2214aff74b94674f,c487db4d4a4ccd0f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:49:56.739 INFO 2981023 --- [ocp-schedules-30,2214aff74b94674f,c487db4d4a4ccd0f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:49:56.753 INFO 2981023 --- [Collect all tenants major compactions2,2214aff74b94674f,7fbb40f3d4bf56fe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:49:56.754 INFO 2981023 --- [Collect all tenants major compactions4,2214aff74b94674f,590d27b8e9a013cc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:49:56.754 INFO 2981023 --- [Collect all tenants major compactions3,2214aff74b94674f,76a1f3d81ab5ba51] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:49:56.755 INFO 2981023 --- [Collect all tenants major compactions6,2214aff74b94674f,9f730163245c83fa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:49:56.756 INFO 2981023 --- [Collect all tenants major compactions5,2214aff74b94674f,16928432dda316fc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:49:56.756 INFO 2981023 --- [Collect all tenants major compactions1,2214aff74b94674f,8b21865caa5467ae] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:49:56.769 INFO 2981023 --- [Collect all tenants major compactions3,2214aff74b94674f,76a1f3d81ab5ba51] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:56.769 INFO 2981023 --- [Collect all tenants major compactions4,2214aff74b94674f,590d27b8e9a013cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:56.770 INFO 2981023 --- [Collect all tenants major compactions5,2214aff74b94674f,16928432dda316fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:56.770 INFO 2981023 --- [Collect all tenants major compactions3,2214aff74b94674f,76a1f3d81ab5ba51] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:49:56.770 INFO 2981023 --- [Collect all tenants major compactions2,2214aff74b94674f,7fbb40f3d4bf56fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:56.770 INFO 2981023 --- [Collect all tenants major compactions4,2214aff74b94674f,590d27b8e9a013cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:49:56.771 INFO 2981023 --- [Collect all tenants major compactions5,2214aff74b94674f,16928432dda316fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:49:56.771 INFO 2981023 --- [Collect all tenants major compactions6,2214aff74b94674f,9f730163245c83fa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:56.772 INFO 2981023 --- [Collect all tenants major compactions2,2214aff74b94674f,7fbb40f3d4bf56fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:49:56.772 INFO 2981023 --- [Collect all tenants major compactions6,2214aff74b94674f,9f730163245c83fa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:49:56.772 INFO 2981023 --- [Collect all tenants major compactions1,2214aff74b94674f,8b21865caa5467ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:49:56.773 INFO 2981023 --- [Collect all tenants major compactions1,2214aff74b94674f,8b21865caa5467ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:49:56.775 INFO 2981023 --- [Collect all tenants major compactions4,2214aff74b94674f,590d27b8e9a013cc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:49:56.775 INFO 2981023 --- [Collect all tenants major compactions4,2214aff74b94674f,590d27b8e9a013cc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:49:56.775 INFO 2981023 --- [Collect all tenants major compactions3,2214aff74b94674f,76a1f3d81ab5ba51] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:49:56.775 INFO 2981023 --- [Collect all tenants major compactions3,2214aff74b94674f,76a1f3d81ab5ba51] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:49:56.775 INFO 2981023 --- [Collect all tenants major compactions6,2214aff74b94674f,9f730163245c83fa] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:49:56.775 INFO 2981023 --- [Collect all tenants major compactions6,2214aff74b94674f,9f730163245c83fa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:49:56.775 INFO 2981023 --- [Collect all tenants major compactions2,2214aff74b94674f,7fbb40f3d4bf56fe] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:49:56.775 INFO 2981023 --- [Collect all tenants major compactions2,2214aff74b94674f,7fbb40f3d4bf56fe] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:49:56.776 INFO 2981023 --- [Collect all tenants major compactions1,2214aff74b94674f,8b21865caa5467ae] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:49:56.776 INFO 2981023 --- [Collect all tenants major compactions1,2214aff74b94674f,8b21865caa5467ae] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:49:56.776 INFO 2981023 --- [Collect all tenants major compactions5,2214aff74b94674f,16928432dda316fc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:49:56.776 INFO 2981023 --- [Collect all tenants major compactions5,2214aff74b94674f,16928432dda316fc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:49:56.777 INFO 2981023 --- [ocp-schedules-30,2214aff74b94674f,c487db4d4a4ccd0f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:49:57.296 INFO 2981023 --- [ocp-async-3,ae427d708ab4d4b0,65413262542666c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:57.763 INFO 2981023 --- [ocp-async-7,679a9674169f653d,336e8a38d6a401c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:49:59.296 INFO 2981023 --- [ocp-async-4,577941d971652407,583e0d5b98721782] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:49:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,980d8737140ed803,0e8b955281d1cfff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048200, interval=60 2024-09-11 17:49:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,980d8737140ed803,0e8b955281d1cfff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:49:59.509 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:49:59.510 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:00.846 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:45:00.0, nextRun=2024-09-11 17:50:00.0, currentDate=Wed Sep 11 17:50:00 HKT 2024 2024-09-11 17:50:00.846 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:45:00.0, nextRunTime=2024-09-11 17:50:00.0} 2024-09-11 17:50:00.850 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:50:00.850 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:50:00.850 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:50:00.850+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:50:00.862 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:45:00.0, nextRun=2024-09-11 17:50:00.0, currentDate=Wed Sep 11 17:50:00 HKT 2024 2024-09-11 17:50:00.862 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:45:00.0, nextRunTime=2024-09-11 17:50:00.0} 2024-09-11 17:50:00.873 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2, 1000006] 2024-09-11 17:50:00.879 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:50:00.880 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:50:00.880 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:50:00.880 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,b7fa28eaf730bdc2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2,1000006]]}}, createTime=null, startTime=2024-09-11T17:50:00.880+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:50:01.301 INFO 2981023 --- [ocp-async-1,0a8aad92452d0593,7b712a99bc55e3c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:01.425 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,95c4494bd857274e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048176, operation:EXECUTE, state: RUNNING 2024-09-11 17:50:01.438 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,95c4494bd857274e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048177, operation:EXECUTE, state: RUNNING 2024-09-11 17:50:01.448 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048176, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048193, task_operation=execute, latest_execution_start_time=2024-09-11T17:50:01.418+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048176}, listMap={}}, executor=10.31.35.10 2024-09-11 17:50:01.461 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:50:01.461 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048177, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048194, task_operation=execute, latest_execution_start_time=2024-09-11T17:50:01.437+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048177}, listMap={cluster_ids=[[1,2,1000006]]}}, executor=10.31.35.10 2024-09-11 17:50:01.465 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:50:01.496 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:50:01.497 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:50:01.509 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:50:01.509 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:50:01.511 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:50:01.511 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:50:01.533 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:50:01.536 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:50:01.536 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1000006 2024-09-11 17:50:01.551 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,d4060fcdcf49dda0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048177, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:50:01.574 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:50:01.603 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:50:01.626 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:50:01.631 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:50:01.631 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:50:01.636 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:50:01.731 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:50:01.776 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:50:01.794 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:50:01.808 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:50:01.852 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:50:01.862 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:50:01.876 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:50:01.882 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:50:01.882 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:50:01.888 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:50:01.937 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048194, status:SUCCESS 2024-09-11 17:50:01.940 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048194, status:SUCCESS 2024-09-11 17:50:01.956 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:50:01.987 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:50:02.000 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:50:02.009 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:50:02.019 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:50:02.025 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:50:02.031 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:50:02.034 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:50:02.034 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:50:02.049 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:50:02.057 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:50:02.069 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,4b24734e3a64c7f7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048176, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:50:02.095 INFO 2981023 --- [ocp-schedules-24,018363a58322c8b1,e33570eb10d10b7a] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:50:02.095 INFO 2981023 --- [ocp-schedules-24,018363a58322c8b1,e33570eb10d10b7a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:50:02.208 INFO 2981023 --- [ocp-schedules-2,22c776b823ca3af2,92ff17e67c23dcc4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:50:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,858bedb505e5a5f5,256cb91d3f60a8b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,858bedb505e5a5f5,256cb91d3f60a8b6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:02.510 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:02.510 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:02.524 INFO 2981023 --- [ocp-schedules-22,39baf05ba9f7aa0e,9f8623f6e275df6c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:50:02.525 INFO 2981023 --- [ocp-schedules-22,39baf05ba9f7aa0e,9f8623f6e275df6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:50:02.535 INFO 2981023 --- [ocp-schedules-22,39baf05ba9f7aa0e,9f8623f6e275df6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:50:02.957 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048193, status:SUCCESS 2024-09-11 17:50:02.958 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048193, status:SUCCESS 2024-09-11 17:50:03.301 INFO 2981023 --- [ocp-async-10,86b982e8180d9045,bb70fde78e093e2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:04.221 INFO 2981023 --- [ocp-schedules-11,03a4dc1efa68be52,133513d858c289d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:50:04.266 WARN 2981023 --- [scanner-13555307-1,03a4dc1efa68be52,7a02b54de5984d2c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:50:04.289 INFO 2981023 --- [alarm-detect-9,03a4dc1efa68be52,9ab3cfb284d0f797] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644739, data is empty. 2024-09-11 17:50:04.293 INFO 2981023 --- [alarm-detect-9,03a4dc1efa68be52,9ab3cfb284d0f797] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644787, data is empty. 2024-09-11 17:50:04.299 INFO 2981023 --- [alarm-detect-9,03a4dc1efa68be52,9ab3cfb284d0f797] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644787, data is empty. 2024-09-11 17:50:04.320 INFO 2981023 --- [alarm-detect-6,03a4dc1efa68be52,12d81b3126a598f0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644739, data is empty. 2024-09-11 17:50:04.320 INFO 2981023 --- [alarm-detect-13,03a4dc1efa68be52,e421f816c52ffb8d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644739, data is empty. 2024-09-11 17:50:04.324 INFO 2981023 --- [alarm-detect-6,03a4dc1efa68be52,12d81b3126a598f0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644787, data is empty. 2024-09-11 17:50:04.325 INFO 2981023 --- [alarm-detect-13,03a4dc1efa68be52,e421f816c52ffb8d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644787, data is empty. 2024-09-11 17:50:04.327 INFO 2981023 --- [alarm-detect-6,03a4dc1efa68be52,12d81b3126a598f0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644787, data is empty. 2024-09-11 17:50:04.331 INFO 2981023 --- [alarm-detect-13,03a4dc1efa68be52,e421f816c52ffb8d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644787, data is empty. 2024-09-11 17:50:04.428 INFO 2981023 --- [alarm-detect-17,03a4dc1efa68be52,c88c6e2823780f9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644739, data is empty. 2024-09-11 17:50:04.434 INFO 2981023 --- [alarm-detect-17,03a4dc1efa68be52,c88c6e2823780f9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644787, data is empty. 2024-09-11 17:50:04.438 INFO 2981023 --- [alarm-detect-17,03a4dc1efa68be52,c88c6e2823780f9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644787, data is empty. 2024-09-11 17:50:04.994 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5da91d685946492d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:40:05.0, nextRun=2024-09-11 17:50:04.0, currentDate=Wed Sep 11 17:50:04 HKT 2024 2024-09-11 17:50:04.994 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5da91d685946492d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-09-11 17:40:05.0, nextRunTime=2024-09-11 17:50:04.0} 2024-09-11 17:50:04.996 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5da91d685946492d] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-09-11 17:50:04.996 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5da91d685946492d] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:50:04.996 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5da91d685946492d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:50:04.996+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:50:05.301 INFO 2981023 --- [ocp-async-6,e24d6cfd9fe2e5a6,b05269cc67cf98df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,48b81e70bd7fcd33,c6b8b4c0f18bb999] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:05.315 INFO 2981023 --- [prometheus-minute-schedule-1,48b81e70bd7fcd33,c6b8b4c0f18bb999] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:05.465 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,d2b7fe39f27af613] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048178, operation:EXECUTE, state: RUNNING 2024-09-11 17:50:05.482 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048178, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048195, task_operation=execute, latest_execution_start_time=2024-09-11T17:50:05.464+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=4048178}, listMap={}}, executor=10.31.35.10 2024-09-11 17:50:05.488 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 17:50:05.488 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 17:50:05.489 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.490 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:50:05.490 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.491 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.491 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.492 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 17:50:05.502 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.502 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.503 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.503 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 17:50:05.510 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:05.511 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:05.511 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.512 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.512 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.513 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 17:50:05.521 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.522 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.522 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.523 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 17:50:05.532 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.532 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.533 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.533 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 17:50:05.538 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.538 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.539 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.539 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 17:50:05.542 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.543 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.543 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.543 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 17:50:05.546 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.546 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.547 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.547 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 17:50:05.551 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.551 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.552 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.552 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 17:50:05.556 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.556 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.556 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.557 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 17:50:05.560 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.560 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.561 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.561 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 17:50:05.567 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.567 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.568 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.568 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 17:50:05.578 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.579 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.579 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.580 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 17:50:05.583 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.583 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.584 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.584 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 17:50:05.588 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.588 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.589 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.589 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 17:50:05.592 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.592 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.593 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.593 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 17:50:05.596 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.596 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.597 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.597 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 17:50:05.600 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.600 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.600 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.601 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 17:50:05.603 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.604 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.604 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.605 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 17:50:05.608 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.608 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.608 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.609 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 17:50:05.611 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.612 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.612 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.613 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 17:50:05.616 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.616 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.616 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.617 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 17:50:05.619 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.620 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.620 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.620 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 17:50:05.623 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.624 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.624 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.625 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 17:50:05.627 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.627 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.628 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.628 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 17:50:05.631 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.631 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.632 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.632 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 17:50:05.635 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.635 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.636 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.636 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 17:50:05.639 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.639 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.639 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.640 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 17:50:05.642 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.643 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.643 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.644 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 17:50:05.647 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.647 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.647 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.648 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 17:50:05.650 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.651 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.651 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.651 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 17:50:05.654 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.654 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.655 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.655 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 17:50:05.658 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.658 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.659 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.659 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 17:50:05.662 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.662 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.662 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.663 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 17:50:05.667 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.668 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.668 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.669 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 17:50:05.672 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.673 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.673 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.673 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 17:50:05.676 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.677 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.677 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.678 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 17:50:05.681 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.681 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.681 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.682 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 17:50:05.684 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.684 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.685 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.685 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 17:50:05.687 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.687 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.688 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.688 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 17:50:05.690 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.691 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.691 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.691 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 17:50:05.694 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.694 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.694 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.695 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 17:50:05.701 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.701 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.702 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.702 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 17:50:05.717 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.717 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.718 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.718 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 17:50:05.722 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.722 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.723 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.723 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 17:50:05.727 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.727 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.728 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.728 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 17:50:05.732 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.732 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.733 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.733 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 17:50:05.736 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.736 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.736 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.737 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 17:50:05.740 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.740 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.740 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.741 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 17:50:05.744 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 17:50:05.744 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 17:50:05.749 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.749 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:50:05.751 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.751 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.752 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.752 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 17:50:05.756 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.756 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:05.757 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.757 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 17:50:05.760 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 17:50:05.768 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,681a3b8a96d1593f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048178, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:50:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:50:05.969 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048195, status:SUCCESS 2024-09-11 17:50:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:05.970 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048195, status:SUCCESS 2024-09-11 17:50:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:50:05.971 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:50:05.983 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:50:05.989 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:50:05.993 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:50:05.998 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:50:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:50:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:50:06.002 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:06.002 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:06.002 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:50:06.002 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:50:06.003 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:06.003 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:06.003 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:06.003 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:50:06.068 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:47:40, start=2024-09-11T17:45:10, end=2024-09-11T17:47:40, updates=128, granule=120000ms, delay=150000ms 2024-09-11 17:50:06.072 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:47:40, target=2024-09-11T17:47:40, granule=120000ms, delay=0ms) 2024-09-11 17:50:06.072 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:47:40, target=2024-09-11T17:39:31, granule=600000ms, delay=489000ms) 2024-09-11 17:50:06.139 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:47:41, start=2024-09-11T17:45:11, end=2024-09-11T17:47:41, updates=306, granule=120000ms, delay=150000ms 2024-09-11 17:50:06.148 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:47:41, target=2024-09-11T17:47:41, granule=120000ms, delay=0ms) 2024-09-11 17:50:06.148 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:47:41, target=2024-09-11T17:40:51, granule=600000ms, delay=410000ms) 2024-09-11 17:50:06.148 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=177ms, period=120000ms 2024-09-11 17:50:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:50:06.337024+08:00, lastProcessTime=2024-09-11T17:49:56.350490+08:00 2024-09-11 17:50:06.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:50:06.337024+08:00 2024-09-11 17:50:07.301 INFO 2981023 --- [ocp-async-9,5abd782887c17c7a,c118b67091255c86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:07.536 INFO 2981023 --- [ocp-schedules-13,e58bc859938a330f,0afdca2d537cbdb8] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:50:07.566 INFO 2981023 --- [ocp-schedules-7,8cdf90362e6ad325,215ef35694d34ad8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:50:07.566+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:50:07.579 INFO 2981023 --- [ocp-schedules-7,8cdf90362e6ad325,215ef35694d34ad8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:50:07.581 INFO 2981023 --- [ocp-schedules-7,8cdf90362e6ad325,215ef35694d34ad8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:50:07.581 INFO 2981023 --- [ocp-schedules-7,8cdf90362e6ad325,215ef35694d34ad8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:50:07.763 INFO 2981023 --- [ocp-async-2,eae0a2a74dcbcc38,eb8162c78897a4b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:50:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,26446dcf56916324,776abb84bde30256] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,26446dcf56916324,776abb84bde30256] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:08.511 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:08.512 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:09.301 INFO 2981023 --- [ocp-async-10,b6ce09f91e30884f,0be33d9962f5b87b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:10.003 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:50:10.003 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:50:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:50:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:50:11.301 INFO 2981023 --- [ocp-async-6,6de8834367dc456c,8f978e417eef5758] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,669c2e92968cfb6a,a8eecb6e8943d323] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,669c2e92968cfb6a,a8eecb6e8943d323] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:11.512 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:11.513 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:12.190 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048140, nodeCount=463, affectRows=463 2024-09-11 17:50:12.210 INFO 2981023 --- [ocp-schedules-6,e1b933295ad60d42,58e46da8d9e2987a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:50:12.660 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,18cdeeeb62dd63cd,3df4d6fb26de8685] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&version=2&ObClusterId=1718689631, client=10.31.35.5, traceId=18cdeeeb62dd63cd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=rtdob,1718689631,2,{"ObRegion":"rtdob","ObCluster":"rtdob","ObRegionId":1718689631,"ObClusterId":1718689631,"Type":"PRIMARY","timestamp":1726048212658649,"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:50:12.665 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,18cdeeeb62dd63cd,3df4d6fb26de8685] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=rtdob, obClusterId=1718689631, rootServerJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, groupName=rtdob, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:50:12.665 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,18cdeeeb62dd63cd,3df4d6fb26de8685] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:50:12.665 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,18cdeeeb62dd63cd,3df4d6fb26de8685] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=rtdob, clusterId=1718689631, affectRows=1 2024-09-11 17:50:12.665 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,18cdeeeb62dd63cd,3df4d6fb26de8685] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.5, traceId=18cdeeeb62dd63cd, duration=5 ms] 2024-09-11 17:50:13.301 INFO 2981023 --- [ocp-async-9,98d67990a577fdc6,09437e4c4f706852] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,16c3e41d391d7937,d49f1b28f67a9418] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,09498d8bc664567c,ffb9e49369d285c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:50:14.318 INFO 2981023 --- [prometheus-minute-schedule-1,16c3e41d391d7937,d49f1b28f67a9418] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:14.429 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048140, nodeCount=807, affectRows=807 2024-09-11 17:50:14.432 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048140, nodeCount=807, validCount=807 2024-09-11 17:50:14.437 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048140, nodeCount=1000, affectRows=1000 2024-09-11 17:50:14.439 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048140, nodeCount=999, validCount=999 2024-09-11 17:50:14.456 INFO 2981023 --- [ocp-schedules-6,f92812a78217839d,5f2f9d7ff9529864] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:50:14.464 INFO 2981023 --- [ocp-schedules-6,f92812a78217839d,5f2f9d7ff9529864] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:50:14.464 INFO 2981023 --- [ocp-schedules-6,f92812a78217839d,5f2f9d7ff9529864] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:50:14.488 INFO 2981023 --- [ocp-schedules-5,8422767f0d6a78d5,678508dcd280c1b9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:50:14.514 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:14.516 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:14.548 WARN 2981023 --- [scanner-13557068-1,8422767f0d6a78d5,a8f5d8a257be5516] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:50:15.061 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048140, nodeCount=463, validCount=463 2024-09-11 17:50:15.301 INFO 2981023 --- [ocp-async-1,820a8b92c00935e0,718a8cdcd86db6eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:16.019 INFO 2981023 --- [ocp-schedules-25,807c5984b5350cfd,47a2b0a7a4c0944c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:50:16.019 INFO 2981023 --- [ocp-schedules-23,c841dfe62eb8c0de,553434c48af7a2ed] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:50:16.020 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:50:16.020 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:50:16.021 INFO 2981023 --- [host-check-clockdiff1,c841dfe62eb8c0de,23f661e81c1ab554] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:50:16.021 INFO 2981023 --- [host-check-clockdiff2,c841dfe62eb8c0de,666f670eaa9f12b1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:50:16.021 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:50:16.023 INFO 2981023 --- [host-check-clockdiff3,c841dfe62eb8c0de,76c60d20890c875e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:50:16.047 INFO 2981023 --- [host-check-clockdiff2,c841dfe62eb8c0de,666f670eaa9f12b1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048216 0 0 , error= 2024-09-11 17:50:16.048 INFO 2981023 --- [host-check-clockdiff3,c841dfe62eb8c0de,76c60d20890c875e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048216 0 0 , error= 2024-09-11 17:50:16.054 INFO 2981023 --- [ocp-schedules-26,f0839ebce5105d7d,d3ed94a264de60db] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:50:16.054 INFO 2981023 --- [host-check-agent-status1,f0839ebce5105d7d,0d43ac8f742899ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:50:16.055 INFO 2981023 --- [host-check-agent-status2,f0839ebce5105d7d,99c1f99eeec9b541] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:50:16.055 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:50:16.055 INFO 2981023 --- [host-check-agent-status3,f0839ebce5105d7d,169c340e698e3455] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:50:16.056 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:50:16.057 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:50:16.059 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:50:16.060 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:50:16.061 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:50:16.067 INFO 2981023 --- [host-check-agent-status1,f0839ebce5105d7d,0d43ac8f742899ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:50:16.101 INFO 2981023 --- [host-check-agent-status1,f0839ebce5105d7d,0d43ac8f742899ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:50:16.187 INFO 2981023 --- [ocp-schedules-5,9448738835592b76,d1ec9317cdd1ce0c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:50:16.187 INFO 2981023 --- [ocp-schedules-4,745ea039788a0ebf,e500394261a0e27f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:50:16.188 INFO 2981023 --- [ocp-schedules-11,510c48d44554e885,2ec0f849e80140df] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:50:16.188 INFO 2981023 --- [ocp-schedules-15,40fec35a9dee4ce1,c165278a83e91608] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:50:16.202 INFO 2981023 --- [ocp-schedules-4,745ea039788a0ebf,e500394261a0e27f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:50:16.204 INFO 2981023 --- [ocp-schedules-5,9448738835592b76,d1ec9317cdd1ce0c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:50:16.207 INFO 2981023 --- [host-check-agent-status2,f0839ebce5105d7d,99c1f99eeec9b541] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:50:16.207 INFO 2981023 --- [host-check-agent-status3,f0839ebce5105d7d,169c340e698e3455] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:50:16.209 INFO 2981023 --- [host-check-agent-status2,f0839ebce5105d7d,99c1f99eeec9b541] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:50:16.215 INFO 2981023 --- [host-check-agent-status3,f0839ebce5105d7d,169c340e698e3455] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:50:16.216 INFO 2981023 --- [ocp-schedules-11,510c48d44554e885,2ec0f849e80140df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:50:16.225 INFO 2981023 --- [ocp-schedules-4,745ea039788a0ebf,e500394261a0e27f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:50:16.256 INFO 2981023 --- [ocp-schedules-14,32483b358dd38d89,3ba505f936ad730b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:50:16.295 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:50:16.296 INFO 2981023 --- [host-check-agent-status2,f0839ebce5105d7d,99c1f99eeec9b541] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:50:16.298 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:50:16.297711572+08:00 2024-09-11 17:50:16.301 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:50:16.298+08:00 2024-09-11 17:50:16.301 INFO 2981023 --- [host-check-agent-status2,f0839ebce5105d7d,99c1f99eeec9b541] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:50:16.301 INFO 2981023 --- [host-check-agent-status2,f0839ebce5105d7d,99c1f99eeec9b541] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:50:16.301 INFO 2981023 --- [host-check-agent-status2,f0839ebce5105d7d,99c1f99eeec9b541] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:50:16.303 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:50:16.298+08:00 and status AVAILABLE 2024-09-11 17:50:16.317 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.317 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:16.318 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:50:16.327 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:50:16.328 INFO 2981023 --- [host-check-agent-status1,f0839ebce5105d7d,0d43ac8f742899ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:50:16.331 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:50:16.332 INFO 2981023 --- [host-check-agent-status1,f0839ebce5105d7d,0d43ac8f742899ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:50:16.332 INFO 2981023 --- [host-check-agent-status1,f0839ebce5105d7d,0d43ac8f742899ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:50:16.332 INFO 2981023 --- [host-check-agent-status1,f0839ebce5105d7d,0d43ac8f742899ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:50:16.334 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.334 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:50:16.335 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.335 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:50:16.335152880+08:00 2024-09-11 17:50:16.336 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:16.337 INFO 2981023 --- [host-check-agent-status3,f0839ebce5105d7d,169c340e698e3455] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:50:16.337 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:16.338 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:50:16.341 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:50:16.339041156+08:00 2024-09-11 17:50:16.342 INFO 2981023 --- [host-check-agent-status3,f0839ebce5105d7d,169c340e698e3455] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:50:16.342 INFO 2981023 --- [host-check-agent-status3,f0839ebce5105d7d,169c340e698e3455] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:50:16.342 INFO 2981023 --- [host-check-agent-status3,f0839ebce5105d7d,169c340e698e3455] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:50:16.350 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:50:16.335+08:00 2024-09-11 17:50:16.352 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:50:16.335+08:00 and status AVAILABLE 2024-09-11 17:50:16.356 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:50:16.341+08:00 2024-09-11 17:50:16.359 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:50:16.341+08:00 and status AVAILABLE 2024-09-11 17:50:16.361 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:50:16.363 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.363 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:16.364 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.365 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:16.366 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:50:16.366 INFO 2981023 --- [ocp-schedules-5,9448738835592b76,d1ec9317cdd1ce0c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:50:16.367 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:50:16.368 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:50:16.375 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:50:16.377 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:50:16.375335+08:00, lastProcessTime=2024-09-11T17:50:06.337024+08:00 2024-09-11 17:50:16.377 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:50:16.382 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:50:16.385 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:50:16.386 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:50:16.388 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.388 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:16.396 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:50:16.406 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.407 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.407 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:50:16.409 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:50:16.428 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:50:16.431 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:50:16.435 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.436 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.437 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:50:16.442 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:50:16.446 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.447 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.447 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:16.448 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:50:16.448 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.450 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:50:16.450 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:50:16.453 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:50:16.457 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:50:16.459 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:50:16.468 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:50:16.468 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:16.469 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:50:16.481 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.487 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.489 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:16.490 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.493 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:50:16.493 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:50:16.493 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:50:16.494 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:50:16.509 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:16.510 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:50:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:16.516 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:50:16.519 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.525 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:50:16.526 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.526 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:16.526 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:50:16.533 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:50:16.536 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:16.536 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:50:16.538 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:50:16.538 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:50:16.542 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:50:16.543 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:50:16.550 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:50:16.551 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.552 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.553 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:16.553 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:50:16.555 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:50:16.557 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:50:16.562 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:50:16.562 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.563 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:50:16.563 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.563 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:50:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:50:16.568 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:50:16.579 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:50:16.589 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:50:16.591 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:50:16.596 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:50:16.600 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.603 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.603 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:16.604 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:50:16.604 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.605 INFO 2981023 --- [host-check-host-status2,807c5984b5350cfd,9b6d36fe211b0bf1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:50:16.607 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:50:16.608 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:50:16.617 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:50:16.620 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:50:16.622 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:16.622 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:50:16.627 INFO 2981023 --- [host-check-host-status3,807c5984b5350cfd,e084b3d475b6fcd6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:50:16.637 INFO 2981023 --- [host-check-host-status1,807c5984b5350cfd,922fc18a38e5030c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:50:16.649 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:16.651 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:50:16.656 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.656 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.657 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:16.658 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:50:16.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.658 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.659 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:50:16.662 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:50:16.664 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:50:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:50:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:50:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:50:16.670 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:50:16.671 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.672 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.672 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:50:16.679 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:50:16.679 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:50:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:50:16.697 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:50:16.375335+08:00 2024-09-11 17:50:16.715 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:50:16.720 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:16.721 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:16.721 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:50:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:50:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:50:16.727 INFO 2981023 --- [ocp-schedules-11,510c48d44554e885,2ec0f849e80140df] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:50:17.301 INFO 2981023 --- [ocp-async-5,91876ad4b170d055,fb594d1c9974010c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,93fd2c4166a67c1a,1a38902467e27b05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:17.315 INFO 2981023 --- [prometheus-minute-schedule-1,93fd2c4166a67c1a,1a38902467e27b05] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:17.517 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:17.517 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:17.763 INFO 2981023 --- [ocp-async-3,61f23e2c1ed28bf5,316f57a5c498fe07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:50:19.301 INFO 2981023 --- [ocp-async-9,72810df2e6060485,6f53a7f18f45de9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:19.755 INFO 2981023 --- [host-check-clockdiff1,c841dfe62eb8c0de,23f661e81c1ab554] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048219 0 0 , error= 2024-09-11 17:50:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,a7f68f596ef3da0f,b4f27bd6b9d88041] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,a7f68f596ef3da0f,b4f27bd6b9d88041] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:20.517 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:20.519 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:21.301 INFO 2981023 --- [ocp-async-3,4200615887735351,5ce481e2eb832459] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:22.213 INFO 2981023 --- [ocp-schedules-1,de01bf4ec56219e5,541d8cecdcdc1087] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:50:23.267 INFO 2981023 --- [ocp-schedules-8,68c1a68c77739fec,807ec81ac8c6eac5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:50:23.267 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:50:23.268 INFO 2981023 --- [ocp-schedules-8,68c1a68c77739fec,807ec81ac8c6eac5] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:50:23.268 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:50:23.271 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:50:23.271 INFO 2981023 --- [ocp-schedules-8,68c1a68c77739fec,807ec81ac8c6eac5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:50:23.272 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:50:23.275 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:50:23.275 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:50:23.275 INFO 2981023 --- [ocp-schedules-8,68c1a68c77739fec,807ec81ac8c6eac5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:50:23.302 INFO 2981023 --- [ocp-async-5,32d4203844318a71,187542b90099ece0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:23.304 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:50:23.306 INFO 2981023 --- [ocp-schedules-8,68c1a68c77739fec,807ec81ac8c6eac5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:50:23.306 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:50:23.308 INFO 2981023 --- [ocp-schedules-8,68c1a68c77739fec,807ec81ac8c6eac5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:50:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,343f694f1b5068dc,cbdd321290c5fa4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,343f694f1b5068dc,cbdd321290c5fa4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:23.519 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:23.520 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:24.300 INFO 2981023 --- [ocp-schedules-27,aeb6fe9b6c840748,c5dd89d14158a2c4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:50:24.691 INFO 2981023 --- [ocp-schedules-21,7e578622edc475cd,3698383aa6cc21e1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:50:24.740 WARN 2981023 --- [scanner-13557328-1,7e578622edc475cd,60d5adaf822af476] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:50:24.762 INFO 2981023 --- [alarm-detect-11,7e578622edc475cd,c5bf8d5bc397ec03] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644787, data is empty. 2024-09-11 17:50:24.767 INFO 2981023 --- [alarm-detect-11,7e578622edc475cd,c5bf8d5bc397ec03] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644787, data is empty. 2024-09-11 17:50:24.771 INFO 2981023 --- [alarm-detect-11,7e578622edc475cd,c5bf8d5bc397ec03] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644787, data is empty. 2024-09-11 17:50:24.786 INFO 2981023 --- [alarm-detect-13,7e578622edc475cd,af6973219e893ba1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644787, data is empty. 2024-09-11 17:50:24.786 INFO 2981023 --- [alarm-detect-19,7e578622edc475cd,5e00eebb75c3561b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644787, data is empty. 2024-09-11 17:50:24.790 INFO 2981023 --- [alarm-detect-13,7e578622edc475cd,af6973219e893ba1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644787, data is empty. 2024-09-11 17:50:24.793 INFO 2981023 --- [alarm-detect-19,7e578622edc475cd,5e00eebb75c3561b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644787, data is empty. 2024-09-11 17:50:24.794 INFO 2981023 --- [alarm-detect-13,7e578622edc475cd,af6973219e893ba1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644787, data is empty. 2024-09-11 17:50:24.798 INFO 2981023 --- [alarm-detect-19,7e578622edc475cd,5e00eebb75c3561b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644787, data is empty. 2024-09-11 17:50:24.905 INFO 2981023 --- [alarm-detect-9,7e578622edc475cd,ac9795623d350e53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644787, data is empty. 2024-09-11 17:50:24.910 INFO 2981023 --- [alarm-detect-9,7e578622edc475cd,ac9795623d350e53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644787, data is empty. 2024-09-11 17:50:24.914 INFO 2981023 --- [alarm-detect-9,7e578622edc475cd,ac9795623d350e53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644787, data is empty. 2024-09-11 17:50:25.302 INFO 2981023 --- [ocp-async-7,cf555e0fe74fcb99,c37ee1244b051119] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,e623f2891d4599e0,845dec87ad333987] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,e623f2891d4599e0,845dec87ad333987] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:26.357 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:50:26.356069+08:00, lastProcessTime=2024-09-11T17:50:16.375335+08:00 2024-09-11 17:50:26.520 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:26.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:26.527 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:50:26.356069+08:00 2024-09-11 17:50:27.302 INFO 2981023 --- [ocp-async-1,fef0f979ba43f75e,1aeda61563dfd23e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:27.763 INFO 2981023 --- [ocp-async-4,07aa6176203713d1,2dea5e66118bafc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:50:29.302 INFO 2981023 --- [ocp-async-5,2515f9afc31e763b,1e163ddd872d5511] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,8ec344a6d2d90914,0235a6280c0f96ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,8ec344a6d2d90914,0235a6280c0f96ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:29.521 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:29.522 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:31.302 INFO 2981023 --- [ocp-async-4,f6e1dcfcda4dc5a3,f8f285ef81b47989] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:32.215 INFO 2981023 --- [ocp-schedules-4,bad4c655f5078a75,bab14bbab2154205] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:50:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,3429c4d0237bfd68,8537428a4a3bbb8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,3429c4d0237bfd68,8537428a4a3bbb8a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:32.522 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:32.523 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:33.302 INFO 2981023 --- [ocp-async-8,811d62eca73296dc,1d490adbe2107ea7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:34.467 INFO 2981023 --- [ocp-schedules-30,2dc779e13474bbf9,7b242376b2923b1f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:50:34.475 INFO 2981023 --- [ocp-schedules-30,2dc779e13474bbf9,7b242376b2923b1f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:50:34.475 INFO 2981023 --- [ocp-schedules-30,2dc779e13474bbf9,7b242376b2923b1f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:50:34.952 INFO 2981023 --- [ocp-schedules-14,1f39bf6a904a2518,9c179d5ddd16db7c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:50:34.965 WARN 2981023 --- [scanner-13557599-1,1f39bf6a904a2518,29b88337f3e3240c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:50:35.302 INFO 2981023 --- [ocp-async-3,463cf41160e09b8b,b40b99d6b4633819] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,8a33ee3e6453fded,24dcd818c225a722] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:35.315 INFO 2981023 --- [prometheus-minute-schedule-1,8a33ee3e6453fded,24dcd818c225a722] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:35.523 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:35.523 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:50:36.337741+08:00, lastProcessTime=2024-09-11T17:50:26.356069+08:00 2024-09-11 17:50:36.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:50:36.337741+08:00 2024-09-11 17:50:37.302 INFO 2981023 --- [ocp-async-5,46c600137924e073,a56e39301e58987f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:37.314 INFO 2981023 --- [prometheus-cache-monitor1,10309bf8ee0d698f,c5122bb28ca16f2c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:50:37.315 INFO 2981023 --- [prometheus-cache-monitor1,10309bf8ee0d698f,c5122bb28ca16f2c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:50:37.763 INFO 2981023 --- [ocp-async-2,5c056135533a6833,7c913386db677d03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:50:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,21726ba3de5f23a4,a14e9ab1754464f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,21726ba3de5f23a4,a14e9ab1754464f4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:38.523 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:38.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:39.303 INFO 2981023 --- [ocp-async-9,a325a9264f703c35,91b277fba75353fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:40.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:50:40.003 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:50:41.303 INFO 2981023 --- [ocp-async-2,7f0e4bbddfaad989,483d0ff80e738e00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,b219f1d77c4dd190,0d9b576493bed20d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:41.315 INFO 2981023 --- [prometheus-minute-schedule-1,b219f1d77c4dd190,0d9b576493bed20d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:41.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:41.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:41.895 INFO 2981023 --- [ocp-schedules-13,718df1c82402edd3,d33700b9beb3472c] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:50:41.915 WARN 2981023 --- [ocp-schedules-13,718df1c82402edd3,d33700b9beb3472c] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:50:42.218 INFO 2981023 --- [ocp-schedules-20,885cc295c712cfed,cd3678cbfd9d83b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:50:43.303 INFO 2981023 --- [ocp-async-6,1c9b10de4b3a8d08,5a27af44ec3dbe5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,689ee486dd102d0d,87fbe899820b36b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,689ee486dd102d0d,87fbe899820b36b3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:44.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:44.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:45.101 INFO 2981023 --- [prometheus-second-schedule-1,557e58d2281c1502,66d724e981fd1761] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:50:45.145 INFO 2981023 --- [ocp-schedules-1,b25935aed5ddef00,cd4ae5ff225a8828] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:50:45.195 WARN 2981023 --- [scanner-13557815-1,b25935aed5ddef00,57d3cd5fd804fa6c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:50:45.212 INFO 2981023 --- [alarm-detect-18,b25935aed5ddef00,0548afefb40c2de9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644787, data is empty. 2024-09-11 17:50:45.216 INFO 2981023 --- [alarm-detect-18,b25935aed5ddef00,0548afefb40c2de9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644787, data is empty. 2024-09-11 17:50:45.222 INFO 2981023 --- [alarm-detect-18,b25935aed5ddef00,0548afefb40c2de9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644787, data is empty. 2024-09-11 17:50:45.238 INFO 2981023 --- [alarm-detect-7,b25935aed5ddef00,10e0b0635a487c02] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644787, data is empty. 2024-09-11 17:50:45.240 INFO 2981023 --- [alarm-detect-16,b25935aed5ddef00,e9e808584275d7d7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644787, data is empty. 2024-09-11 17:50:45.243 INFO 2981023 --- [alarm-detect-7,b25935aed5ddef00,10e0b0635a487c02] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644787, data is empty. 2024-09-11 17:50:45.244 INFO 2981023 --- [alarm-detect-16,b25935aed5ddef00,e9e808584275d7d7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644787, data is empty. 2024-09-11 17:50:45.249 INFO 2981023 --- [alarm-detect-16,b25935aed5ddef00,e9e808584275d7d7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644787, data is empty. 2024-09-11 17:50:45.249 INFO 2981023 --- [alarm-detect-7,b25935aed5ddef00,10e0b0635a487c02] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644787, data is empty. 2024-09-11 17:50:45.276 INFO 2981023 --- [alarm-detect-13,b25935aed5ddef00,499fe941c653e9e8] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007103] will be resolved. 2024-09-11 17:50:45.276 INFO 2981023 --- [alarm-detect-13,b25935aed5ddef00,499fe941c653e9e8] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007103] will be resolved. 2024-09-11 17:50:45.303 INFO 2981023 --- [ocp-async-4,935cd8589f452aa7,a27467aed515d07c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:45.321 INFO 2981023 --- [alarm-detect-13,b25935aed5ddef00,499fe941c653e9e8] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 17:50:45.363 INFO 2981023 --- [alarm-detect-13,b25935aed5ddef00,b68965e1dbc3ab0a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644787, data is empty. 2024-09-11 17:50:45.369 INFO 2981023 --- [alarm-detect-13,b25935aed5ddef00,b68965e1dbc3ab0a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644787, data is empty. 2024-09-11 17:50:45.376 INFO 2981023 --- [alarm-detect-13,b25935aed5ddef00,b68965e1dbc3ab0a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644787, data is empty. 2024-09-11 17:50:46.018 INFO 2981023 --- [ocp-schedules-20,2c2da95e2684fabc,1f3d30641bc1a853] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:50:46.018 INFO 2981023 --- [ocp-schedules-11,831bf35649f8e10f,66ed3bb2cfb43498] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:50:46.019 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:50:46.019 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:50:46.019 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:50:46.020 INFO 2981023 --- [host-check-clockdiff1,831bf35649f8e10f,3908b03484197931] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:50:46.020 INFO 2981023 --- [host-check-clockdiff2,831bf35649f8e10f,5be00d8e1a98fe33] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:50:46.020 INFO 2981023 --- [host-check-clockdiff3,831bf35649f8e10f,2d5c016d3dfe9184] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:50:46.026 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:50:46.027 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:50:46.027 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:50:46.028 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:50:46.028 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:50:46.028 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:50:46.035 INFO 2981023 --- [ocp-schedules-19,0e31a7bb675d2a6f,56824b9ebd13f3b5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:50:46.036 INFO 2981023 --- [host-check-agent-status2,0e31a7bb675d2a6f,cce9199af3e8b334] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:50:46.039 INFO 2981023 --- [host-check-clockdiff3,831bf35649f8e10f,2d5c016d3dfe9184] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048246 0 0 , error= 2024-09-11 17:50:46.039 INFO 2981023 --- [host-check-agent-status3,0e31a7bb675d2a6f,aa5d61b73e4fb802] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:50:46.036 INFO 2981023 --- [host-check-agent-status1,0e31a7bb675d2a6f,66625338d0a239a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:50:46.047 INFO 2981023 --- [host-check-agent-status3,0e31a7bb675d2a6f,aa5d61b73e4fb802] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:50:46.048 INFO 2981023 --- [host-check-agent-status2,0e31a7bb675d2a6f,cce9199af3e8b334] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:50:46.048 INFO 2981023 --- [host-check-agent-status1,0e31a7bb675d2a6f,66625338d0a239a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:50:46.049 INFO 2981023 --- [host-check-agent-status1,0e31a7bb675d2a6f,66625338d0a239a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:50:46.049 INFO 2981023 --- [host-check-agent-status3,0e31a7bb675d2a6f,aa5d61b73e4fb802] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:50:46.050 INFO 2981023 --- [host-check-agent-status2,0e31a7bb675d2a6f,cce9199af3e8b334] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:50:46.062 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:50:46.062 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:50:46.064 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:50:46.063643878+08:00 2024-09-11 17:50:46.064 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:50:46.064+08:00 2024-09-11 17:50:46.064 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:50:46.064614217+08:00 2024-09-11 17:50:46.065 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:50:46.064+08:00 2024-09-11 17:50:46.066 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:50:46.064+08:00 and status AVAILABLE 2024-09-11 17:50:46.066 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:50:46.068 INFO 2981023 --- [host-check-agent-status1,0e31a7bb675d2a6f,66625338d0a239a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:50:46.069 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:50:46.068348677+08:00 2024-09-11 17:50:46.070 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:50:46.064+08:00 and status AVAILABLE 2024-09-11 17:50:46.070 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:50:46.069+08:00 2024-09-11 17:50:46.071 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:50:46.069+08:00 and status AVAILABLE 2024-09-11 17:50:46.076 INFO 2981023 --- [host-check-agent-status1,0e31a7bb675d2a6f,66625338d0a239a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:50:46.076 INFO 2981023 --- [host-check-agent-status1,0e31a7bb675d2a6f,66625338d0a239a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:50:46.076 INFO 2981023 --- [host-check-agent-status1,0e31a7bb675d2a6f,66625338d0a239a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:50:46.077 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:50:46.079 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:50:46.084 INFO 2981023 --- [host-check-agent-status2,0e31a7bb675d2a6f,cce9199af3e8b334] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:50:46.087 INFO 2981023 --- [host-check-agent-status2,0e31a7bb675d2a6f,cce9199af3e8b334] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:50:46.087 INFO 2981023 --- [host-check-agent-status2,0e31a7bb675d2a6f,cce9199af3e8b334] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:50:46.087 INFO 2981023 --- [host-check-agent-status2,0e31a7bb675d2a6f,cce9199af3e8b334] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:50:46.092 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:50:46.111 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:50:46.111 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:50:46.112 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:50:46.113 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:50:46.123 INFO 2981023 --- [host-check-agent-status3,0e31a7bb675d2a6f,aa5d61b73e4fb802] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:50:46.127 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:50:46.127 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:50:46.132 INFO 2981023 --- [host-check-agent-status3,0e31a7bb675d2a6f,aa5d61b73e4fb802] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:50:46.132 INFO 2981023 --- [host-check-agent-status3,0e31a7bb675d2a6f,aa5d61b73e4fb802] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:50:46.132 INFO 2981023 --- [host-check-agent-status3,0e31a7bb675d2a6f,aa5d61b73e4fb802] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:50:46.133 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:50:46.141 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:50:46.150 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:50:46.157 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:50:46.158 INFO 2981023 --- [ocp-schedules-23,26df46fab5451b4b,ab2d55bb5c103117] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:50:46.161 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:50:46.167 INFO 2981023 --- [ocp-schedules-23,26df46fab5451b4b,ab2d55bb5c103117] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:50:46.184 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:50:46.187 INFO 2981023 --- [ocp-schedules-8,675269ffcd97ce43,724741d0eedf2511] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:50:46.188 INFO 2981023 --- [ocp-schedules-17,a6591a2242ea13f9,a1fb61ce0049685b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:50:46.188 INFO 2981023 --- [ocp-schedules-28,f12682d3952a39db,eaf8ef8734980c2d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:50:46.188 INFO 2981023 --- [ocp-schedules-2,a592595095a76fb7,0587444148586472] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:50:46.193 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:50:46.197 INFO 2981023 --- [ocp-schedules-17,a6591a2242ea13f9,a1fb61ce0049685b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:50:46.197 INFO 2981023 --- [ocp-schedules-8,675269ffcd97ce43,724741d0eedf2511] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:50:46.200 INFO 2981023 --- [ocp-schedules-28,f12682d3952a39db,eaf8ef8734980c2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:50:46.209 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:50:46.212 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:50:46.217 INFO 2981023 --- [ocp-schedules-22,ca2a3e7eb7ee70bd,c7fd176f577182bc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:50:46.220 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:50:46.233 INFO 2981023 --- [ocp-schedules-17,a6591a2242ea13f9,a1fb61ce0049685b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:50:46.244 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:50:46.279 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:50:46.281 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.281 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.282 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:46.282 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:50:46.283 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:46.284 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:50:46.287 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:50:46.287 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:50:46.293 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.293 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:50:46.294 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:50:46.294 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.305 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:50:46.337 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:50:46.339 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:50:46.342 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:50:46.342 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.342 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:46.349 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:50:46.355 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:50:46.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:50:46.351053+08:00, lastProcessTime=2024-09-11T17:50:36.337741+08:00 2024-09-11 17:50:46.355 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.356 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:46.356 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:50:46.358 INFO 2981023 --- [ocp-schedules-8,675269ffcd97ce43,724741d0eedf2511] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:50:46.359 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.360 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:46.362 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:50:46.381 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.382 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.382 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:50:46.385 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.386 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:46.393 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:50:46.395 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.395 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.396 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:50:46.396 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:50:46.396 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.399 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:50:46.419 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:50:46.421 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:50:46.423 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:50:46.423 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:50:46.426 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.426 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.426 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:46.428 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:50:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:50:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:50:46.433 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:50:46.433 INFO 2981023 --- [host-check-host-status1,2c2da95e2684fabc,5b2a67ff9d9d405b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:50:46.435 INFO 2981023 --- [host-check-host-status3,2c2da95e2684fabc,7a4a1b3fc3031d76] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:50:46.438 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.439 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.440 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:50:46.443 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:46.443 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:50:46.455 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:50:46.461 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.465 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:46.469 INFO 2981023 --- [host-check-host-status2,2c2da95e2684fabc,bd45109abbb616c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:50:46.470 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.471 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.472 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:50:46.472 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:50:46.472 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.473 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:50:46.473 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:50:46.474 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.474 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.478 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.478 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.479 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:46.480 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.481 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:50:46.482 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.482 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:50:46.482 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:46.483 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:50:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:50:46.486 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:50:46.491 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:50:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:50:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:50:46.503 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:50:46.508 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:50:46.510 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:46.519 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.520 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.521 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:46.521 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:50:46.523 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.524 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.524 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:50:46.525 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:50:46.525 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:50:46.526 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.537 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:50:46.538 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:50:46.541 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.541 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:50:46.543 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:50:46.557 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:50:46.576 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.577 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.577 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:50:46.578 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:50:46.578 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.581 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:50:46.581 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:50:46.582 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:50:46.585 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:50:46.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:46.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:50:46.592 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.593 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.593 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:50:46.597 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.598 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.599 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:50:46.599 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:50:46.599 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:50:46.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:50:46.603 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:50:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.614 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:50:46.614 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:50:46.617 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:50:46.620 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:50:46.621 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:50:46.622 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:50:46.630 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.631 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.632 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:50:46.646 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:50:46.646 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:50:46.647 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:50:46.651 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:50:46.654 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:50:46.656 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:50:46.351053+08:00 2024-09-11 17:50:46.669 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:50:46.675 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:50:46.675 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:50:46.678 INFO 2981023 --- [ocp-schedules-10,e150d40ae7f9d49e,084a2fc52754d6e4] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:50:46.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:46.683 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:46.683 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:50:46.689 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:50:46.689 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:50:46.689 INFO 2981023 --- [ocp-schedules-28,f12682d3952a39db,eaf8ef8734980c2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:50:46.982 INFO 2981023 --- [host-check-clockdiff1,831bf35649f8e10f,3908b03484197931] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048246 0 0 , error= 2024-09-11 17:50:47.303 INFO 2981023 --- [ocp-async-3,acdd9fefea3efac7,018cb11e714a61db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,15fbe5fa07dc938a,aa928b0f9331f6f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:47.315 INFO 2981023 --- [prometheus-minute-schedule-1,15fbe5fa07dc938a,aa928b0f9331f6f5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:47.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:47.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:47.763 INFO 2981023 --- [ocp-async-10,30d4bf602101cb54,5f72feabe8e74d2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:50:47.808 INFO 2981023 --- [host-check-clockdiff2,831bf35649f8e10f,5be00d8e1a98fe33] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048247 0 0 , error= 2024-09-11 17:50:49.303 INFO 2981023 --- [ocp-async-1,f11035be281306f9,0a374b35ad777c9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:50:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:50:50.260 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:50:50.264 INFO 2981023 --- [ocp-schedules-20,51b2ace38b6e2a06,ff98f4f8b783e0ed] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.266 INFO 2981023 --- [ocp-schedules-4,be22677965abd567,e549815cdf78e825] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:50:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,b752b41a1b093229,20d61c257384b3c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,b752b41a1b093229,20d61c257384b3c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:50.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:50.527 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:51.303 INFO 2981023 --- [ocp-async-10,2c1104f57f9acc9f,f25a79ff1dde6b0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:51.569 INFO 2981023 --- [ocp-schedules-22,e327125a96b31d90,4e64555483316824] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:50:51.570 INFO 2981023 --- [ocp-schedules-22,e327125a96b31d90,4e64555483316824] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:50:51.571 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:50:51.571 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:50:51.571 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:50:51.571 INFO 2981023 --- [ocp-schedules-22,e327125a96b31d90,4e64555483316824] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:50:52.221 INFO 2981023 --- [ocp-schedules-18,ac259c0d5763384b,15dfb4a406589a10] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:50:53.162 INFO 2981023 --- [ocp-schedules-25,706a0daa4134d246,bd3c59f9b9dc4b1b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:50:53.162 INFO 2981023 --- [ocp-schedules-25,706a0daa4134d246,bd3c59f9b9dc4b1b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:50:53.303 INFO 2981023 --- [ocp-async-3,2bdcb3c64c92dc7d,cc56ceaa4003d15f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,7225c166a000a48e,dc335096a9e4b6df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,7225c166a000a48e,dc335096a9e4b6df] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:53.523 INFO 2981023 --- [ocp-schedules-30,0ee0cdc20ac87fc7,c2b7943d0a5724c3] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:50:53.527 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:53.528 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:53.543 WARN 2981023 --- [ocp-schedules-30,0ee0cdc20ac87fc7,c2b7943d0a5724c3] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:50:53.791 INFO 2981023 --- [ocp-schedules-18,4a9f098366696de9,aa1f2da3252db375] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:50:54.302 INFO 2981023 --- [ocp-schedules-23,6eeaf2f53fa56447,c8957af46d140fd3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:50:54.478 INFO 2981023 --- [ocp-schedules-10,e548455fbad90123,7a16ff345c5e3168] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:50:54.489 INFO 2981023 --- [ocp-schedules-10,e548455fbad90123,7a16ff345c5e3168] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:50:54.489 INFO 2981023 --- [ocp-schedules-10,e548455fbad90123,7a16ff345c5e3168] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:50:54.923 INFO 2981023 --- [ocp-schedules-25,faac9ef05e3eab36,3b471fb6eb46ad82] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:50:54.923+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:50:54.928 INFO 2981023 --- [ocp-schedules-25,faac9ef05e3eab36,3b471fb6eb46ad82] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:50:55.073 INFO 2981023 --- [ocp-schedules-25,faac9ef05e3eab36,3b471fb6eb46ad82] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 45 to SqlAggregateData 2024-09-11 17:50:55.074 INFO 2981023 --- [ocp-schedules-25,faac9ef05e3eab36,3b471fb6eb46ad82] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:50:55.304 INFO 2981023 --- [ocp-async-6,4db165ce4be2f99c,14ade8c034e46488] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:55.408 INFO 2981023 --- [ocp-schedules-29,3bb6c5653fe5ac32,410f67470b47d4da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:50:55.453 WARN 2981023 --- [scanner-13558176-1,3bb6c5653fe5ac32,3a7453d47500e5a2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:50:55.625 INFO 2981023 --- [ocp-schedules-15,874065fbecc8ea4c,2b10e3cedc4bf9dd] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:50:55.626 INFO 2981023 --- [ocp-schedules-15,874065fbecc8ea4c,2b10e3cedc4bf9dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:50:55.661 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:50:55.661 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:50:55.661 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.661 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:50:55.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.664 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:50:55.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:50:55.665 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:50:55.669 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.669 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.670 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.674 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.675 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.680 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.680 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:50:55.681 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.681 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.684 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.685 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.688 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.688 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:50:55.691 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.692 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.692 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:50:55.692 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3871D6DA-0-0] 2024-09-11 17:50:55.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:50:55.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:50:55.694 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.695 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:50:55.698 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.699 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.704 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.705 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:50:55.706 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.706 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.706 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:50:55.706 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:50:55.707 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.708 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:50:55.708 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.709 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:50:55.709 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:50:55.713 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:50:55.715 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.716 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:50:55.716 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:50:55.716 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3871D6E4-0-0] 2024-09-11 17:50:55.716 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:50:55.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:50:55.722 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.722 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:50:55.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.724 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:50:55.724 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.724 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:50:55.725 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.725 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:50:55.725 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.725 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:50:55.726 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.726 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:50:55.727 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:50:55.727 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.727 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.727 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:55.728 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:50:55.728 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:50:55.728 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.728 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:50:55.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:50:55.729 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.729 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.730 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:50:55.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:50:55.730 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:50:55.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.731 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:50:55.731 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:50:55.732 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:50:55.733 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:50:55.745 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:50:55.748 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:55.748 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:50:55.748 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:50:55.748 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:50:55.748 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:50:55.749 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.749 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.750 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:50:55.750 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:50:55.756 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:50:55.757 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:50:55.757 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.758 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:50:55.761 INFO 2981023 --- [ocp-schedules-16,3ecadc47b36dcdda,cd2b1b39fb8e6ea0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:50:55.761 INFO 2981023 --- [ocp-schedules-16,3ecadc47b36dcdda,cd2b1b39fb8e6ea0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:50:55.762 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:50:55.763 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:50:55.763 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.764 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.764 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:50:55.764 INFO 2981023 --- [ocp-schedules-16,3ecadc47b36dcdda,cd2b1b39fb8e6ea0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:50:55.764 INFO 2981023 --- [ocp-schedules-16,3ecadc47b36dcdda,cd2b1b39fb8e6ea0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:50:55.765 INFO 2981023 --- [ocp-schedules-16,3ecadc47b36dcdda,cd2b1b39fb8e6ea0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:50:55.765 INFO 2981023 --- [ocp-schedules-16,3ecadc47b36dcdda,cd2b1b39fb8e6ea0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:50:55.766 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:50:55.766 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:55.766 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:50:55.773 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:50:55.775 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:50:55.775 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:50:55.775 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:50:55.777 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:50:55.779 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:50:55.779 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:50:55.779 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:50:55.781 INFO 2981023 --- [ocp-schedules-15,874065fbecc8ea4c,2b10e3cedc4bf9dd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:50:55.868 INFO 2981023 --- [ocp-schedules-14,92af0a404afecf8c,dd5091802d7cce1f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:50:55.874 INFO 2981023 --- [ocp-schedules-14,92af0a404afecf8c,dd5091802d7cce1f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:50:55.874 INFO 2981023 --- [ocp-schedules-14,92af0a404afecf8c,dd5091802d7cce1f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:50:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,d76324860eb790c4,be6069ce34a06e24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,d76324860eb790c4,be6069ce34a06e24] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:56.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:50:56.353394+08:00, lastProcessTime=2024-09-11T17:50:46.351053+08:00 2024-09-11 17:50:56.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:50:56.353394+08:00 2024-09-11 17:50:56.528 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:56.529 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:50:56.777 INFO 2981023 --- [ocp-schedules-4,5c8229443ef6b967,ed6e04cf311f0de5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:50:56.777 INFO 2981023 --- [ocp-schedules-4,5c8229443ef6b967,ed6e04cf311f0de5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:50:56.791 INFO 2981023 --- [Collect all tenants major compactions2,5c8229443ef6b967,ab7871923349cc6e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:50:56.791 INFO 2981023 --- [Collect all tenants major compactions1,5c8229443ef6b967,fd012fecd1e593e1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:50:56.791 INFO 2981023 --- [Collect all tenants major compactions3,5c8229443ef6b967,b32d0b0fe927109d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:50:56.791 INFO 2981023 --- [Collect all tenants major compactions4,5c8229443ef6b967,6cea0f38b55ca716] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:50:56.793 INFO 2981023 --- [Collect all tenants major compactions6,5c8229443ef6b967,bef771539a279985] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:50:56.793 INFO 2981023 --- [Collect all tenants major compactions5,5c8229443ef6b967,7617c21fab79857a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:50:56.803 INFO 2981023 --- [Collect all tenants major compactions3,5c8229443ef6b967,b32d0b0fe927109d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:56.803 INFO 2981023 --- [Collect all tenants major compactions2,5c8229443ef6b967,ab7871923349cc6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:56.803 INFO 2981023 --- [Collect all tenants major compactions1,5c8229443ef6b967,fd012fecd1e593e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:56.804 INFO 2981023 --- [Collect all tenants major compactions3,5c8229443ef6b967,b32d0b0fe927109d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:50:56.804 INFO 2981023 --- [Collect all tenants major compactions2,5c8229443ef6b967,ab7871923349cc6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:50:56.805 INFO 2981023 --- [Collect all tenants major compactions1,5c8229443ef6b967,fd012fecd1e593e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:50:56.806 INFO 2981023 --- [Collect all tenants major compactions5,5c8229443ef6b967,7617c21fab79857a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:56.806 INFO 2981023 --- [Collect all tenants major compactions4,5c8229443ef6b967,6cea0f38b55ca716] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:56.807 INFO 2981023 --- [Collect all tenants major compactions5,5c8229443ef6b967,7617c21fab79857a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:50:56.808 INFO 2981023 --- [Collect all tenants major compactions6,5c8229443ef6b967,bef771539a279985] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:50:56.808 INFO 2981023 --- [Collect all tenants major compactions4,5c8229443ef6b967,6cea0f38b55ca716] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:50:56.808 INFO 2981023 --- [Collect all tenants major compactions1,5c8229443ef6b967,fd012fecd1e593e1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:50:56.808 INFO 2981023 --- [Collect all tenants major compactions2,5c8229443ef6b967,ab7871923349cc6e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:50:56.808 INFO 2981023 --- [Collect all tenants major compactions1,5c8229443ef6b967,fd012fecd1e593e1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:50:56.808 INFO 2981023 --- [Collect all tenants major compactions2,5c8229443ef6b967,ab7871923349cc6e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:50:56.809 INFO 2981023 --- [Collect all tenants major compactions3,5c8229443ef6b967,b32d0b0fe927109d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:50:56.809 INFO 2981023 --- [Collect all tenants major compactions3,5c8229443ef6b967,b32d0b0fe927109d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:50:56.809 INFO 2981023 --- [Collect all tenants major compactions6,5c8229443ef6b967,bef771539a279985] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:50:56.809 INFO 2981023 --- [Collect all tenants major compactions5,5c8229443ef6b967,7617c21fab79857a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:50:56.809 INFO 2981023 --- [Collect all tenants major compactions5,5c8229443ef6b967,7617c21fab79857a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:50:56.810 INFO 2981023 --- [Collect all tenants major compactions4,5c8229443ef6b967,6cea0f38b55ca716] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:50:56.810 INFO 2981023 --- [Collect all tenants major compactions4,5c8229443ef6b967,6cea0f38b55ca716] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:50:56.811 INFO 2981023 --- [Collect all tenants major compactions6,5c8229443ef6b967,bef771539a279985] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:50:56.811 INFO 2981023 --- [Collect all tenants major compactions6,5c8229443ef6b967,bef771539a279985] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:50:56.811 INFO 2981023 --- [ocp-schedules-4,5c8229443ef6b967,ed6e04cf311f0de5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:50:57.304 INFO 2981023 --- [ocp-async-10,d3e9641aa2a01b27,aed2da8bfe17cb7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:57.764 INFO 2981023 --- [ocp-async-1,7b8f9888666bb7ae,55f6be890a9b9ef1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:50:59.304 INFO 2981023 --- [ocp-async-7,35c7c1a61f373f24,94f6955185f8586c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:50:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,3f911943ed19664e,ca4417bab00af2aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048260, interval=60 2024-09-11 17:50:59.315 INFO 2981023 --- [prometheus-minute-schedule-1,3f911943ed19664e,ca4417bab00af2aa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:50:59.529 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:50:59.529 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:01.304 INFO 2981023 --- [ocp-async-1,bcb4a983836d524c,7800a4d477fbb298] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:02.223 INFO 2981023 --- [ocp-schedules-4,567718303fa9fc41,040346b7b45d3631] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:51:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,59f9c350e2c4d480,2c28b6a40f00dd79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,59f9c350e2c4d480,2c28b6a40f00dd79] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:02.529 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:02.530 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:03.304 INFO 2981023 --- [ocp-async-8,f21c8dbd5499a9a2,c543996fdc1a261b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:05.304 INFO 2981023 --- [ocp-async-3,bd9651c250a9b8f5,e0ba3a30ad1537e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,179937a00766d780,1a5e4405de85e902] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,179937a00766d780,1a5e4405de85e902] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:05.472 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:47:05.127, end=2024-09-11T17:49:05.468} 2024-09-11 17:51:05.475 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName= 2024-09-11 17:51:05.475 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:51:05.477 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:05.127, end=2024-09-11T17:49:05.468},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:51:05.477 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:51:05.477 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:51:05.487 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:51:05.506 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865506000, endTimeUs=1726048145468000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=60D9BD7285B8D2D6AE89F7C622A3BE2E, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:51:05.506Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.506 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865506000, endTimeUs=1726048145468000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=E9E2014C8CE705871C555597A6A32456, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T09:51:05.506Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.6:2882}, serverIpMap={10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.509 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:51:05.509 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.510 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 17:51:05.510 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.515 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.515 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.515 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.515 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=60D9BD7285B8D2D6AE89F7C622A3BE2E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.517 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.517 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.517 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.517 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.523 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.523 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=60D9BD7285B8D2D6AE89F7C622A3BE2E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.523 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.523 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.524 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:51:05.524 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.506, end=2024-09-11T17:49:05.468}, clusterId=1000006, tenantId=1000006, sqlId=E9E2014C8CE705871C555597A6A32456, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.524 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.528 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.506, end=2024-09-11T17:49:05.468}, clusterId=1000006, tenantId=1000006, sqlId=60D9BD7285B8D2D6AE89F7C622A3BE2E, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.528 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:51:05.529 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:05.127, end=2024-09-11T17:49:05.468},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:51:05.529 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:51:05.529 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:51:05.530 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:05.530 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:51:05.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:05.531 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:05.127, end=2024-09-11T17:49:05.468},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:51:05.531 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:51:05.531 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:51:05.532 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:51:05.535 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:05.127, end=2024-09-11T17:49:05.468},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:51:05.535 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:51:05.535 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:51:05.545 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:51:05.558 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865558000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.558Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.560 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:51:05.560 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.564 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.564 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.566 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.566 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.571 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.571 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.572 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:51:05.572 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.558, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.572 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:51:05.574 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:05.127, end=2024-09-11T17:49:05.468},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:51:05.574 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:51:05.574 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:51:05.594 INFO 2981023 --- [ocp-schedules-23,505b18511b3a7899,c1a04fece079b625] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:51:05.617 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865617000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.617Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.623 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865623000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.623Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.623 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865623000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.623Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.630 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865630000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.630Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.641 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865641000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.641Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.648 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865648000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.648Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.652 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865652000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.652Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.654 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865654000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.654Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.665 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:51:05.665 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.669 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.669 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.670 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:51:05.670 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.674 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 63 2024-09-11 17:51:05.674 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.677 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.677 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.678 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.678 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.684 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.684 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.688 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:51:05.688 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.696 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.696 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.699 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.699 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.702 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.702 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.703 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.703 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.704 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.704 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.705 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.705 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.705 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.705 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.705 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.705 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.706 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.706 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.706 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.706 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.707 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.707 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.711 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.711 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.712 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.712 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.715 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.715 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.716 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.716 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.717 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.717 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.717 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.717 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.718 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.718 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.721 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.721 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.722 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.722 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.727 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.728 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.728 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.732 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.732 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.733 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.733 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.733 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.733 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.733 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.734 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.623, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.734 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.735 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.735 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.737 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.738 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.648, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.740 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.617, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.740 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.744 WARN 2981023 --- [scanner-13558408-1,505b18511b3a7899,2422294aa5313ed9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:51:05.744 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.749 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.623, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.749 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.63, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.751 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.652, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.757 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.654, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.768 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.641, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.797 INFO 2981023 --- [alarm-detect-10,505b18511b3a7899,0f440955e162415c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644787, data is empty. 2024-09-11 17:51:05.804 INFO 2981023 --- [alarm-detect-10,505b18511b3a7899,0f440955e162415c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644832, data is empty. 2024-09-11 17:51:05.805 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865805000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.805Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.806 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865806000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.806Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.807 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865807000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.807Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.812 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865812000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.812Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.820 INFO 2981023 --- [alarm-detect-10,505b18511b3a7899,0f440955e162415c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644832, data is empty. 2024-09-11 17:51:05.821 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.821 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.826 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865826000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.826Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.830 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.830 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.836 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.836 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.836 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865836000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.836Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.839 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865839000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.839Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.840 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:51:05.840 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.844 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.844 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.845 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961865845000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:51:05.845Z, endTime=2024-09-11T09:49:05.468Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:51:05.855 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:51:05.855 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.858 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.858 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.860 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.860 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.862 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:51:05.862 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:51:05.864 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.864 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.864 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.864 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.865 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.865 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.866 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.866 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.869 INFO 2981023 --- [alarm-detect-10,505b18511b3a7899,35989de09d27b671] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644787, data is empty. 2024-09-11 17:51:05.870 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.870 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.872 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.872 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.872 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.872 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.873 INFO 2981023 --- [alarm-detect-11,505b18511b3a7899,de55cdb150001ece] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644787, data is empty. 2024-09-11 17:51:05.873 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.873 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.877 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.877 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.883 INFO 2981023 --- [alarm-detect-10,505b18511b3a7899,35989de09d27b671] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644832, data is empty. 2024-09-11 17:51:05.883 INFO 2981023 --- [alarm-detect-11,505b18511b3a7899,de55cdb150001ece] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644832, data is empty. 2024-09-11 17:51:05.883 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.883 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.884 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.884 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.884 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.884 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.890 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.890 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.891 INFO 2981023 --- [alarm-detect-11,505b18511b3a7899,de55cdb150001ece] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644832, data is empty. 2024-09-11 17:51:05.892 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.892 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.895 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.895 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.897 INFO 2981023 --- [alarm-detect-10,505b18511b3a7899,35989de09d27b671] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644832, data is empty. 2024-09-11 17:51:05.897 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.897 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.897 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.897 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:51:05.897 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:51:05.901 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.904 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.807, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.904 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.904 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.906 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.906 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.909 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.909 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.909 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.912 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.912 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.806, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.916 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.917 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.917 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.918 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:51:05.918 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048145468000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 17:51:05.919 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.920 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.812, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.920 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.920 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.805, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.921 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:51:05.924 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.836, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.926 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.839, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.926 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.845, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.927 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:51:05.826, end=2024-09-11T17:49:05.468}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:51:05.928 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:51:05.929 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:05.127, end=2024-09-11T17:49:05.468},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:51:05.929 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:51:05.929 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:51:05.933 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:51:06.039 INFO 2981023 --- [alarm-detect-14,505b18511b3a7899,dbc388b1c83fc428] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644787, data is empty. 2024-09-11 17:51:06.041 INFO 2981023 --- [alarm-detect-14,505b18511b3a7899,dbc388b1c83fc428] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644832, data is empty. 2024-09-11 17:51:06.049 INFO 2981023 --- [alarm-detect-14,505b18511b3a7899,dbc388b1c83fc428] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644832, data is empty. 2024-09-11 17:51:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:51:06.337563+08:00, lastProcessTime=2024-09-11T17:50:56.353394+08:00 2024-09-11 17:51:06.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:51:06.337563+08:00 2024-09-11 17:51:07.304 INFO 2981023 --- [ocp-async-5,5069de77ded2d8f0,324d83095ea4c596] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:07.551 INFO 2981023 --- [ocp-schedules-28,1a53c0cd12a720aa,43439f61fdb0377c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:51:07.594 INFO 2981023 --- [ocp-schedules-4,ffdcd252d1df1a42,bd0df63eff354dee] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:51:07.594+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:51:07.595 INFO 2981023 --- [ocp-schedules-4,ffdcd252d1df1a42,bd0df63eff354dee] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:51:07.596 INFO 2981023 --- [ocp-schedules-4,ffdcd252d1df1a42,bd0df63eff354dee] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:51:07.596 INFO 2981023 --- [ocp-schedules-4,ffdcd252d1df1a42,bd0df63eff354dee] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:51:07.764 INFO 2981023 --- [ocp-async-2,9f1711318f1c106b,0293c538c093a0eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:51:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,525791031389cfb1,ad08ce8ede0a3b52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,525791031389cfb1,ad08ce8ede0a3b52] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:08.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:08.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:09.304 INFO 2981023 --- [ocp-async-10,93afdb84590ce1ab,0da1fb2f122a8486] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:51:10.003 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:51:10.913 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:51:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:51:11.266 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048200, nodeCount=194, affectRows=194 2024-09-11 17:51:11.305 INFO 2981023 --- [ocp-async-2,b4e851e721ff31a1,3af7b876dd0706fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,a099e4e93f96ca74,04f0e217518297a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,a099e4e93f96ca74,04f0e217518297a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:11.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:11.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:12.227 INFO 2981023 --- [ocp-schedules-3,2bc8d9c8ada0d8c6,5a812a4ba45f8c09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:51:12.466 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048200, nodeCount=194, validCount=194 2024-09-11 17:51:12.506 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048200, nodeCount=269, affectRows=269 2024-09-11 17:51:13.305 INFO 2981023 --- [ocp-async-9,cd87de22b41e5edc,1a2615756111cbe3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:13.441 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048200, nodeCount=269, validCount=269 2024-09-11 17:51:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,67e2173bca3e5017,8d386091fe2ec661] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,bab7c299ebdbd768,fd82f4550a15b608] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:51:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,67e2173bca3e5017,8d386091fe2ec661] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:14.491 INFO 2981023 --- [ocp-schedules-4,e8bf0158116c5f2d,a0a81eca6adb506c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:51:14.520 INFO 2981023 --- [ocp-schedules-4,e8bf0158116c5f2d,a0a81eca6adb506c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:51:14.520 INFO 2981023 --- [ocp-schedules-4,e8bf0158116c5f2d,a0a81eca6adb506c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:51:14.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:14.533 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:14.593 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048200, nodeCount=806, affectRows=806 2024-09-11 17:51:14.593 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048200, nodeCount=1000, affectRows=1000 2024-09-11 17:51:14.595 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048200, nodeCount=806, validCount=806 2024-09-11 17:51:14.595 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048200, nodeCount=999, validCount=999 2024-09-11 17:51:15.305 INFO 2981023 --- [ocp-async-1,f1854f8b0609abe4,5bee245dd946aaf6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:16.015 INFO 2981023 --- [ocp-schedules-30,5212b624f9623c17,58c4a22c31f97e97] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:51:16.015 INFO 2981023 --- [host-check-clockdiff1,5212b624f9623c17,72b0c57d607fa4b1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:51:16.016 INFO 2981023 --- [host-check-clockdiff2,5212b624f9623c17,1b20f96483cc0a82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:51:16.016 INFO 2981023 --- [host-check-clockdiff3,5212b624f9623c17,aa5d55625cabc1bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:51:16.025 INFO 2981023 --- [ocp-schedules-4,3c859c833d9aae0e,c3829101b17026a8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:51:16.027 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:51:16.027 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:51:16.027 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:51:16.035 INFO 2981023 --- [host-check-clockdiff2,5212b624f9623c17,1b20f96483cc0a82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048276 0 0 , error= 2024-09-11 17:51:16.035 INFO 2981023 --- [host-check-clockdiff3,5212b624f9623c17,aa5d55625cabc1bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048276 0 0 , error= 2024-09-11 17:51:16.036 INFO 2981023 --- [ocp-schedules-25,4e0cae3faf4f4d85,1a9a237654c66c81] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:51:16.036 INFO 2981023 --- [host-check-agent-status1,4e0cae3faf4f4d85,e582937c99f74564] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:51:16.036 INFO 2981023 --- [host-check-agent-status2,4e0cae3faf4f4d85,551e6df01cb22b02] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:51:16.038 INFO 2981023 --- [host-check-agent-status3,4e0cae3faf4f4d85,3a0527132e52d4ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:51:16.041 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:51:16.041 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:51:16.041 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:51:16.043 INFO 2981023 --- [host-check-agent-status3,4e0cae3faf4f4d85,3a0527132e52d4ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:51:16.045 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:51:16.045 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:51:16.046 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:51:16.046 INFO 2981023 --- [host-check-agent-status3,4e0cae3faf4f4d85,3a0527132e52d4ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:51:16.048 INFO 2981023 --- [host-check-agent-status2,4e0cae3faf4f4d85,551e6df01cb22b02] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:51:16.049 INFO 2981023 --- [host-check-agent-status1,4e0cae3faf4f4d85,e582937c99f74564] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:51:16.049 INFO 2981023 --- [host-check-agent-status2,4e0cae3faf4f4d85,551e6df01cb22b02] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:51:16.050 INFO 2981023 --- [host-check-agent-status1,4e0cae3faf4f4d85,e582937c99f74564] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:51:16.073 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:51:16.073 INFO 2981023 --- [host-check-agent-status1,4e0cae3faf4f4d85,e582937c99f74564] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:51:16.074 INFO 2981023 --- [host-check-agent-status3,4e0cae3faf4f4d85,3a0527132e52d4ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:51:16.078 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:51:16.078 INFO 2981023 --- [host-check-agent-status1,4e0cae3faf4f4d85,e582937c99f74564] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:51:16.078 INFO 2981023 --- [host-check-agent-status1,4e0cae3faf4f4d85,e582937c99f74564] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:51:16.078 INFO 2981023 --- [host-check-agent-status1,4e0cae3faf4f4d85,e582937c99f74564] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:51:16.079 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:51:16.079 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:51:16.079356673+08:00 2024-09-11 17:51:16.080 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:51:16.079884164+08:00 2024-09-11 17:51:16.080 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:51:16.079+08:00 2024-09-11 17:51:16.080 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:51:16.080+08:00 2024-09-11 17:51:16.081 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:51:16.081803413+08:00 2024-09-11 17:51:16.082 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:51:16.082+08:00 2024-09-11 17:51:16.084 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:51:16.082+08:00 and status AVAILABLE 2024-09-11 17:51:16.085 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:51:16.079+08:00 and status AVAILABLE 2024-09-11 17:51:16.086 INFO 2981023 --- [ocp-schedules-22,cc7d0d44b6bd6c47,85d05e4d27ccab5b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:51:16.088 INFO 2981023 --- [host-check-agent-status3,4e0cae3faf4f4d85,3a0527132e52d4ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:51:16.088 INFO 2981023 --- [host-check-agent-status3,4e0cae3faf4f4d85,3a0527132e52d4ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:51:16.088 INFO 2981023 --- [host-check-agent-status3,4e0cae3faf4f4d85,3a0527132e52d4ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:51:16.091 INFO 2981023 --- [host-check-agent-status2,4e0cae3faf4f4d85,551e6df01cb22b02] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:51:16.092 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:51:16.094 INFO 2981023 --- [host-check-agent-status2,4e0cae3faf4f4d85,551e6df01cb22b02] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:51:16.094 INFO 2981023 --- [host-check-agent-status2,4e0cae3faf4f4d85,551e6df01cb22b02] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:51:16.094 INFO 2981023 --- [host-check-agent-status2,4e0cae3faf4f4d85,551e6df01cb22b02] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:51:16.094 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:51:16.080+08:00 and status AVAILABLE 2024-09-11 17:51:16.101 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:51:16.113 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:51:16.114 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:51:16.114 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:51:16.116 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:51:16.125 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:51:16.141 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:51:16.143 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:51:16.150 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:51:16.158 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:51:16.162 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:51:16.187 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:51:16.191 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:51:16.196 INFO 2981023 --- [ocp-schedules-8,a56f4c6489b76886,dc791cd071aeb3f8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:51:16.196 INFO 2981023 --- [ocp-schedules-14,063c643dff946ac8,0848e71a58083035] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:51:16.197 INFO 2981023 --- [ocp-schedules-26,99a810a6fab1392c,5c985decbcdd935d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:51:16.197 INFO 2981023 --- [ocp-schedules-28,a36f63fca2f94104,d8f736d9683bbf34] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:51:16.198 INFO 2981023 --- [ocp-schedules-14,063c643dff946ac8,0848e71a58083035] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:51:16.205 INFO 2981023 --- [ocp-schedules-8,a56f4c6489b76886,dc791cd071aeb3f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:51:16.206 INFO 2981023 --- [ocp-schedules-14,063c643dff946ac8,0848e71a58083035] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:51:16.212 WARN 2981023 --- [scanner-13560164-1,cc7d0d44b6bd6c47,3a8acb52b3c17c1a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:51:16.223 INFO 2981023 --- [ocp-schedules-26,99a810a6fab1392c,5c985decbcdd935d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:51:16.231 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:51:16.246 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:51:16.259 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:51:16.264 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:51:16.276 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:51:16.284 INFO 2981023 --- [ocp-schedules-12,64c72ff203dbd537,ed721ef10ee93383] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:51:16.286 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:51:16.293 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:51:16.300 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:51:16.302 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:51:16.313 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:51:16.328 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:51:16.335 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:51:16.337 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:51:16.339 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:51:16.338858+08:00, lastProcessTime=2024-09-11T17:51:06.337563+08:00 2024-09-11 17:51:16.340 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:51:16.345 INFO 2981023 --- [host-check-host-status1,3c859c833d9aae0e,0a305dc06f85e200] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:51:16.349 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:51:16.369 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:51:16.374 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:51:16.399 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:51:16.420 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.422 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:16.422 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:51:16.431 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:51:16.431 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.432 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:16.433 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:51:16.439 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.439 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:16.442 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.443 INFO 2981023 --- [host-check-host-status3,3c859c833d9aae0e,ef2ee315cfebe1d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:51:16.444 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:16.446 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:51:16.447 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.448 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:16.488 INFO 2981023 --- [ocp-schedules-8,a56f4c6489b76886,dc791cd071aeb3f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:51:16.496 INFO 2981023 --- [host-check-host-status2,3c859c833d9aae0e,72e4856e03b09124] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:51:16.552 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.553 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.554 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:51:16.556 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.558 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.558 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:51:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:16.603 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:51:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.604 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.604 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.605 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.605 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:51:16.605 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.606 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:16.606 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:51:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:51:16.607 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.611 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:51:16.611 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:51:16.617 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:16.617 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:51:16.633 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:16.633 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:51:16.645 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.646 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.647 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:16.647 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:51:16.648 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.650 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:51:16.651 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:16.651 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:51:16.655 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:16.657 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:16.657 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:51:16.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.659 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.659 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:16.660 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:51:16.663 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.664 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:51:16.664 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.664 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:16.665 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:51:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:51:16.668 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:51:16.669 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:51:16.670 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:16.672 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:51:16.674 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:51:16.678 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.678 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.679 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:16.679 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:51:16.682 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:51:16.685 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:51:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:51:16.687 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:51:16.692 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:51:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:51:16.735 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:51:16.338858+08:00 2024-09-11 17:51:16.738 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:51:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.747 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:51:16.747 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:51:16.753 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:51:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:51:16.753 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.753 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:51:16.757 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:51:16.759 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:51:16.759 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:51:16.763 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:16.763 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:16.764 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:51:16.769 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:51:16.769 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726047684472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:51:16.769 INFO 2981023 --- [ocp-schedules-26,99a810a6fab1392c,5c985decbcdd935d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:51:16.963 INFO 2981023 --- [host-check-clockdiff1,5212b624f9623c17,72b0c57d607fa4b1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048276 0 0 , error= 2024-09-11 17:51:17.305 INFO 2981023 --- [ocp-async-4,b6b4707b509bb3e4,b535482f785867cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,665540e8ec88617d,6c9aad54628ae358] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,665540e8ec88617d,6c9aad54628ae358] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:17.533 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:17.533 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:17.764 INFO 2981023 --- [ocp-async-8,586faa185b4e9905,0e94f67799fd4ea0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:51:19.305 INFO 2981023 --- [ocp-async-5,e05f73918af578a3,69fea9ef1a7559a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,08160d89a37f5c1b,24de423306fc5e14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,08160d89a37f5c1b,24de423306fc5e14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:20.534 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:20.535 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:21.305 INFO 2981023 --- [ocp-async-1,7f6c23e8cfe3933b,f446ce68e145fe12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:22.229 INFO 2981023 --- [ocp-schedules-26,a9c6b4c0ea6389a9,170c925c26b4cf36] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:51:23.305 INFO 2981023 --- [ocp-async-3,ed369d7a807b906a,e809fb5e4d22ecef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:23.311 INFO 2981023 --- [ocp-schedules-27,d402e9fea534578c,1428c2c09dbd16d2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:51:23.311 INFO 2981023 --- [ocp-schedules-27,d402e9fea534578c,1428c2c09dbd16d2] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:51:23.311 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:51:23.312 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:51:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,8d31dd04a66df80c,a6f0a5a459de639e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:23.315 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:51:23.318 INFO 2981023 --- [ocp-schedules-27,d402e9fea534578c,1428c2c09dbd16d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:51:23.322 INFO 2981023 --- [ocp-schedules-27,d402e9fea534578c,1428c2c09dbd16d2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:51:23.323 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:51:23.325 INFO 2981023 --- [prometheus-minute-schedule-1,8d31dd04a66df80c,a6f0a5a459de639e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:23.330 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:51:23.344 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:51:23.355 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:51:23.359 INFO 2981023 --- [ocp-schedules-27,d402e9fea534578c,1428c2c09dbd16d2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:51:23.360 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:51:23.362 INFO 2981023 --- [ocp-schedules-27,d402e9fea534578c,1428c2c09dbd16d2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:51:23.535 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:23.536 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:24.305 INFO 2981023 --- [ocp-schedules-5,50c728809ca96db5,ae32ee6f504140d8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:51:24.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7d108c1e5c9df074,b972058f0a69329e] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&version=2&ObClusterId=1719994443, client=10.31.35.6, traceId=7d108c1e5c9df074, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkrtd,1719994443,2,{"ObRegion":"linkrtd","ObCluster":"linkrtd","ObRegionId":1719994443,"ObClusterId":1719994443,"Type":"PRIMARY","timestamp":1726048284469627,"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:51:24.480 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7d108c1e5c9df074,b972058f0a69329e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkrtd, obClusterId=1719994443, rootServerJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, groupName=linkrtd, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:51:24.480 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7d108c1e5c9df074,b972058f0a69329e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:51:24.480 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7d108c1e5c9df074,b972058f0a69329e] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkrtd, clusterId=1719994443, affectRows=1 2024-09-11 17:51:24.480 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7d108c1e5c9df074,b972058f0a69329e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.6, traceId=7d108c1e5c9df074, duration=10 ms] 2024-09-11 17:51:25.305 INFO 2981023 --- [ocp-async-9,b342dc3ff02d2ce5,38c6d8bc8a20f92c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,7f719b232444ea8b,301c0d4fdf359c4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,7f719b232444ea8b,301c0d4fdf359c4a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:51:26.351356+08:00, lastProcessTime=2024-09-11T17:51:16.338858+08:00 2024-09-11 17:51:26.484 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:51:26.351356+08:00 2024-09-11 17:51:26.536 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:26.536 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:26.646 INFO 2981023 --- [ocp-schedules-2,cd4f057766d0a5e4,853ccfa2443476f2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:51:26.713 WARN 2981023 --- [scanner-13560421-1,cd4f057766d0a5e4,ec4156bbf74e29aa] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:51:26.725 INFO 2981023 --- [alarm-detect-16,cd4f057766d0a5e4,2fab4ba8ead1397f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644832, data is empty. 2024-09-11 17:51:26.728 INFO 2981023 --- [alarm-detect-16,cd4f057766d0a5e4,2fab4ba8ead1397f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644832, data is empty. 2024-09-11 17:51:26.731 INFO 2981023 --- [alarm-detect-16,cd4f057766d0a5e4,2fab4ba8ead1397f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644832, data is empty. 2024-09-11 17:51:26.757 INFO 2981023 --- [alarm-detect-17,cd4f057766d0a5e4,5d16f23a710c51ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644832, data is empty. 2024-09-11 17:51:26.763 INFO 2981023 --- [alarm-detect-17,cd4f057766d0a5e4,5d16f23a710c51ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644832, data is empty. 2024-09-11 17:51:26.765 INFO 2981023 --- [alarm-detect-11,cd4f057766d0a5e4,d09e6a8ae59fc994] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644832, data is empty. 2024-09-11 17:51:26.766 INFO 2981023 --- [alarm-detect-17,cd4f057766d0a5e4,5d16f23a710c51ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644832, data is empty. 2024-09-11 17:51:26.768 INFO 2981023 --- [alarm-detect-11,cd4f057766d0a5e4,d09e6a8ae59fc994] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644832, data is empty. 2024-09-11 17:51:26.775 INFO 2981023 --- [alarm-detect-11,cd4f057766d0a5e4,d09e6a8ae59fc994] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644832, data is empty. 2024-09-11 17:51:26.886 INFO 2981023 --- [alarm-detect-13,cd4f057766d0a5e4,6e844acccedc2835] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644832, data is empty. 2024-09-11 17:51:26.889 INFO 2981023 --- [alarm-detect-13,cd4f057766d0a5e4,6e844acccedc2835] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644832, data is empty. 2024-09-11 17:51:26.894 INFO 2981023 --- [alarm-detect-13,cd4f057766d0a5e4,6e844acccedc2835] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644832, data is empty. 2024-09-11 17:51:27.305 INFO 2981023 --- [ocp-async-8,d2ec39a678183abc,87fdf0b911559480] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:27.764 INFO 2981023 --- [ocp-async-7,4466f23d46ab0b52,44b7ebfbb85d7c38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:51:29.305 INFO 2981023 --- [ocp-async-3,a77bb7f1873afb0d,d9fef18a629fed2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,fe15887cd376cc36,8090fc0c43bf7aba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,fe15887cd376cc36,8090fc0c43bf7aba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:29.536 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:29.537 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:31.306 INFO 2981023 --- [ocp-async-7,e2aebddb5d1b00a4,86910eaf31b9a519] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:32.232 INFO 2981023 --- [ocp-schedules-14,bf8d4d48d3a966f8,0a8fd69a9af1953e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:51:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,8a9206f95f483b55,9a74504fbe7a0a3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:32.315 INFO 2981023 --- [prometheus-minute-schedule-1,8a9206f95f483b55,9a74504fbe7a0a3d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:32.537 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:32.537 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:33.306 INFO 2981023 --- [ocp-async-8,dfc407e8f3e43d95,53719c63c76df323] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:33.785 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:51:33.785 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:51:33.787 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:51:33.787 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:51:33.787 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:51:33.788 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:51:33.791 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:51:33.791 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:51:33.791 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:51:33.791 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:51:33.815 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:51:33.819 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:51:33.819 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:51:33.819 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:51:33.819 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:51:33.819 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:51:33.823 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:51:34.065 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:51:34.375 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.584209541S 2024-09-11 17:51:34.375 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:51:34.427 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:51:34.427 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:51:34.445 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:51:34.482 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:51:34.482 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:51:34.482 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:51:34.482 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:51:34.482 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:51:34.482 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:51:34.487 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:51:34.525 INFO 2981023 --- [ocp-schedules-17,3b13385af22f8d0a,cd3b5df579f54288] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:51:34.533 INFO 2981023 --- [ocp-schedules-17,3b13385af22f8d0a,cd3b5df579f54288] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:51:34.533 INFO 2981023 --- [ocp-schedules-17,3b13385af22f8d0a,cd3b5df579f54288] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:51:34.567 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] o.s.boot.SpringApplication : Started application in 0.779 seconds (JVM running for 84094.44) 2024-09-11 17:51:34.567 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.779843193S 2024-09-11 17:51:34.766 INFO 2981023 --- [ocp-schedules-26,cf82394a1152907c,6b98b23ef3075018] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:51:35.306 INFO 2981023 --- [ocp-async-10,653bf20e8e5074d4,f8415d8937ee3b64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,85c46709247603ae,bc6377b7d935b77c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,85c46709247603ae,bc6377b7d935b77c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:35.538 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:35.538 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:51:36.337187+08:00, lastProcessTime=2024-09-11T17:51:26.351356+08:00 2024-09-11 17:51:36.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:51:36.337187+08:00 2024-09-11 17:51:36.936 INFO 2981023 --- [ocp-schedules-2,e89c3d3e0dd58247,a76fc6be8f5de1ae] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:51:36.959 WARN 2981023 --- [scanner-13560698-1,e89c3d3e0dd58247,a7ef89c0f20a783a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:51:37.306 INFO 2981023 --- [ocp-async-2,1bafe2bc0bd759f9,bc5e40b7f0487a95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:37.314 INFO 2981023 --- [prometheus-cache-monitor1,686a3c29aec0d4f5,15eaa207690783e2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:51:37.315 INFO 2981023 --- [prometheus-cache-monitor1,686a3c29aec0d4f5,15eaa207690783e2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:51:37.764 INFO 2981023 --- [ocp-async-4,29d0bef9521114af,3ab1a188b173dccf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:51:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,850709b16802dbb4,c1a7e15a9e97334c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,850709b16802dbb4,c1a7e15a9e97334c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:38.538 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:38.539 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:39.306 INFO 2981023 --- [ocp-async-6,fc9f10f61becb2e1,4bebf6f14406dd84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:40.003 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:51:40.003 INFO 2981023 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:51:41.306 INFO 2981023 --- [ocp-async-1,545eb9fb7ae1ad24,93e557eb998dee0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,380e1169b96a904b,44153381c7fd693b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,380e1169b96a904b,44153381c7fd693b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:41.539 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:41.539 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:42.050 INFO 2981023 --- [ocp-schedules-15,32e73476df405976,26d5a8c2a1b6da5e] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:51:42.069 WARN 2981023 --- [ocp-schedules-15,32e73476df405976,26d5a8c2a1b6da5e] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:51:42.235 INFO 2981023 --- [ocp-schedules-24,6398cdc11d70ffdd,9fc6c82a23888df7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:51:43.306 INFO 2981023 --- [ocp-async-3,4bd7033eec0f98c3,a2bfca35681ea711] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,33135f552bcd2c2a,138a584d0a9b59e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,33135f552bcd2c2a,138a584d0a9b59e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:44.539 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:44.540 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:45.101 INFO 2981023 --- [prometheus-second-schedule-1,09794ab293509e7e,0db374ecb37369c0] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:51:45.306 INFO 2981023 --- [ocp-async-7,5f20f3675f9e9f3b,092b0e6e36660254] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:46.020 INFO 2981023 --- [ocp-schedules-9,bf09bb1add710fe5,970c8887662fae80] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:51:46.023 INFO 2981023 --- [ocp-schedules-15,b602697a4fbdfaac,ff677729ab427aff] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:51:46.024 INFO 2981023 --- [host-check-clockdiff1,bf09bb1add710fe5,8331941a926c54b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:51:46.025 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:51:46.025 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:51:46.026 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:51:46.027 INFO 2981023 --- [host-check-clockdiff2,bf09bb1add710fe5,fba98248831cadd7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:51:46.028 INFO 2981023 --- [ocp-schedules-3,bd06f7bd48ee782d,b09625b6f1f0a24e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:51:46.028 INFO 2981023 --- [host-check-clockdiff3,bf09bb1add710fe5,150d3910c80f6336] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:51:46.031 INFO 2981023 --- [host-check-agent-status1,bd06f7bd48ee782d,269a65a70f12064f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:51:46.032 INFO 2981023 --- [host-check-agent-status2,bd06f7bd48ee782d,d345abd63010cdeb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:51:46.032 INFO 2981023 --- [host-check-agent-status3,bd06f7bd48ee782d,7313269a8f558356] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:51:46.034 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:51:46.036 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:51:46.036 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:51:46.037 INFO 2981023 --- [host-check-clockdiff2,bf09bb1add710fe5,fba98248831cadd7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048306 0 0 , error= 2024-09-11 17:51:46.041 INFO 2981023 --- [host-check-clockdiff3,bf09bb1add710fe5,150d3910c80f6336] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048306 0 0 , error= 2024-09-11 17:51:46.041 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:51:46.042 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:51:46.042 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:51:46.044 INFO 2981023 --- [host-check-agent-status3,bd06f7bd48ee782d,7313269a8f558356] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:51:46.045 INFO 2981023 --- [host-check-agent-status2,bd06f7bd48ee782d,d345abd63010cdeb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:51:46.045 INFO 2981023 --- [host-check-agent-status3,bd06f7bd48ee782d,7313269a8f558356] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:51:46.046 INFO 2981023 --- [host-check-agent-status1,bd06f7bd48ee782d,269a65a70f12064f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:51:46.047 INFO 2981023 --- [host-check-agent-status2,bd06f7bd48ee782d,d345abd63010cdeb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:51:46.048 INFO 2981023 --- [host-check-agent-status1,bd06f7bd48ee782d,269a65a70f12064f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:51:46.064 INFO 2981023 --- [host-check-agent-status3,bd06f7bd48ee782d,7313269a8f558356] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:51:46.065 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:51:46.065 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:51:46.065 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:51:46.066 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:51:46.066474010+08:00 2024-09-11 17:51:46.067 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:51:46.066835939+08:00 2024-09-11 17:51:46.067 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:51:46.067250648+08:00 2024-09-11 17:51:46.067 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:51:46.066+08:00 2024-09-11 17:51:46.068 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:51:46.067+08:00 2024-09-11 17:51:46.069 INFO 2981023 --- [host-check-agent-status3,bd06f7bd48ee782d,7313269a8f558356] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:51:46.069 INFO 2981023 --- [host-check-agent-status3,bd06f7bd48ee782d,7313269a8f558356] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:51:46.069 INFO 2981023 --- [host-check-agent-status3,bd06f7bd48ee782d,7313269a8f558356] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:51:46.069 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:51:46.067+08:00 2024-09-11 17:51:46.073 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:51:46.066+08:00 and status AVAILABLE 2024-09-11 17:51:46.074 INFO 2981023 --- [host-check-agent-status1,bd06f7bd48ee782d,269a65a70f12064f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:51:46.076 INFO 2981023 --- [host-check-agent-status1,bd06f7bd48ee782d,269a65a70f12064f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:51:46.076 INFO 2981023 --- [host-check-agent-status1,bd06f7bd48ee782d,269a65a70f12064f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:51:46.076 INFO 2981023 --- [host-check-agent-status1,bd06f7bd48ee782d,269a65a70f12064f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:51:46.078 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:51:46.079 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:51:46.067+08:00 and status AVAILABLE 2024-09-11 17:51:46.081 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:51:46.067+08:00 and status AVAILABLE 2024-09-11 17:51:46.084 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:51:46.085 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:51:46.097 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:51:46.099 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:51:46.105 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:51:46.106 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:51:46.106 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:51:46.113 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:51:46.114 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:51:46.122 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:51:46.124 INFO 2981023 --- [host-check-agent-status2,bd06f7bd48ee782d,d345abd63010cdeb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:51:46.128 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:51:46.144 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:51:46.144 INFO 2981023 --- [host-check-agent-status2,bd06f7bd48ee782d,d345abd63010cdeb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:51:46.144 INFO 2981023 --- [host-check-agent-status2,bd06f7bd48ee782d,d345abd63010cdeb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:51:46.144 INFO 2981023 --- [host-check-agent-status2,bd06f7bd48ee782d,d345abd63010cdeb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:51:46.153 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:51:46.161 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:51:46.161 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:51:46.183 INFO 2981023 --- [ocp-schedules-24,bd17b4c7fe86ba3d,cab65b78e86b4bc1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:51:46.187 INFO 2981023 --- [ocp-schedules-14,647334b033cad5fa,5ab0fba2f9a610b4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:51:46.187 INFO 2981023 --- [ocp-schedules-21,475be917f95d4fe0,a6691996250374cf] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:51:46.188 INFO 2981023 --- [ocp-schedules-2,e8cdcdd9fdac9bf8,9c3a90c8bfcd2c5d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:51:46.188 INFO 2981023 --- [ocp-schedules-18,2e22e9c3a00a32d2,a1e6f9f5cdfdffca] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:51:46.193 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:51:46.197 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:51:46.199 INFO 2981023 --- [ocp-schedules-14,647334b033cad5fa,5ab0fba2f9a610b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:51:46.199 INFO 2981023 --- [ocp-schedules-21,475be917f95d4fe0,a6691996250374cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:51:46.206 INFO 2981023 --- [ocp-schedules-2,e8cdcdd9fdac9bf8,9c3a90c8bfcd2c5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:51:46.210 INFO 2981023 --- [ocp-schedules-24,bd17b4c7fe86ba3d,cab65b78e86b4bc1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:51:46.227 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:51:46.234 INFO 2981023 --- [ocp-schedules-27,63e6c86a000d0b97,b920993241aa735b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:51:46.234 INFO 2981023 --- [ocp-schedules-27,63e6c86a000d0b97,b920993241aa735b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:51:46.237 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:51:46.248 INFO 2981023 --- [ocp-schedules-21,475be917f95d4fe0,a6691996250374cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:51:46.273 INFO 2981023 --- [ocp-schedules-20,cdb1d1ef8b524649,2b9dc4e00925f6a5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:51:46.275 INFO 2981023 --- [ocp-schedules-20,cdb1d1ef8b524649,2b9dc4e00925f6a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:51:46.280 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.280 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.281 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:46.281 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:51:46.281 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:46.282 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:51:46.282 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:51:46.285 INFO 2981023 --- [ocp-schedules-10,e8745e99963eb53b,2393d5373fdcbfb5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:51:46.289 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:51:46.295 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.296 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.296 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:51:46.296 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:46.297 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:51:46.297 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:51:46.297 INFO 2981023 --- [ocp-schedules-14,647334b033cad5fa,5ab0fba2f9a610b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:51:46.297 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.297 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:51:46.312 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:51:46.314 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:51:46.317 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:51:46.329 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.329 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:51:46.330 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:46.331 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:51:46.331 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.331 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:46.336 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.336 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.336 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:46.337 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:51:46.337 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.337 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.338 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:51:46.338 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:51:46.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,ac796a6a167020f3,ed41faca801e6e05] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=ac796a6a167020f3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:51:46.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,f294c384f40a6e8e,1ea0ba9f965ab9c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=f294c384f40a6e8e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:51:46.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,f294c384f40a6e8e,1ea0ba9f965ab9c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=f294c384f40a6e8e, duration=1 ms] 2024-09-11 17:51:46.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,ac796a6a167020f3,ed41faca801e6e05] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=ac796a6a167020f3, duration=1 ms] 2024-09-11 17:51:46.345 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:51:46.343469+08:00, lastProcessTime=2024-09-11T17:51:36.337187+08:00 2024-09-11 17:51:46.347 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.348 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:51:46.355 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.356 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:51:46.361 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.361 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.364 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:51:46.367 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:51:46.369 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,bf23876858ede504,a5a5c25634c73828] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&ObClusterId=1719994443&version=2, client=10.31.35.10, traceId=bf23876858ede504, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkrtd,1719994443,false,2,] 2024-09-11 17:51:46.369 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,bf23876858ede504,a5a5c25634c73828] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=bf23876858ede504, duration=1 ms] 2024-09-11 17:51:46.383 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.384 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:51:46.388 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.391 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:51:46.391 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.392 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.392 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:51:46.394 INFO 2981023 --- [ocp-schedules-20,cdb1d1ef8b524649,2b9dc4e00925f6a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:51:46.401 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:51:46.403 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.404 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:51:46.405 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.405 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:51:46.406 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:51:46.406 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.410 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:51:46.410 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:51:46.412 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.416 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.418 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:51:46.427 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:51:46.427 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:46.428 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:51:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.431 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:51:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:51:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:51:46.436 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:51:46.438 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:51:46.443 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:51:46.450 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:46.450 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:51:46.450 INFO 2981023 --- [host-check-host-status1,b602697a4fbdfaac,ea6045f6ab0e5daa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:51:46.455 INFO 2981023 --- [host-check-host-status2,b602697a4fbdfaac,86564edb3192cb98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:51:46.461 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:51:46.477 INFO 2981023 --- [host-check-host-status3,b602697a4fbdfaac,8cce41aadf3546bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:51:46.479 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.481 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:46.483 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.484 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:46.485 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.487 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:51:46.488 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:51:46.505 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.505 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.505 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:46.506 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.507 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:51:46.513 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:51:46.518 INFO 2981023 --- [pool-10-thread-3,db353763a018c7c6,32ccc63d4107e625] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:51:46.518 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:46.518 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:51:46.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:51:46.526 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:51:46.529 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.530 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.530 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:51:46.531 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:51:46.531 INFO 2981023 --- [pool-10-thread-4,db353763a018c7c6,2e8fb1f66459ff35] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:51:46.535 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.535 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:51:46.536 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:51:46.536 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.540 INFO 2981023 --- [pool-10-thread-3,db353763a018c7c6,32ccc63d4107e625] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:51:46.540 INFO 2981023 --- [pool-10-thread-3,db353763a018c7c6,32ccc63d4107e625] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:51:46.541 INFO 2981023 --- [pool-10-thread-2,db353763a018c7c6,5c5bbef4b5ae358b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:51:46.541 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:51:46.541 INFO 2981023 --- [pool-10-thread-1,db353763a018c7c6,334cd2d05fe34b15] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:51:46.556 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:51:46.556 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:51:46.557 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:51:46.557 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:51:46.558 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,838e7276464ac79b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:51:46.558 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,838e7276464ac79b] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:51:46.559 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:46.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:51:46.579 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.580 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:51:46.580 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.580 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:51:46.585 WARN 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,838e7276464ac79b] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:51:46.590 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.591 INFO 2981023 --- [pool-10-thread-2,db353763a018c7c6,5c5bbef4b5ae358b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:51:46.591 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.592 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:46.593 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:51:46.597 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.598 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:51:46.600 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:51:46.605 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.605 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:51:46.614 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.615 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:51:46.616 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.617 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:51:46.617 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:51:46.618 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:51:46.619 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.633 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.634 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:51:46.635 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.636 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:51:46.640 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:51:46.640 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:51:46.652 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.653 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.654 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:51:46.656 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:51:46.656 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.658 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:51:46.659 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:51:46.661 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.662 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:51:46.663 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.664 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:51:46.665 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:51:46.666 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.667 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:51:46.668 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:51:46.668 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.672 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:51:46.673 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:51:46.674 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:46.674 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:51:46.677 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.677 INFO 2981023 --- [ForkJoinPool-6-worker-4,db353763a018c7c6,1d16333bcac237c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.679 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:51:46, 2024-09-11 17:51:46] 2024-09-11 17:51:46.689 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:51:46.690 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:51:46.693 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:51:46.694 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.696 INFO 2981023 --- [ForkJoinPool-6-worker-4,db353763a018c7c6,1d16333bcac237c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:51:46.697 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:51:46.697 INFO 2981023 --- [ForkJoinPool-6-worker-9,db353763a018c7c6,1fcbc74a9d1fd4fd] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:51:46.697 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:51:46.698 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:51:46, 2024-09-11 17:51:46] 2024-09-11 17:51:46.699 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.700 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.700 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.701 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:51:46.702 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:51:46.703 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:51:46.704 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.706 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:51:46.715 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:51:46.724 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.725 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:51:46.725 INFO 2981023 --- [ForkJoinPool-6-worker-4,db353763a018c7c6,1d16333bcac237c8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:51:46.726 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.726 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:51:46.726 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:51:46.726 INFO 2981023 --- [ForkJoinPool-6-worker-4,db353763a018c7c6,1d16333bcac237c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.726 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:51:46.727 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:51:46.727 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.727 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.728 INFO 2981023 --- [ForkJoinPool-6-worker-4,db353763a018c7c6,1d16333bcac237c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:51:46.728 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:51:46, 2024-09-11 17:51:46] 2024-09-11 17:51:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-4,db353763a018c7c6,1d16333bcac237c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.731 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:51:46.731 INFO 2981023 --- [ForkJoinPool-6-worker-7,db353763a018c7c6,24e8d56541039299] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:51:46.732 INFO 2981023 --- [pool-10-thread-1,db353763a018c7c6,334cd2d05fe34b15] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:51:46.732 INFO 2981023 --- [ForkJoinPool-6-worker-4,db353763a018c7c6,1d16333bcac237c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:51:46.742 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:51:46.754 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.757 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:51:46.758 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.759 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:51:46.759 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.764 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.769 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:51:46.770 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:51:46.771 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.773 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:51:46.802 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:51:46.812 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.814 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.815 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:51:46.816 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:51:46.816 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:51:46.821 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:46.822 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:51:46.822 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.822 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:51:46.822 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.824 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:51:46.824 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:51:46.825 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:51:46.826 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:46.827 INFO 2981023 --- [ForkJoinPool-6-worker-0,db353763a018c7c6,e0d69ad06cbd5ccc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:51:46.829 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:51:46.831 INFO 2981023 --- [pool-10-thread-4,db353763a018c7c6,2e8fb1f66459ff35] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:51:46.837 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:51:46.837 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:51:46.839 INFO 2981023 --- [ocp-schedules-2,e8cdcdd9fdac9bf8,9c3a90c8bfcd2c5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:51:46.841 INFO 2981023 --- [ocp-schedules-4,5e6b60d8988c4f32,f4b1d092185f77c4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:51:46.841 INFO 2981023 --- [ocp-schedules-4,5e6b60d8988c4f32,f4b1d092185f77c4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:51:46.848 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:51:46.849 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:51:46.343469+08:00 2024-09-11 17:51:46.855 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:51:46.859 INFO 2981023 --- [ocp-schedules-6,b08e6a65bd2c3690,e08ed1e4cdcac170] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:51:46.919 INFO 2981023 --- [pool-10-thread-2,db353763a018c7c6,5c5bbef4b5ae358b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:51:46.922 INFO 2981023 --- [pool-10-thread-2,db353763a018c7c6,5c5bbef4b5ae358b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:51:47.117 INFO 2981023 --- [ocp-schedules-5,a2f5a73526285f3f,349a9ffbbf2aaf05] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:51:47.155 WARN 2981023 --- [scanner-13560904-1,a2f5a73526285f3f,e41c804fbc7ab4e8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:51:47.176 INFO 2981023 --- [alarm-detect-18,a2f5a73526285f3f,3d996de07dac87fe] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644832, data is empty. 2024-09-11 17:51:47.180 INFO 2981023 --- [alarm-detect-18,a2f5a73526285f3f,3d996de07dac87fe] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644832, data is empty. 2024-09-11 17:51:47.183 INFO 2981023 --- [alarm-detect-18,a2f5a73526285f3f,3d996de07dac87fe] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644832, data is empty. 2024-09-11 17:51:47.201 INFO 2981023 --- [alarm-detect-15,a2f5a73526285f3f,29317878f78a314b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644832, data is empty. 2024-09-11 17:51:47.203 INFO 2981023 --- [alarm-detect-14,a2f5a73526285f3f,f660a834b0e521d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644832, data is empty. 2024-09-11 17:51:47.210 INFO 2981023 --- [alarm-detect-15,a2f5a73526285f3f,29317878f78a314b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644832, data is empty. 2024-09-11 17:51:47.212 INFO 2981023 --- [alarm-detect-14,a2f5a73526285f3f,f660a834b0e521d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644832, data is empty. 2024-09-11 17:51:47.215 INFO 2981023 --- [alarm-detect-15,a2f5a73526285f3f,29317878f78a314b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644832, data is empty. 2024-09-11 17:51:47.215 INFO 2981023 --- [alarm-detect-14,a2f5a73526285f3f,f660a834b0e521d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644832, data is empty. 2024-09-11 17:51:47.308 INFO 2981023 --- [ocp-async-10,a4f20f567d0970b8,be9f574bae4b84d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,826cae464a77ab46,653a03d381d932db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:47.318 INFO 2981023 --- [prometheus-minute-schedule-1,826cae464a77ab46,653a03d381d932db] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:47.329 INFO 2981023 --- [alarm-detect-9,a2f5a73526285f3f,8c32dcef6e74f776] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644832, data is empty. 2024-09-11 17:51:47.337 INFO 2981023 --- [alarm-detect-9,a2f5a73526285f3f,8c32dcef6e74f776] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644832, data is empty. 2024-09-11 17:51:47.341 INFO 2981023 --- [alarm-detect-9,a2f5a73526285f3f,8c32dcef6e74f776] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644832, data is empty. 2024-09-11 17:51:47.541 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:47.541 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:47.764 INFO 2981023 --- [ocp-async-4,198bc1d107ef1306,97524a1514ecd1fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:51:48.556 INFO 2981023 --- [host-check-clockdiff1,bf09bb1add710fe5,8331941a926c54b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048308 0 0 , error= 2024-09-11 17:51:49.307 INFO 2981023 --- [ocp-async-5,b53eb350b977bb63,20187a50e0de8bac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:51:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:51:50.267 INFO 2981023 --- [ocp-schedules-20,bf84777147e9848d,b0c9a248846f6d73] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:51:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,4033f56664051c95,d88118bee56f7cc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,4033f56664051c95,d88118bee56f7cc1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:50.541 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:50.543 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:51.308 INFO 2981023 --- [ocp-async-4,350bf4d02a49ac3a,e795b15ade96d0c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:51.571 INFO 2981023 --- [ocp-schedules-10,1aee8aa028a68468,b7d02addaec83604] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:51:51.572 INFO 2981023 --- [ocp-schedules-10,1aee8aa028a68468,b7d02addaec83604] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:51:51.572 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:51:51.572 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:51:51.573 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:51:51.573 INFO 2981023 --- [ocp-schedules-10,1aee8aa028a68468,b7d02addaec83604] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:51:52.237 INFO 2981023 --- [ocp-schedules-11,9cb1d2a3759c1b6c,0b026b8c50e0a78c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:51:53.168 INFO 2981023 --- [ocp-schedules-22,75150de4b1528291,97254cacf5fd726b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:51:53.168 INFO 2981023 --- [ocp-schedules-22,75150de4b1528291,97254cacf5fd726b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:51:53.308 INFO 2981023 --- [ocp-async-10,b6cb85f3d29261d7,d077ce39e64ec5f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,45595c7af2e1b3c5,609aec8c8099cdac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,45595c7af2e1b3c5,609aec8c8099cdac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:53.543 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:53.544 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:53.573 INFO 2981023 --- [ocp-schedules-15,ffc1840ddc0d3798,d5012804c6ad04b0] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:51:53.592 WARN 2981023 --- [ocp-schedules-15,ffc1840ddc0d3798,d5012804c6ad04b0] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:51:54.254 INFO 2981023 --- [ocp-schedules-4,4fdc292bfe3a8e3d,b6203a8235713a22] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:51:54.308 INFO 2981023 --- [ocp-schedules-9,d1f5c152d8f90c81,2365a32ebc48e732] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:51:54.536 INFO 2981023 --- [ocp-schedules-27,cd4a1a2c51e8e4ac,a22df178beaf19b1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:51:54.545 INFO 2981023 --- [ocp-schedules-27,cd4a1a2c51e8e4ac,a22df178beaf19b1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:51:54.545 INFO 2981023 --- [ocp-schedules-27,cd4a1a2c51e8e4ac,a22df178beaf19b1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:51:54.887 INFO 2981023 --- [ocp-schedules-12,55ff11be342a051e,eddf5fd789cbb4a7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:51:54.887 INFO 2981023 --- [ocp-schedules-12,55ff11be342a051e,eddf5fd789cbb4a7] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:51:54.892 INFO 2981023 --- [ocp-schedules-12,55ff11be342a051e,eddf5fd789cbb4a7] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:51:54.892 INFO 2981023 --- [ocp-schedules-12,55ff11be342a051e,eddf5fd789cbb4a7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:51:54.892 INFO 2981023 --- [ocp-schedules-12,55ff11be342a051e,eddf5fd789cbb4a7] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:51:54.902 INFO 2981023 --- [ocp-schedules-12,55ff11be342a051e,eddf5fd789cbb4a7] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:51:54.902 INFO 2981023 --- [ocp-schedules-12,55ff11be342a051e,eddf5fd789cbb4a7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:51:54.902 INFO 2981023 --- [ocp-schedules-12,55ff11be342a051e,eddf5fd789cbb4a7] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:51:54.914 INFO 2981023 --- [ocp-schedules-12,55ff11be342a051e,eddf5fd789cbb4a7] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:51:55.104 INFO 2981023 --- [ocp-schedules-15,5a174c6cde6b3548,f23217696df36167] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:51:55.104+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:51:55.108 INFO 2981023 --- [ocp-schedules-15,5a174c6cde6b3548,f23217696df36167] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:51:55.247 INFO 2981023 --- [ocp-schedules-15,5a174c6cde6b3548,f23217696df36167] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 50 to SqlAggregateData 2024-09-11 17:51:55.248 INFO 2981023 --- [ocp-schedules-15,5a174c6cde6b3548,f23217696df36167] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:51:55.308 INFO 2981023 --- [ocp-async-3,76dfb9df1f094d77,c732e47f63d3a65d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:55.765 INFO 2981023 --- [ocp-schedules-10,89599e6b5c70dbb9,c28ac0564ab8624e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:51:55.765 INFO 2981023 --- [ocp-schedules-10,89599e6b5c70dbb9,c28ac0564ab8624e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:51:55.768 INFO 2981023 --- [ocp-schedules-10,89599e6b5c70dbb9,c28ac0564ab8624e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:51:55.768 INFO 2981023 --- [ocp-schedules-10,89599e6b5c70dbb9,c28ac0564ab8624e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:51:55.770 INFO 2981023 --- [ocp-schedules-10,89599e6b5c70dbb9,c28ac0564ab8624e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:51:55.770 INFO 2981023 --- [ocp-schedules-10,89599e6b5c70dbb9,c28ac0564ab8624e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:51:55.781 INFO 2981023 --- [ocp-schedules-6,faab1ab709566097,52630d5f40a8cfb8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:51:55.782 INFO 2981023 --- [ocp-schedules-6,faab1ab709566097,52630d5f40a8cfb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:51:55.823 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:51:55.824 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:51:55.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.825 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:51:55.826 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.827 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:51:55.828 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:51:55.829 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:51:55.833 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.833 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.834 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.834 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.834 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.834 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.840 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.840 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.845 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.845 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.846 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.846 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.847 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.847 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:51:55.860 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.860 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.861 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:51:55.861 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE39A1D755-0-0] 2024-09-11 17:51:55.861 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:51:55.861 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.862 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:51:55.863 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.863 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.863 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:51:55.865 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.875 INFO 2981023 --- [ocp-schedules-27,d387b451ab83c31f,6abb578a520dcbfc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:51:55.875 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.876 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:51:55.876 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.877 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:51:55.878 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.879 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:51:55.879 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:51:55.882 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.883 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:51:55.884 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.885 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:51:55.893 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.894 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:51:55.897 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:51:55.898 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.899 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:51:55.899 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:51:55.900 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE39A1D75F-0-0] 2024-09-11 17:51:55.900 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:51:55.902 INFO 2981023 --- [ocp-schedules-27,d387b451ab83c31f,6abb578a520dcbfc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:51:55.902 INFO 2981023 --- [ocp-schedules-27,d387b451ab83c31f,6abb578a520dcbfc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:51:55.908 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.908 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:51:55.909 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.909 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.909 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:51:55.910 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:51:55.910 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.910 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.910 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:51:55.911 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:51:55.911 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.911 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.911 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:55.912 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:51:55.912 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.912 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:51:55.913 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:51:55.913 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.913 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.913 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.913 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:51:55.914 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.914 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:51:55.914 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.914 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:51:55.914 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:51:55.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.916 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:51:55.916 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:51:55.916 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.917 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:51:55.917 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.918 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:51:55.918 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.919 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:51:55.919 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:51:55.922 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:51:55.927 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:51:55.929 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:55.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.931 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:51:55.931 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:51:55.931 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:51:55.931 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:51:55.935 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:51:55.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:51:55.942 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:51:55.943 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:51:55.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:51:55.945 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:51:55.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.945 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:51:55.945 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.952 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:51:55.953 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:51:55.954 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:55.954 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:51:55.956 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:51:55.958 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:51:55.958 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:51:55.958 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:51:55.960 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:51:55.962 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:51:55.962 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:51:55.962 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:51:55.962 INFO 2981023 --- [ocp-schedules-6,faab1ab709566097,52630d5f40a8cfb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:51:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,65df4171eaee3a6c,2d49fc37106f324a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,65df4171eaee3a6c,2d49fc37106f324a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:51:56.351057+08:00, lastProcessTime=2024-09-11T17:51:46.343469+08:00 2024-09-11 17:51:56.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:51:56.351057+08:00 2024-09-11 17:51:56.544 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:56.544 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:51:56.811 INFO 2981023 --- [ocp-schedules-15,31aaae10b8bd8915,ae82dfa5b6b99cfe] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:51:56.811 INFO 2981023 --- [ocp-schedules-15,31aaae10b8bd8915,ae82dfa5b6b99cfe] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:51:56.824 INFO 2981023 --- [Collect all tenants major compactions3,31aaae10b8bd8915,fc799e9fc5f68702] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:51:56.825 INFO 2981023 --- [Collect all tenants major compactions6,31aaae10b8bd8915,be6994fc337b6510] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:51:56.826 INFO 2981023 --- [Collect all tenants major compactions5,31aaae10b8bd8915,c1b46a4514155bd5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:51:56.826 INFO 2981023 --- [Collect all tenants major compactions1,31aaae10b8bd8915,c0c4a5c7ca2efb1d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:51:56.827 INFO 2981023 --- [Collect all tenants major compactions4,31aaae10b8bd8915,9ac1a0f0fdc5d20a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:51:56.828 INFO 2981023 --- [Collect all tenants major compactions2,31aaae10b8bd8915,b24a270536924f19] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:51:56.842 INFO 2981023 --- [Collect all tenants major compactions4,31aaae10b8bd8915,9ac1a0f0fdc5d20a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:56.843 INFO 2981023 --- [Collect all tenants major compactions4,31aaae10b8bd8915,9ac1a0f0fdc5d20a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:51:56.844 INFO 2981023 --- [Collect all tenants major compactions3,31aaae10b8bd8915,fc799e9fc5f68702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:56.845 INFO 2981023 --- [Collect all tenants major compactions3,31aaae10b8bd8915,fc799e9fc5f68702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:51:56.845 INFO 2981023 --- [Collect all tenants major compactions5,31aaae10b8bd8915,c1b46a4514155bd5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:56.846 INFO 2981023 --- [Collect all tenants major compactions2,31aaae10b8bd8915,b24a270536924f19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:56.846 INFO 2981023 --- [Collect all tenants major compactions5,31aaae10b8bd8915,c1b46a4514155bd5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:51:56.848 INFO 2981023 --- [Collect all tenants major compactions3,31aaae10b8bd8915,fc799e9fc5f68702] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:51:56.848 INFO 2981023 --- [Collect all tenants major compactions4,31aaae10b8bd8915,9ac1a0f0fdc5d20a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:51:56.848 INFO 2981023 --- [Collect all tenants major compactions2,31aaae10b8bd8915,b24a270536924f19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:51:56.848 INFO 2981023 --- [Collect all tenants major compactions6,31aaae10b8bd8915,be6994fc337b6510] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:56.850 INFO 2981023 --- [Collect all tenants major compactions6,31aaae10b8bd8915,be6994fc337b6510] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:51:56.850 INFO 2981023 --- [Collect all tenants major compactions1,31aaae10b8bd8915,c0c4a5c7ca2efb1d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:51:56.850 INFO 2981023 --- [Collect all tenants major compactions5,31aaae10b8bd8915,c1b46a4514155bd5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:51:56.850 INFO 2981023 --- [Collect all tenants major compactions2,31aaae10b8bd8915,b24a270536924f19] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:51:56.851 INFO 2981023 --- [Collect all tenants major compactions3,31aaae10b8bd8915,fc799e9fc5f68702] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:51:56.852 INFO 2981023 --- [Collect all tenants major compactions6,31aaae10b8bd8915,be6994fc337b6510] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:51:56.852 INFO 2981023 --- [Collect all tenants major compactions6,31aaae10b8bd8915,be6994fc337b6510] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:51:56.853 INFO 2981023 --- [Collect all tenants major compactions1,31aaae10b8bd8915,c0c4a5c7ca2efb1d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:51:56.853 INFO 2981023 --- [Collect all tenants major compactions4,31aaae10b8bd8915,9ac1a0f0fdc5d20a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:51:56.854 INFO 2981023 --- [Collect all tenants major compactions1,31aaae10b8bd8915,c0c4a5c7ca2efb1d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:51:56.854 INFO 2981023 --- [Collect all tenants major compactions2,31aaae10b8bd8915,b24a270536924f19] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:51:56.854 INFO 2981023 --- [Collect all tenants major compactions1,31aaae10b8bd8915,c0c4a5c7ca2efb1d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:51:56.856 INFO 2981023 --- [Collect all tenants major compactions5,31aaae10b8bd8915,c1b46a4514155bd5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:51:56.856 INFO 2981023 --- [ocp-schedules-15,31aaae10b8bd8915,ae82dfa5b6b99cfe] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:51:57.308 INFO 2981023 --- [ocp-async-8,c7502df3ab5f7189,dc7fa68622ae7460] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:57.386 INFO 2981023 --- [ocp-schedules-6,0ffb4a6134e82dc6,02383ca84fd8dfb7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:51:57.434 WARN 2981023 --- [scanner-13561247-1,0ffb4a6134e82dc6,70d06bfb254e5d7f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:51:57.764 INFO 2981023 --- [ocp-async-3,6b9f7216a40f126e,6233c7cd25370bc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:51:59.308 INFO 2981023 --- [ocp-async-10,8914181382798716,2da790096bcd724a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:51:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,0cf4365de7cc37c8,e5d36462e0fd7315] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048320, interval=60 2024-09-11 17:51:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,0cf4365de7cc37c8,e5d36462e0fd7315] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:51:59.544 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:51:59.545 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:01.313 INFO 2981023 --- [ocp-async-3,7ebb3fe32ba30c15,ef59cb36aecb08db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:02.240 INFO 2981023 --- [ocp-schedules-13,605eb61caa81c471,7dad1685c9edb4a8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:52:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,c86f15f5688bb1a7,f8c9fa97865de87b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,c86f15f5688bb1a7,f8c9fa97865de87b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:02.535 INFO 2981023 --- [ocp-schedules-8,e96209912d0022d0,0cc62227471d3176] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:52:02.536 INFO 2981023 --- [ocp-schedules-8,e96209912d0022d0,0cc62227471d3176] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:02.545 INFO 2981023 --- [ocp-schedules-8,e96209912d0022d0,0cc62227471d3176] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:52:02.545 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:02.545 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:03.308 INFO 2981023 --- [ocp-async-4,59a72a8429a8c19e,eb6bde39d49d8235] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:05.308 INFO 2981023 --- [ocp-async-2,8581b990b1e27e23,0378e4fb256bee31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,e796542ad9b03583,d435965fbea6a9b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,e796542ad9b03583,d435965fbea6a9b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:05.546 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:05.546 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:52:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:52:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:52:05.982 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:52:05.987 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:52:05.993 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:52:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:52:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:52:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:52:06.001 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T17:49:41, target=2024-09-11T17:47:41, granule=120000ms, delay=120000ms) 2024-09-11 17:52:06.001 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:47:41, target=2024-09-11T17:40:51, granule=600000ms, delay=410000ms) 2024-09-11 17:52:06.004 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T17:49:40, target=2024-09-11T17:47:40, granule=120000ms, delay=120000ms) 2024-09-11 17:52:06.004 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:47:40, target=2024-09-11T17:39:31, granule=600000ms, delay=489000ms) 2024-09-11 17:52:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=34ms, period=120000ms 2024-09-11 17:52:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:52:06.336986+08:00, lastProcessTime=2024-09-11T17:51:56.351057+08:00 2024-09-11 17:52:06.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:52:06.336986+08:00 2024-09-11 17:52:07.309 INFO 2981023 --- [ocp-async-7,a61cfe1c9cdf3b11,b31fc3a0022312a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:07.563 INFO 2981023 --- [ocp-schedules-2,e1f403a622c9bd5b,9cae9291d0b9825d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:52:07.576 INFO 2981023 --- [ocp-schedules-22,4b965dd1f6547c48,15422959f623cfaf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:52:07.618 INFO 2981023 --- [ocp-schedules-12,167e06e3fa5b437a,dccd799e48140a0c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:52:07.618+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:52:07.619 INFO 2981023 --- [ocp-schedules-12,167e06e3fa5b437a,dccd799e48140a0c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:52:07.620 INFO 2981023 --- [ocp-schedules-12,167e06e3fa5b437a,dccd799e48140a0c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:52:07.621 INFO 2981023 --- [ocp-schedules-12,167e06e3fa5b437a,dccd799e48140a0c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:52:07.655 INFO 2981023 --- [alarm-detect-15,4b965dd1f6547c48,1893dc57e0b2191e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644832, data is empty. 2024-09-11 17:52:07.663 INFO 2981023 --- [alarm-detect-15,4b965dd1f6547c48,1893dc57e0b2191e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644882, data is empty. 2024-09-11 17:52:07.667 INFO 2981023 --- [alarm-detect-15,4b965dd1f6547c48,1893dc57e0b2191e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644882, data is empty. 2024-09-11 17:52:07.675 WARN 2981023 --- [scanner-13561488-1,4b965dd1f6547c48,461346aca06d894d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:52:07.692 INFO 2981023 --- [alarm-detect-13,4b965dd1f6547c48,bfd9e135bcb6f93b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644832, data is empty. 2024-09-11 17:52:07.693 INFO 2981023 --- [alarm-detect-9,4b965dd1f6547c48,d190f4508b003f50] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644832, data is empty. 2024-09-11 17:52:07.696 INFO 2981023 --- [alarm-detect-13,4b965dd1f6547c48,bfd9e135bcb6f93b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644882, data is empty. 2024-09-11 17:52:07.698 INFO 2981023 --- [alarm-detect-9,4b965dd1f6547c48,d190f4508b003f50] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644882, data is empty. 2024-09-11 17:52:07.700 INFO 2981023 --- [alarm-detect-13,4b965dd1f6547c48,bfd9e135bcb6f93b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644882, data is empty. 2024-09-11 17:52:07.702 INFO 2981023 --- [alarm-detect-9,4b965dd1f6547c48,d190f4508b003f50] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644882, data is empty. 2024-09-11 17:52:07.742 INFO 2981023 --- [alarm-detect-6,4b965dd1f6547c48,f7fb5bec767328df] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726048290 ~ 1726048317 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726048305, value=1.682)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.682, values={1726048305=1.682}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:52:07.742 INFO 2981023 --- [alarm-detect-6,4b965dd1f6547c48,f7fb5bec767328df] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.10-3, currentState=Pending 2024-09-11 17:52:07.765 INFO 2981023 --- [ocp-async-1,1577cb755a86e1e0,dd17d5567b00cf59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:52:07.814 INFO 2981023 --- [alarm-detect-1,4b965dd1f6547c48,f368be277613572f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644832, data is empty. 2024-09-11 17:52:07.816 INFO 2981023 --- [alarm-detect-1,4b965dd1f6547c48,f368be277613572f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644882, data is empty. 2024-09-11 17:52:07.822 INFO 2981023 --- [alarm-detect-1,4b965dd1f6547c48,f368be277613572f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644882, data is empty. 2024-09-11 17:52:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,7da54b4762950259,5a75905360970811] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:08.315 INFO 2981023 --- [prometheus-minute-schedule-1,7da54b4762950259,5a75905360970811] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:08.546 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:08.547 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:09.309 INFO 2981023 --- [ocp-async-4,50c77fa70dc7bdf9,d5b2401afa4e4c1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:09.734 INFO 2981023 --- [ocp-schedules-19,ce4f42feca187b49,d64a7b27e017295e] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-09-11 17:52:09.735 INFO 2981023 --- [ocp-schedules-19,ce4f42feca187b49,d64a7b27e017295e] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-09-11 17:52:09.735 INFO 2981023 --- [ocp-schedules-19,ce4f42feca187b49,d64a7b27e017295e] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=11634 2024-09-11 17:52:09.912 INFO 2981023 --- [ocp-schedules-19,ce4f42feca187b49,d64a7b27e017295e] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=11634 2024-09-11 17:52:10.002 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:52:10.002 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:52:10.918 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:52:10.920 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:52:11.309 INFO 2981023 --- [ocp-async-1,a85895f4d8218b50,b404f5028b250395] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:11.314 INFO 2981023 --- [prometheus-minute-schedule-1,fa813fc4097d6425,753b6f0b35d83d4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:11.319 INFO 2981023 --- [prometheus-minute-schedule-1,fa813fc4097d6425,753b6f0b35d83d4c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:11.547 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:11.548 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:11.780 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048260, nodeCount=806, affectRows=806 2024-09-11 17:52:11.783 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048260, nodeCount=806, validCount=806 2024-09-11 17:52:11.790 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048260, nodeCount=1000, affectRows=1000 2024-09-11 17:52:11.793 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048260, nodeCount=999, validCount=999 2024-09-11 17:52:12.243 INFO 2981023 --- [ocp-schedules-13,240ff1416a539d6d,bbf9fc9f12ed44da] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:52:12.879 INFO 2981023 --- [ocp-schedules-14,68121fca93c9fd63,c7c7db71f20ff763] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:52:12.889 INFO 2981023 --- [ocp-schedules-14,68121fca93c9fd63,c7c7db71f20ff763] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:12.893 INFO 2981023 --- [ocp-schedules-14,68121fca93c9fd63,c7c7db71f20ff763] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:12.950 INFO 2981023 --- [ocp-schedules-14,68121fca93c9fd63,c7c7db71f20ff763] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:12.951 INFO 2981023 --- [ocp-schedules-14,68121fca93c9fd63,c7c7db71f20ff763] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:12.962 INFO 2981023 --- [ocp-schedules-14,68121fca93c9fd63,c7c7db71f20ff763] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:12.963 INFO 2981023 --- [ocp-schedules-14,68121fca93c9fd63,c7c7db71f20ff763] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:12.974 INFO 2981023 --- [ocp-schedules-14,68121fca93c9fd63,c7c7db71f20ff763] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=3, cachedSize=5 2024-09-11 17:52:13.309 INFO 2981023 --- [ocp-async-9,b5dcc25b2f0323fb,4663c05f0ebcb169] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:13.531 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048260, nodeCount=463, affectRows=463 2024-09-11 17:52:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,248cf9e2422514bf,61d17e31628e45c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,87de120eb8693bd4,7a402c69762603c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:52:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,248cf9e2422514bf,61d17e31628e45c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:14.547 INFO 2981023 --- [ocp-schedules-19,0183724cd9067dc0,d806c74463955f34] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:52:14.548 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:14.548 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:14.552 INFO 2981023 --- [ocp-schedules-19,0183724cd9067dc0,d806c74463955f34] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:52:14.552 INFO 2981023 --- [ocp-schedules-19,0183724cd9067dc0,d806c74463955f34] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:52:15.217 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048260, nodeCount=463, validCount=463 2024-09-11 17:52:15.309 INFO 2981023 --- [ocp-async-8,667ad32e67aa86f4,88221af633d5f09c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:16.024 INFO 2981023 --- [ocp-schedules-11,cdb530ce0d294d5c,6b377f1deb06f938] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:52:16.024 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:52:16.024 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:52:16.025 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:52:16.030 INFO 2981023 --- [ocp-schedules-26,7c8873866ffcd32a,ba6ed533f5957505] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:52:16.031 INFO 2981023 --- [host-check-clockdiff2,7c8873866ffcd32a,b6342f39cd225d96] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:52:16.039 INFO 2981023 --- [host-check-clockdiff3,7c8873866ffcd32a,4a43b13c55251f74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:52:16.040 INFO 2981023 --- [host-check-clockdiff1,7c8873866ffcd32a,bc11dc7808013456] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:52:16.049 INFO 2981023 --- [host-check-clockdiff3,7c8873866ffcd32a,4a43b13c55251f74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048336 0 0 , error= 2024-09-11 17:52:16.050 INFO 2981023 --- [host-check-clockdiff2,7c8873866ffcd32a,b6342f39cd225d96] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048336 0 0 , error= 2024-09-11 17:52:16.053 INFO 2981023 --- [ocp-schedules-23,97221113dd4cba9b,698f8a8531271ed5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:52:16.054 INFO 2981023 --- [host-check-agent-status1,97221113dd4cba9b,43e878370e58f000] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:52:16.054 INFO 2981023 --- [host-check-agent-status2,97221113dd4cba9b,d5c7bc6349fd4f30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:52:16.054 INFO 2981023 --- [host-check-agent-status3,97221113dd4cba9b,cd0ba0309064fb88] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:52:16.060 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:16.061 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:16.061 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:16.063 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:16.066 INFO 2981023 --- [host-check-agent-status2,97221113dd4cba9b,d5c7bc6349fd4f30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:16.067 INFO 2981023 --- [host-check-agent-status2,97221113dd4cba9b,d5c7bc6349fd4f30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:16.067 INFO 2981023 --- [host-check-agent-status1,97221113dd4cba9b,43e878370e58f000] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:16.068 INFO 2981023 --- [host-check-agent-status1,97221113dd4cba9b,43e878370e58f000] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:16.078 INFO 2981023 --- [host-check-agent-status3,97221113dd4cba9b,cd0ba0309064fb88] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:16.079 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:16.079 INFO 2981023 --- [host-check-agent-status3,97221113dd4cba9b,cd0ba0309064fb88] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:16.080 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:16.083 INFO 2981023 --- [host-check-agent-status1,97221113dd4cba9b,43e878370e58f000] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:52:16.092 INFO 2981023 --- [host-check-agent-status1,97221113dd4cba9b,43e878370e58f000] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:52:16.092 INFO 2981023 --- [host-check-agent-status1,97221113dd4cba9b,43e878370e58f000] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:52:16.092 INFO 2981023 --- [host-check-agent-status1,97221113dd4cba9b,43e878370e58f000] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:52:16.102 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:52:16.102 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:52:16.103 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:52:16.103011938+08:00 2024-09-11 17:52:16.103 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:52:16.103+08:00 2024-09-11 17:52:16.105 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:52:16.103+08:00 and status AVAILABLE 2024-09-11 17:52:16.110 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:52:16.109772461+08:00 2024-09-11 17:52:16.111 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:52:16.110+08:00 2024-09-11 17:52:16.114 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:52:16.110+08:00 and status AVAILABLE 2024-09-11 17:52:16.116 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:52:16.116 INFO 2981023 --- [host-check-agent-status3,97221113dd4cba9b,cd0ba0309064fb88] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:52:16.118 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:52:16.118061901+08:00 2024-09-11 17:52:16.120 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:52:16.118+08:00 2024-09-11 17:52:16.120 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:52:16.121 INFO 2981023 --- [host-check-agent-status3,97221113dd4cba9b,cd0ba0309064fb88] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:52:16.121 INFO 2981023 --- [host-check-agent-status3,97221113dd4cba9b,cd0ba0309064fb88] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:52:16.121 INFO 2981023 --- [host-check-agent-status3,97221113dd4cba9b,cd0ba0309064fb88] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:52:16.122 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:52:16.118+08:00 and status AVAILABLE 2024-09-11 17:52:16.132 INFO 2981023 --- [host-check-agent-status2,97221113dd4cba9b,d5c7bc6349fd4f30] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:52:16.135 INFO 2981023 --- [host-check-agent-status2,97221113dd4cba9b,d5c7bc6349fd4f30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:52:16.135 INFO 2981023 --- [host-check-agent-status2,97221113dd4cba9b,d5c7bc6349fd4f30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:52:16.135 INFO 2981023 --- [host-check-agent-status2,97221113dd4cba9b,d5c7bc6349fd4f30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:52:16.137 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:52:16.143 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:16.143 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:16.145 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:16.145 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:52:16.145 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:16.162 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:52:16.171 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:16.175 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:16.202 INFO 2981023 --- [ocp-schedules-8,cf7bc508f94735d0,5d05fb56b8ef9e3f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:52:16.202 INFO 2981023 --- [ocp-schedules-19,2fa6609e64f3a1ba,c9235757c8b87743] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:52:16.203 INFO 2981023 --- [ocp-schedules-19,2fa6609e64f3a1ba,c9235757c8b87743] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:16.204 INFO 2981023 --- [ocp-schedules-8,cf7bc508f94735d0,5d05fb56b8ef9e3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:16.209 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:52:16.210 INFO 2981023 --- [ocp-schedules-16,8ff6a2ff7107e014,1efe208bcaad509c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:52:16.210 INFO 2981023 --- [ocp-schedules-17,b582f2d7b17ba145,b5541c751c8a4b60] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:52:16.221 INFO 2981023 --- [ocp-schedules-16,8ff6a2ff7107e014,1efe208bcaad509c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:16.223 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:52:16.236 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.237 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:16.237 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:52:16.240 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:52:16.252 INFO 2981023 --- [ocp-schedules-15,b9f339ff968e4d74,16b4833034937352] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:52:16.254 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:52:16.256 INFO 2981023 --- [ocp-schedules-8,cf7bc508f94735d0,5d05fb56b8ef9e3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:52:16.257 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:52:16.261 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:52:16.281 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:52:16.286 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.287 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:16.288 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:52:16.289 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:52:16.301 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.301 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:16.303 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:52:16.303 INFO 2981023 --- [ocp-schedules-19,2fa6609e64f3a1ba,c9235757c8b87743] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:52:16.308 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:52:16.309 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.310 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:16.310 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.310 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.310 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:16.310 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:16.324 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:52:16.327 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:52:16.334 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:52:16.343 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:52:16.344 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:52:16.343558+08:00, lastProcessTime=2024-09-11T17:52:06.336986+08:00 2024-09-11 17:52:16.353 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:52:16.361 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.363 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.364 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:52:16.364 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:52:16.366 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:52:16.372 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.373 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.373 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:52:16.376 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:52:16.384 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:52:16.395 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:52:16.397 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.397 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.398 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:16.398 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:52:16.404 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.407 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:52:16.408 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:52:16.410 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:52:16.422 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:16.422 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:52:16.425 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:52:16.426 INFO 2981023 --- [host-check-host-status1,cdb530ce0d294d5c,01416a4a6fa05893] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:52:16.427 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:52:16.433 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.434 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.434 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:16.435 INFO 2981023 --- [host-check-host-status2,cdb530ce0d294d5c,32070de656750124] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:52:16.435 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:52:16.437 INFO 2981023 --- [host-check-host-status3,cdb530ce0d294d5c,8f203a6fba388b9c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:52:16.437 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.440 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:52:16.442 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:52:16.445 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:52:16.455 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.455 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.455 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.456 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:52:16.456 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.457 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:16.457 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:52:16.461 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:52:16.464 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:16.465 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:16.465 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:16.473 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.474 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.474 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:16.475 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:52:16.476 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.476 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.477 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:16.477 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:52:16.478 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.478 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:52:16.480 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:52:16.481 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:52:16.481 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:16.482 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:16.484 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:52:16.487 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:16.487 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:52:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:52:16.503 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:16.509 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.511 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.511 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:16.512 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:52:16.515 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:52:16.518 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:16.519 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:16.534 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:52:16.553 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:52:16.561 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.561 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.562 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:52:16.563 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:52:16.563 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:52:16.570 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.570 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:16.570 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.570 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:52:16.570 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:52:16.571 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:52:16.571 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:52:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:52:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:52:16.578 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:52:16.578 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:52:16.343558+08:00 2024-09-11 17:52:16.578 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:52:16.578 INFO 2981023 --- [ocp-schedules-16,8ff6a2ff7107e014,1efe208bcaad509c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:52:17.310 INFO 2981023 --- [ocp-async-6,5c2384c799146b8c,1a2e144b6a71a3e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,5e9306df396c7167,9b18dbd8f2eb82e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:17.317 INFO 2981023 --- [prometheus-minute-schedule-1,5e9306df396c7167,9b18dbd8f2eb82e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:17.548 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:17.549 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:17.765 INFO 2981023 --- [ocp-async-3,6bc987282c4d76d6,04c098d8ca4915ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:52:17.871 INFO 2981023 --- [ocp-schedules-10,025556bfc9c55fe2,55ba97afc078143f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:52:17.918 WARN 2981023 --- [scanner-13563277-1,025556bfc9c55fe2,f202cfa44749d9c0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:52:18.572 INFO 2981023 --- [host-check-clockdiff1,7c8873866ffcd32a,bc11dc7808013456] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048338 0 0 , error= 2024-09-11 17:52:19.309 INFO 2981023 --- [ocp-async-10,8be2d91d6c0c39c0,775e6e91d4ef04a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,2638576a472dc3d9,02a48ab2a3803c46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,2638576a472dc3d9,02a48ab2a3803c46] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:20.549 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:20.551 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:21.309 INFO 2981023 --- [ocp-async-3,485e40cc6b8f9522,f4a762a1975e36b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:22.246 INFO 2981023 --- [ocp-schedules-13,473e7c4b23ad7094,989a148cd912ffdb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:52:23.309 INFO 2981023 --- [ocp-async-2,d17636d6c2452908,a64195b0b0c123be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,e703d76bde70ad0a,83f3ad4d48226a54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,e703d76bde70ad0a,83f3ad4d48226a54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:23.365 INFO 2981023 --- [ocp-schedules-7,ec31a8745ae5230e,b3f6118f3bb356cb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:52:23.365 INFO 2981023 --- [ocp-schedules-7,ec31a8745ae5230e,b3f6118f3bb356cb] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:52:23.365 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:52:23.365 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:52:23.368 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:23.369 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:23.369 INFO 2981023 --- [ocp-schedules-7,ec31a8745ae5230e,b3f6118f3bb356cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:23.372 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:23.372 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:23.372 INFO 2981023 --- [ocp-schedules-7,ec31a8745ae5230e,b3f6118f3bb356cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:23.396 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:52:23.397 INFO 2981023 --- [ocp-schedules-7,ec31a8745ae5230e,b3f6118f3bb356cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:52:23.397 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:52:23.399 INFO 2981023 --- [ocp-schedules-7,ec31a8745ae5230e,b3f6118f3bb356cb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:52:23.551 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:23.551 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:24.311 INFO 2981023 --- [ocp-schedules-27,4a6fa03b4660b0db,e90e75edc313c521] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:52:25.310 INFO 2981023 --- [ocp-async-9,a0a3ea6bc02bfb47,45efd4324c3586ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,a249dce983824614,07a616528205dec4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,a249dce983824614,07a616528205dec4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:52:26.352396+08:00, lastProcessTime=2024-09-11T17:52:16.343558+08:00 2024-09-11 17:52:26.483 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:52:26.352396+08:00 2024-09-11 17:52:26.551 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:26.552 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:27.310 INFO 2981023 --- [ocp-async-10,2fe873e7d72e438f,1a059c4db98f2d73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:27.765 INFO 2981023 --- [ocp-async-7,a0b0314f9c595f32,8cbb76ae1176a7ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:52:28.062 INFO 2981023 --- [ocp-schedules-14,61cdf0d4c8e4d132,fcb884a93e745966] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:52:28.074 WARN 2981023 --- [scanner-13563469-1,61cdf0d4c8e4d132,64d0a29925c1e108] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:52:28.122 INFO 2981023 --- [alarm-detect-1,61cdf0d4c8e4d132,a80b16befae100d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644882, data is empty. 2024-09-11 17:52:28.129 INFO 2981023 --- [alarm-detect-1,61cdf0d4c8e4d132,a80b16befae100d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644882, data is empty. 2024-09-11 17:52:28.133 INFO 2981023 --- [alarm-detect-1,61cdf0d4c8e4d132,a80b16befae100d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644882, data is empty. 2024-09-11 17:52:28.154 INFO 2981023 --- [alarm-detect-4,61cdf0d4c8e4d132,fa986fddc7277c84] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644882, data is empty. 2024-09-11 17:52:28.158 INFO 2981023 --- [alarm-detect-10,61cdf0d4c8e4d132,a0c100a82945c696] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644882, data is empty. 2024-09-11 17:52:28.163 INFO 2981023 --- [alarm-detect-10,61cdf0d4c8e4d132,a0c100a82945c696] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644882, data is empty. 2024-09-11 17:52:28.165 INFO 2981023 --- [alarm-detect-4,61cdf0d4c8e4d132,fa986fddc7277c84] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644882, data is empty. 2024-09-11 17:52:28.171 INFO 2981023 --- [alarm-detect-10,61cdf0d4c8e4d132,a0c100a82945c696] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644882, data is empty. 2024-09-11 17:52:28.174 INFO 2981023 --- [alarm-detect-4,61cdf0d4c8e4d132,fa986fddc7277c84] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644882, data is empty. 2024-09-11 17:52:28.257 INFO 2981023 --- [alarm-detect-10,61cdf0d4c8e4d132,31f7ea9966b78b73] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644882, data is empty. 2024-09-11 17:52:28.262 INFO 2981023 --- [alarm-detect-10,61cdf0d4c8e4d132,31f7ea9966b78b73] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644882, data is empty. 2024-09-11 17:52:28.266 INFO 2981023 --- [alarm-detect-10,61cdf0d4c8e4d132,31f7ea9966b78b73] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644882, data is empty. 2024-09-11 17:52:29.310 INFO 2981023 --- [ocp-async-6,440d6e694a257018,7818cdaa67ec3c16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,012be076cbbb2b33,bd6a00d4eb146bcd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,012be076cbbb2b33,bd6a00d4eb146bcd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:29.552 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:29.553 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:31.310 INFO 2981023 --- [ocp-async-1,47ab6e153248179a,85804e01f0b62cec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:32.248 INFO 2981023 --- [ocp-schedules-10,43b3dca873bb3dca,78bf1700effc15f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:52:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,9395ec3129599ede,6f8611a96340b042] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,9395ec3129599ede,6f8611a96340b042] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:32.553 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:32.553 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:33.310 INFO 2981023 --- [ocp-async-5,54d7883ad7a48fd4,b1f49285a376c6ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:34.555 INFO 2981023 --- [ocp-schedules-22,8b7895fa4d1ae2fd,f1e485e655e5f21f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:52:34.564 INFO 2981023 --- [ocp-schedules-22,8b7895fa4d1ae2fd,f1e485e655e5f21f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:52:34.564 INFO 2981023 --- [ocp-schedules-22,8b7895fa4d1ae2fd,f1e485e655e5f21f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:52:35.310 INFO 2981023 --- [ocp-async-3,293f2072cacb8769,9ddbe7296b2450c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,22be2826692a2b13,f6f100077e5945ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,22be2826692a2b13,f6f100077e5945ba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:35.554 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:35.554 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:52:36.336964+08:00, lastProcessTime=2024-09-11T17:52:26.352396+08:00 2024-09-11 17:52:36.472 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:52:36.336964+08:00 2024-09-11 17:52:37.310 INFO 2981023 --- [ocp-async-7,f3fe7a52be38f45b,b29120f57a1bd5ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:37.314 INFO 2981023 --- [prometheus-cache-monitor1,f52861e2349b12f9,09ee13f766ae29f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:52:37.314 INFO 2981023 --- [prometheus-cache-monitor1,f52861e2349b12f9,09ee13f766ae29f5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:52:37.765 INFO 2981023 --- [ocp-async-3,458b4447110e8f8c,fb298e6ba646527f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:52:38.305 INFO 2981023 --- [ocp-schedules-16,41747198e87f04cb,0f85b1d0dcd8110f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:52:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,32d8512dd2beb48c,39d2731e8ee8b931] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:38.318 INFO 2981023 --- [prometheus-minute-schedule-1,32d8512dd2beb48c,39d2731e8ee8b931] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:38.346 WARN 2981023 --- [scanner-13563798-1,41747198e87f04cb,e7650083d7eb278e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:52:38.554 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:38.555 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:39.310 INFO 2981023 --- [ocp-async-5,93eb07a0187faf90,372a04bf3edb5611] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:40.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:52:40.003 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:52:41.311 INFO 2981023 --- [ocp-async-4,3e9cf26e86027115,f2f30a95dbc21874] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,db23d5dc68a75c22,e7fe4d428180240a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,db23d5dc68a75c22,e7fe4d428180240a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:41.555 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:41.555 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:42.200 INFO 2981023 --- [ocp-schedules-23,1133858ae1fbf833,071c7758c271831b] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:52:42.219 WARN 2981023 --- [ocp-schedules-23,1133858ae1fbf833,071c7758c271831b] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:52:42.251 INFO 2981023 --- [ocp-schedules-25,9f263c7d46e5ab94,7ba2801e2636a23f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:52:43.311 INFO 2981023 --- [ocp-async-7,e2f8fc3b42cb0c90,440e9f38d932c117] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,dc4dfb1cee557cf2,4e775d1e0ac607f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,dc4dfb1cee557cf2,4e775d1e0ac607f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:44.556 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:44.556 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:45.101 INFO 2981023 --- [prometheus-second-schedule-1,9c35b7bb436f7d56,b7977f96e5d8b0f6] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:52:45.256 INFO 2981023 --- [ocp-schedules-25,9a1e08db639c1757,04a0a25e1e730ed4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-09-11 17:52:45.257 INFO 2981023 --- [ocp-schedules-25,9a1e08db639c1757,04a0a25e1e730ed4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:45.283 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:45.311 INFO 2981023 --- [ocp-async-1,8f7c4d70da65d3b9,53a63148314f6779] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:45.311 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:21:42.083265, 2024-09-11 17:52:45.261, 1000] 2024-09-11 17:52:45.315 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000006, startTime 2024-09-11 17:21:42.083265, endTime 2024-09-11 17:52:45.261 2024-09-11 17:52:45.326 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:45.328 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:45.329 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:45.330 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:45.331 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 02:03:06.281634, 2024-09-11 17:52:45.316, 1000] 2024-09-11 17:52:45.332 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 10:40:18.658505, 2024-09-11 17:52:45.315, 1000] 2024-09-11 17:52:45.332 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-09-11 02:03:06.281634, endTime 2024-09-11 17:52:45.316 2024-09-11 17:52:45.334 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2024-09-11 10:40:18.658505, endTime 2024-09-11 17:52:45.315 2024-09-11 17:52:45.350 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:21:42.083371, 2024-09-11 17:52:45.319, 1000] 2024-09-11 17:52:45.350 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000006, startTime 2024-09-11 17:21:42.083371, endTime 2024-09-11 17:52:45.319 2024-09-11 17:52:45.374 INFO 2981023 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:45.374 INFO 2981023 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:45.374 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:45.375 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:45.404 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 16:48:30.801707, 2024-09-11 17:52:45.367, 1000] 2024-09-11 17:52:45.404 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 08:00:10.321716, 2024-09-11 17:52:45.367, 1000] 2024-09-11 17:52:45.404 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2024-09-11 16:48:30.801707, endTime 2024-09-11 17:52:45.367 2024-09-11 17:52:45.404 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-09-11 08:00:10.321716, endTime 2024-09-11 17:52:45.367 2024-09-11 17:52:45.405 INFO 2981023 --- [ocp-schedules-25,9a1e08db639c1757,04a0a25e1e730ed4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, tryCount=3, successCount=3 2024-09-11 17:52:46.023 INFO 2981023 --- [ocp-schedules-23,84fea6c81533182f,c47992cd3406a103] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:52:46.024 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:52:46.024 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:52:46.024 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:52:46.027 INFO 2981023 --- [ocp-schedules-6,2536cc07fcb064a9,fff5c24ced9235e5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:52:46.027 INFO 2981023 --- [host-check-agent-status1,2536cc07fcb064a9,21891d262f1116d0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:52:46.027 INFO 2981023 --- [host-check-agent-status2,2536cc07fcb064a9,99e5ff930c597233] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:52:46.028 INFO 2981023 --- [host-check-agent-status3,2536cc07fcb064a9,e0f45ad47a697cd1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:52:46.028 INFO 2981023 --- [ocp-schedules-12,2e7c20f98036cc36,81bd4e51ad8beead] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:52:46.029 INFO 2981023 --- [host-check-clockdiff1,2e7c20f98036cc36,df0ebf10ed05c20e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:52:46.029 INFO 2981023 --- [host-check-clockdiff3,2e7c20f98036cc36,13edac5c0983dadf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:52:46.029 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:46.029 INFO 2981023 --- [host-check-clockdiff2,2e7c20f98036cc36,a80c14c189ed10d6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:52:46.031 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:46.032 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:46.034 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:46.034 INFO 2981023 --- [host-check-agent-status2,2536cc07fcb064a9,99e5ff930c597233] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:46.034 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:46.036 INFO 2981023 --- [host-check-agent-status2,2536cc07fcb064a9,99e5ff930c597233] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:46.037 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:46.038 INFO 2981023 --- [host-check-agent-status1,2536cc07fcb064a9,21891d262f1116d0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:46.039 INFO 2981023 --- [host-check-agent-status3,2536cc07fcb064a9,e0f45ad47a697cd1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:46.042 INFO 2981023 --- [host-check-agent-status3,2536cc07fcb064a9,e0f45ad47a697cd1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:46.042 INFO 2981023 --- [host-check-clockdiff3,2e7c20f98036cc36,13edac5c0983dadf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048366 0 0 , error= 2024-09-11 17:52:46.043 INFO 2981023 --- [host-check-agent-status1,2536cc07fcb064a9,21891d262f1116d0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:46.051 INFO 2981023 --- [host-check-clockdiff2,2e7c20f98036cc36,a80c14c189ed10d6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048366 0 0 , error= 2024-09-11 17:52:46.061 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:52:46.062 INFO 2981023 --- [host-check-agent-status2,2536cc07fcb064a9,99e5ff930c597233] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:52:46.063 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:52:46.064025927+08:00 2024-09-11 17:52:46.064 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:52:46.063+08:00 2024-09-11 17:52:46.064 INFO 2981023 --- [host-check-agent-status3,2536cc07fcb064a9,e0f45ad47a697cd1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:52:46.065 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:52:46.063+08:00 and status AVAILABLE 2024-09-11 17:52:46.066 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:52:46.067 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:52:46.067 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:52:46.067646394+08:00 2024-09-11 17:52:46.068 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:52:46.067+08:00 2024-09-11 17:52:46.068 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:52:46.068841877+08:00 2024-09-11 17:52:46.069 INFO 2981023 --- [host-check-agent-status3,2536cc07fcb064a9,e0f45ad47a697cd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:52:46.069 INFO 2981023 --- [host-check-agent-status3,2536cc07fcb064a9,e0f45ad47a697cd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:52:46.069 INFO 2981023 --- [host-check-agent-status3,2536cc07fcb064a9,e0f45ad47a697cd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:52:46.069 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:52:46.068+08:00 2024-09-11 17:52:46.069 INFO 2981023 --- [host-check-agent-status1,2536cc07fcb064a9,21891d262f1116d0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:52:46.069 INFO 2981023 --- [host-check-agent-status2,2536cc07fcb064a9,99e5ff930c597233] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:52:46.069 INFO 2981023 --- [host-check-agent-status2,2536cc07fcb064a9,99e5ff930c597233] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:52:46.069 INFO 2981023 --- [host-check-agent-status2,2536cc07fcb064a9,99e5ff930c597233] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:52:46.070 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:52:46.067+08:00 and status AVAILABLE 2024-09-11 17:52:46.070 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:52:46.068+08:00 and status AVAILABLE 2024-09-11 17:52:46.072 INFO 2981023 --- [host-check-agent-status1,2536cc07fcb064a9,21891d262f1116d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:52:46.072 INFO 2981023 --- [host-check-agent-status1,2536cc07fcb064a9,21891d262f1116d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:52:46.072 INFO 2981023 --- [host-check-agent-status1,2536cc07fcb064a9,21891d262f1116d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:52:46.073 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:52:46.076 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:52:46.078 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:46.079 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:46.082 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:46.084 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:46.088 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:52:46.101 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:52:46.101 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:52:46.106 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:46.107 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:46.120 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:52:46.126 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:52:46.135 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:52:46.141 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:52:46.144 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:52:46.162 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:52:46.173 INFO 2981023 --- [ocp-schedules-21,a18d40d6416fe0ea,655f0be6867c2954] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:52:46.187 INFO 2981023 --- [ocp-schedules-13,6b36e27c5342ec60,94093160166eb9f8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:52:46.188 INFO 2981023 --- [ocp-schedules-15,cd9585eaad74ae98,9caa70c13b9bc312] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:52:46.192 INFO 2981023 --- [ocp-schedules-13,6b36e27c5342ec60,94093160166eb9f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:46.193 INFO 2981023 --- [ocp-schedules-15,cd9585eaad74ae98,9caa70c13b9bc312] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:46.194 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:52:46.196 INFO 2981023 --- [ocp-schedules-2,a9b7a0980e5f5890,c8bd7d48af6a94a8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:52:46.196 INFO 2981023 --- [ocp-schedules-16,7308943b81866950,9e853884ea960ca4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:52:46.197 INFO 2981023 --- [ocp-schedules-21,a18d40d6416fe0ea,655f0be6867c2954] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:52:46.199 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:52:46.208 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:52:46.210 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:52:46.226 INFO 2981023 --- [ocp-schedules-2,a9b7a0980e5f5890,c8bd7d48af6a94a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:46.231 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:52:46.233 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:52:46.240 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:52:46.248 INFO 2981023 --- [ocp-schedules-30,09241fae79e906ef,839a0a0b2b5072ef] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:52:46.252 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.252 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:52:46.253 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:46.253 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.254 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:52:46.254 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:52:46.255 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:52:46.255 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.274 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:52:46.280 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.280 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:52:46.280 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:46.281 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:52:46.281 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:52:46.294 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.294 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:52:46.295 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:46.295 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:52:46.306 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:52:46.318 INFO 2981023 --- [host-check-host-status1,84fea6c81533182f,fdd4280b22db836a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:52:46.323 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:52:46.328 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.329 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.330 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:52:46.330 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:52:46.330 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.334 INFO 2981023 --- [ocp-schedules-15,cd9585eaad74ae98,9caa70c13b9bc312] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:52:46.341 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:52:46.342 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:52:46.330845+08:00, lastProcessTime=2024-09-11T17:52:36.336964+08:00 2024-09-11 17:52:46.342 INFO 2981023 --- [ocp-schedules-13,6b36e27c5342ec60,94093160166eb9f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:52:46.348 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.348 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.348 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:46.350 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:52:46.351 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:52:46.352 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:46.359 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:52:46.362 INFO 2981023 --- [host-check-host-status2,84fea6c81533182f,547538f8fc71bb4b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:52:46.365 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.365 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:46.371 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:52:46.373 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:52:46.392 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.395 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.396 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.396 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:52:46.405 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.406 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.406 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:52:46.407 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:52:46.407 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:52:46.407 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.408 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:52:46.409 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:52:46.409 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.414 INFO 2981023 --- [host-check-host-status3,84fea6c81533182f,e21ad08c5d78d9c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:52:46.415 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:52:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.433 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.433 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.434 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:52:46.434 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:52:46.441 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.441 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:46.442 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:52:46.442 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.445 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:52:46.446 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:52:46.451 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.452 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.453 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:52:46.453 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:52:46.454 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.459 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:46.459 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:52:46.469 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:52:46.474 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:46.476 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.477 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.478 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:46.479 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.481 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:52:46.482 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:52:46.484 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.485 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.485 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:46.486 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:52:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.488 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:52:46.489 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:52:46.491 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:52:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:52:46.493 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:52:46.494 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:52:46.510 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.511 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.511 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:52:46.511 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:52:46.512 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.512 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:46.514 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:52:46.514 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:52:46.514 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.514 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.515 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:52:46.515 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:52:46.515 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:52:46.516 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.522 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:46.522 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:52:46.536 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:52:46.542 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.543 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.546 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.546 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:46.547 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:52:46.551 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:52:46.554 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:52:46.555 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.555 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:52:46.558 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:52:46.567 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:52:46.572 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:52:46.577 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:52:46.577 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:52:46.581 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:52:46.583 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:52:46.596 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:52:46.598 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:52:46.600 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:52:46.604 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:52:46.604 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:52:46.330845+08:00 2024-09-11 17:52:46.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.607 INFO 2981023 --- [ocp-schedules-19,35e91c6b8505f6e7,885ab9e7b225f978] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:52:46.608 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.608 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:52:46.613 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:52:46.613 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:52:46.628 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:52:46.635 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.635 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.636 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:52:46.637 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:52:46.643 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:52:46.643 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:52:46.644 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:46.644 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:46.645 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:52:46.651 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:52:46.651 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:52:46.652 INFO 2981023 --- [ocp-schedules-2,a9b7a0980e5f5890,c8bd7d48af6a94a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:52:46.979 INFO 2981023 --- [host-check-clockdiff1,2e7c20f98036cc36,df0ebf10ed05c20e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048366 0 0 , error= 2024-09-11 17:52:47.311 INFO 2981023 --- [ocp-async-4,e25f846630e547fc,b76e48cdd484d638] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,59dcc380280e8245,06c56d1e9ed582b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,59dcc380280e8245,06c56d1e9ed582b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:47.463 INFO 2981023 --- [ocp-schedules-30,688d19b73dc4a673,53f808738a9f7d28] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 2 2024-09-11 17:52:47.464 INFO 2981023 --- [ocp-schedules-30,688d19b73dc4a673,53f808738a9f7d28] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-09-11 17:52:47.464 INFO 2981023 --- [ocp-schedules-30,688d19b73dc4a673,53f808738a9f7d28] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 2 2024-09-11 17:52:47.556 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:47.557 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:47.765 INFO 2981023 --- [ocp-async-1,455774a841b3dddd,dca0b234400dc1aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:52:48.502 INFO 2981023 --- [ocp-schedules-27,9e72e5df4bf5f36a,fa2d1ec0e3f3aedc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:52:48.570 INFO 2981023 --- [alarm-detect-16,9e72e5df4bf5f36a,d4faf66bc6123781] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644882, data is empty. 2024-09-11 17:52:48.574 INFO 2981023 --- [alarm-detect-16,9e72e5df4bf5f36a,d4faf66bc6123781] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644882, data is empty. 2024-09-11 17:52:48.580 INFO 2981023 --- [alarm-detect-16,9e72e5df4bf5f36a,d4faf66bc6123781] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644882, data is empty. 2024-09-11 17:52:48.592 WARN 2981023 --- [scanner-13564041-1,9e72e5df4bf5f36a,7d741caa21129928] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:52:48.598 INFO 2981023 --- [alarm-detect-15,9e72e5df4bf5f36a,a432c5cbb1955d30] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644882, data is empty. 2024-09-11 17:52:48.603 INFO 2981023 --- [alarm-detect-15,9e72e5df4bf5f36a,a432c5cbb1955d30] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644882, data is empty. 2024-09-11 17:52:48.605 INFO 2981023 --- [alarm-detect-11,9e72e5df4bf5f36a,289ac3db11167c09] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644882, data is empty. 2024-09-11 17:52:48.610 INFO 2981023 --- [alarm-detect-11,9e72e5df4bf5f36a,289ac3db11167c09] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644882, data is empty. 2024-09-11 17:52:48.610 INFO 2981023 --- [alarm-detect-15,9e72e5df4bf5f36a,a432c5cbb1955d30] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644882, data is empty. 2024-09-11 17:52:48.618 INFO 2981023 --- [alarm-detect-11,9e72e5df4bf5f36a,289ac3db11167c09] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644882, data is empty. 2024-09-11 17:52:48.717 INFO 2981023 --- [alarm-detect-19,9e72e5df4bf5f36a,6a34522a2b8b8e16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644882, data is empty. 2024-09-11 17:52:48.724 INFO 2981023 --- [alarm-detect-19,9e72e5df4bf5f36a,6a34522a2b8b8e16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644882, data is empty. 2024-09-11 17:52:48.730 INFO 2981023 --- [alarm-detect-19,9e72e5df4bf5f36a,6a34522a2b8b8e16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644882, data is empty. 2024-09-11 17:52:49.311 INFO 2981023 --- [ocp-async-9,12bd3835645c562e,d1e450926b30ecc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:52:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:52:50.269 INFO 2981023 --- [ocp-schedules-23,1dffb68e3495a965,7f88e98c05dff20c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:52:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,fa059ad9feff77ca,3529f733408a170b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,fa059ad9feff77ca,3529f733408a170b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:50.557 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:50.557 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:51.311 INFO 2981023 --- [ocp-async-8,b169deb17eed6dfb,341daed93e0239d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:51.573 INFO 2981023 --- [ocp-schedules-16,05934f612bdcdcad,19078a501c0ca860] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:52:51.574 INFO 2981023 --- [ocp-schedules-16,05934f612bdcdcad,19078a501c0ca860] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:51.575 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:52:51.575 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:52:51.575 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:52:51.575 INFO 2981023 --- [ocp-schedules-16,05934f612bdcdcad,19078a501c0ca860] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:52:52.253 INFO 2981023 --- [ocp-schedules-3,ed6d86b61563cea3,2cd1ec6f73b2ee31] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:52:53.175 INFO 2981023 --- [ocp-schedules-17,3de6c36a1d6e11d8,bbb11b7b3377399a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:52:53.175 INFO 2981023 --- [ocp-schedules-17,3de6c36a1d6e11d8,bbb11b7b3377399a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:52:53.311 INFO 2981023 --- [ocp-async-4,021e746a134bd34f,8258e71b1c63a377] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,9b936f572ad39bb6,4e65cc4d101596a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,9b936f572ad39bb6,4e65cc4d101596a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:53.557 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:53.558 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:53.619 INFO 2981023 --- [ocp-schedules-9,6eb0bbde1937aee6,58b65aa9e51cc218] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:52:53.636 WARN 2981023 --- [ocp-schedules-9,6eb0bbde1937aee6,58b65aa9e51cc218] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:52:54.313 INFO 2981023 --- [ocp-schedules-16,5009d34b01d7e3bd,7b1f9cf9d5324bbc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:52:54.566 INFO 2981023 --- [ocp-schedules-24,673383a66ca403ef,990a215832af5618] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:52:54.580 INFO 2981023 --- [ocp-schedules-24,673383a66ca403ef,990a215832af5618] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:52:54.580 INFO 2981023 --- [ocp-schedules-24,673383a66ca403ef,990a215832af5618] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:52:54.705 INFO 2981023 --- [ocp-schedules-27,6a60711645a43c04,f0d7ec2fd52155cb] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:52:55.262 INFO 2981023 --- [ocp-schedules-20,03cffde212a91b23,4419f059689f3429] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:52:55.262+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:52:55.266 INFO 2981023 --- [ocp-schedules-20,03cffde212a91b23,4419f059689f3429] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:52:55.311 INFO 2981023 --- [ocp-async-7,c846762b1dfae9ba,75c2e6db0a062d98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:55.398 INFO 2981023 --- [ocp-schedules-20,03cffde212a91b23,4419f059689f3429] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 47 to SqlAggregateData 2024-09-11 17:52:55.400 INFO 2981023 --- [ocp-schedules-20,03cffde212a91b23,4419f059689f3429] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:52:55.772 INFO 2981023 --- [ocp-schedules-16,883f0fd480e36084,11992becdb782f13] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:52:55.772 INFO 2981023 --- [ocp-schedules-16,883f0fd480e36084,11992becdb782f13] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:52:55.773 INFO 2981023 --- [ocp-schedules-16,883f0fd480e36084,11992becdb782f13] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:52:55.773 INFO 2981023 --- [ocp-schedules-16,883f0fd480e36084,11992becdb782f13] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:52:55.773 INFO 2981023 --- [ocp-schedules-16,883f0fd480e36084,11992becdb782f13] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:52:55.773 INFO 2981023 --- [ocp-schedules-16,883f0fd480e36084,11992becdb782f13] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:52:55.902 INFO 2981023 --- [ocp-schedules-14,133850241b6ad278,77c091137feea093] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:52:55.908 INFO 2981023 --- [ocp-schedules-14,133850241b6ad278,77c091137feea093] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:52:55.908 INFO 2981023 --- [ocp-schedules-14,133850241b6ad278,77c091137feea093] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:52:55.962 INFO 2981023 --- [ocp-schedules-8,069c91100c6d07b2,3c073d1cc3bed774] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:52:55.963 INFO 2981023 --- [ocp-schedules-8,069c91100c6d07b2,3c073d1cc3bed774] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:52:55.993 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:52:55.993 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:55.994 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:52:55.994 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:55.995 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:52:55.995 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:55.996 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:52:55.997 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:52:55.999 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:52:56.001 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.001 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.002 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.003 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.006 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.007 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.012 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.012 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:52:56.014 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.014 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.017 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.017 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.019 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.019 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:52:56.024 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.025 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:52:56.025 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.025 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.026 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:52:56.026 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3AD1D6E7-0-0] 2024-09-11 17:52:56.026 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:52:56.026 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.027 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:52:56.030 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.030 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.031 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.032 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:52:56.037 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.038 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:52:56.038 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.039 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:52:56.039 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.040 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:52:56.040 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.045 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:52:56.046 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:52:56.049 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.049 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:52:56.050 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.050 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.050 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:52:56.051 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:52:56.051 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:52:56.051 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:52:56.051 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3AD1D6F1-0-0] 2024-09-11 17:52:56.051 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.051 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:52:56.052 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:52:56.052 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.053 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:52:56.053 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.054 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:52:56.055 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.055 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:52:56.056 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.056 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:52:56.057 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.058 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:52:56.058 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.059 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:52:56.059 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:52:56.059 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.060 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:52:56.060 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.061 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:52:56.061 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.062 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:52:56.062 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:52:56.063 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.063 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:52:56.064 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.065 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:52:56.065 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:56.065 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.066 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:52:56.066 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:52:56.067 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.067 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.068 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:52:56.081 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:56.082 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:52:56.082 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.083 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:52:56.086 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:52:56.086 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:52:56.086 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:52:56.086 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:52:56.087 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.088 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:52:56.089 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:52:56.091 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:52:56.091 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.091 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:52:56.098 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:52:56.099 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:52:56.099 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.100 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.100 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:52:56.101 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:52:56.101 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.102 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:52:56.105 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:52:56.107 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:52:56.107 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:52:56.107 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:52:56.112 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:52:56.113 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:52:56.113 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:52:56.113 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:52:56.113 INFO 2981023 --- [ocp-schedules-8,069c91100c6d07b2,3c073d1cc3bed774] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:52:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,92ec6e5be361d129,6f83f108bd3eefa0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,92ec6e5be361d129,6f83f108bd3eefa0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:56.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:52:56.337009+08:00, lastProcessTime=2024-09-11T17:52:46.330845+08:00 2024-09-11 17:52:56.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:52:56.337009+08:00 2024-09-11 17:52:56.558 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:56.559 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:52:56.856 INFO 2981023 --- [ocp-schedules-26,ec15ba0fea611784,87241af22817758d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:52:56.856 INFO 2981023 --- [ocp-schedules-26,ec15ba0fea611784,87241af22817758d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:52:56.871 INFO 2981023 --- [Collect all tenants major compactions3,ec15ba0fea611784,24c1f033059b4384] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:52:56.871 INFO 2981023 --- [Collect all tenants major compactions2,ec15ba0fea611784,f12c438992929eef] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:52:56.872 INFO 2981023 --- [Collect all tenants major compactions6,ec15ba0fea611784,952c57394df66e98] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:52:56.872 INFO 2981023 --- [Collect all tenants major compactions4,ec15ba0fea611784,2d1a3e35d91183ae] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:52:56.873 INFO 2981023 --- [Collect all tenants major compactions1,ec15ba0fea611784,d8da75c1811985cc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:52:56.874 INFO 2981023 --- [Collect all tenants major compactions5,ec15ba0fea611784,e26bc13beba0feb6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:52:56.884 INFO 2981023 --- [Collect all tenants major compactions3,ec15ba0fea611784,24c1f033059b4384] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.884 INFO 2981023 --- [Collect all tenants major compactions4,ec15ba0fea611784,2d1a3e35d91183ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.885 INFO 2981023 --- [Collect all tenants major compactions3,ec15ba0fea611784,24c1f033059b4384] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:52:56.885 INFO 2981023 --- [Collect all tenants major compactions6,ec15ba0fea611784,952c57394df66e98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.885 INFO 2981023 --- [Collect all tenants major compactions1,ec15ba0fea611784,d8da75c1811985cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.885 INFO 2981023 --- [Collect all tenants major compactions5,ec15ba0fea611784,e26bc13beba0feb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.885 INFO 2981023 --- [Collect all tenants major compactions2,ec15ba0fea611784,f12c438992929eef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:52:56.886 INFO 2981023 --- [Collect all tenants major compactions4,ec15ba0fea611784,2d1a3e35d91183ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:52:56.886 INFO 2981023 --- [Collect all tenants major compactions5,ec15ba0fea611784,e26bc13beba0feb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:52:56.886 INFO 2981023 --- [Collect all tenants major compactions6,ec15ba0fea611784,952c57394df66e98] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:52:56.887 INFO 2981023 --- [Collect all tenants major compactions2,ec15ba0fea611784,f12c438992929eef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:52:56.888 INFO 2981023 --- [Collect all tenants major compactions1,ec15ba0fea611784,d8da75c1811985cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:52:56.889 INFO 2981023 --- [Collect all tenants major compactions4,ec15ba0fea611784,2d1a3e35d91183ae] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:52:56.889 INFO 2981023 --- [Collect all tenants major compactions4,ec15ba0fea611784,2d1a3e35d91183ae] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:52:56.889 INFO 2981023 --- [Collect all tenants major compactions2,ec15ba0fea611784,f12c438992929eef] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:52:56.889 INFO 2981023 --- [Collect all tenants major compactions2,ec15ba0fea611784,f12c438992929eef] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:52:56.890 INFO 2981023 --- [Collect all tenants major compactions3,ec15ba0fea611784,24c1f033059b4384] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:52:56.890 INFO 2981023 --- [Collect all tenants major compactions3,ec15ba0fea611784,24c1f033059b4384] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:52:56.890 INFO 2981023 --- [Collect all tenants major compactions5,ec15ba0fea611784,e26bc13beba0feb6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:52:56.890 INFO 2981023 --- [Collect all tenants major compactions5,ec15ba0fea611784,e26bc13beba0feb6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:52:56.890 INFO 2981023 --- [Collect all tenants major compactions6,ec15ba0fea611784,952c57394df66e98] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:52:56.890 INFO 2981023 --- [Collect all tenants major compactions6,ec15ba0fea611784,952c57394df66e98] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:52:56.891 INFO 2981023 --- [Collect all tenants major compactions1,ec15ba0fea611784,d8da75c1811985cc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:52:56.891 INFO 2981023 --- [Collect all tenants major compactions1,ec15ba0fea611784,d8da75c1811985cc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:52:56.891 INFO 2981023 --- [ocp-schedules-26,ec15ba0fea611784,87241af22817758d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:52:57.312 INFO 2981023 --- [ocp-async-1,1f9d83b69af88850,3844411e046419e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:57.765 INFO 2981023 --- [ocp-async-9,074141a031d3d8a0,fa3a8f973f584206] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:52:58.615 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.620 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, o 2024-09-11 17:52:58.622 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.623 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.623 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.624 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.628 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.631 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.634 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.637 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.637 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.638 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.639 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.640 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.642 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.644 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.647 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.649 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.653 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.656 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.659 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.662 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.663 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.664 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.664 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.665 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:46:58.542, end=2024-09-11T17:51:58.611} 2024-09-11 17:52:58.665 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:52:58.757 INFO 2981023 --- [ocp-schedules-16,3ded724393fde5df,4dc9b93587bbaa67] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:52:58.800 WARN 2981023 --- [scanner-13564302-1,3ded724393fde5df,78b8edeb8cec0a85] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:52:59.312 INFO 2981023 --- [ocp-async-6,358230f07bdb16f6,dc09a607783eae95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:52:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,422156bcaaba6dcc,c01880ffb7b5b865] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048380, interval=60 2024-09-11 17:52:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,422156bcaaba6dcc,c01880ffb7b5b865] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:52:59.559 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:52:59.560 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:01.314 INFO 2981023 --- [ocp-async-7,881a3df5b055b2f4,ff6ebc363513eedb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:02.255 INFO 2981023 --- [ocp-schedules-21,a8483f43b27da6ea,f418007b4178e8cc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:53:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,29b54d2e37526947,3f6a8d73113af898] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,29b54d2e37526947,3f6a8d73113af898] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:02.560 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:02.561 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:03.314 INFO 2981023 --- [ocp-async-1,f9d317746bad1b2c,c1d0ec487497ebab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,d9440770334d8253,8f619caf757376fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:05.314 INFO 2981023 --- [ocp-async-4,d399c54d70b7bb21,f81874e28f581828] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,d9440770334d8253,8f619caf757376fb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:05.561 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:05.562 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:05.955 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:49:05.468, end=2024-09-11T17:51:05.949} 2024-09-11 17:53:05.961 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, o 2024-09-11 17:53:05.961 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:53:05.963 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 38 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:49:05.468, end=2024-09-11T17:51:05.949},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:53:05.989 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:53:05.989 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:53:06.022 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:53:06.040 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986040000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.040Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.042 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:53:06.042 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.047 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.047 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.049 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.049 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.053 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.053 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.054 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:53:06.054 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.04, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.055 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:53:06.056 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:49:05.468, end=2024-09-11T17:51:05.949},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:53:06.056 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:53:06.056 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:53:06.057 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:53:06.062 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:49:05.468, end=2024-09-11T17:51:05.949},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:53:06.063 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:53:06.063 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:53:06.074 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:53:06.074 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:53:06.074 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:49:05.468, end=2024-09-11T17:51:05.949},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:53:06.074 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:53:06.075 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:53:06.076 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:53:06.076 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:49:05.468, end=2024-09-11T17:51:05.949},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:53:06.076 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:53:06.076 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:53:06.077 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:53:06.081 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:49:05.468, end=2024-09-11T17:51:05.949},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:53:06.084 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:53:06.084 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:53:06.127 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986127000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.127Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.128 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986128000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.128Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.133 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986133000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.133Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.138 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986138000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.138Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.141 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986141000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.141Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.142 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986142000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.142Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.144 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986144000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.144Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.144 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986144000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.144Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.158 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:53:06.158 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.160 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:53:06.160 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.163 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 63 2024-09-11 17:53:06.163 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.166 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:53:06.166 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.174 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:53:06.174 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.175 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:53:06.175 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.177 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.177 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.177 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.177 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.177 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.177 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.181 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:53:06.181 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.181 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.181 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.182 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.182 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.183 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:53:06.183 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.183 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.183 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.185 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.185 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.185 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.185 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.191 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.191 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.192 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.192 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.192 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.192 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.194 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.194 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.194 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.194 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.195 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.197 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.197 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.198 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.198 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.199 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.199 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.199 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.199 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.200 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.201 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.201 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.201 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.202 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.202 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.202 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.127, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.203 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.206 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.206 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.207 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.207 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.207 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.207 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.209 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.133, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.210 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.211 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.213 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.128, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.214 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.138, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.215 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.142, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.217 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.141, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.218 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.218 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.219 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.219 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.222 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.222 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.228 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.144, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.228 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.144, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.233 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986233000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.233Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.243 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:53:06.243 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.247 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986247000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.247Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.249 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986249000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.249Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.258 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986258000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84D9CB28179A8A6A515E8623A68C21C2, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.258Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.260 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986260000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.260Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.260 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.260 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.261 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986261000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.261Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.262 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:53:06.262 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.264 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:53:06.264 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.267 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 20 2024-09-11 17:53:06.267 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.270 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.270 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.274 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:53:06.274 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.274 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.276 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.276 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.279 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:53:06.279 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.280 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.280 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.280 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.280 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.280 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.233, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.281 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986281000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.281Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.285 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725961986285000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:53:06.285Z, endTime=2024-09-11T09:51:05.949Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:53:06.287 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.287 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84D9CB28179A8A6A515E8623A68C21C2, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.287 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.287 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.291 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:53:06.291 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.292 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.292 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.294 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.294 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.294 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.294 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.295 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.295 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.296 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:53:06.296 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:53:06.297 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.297 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.298 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.298 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.298 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.299 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.299 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.303 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.249, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.303 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.305 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.305 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.306 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.306 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.307 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.307 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.307 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.307 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84D9CB28179A8A6A515E8623A68C21C2, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.309 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.310 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.311 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.247, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.311 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.311 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.312 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.312 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.315 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.316 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:53:06.316 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:53:06.316 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.316 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.317 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.26, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.317 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.318 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.258, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=84D9CB28179A8A6A515E8623A68C21C2, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.320 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.261, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.321 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.281, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.323 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:53:06.323 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048265949000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:53:06.324 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:53:06.326 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:53:06.285, end=2024-09-11T17:51:05.949}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:53:06.326 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:53:06.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:53:06.351022+08:00, lastProcessTime=2024-09-11T17:52:56.337009+08:00 2024-09-11 17:53:06.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:53:06.351022+08:00 2024-09-11 17:53:07.314 INFO 2981023 --- [ocp-async-10,9788cba37381cc0a,90bfc591bc6a3800] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:07.577 INFO 2981023 --- [ocp-schedules-21,c0938c9df03337b4,a313817636166731] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:53:07.633 INFO 2981023 --- [ocp-schedules-18,66296ffcf204dc1f,6789e97e53d3ae0f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:53:07.633+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:53:07.634 INFO 2981023 --- [ocp-schedules-18,66296ffcf204dc1f,6789e97e53d3ae0f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:53:07.637 INFO 2981023 --- [ocp-schedules-18,66296ffcf204dc1f,6789e97e53d3ae0f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:53:07.637 INFO 2981023 --- [ocp-schedules-18,66296ffcf204dc1f,6789e97e53d3ae0f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:53:07.765 INFO 2981023 --- [ocp-async-2,090457f010220f7e,e6c304ee689d0512] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:53:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,1097b7253e4f8d21,8aacb86c39d59d88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,1097b7253e4f8d21,8aacb86c39d59d88] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:08.562 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:08.562 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:08.947 INFO 2981023 --- [ocp-schedules-30,3111eca267dcfc17,f84775de68e5c55b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:53:08.987 WARN 2981023 --- [scanner-13564568-1,3111eca267dcfc17,d048484f06dcbeca] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:53:09.019 INFO 2981023 --- [alarm-detect-15,3111eca267dcfc17,4650092010b39174] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644882, data is empty. 2024-09-11 17:53:09.022 INFO 2981023 --- [alarm-detect-15,3111eca267dcfc17,4650092010b39174] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644929, data is empty. 2024-09-11 17:53:09.026 INFO 2981023 --- [alarm-detect-15,3111eca267dcfc17,4650092010b39174] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644929, data is empty. 2024-09-11 17:53:09.053 INFO 2981023 --- [alarm-detect-19,3111eca267dcfc17,6a20072bf88525d0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644882, data is empty. 2024-09-11 17:53:09.053 INFO 2981023 --- [alarm-detect-18,3111eca267dcfc17,2a6c13f67baf6557] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644882, data is empty. 2024-09-11 17:53:09.058 INFO 2981023 --- [alarm-detect-18,3111eca267dcfc17,2a6c13f67baf6557] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644929, data is empty. 2024-09-11 17:53:09.059 INFO 2981023 --- [alarm-detect-19,3111eca267dcfc17,6a20072bf88525d0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644929, data is empty. 2024-09-11 17:53:09.063 INFO 2981023 --- [alarm-detect-18,3111eca267dcfc17,2a6c13f67baf6557] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644929, data is empty. 2024-09-11 17:53:09.063 INFO 2981023 --- [alarm-detect-19,3111eca267dcfc17,6a20072bf88525d0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644929, data is empty. 2024-09-11 17:53:09.168 INFO 2981023 --- [alarm-detect-19,3111eca267dcfc17,2f2bff4774767fe8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644882, data is empty. 2024-09-11 17:53:09.176 INFO 2981023 --- [alarm-detect-19,3111eca267dcfc17,2f2bff4774767fe8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644929, data is empty. 2024-09-11 17:53:09.179 INFO 2981023 --- [alarm-detect-19,3111eca267dcfc17,2f2bff4774767fe8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644929, data is empty. 2024-09-11 17:53:09.314 INFO 2981023 --- [ocp-async-8,7278db49cf5dd0dd,4d66aae47f83fcbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:10.003 INFO 2981023 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:53:10.004 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:53:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:53:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:53:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,d2fe2d9209d50e9a,8fd530702cd95678] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:11.314 INFO 2981023 --- [ocp-async-3,4bd630b01ba4ee40,a525314be414014e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,d2fe2d9209d50e9a,8fd530702cd95678] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:11.562 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:11.563 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:11.975 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048320, nodeCount=806, affectRows=806 2024-09-11 17:53:11.978 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048320, nodeCount=806, validCount=806 2024-09-11 17:53:11.985 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048320, nodeCount=1000, affectRows=1000 2024-09-11 17:53:11.988 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048320, nodeCount=999, validCount=999 2024-09-11 17:53:12.258 INFO 2981023 --- [ocp-schedules-4,2621037f8d109337,6d2d5668b25ff5ed] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:53:13.314 INFO 2981023 --- [ocp-async-9,f5eadc26295df476,e94ea2cdd725b1ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,d2dd444103d73c9f,ae12090470affc0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,d4c351654df11eec,75cd6eef18220055] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:53:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,d2dd444103d73c9f,ae12090470affc0b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:14.563 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:14.563 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:14.583 INFO 2981023 --- [ocp-schedules-15,b250dba83fea68d8,5f46897e88dd4ab4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:53:14.592 INFO 2981023 --- [ocp-schedules-15,b250dba83fea68d8,5f46897e88dd4ab4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:53:14.592 INFO 2981023 --- [ocp-schedules-15,b250dba83fea68d8,5f46897e88dd4ab4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:53:15.163 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048320, nodeCount=463, affectRows=463 2024-09-11 17:53:15.314 INFO 2981023 --- [ocp-async-5,ee14aa6595ada489,8e50665e21e5db12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:16.018 INFO 2981023 --- [ocp-schedules-17,fcdface553097846,e4fb56810d447626] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:53:16.024 INFO 2981023 --- [host-check-clockdiff1,fcdface553097846,1110aad47899abac] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:53:16.025 INFO 2981023 --- [ocp-schedules-19,e706a7fc13b37fb3,5d3991923d301e45] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:53:16.025 INFO 2981023 --- [host-check-clockdiff3,fcdface553097846,e7049d5ad45bf32c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:53:16.027 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:53:16.028 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:53:16.028 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:53:16.036 INFO 2981023 --- [host-check-clockdiff2,fcdface553097846,d27c5dde33d873ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:53:16.048 INFO 2981023 --- [host-check-clockdiff3,fcdface553097846,e7049d5ad45bf32c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048396 0 0 , error= 2024-09-11 17:53:16.052 INFO 2981023 --- [ocp-schedules-28,9ca646f219bc6381,5effcb869050781a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:53:16.059 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:53:16.063 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:53:16.063 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:53:16.064 INFO 2981023 --- [host-check-agent-status1,9ca646f219bc6381,344733639f56f4fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:53:16.064 INFO 2981023 --- [host-check-agent-status2,9ca646f219bc6381,177d553b431cd68b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:53:16.064 INFO 2981023 --- [host-check-agent-status3,9ca646f219bc6381,da9ae12816064c0d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:53:16.064 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:53:16.065 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:53:16.065 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:53:16.070 INFO 2981023 --- [host-check-agent-status3,9ca646f219bc6381,da9ae12816064c0d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:53:16.072 INFO 2981023 --- [host-check-agent-status3,9ca646f219bc6381,da9ae12816064c0d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:53:16.178 INFO 2981023 --- [host-check-agent-status2,9ca646f219bc6381,177d553b431cd68b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:53:16.179 INFO 2981023 --- [host-check-agent-status1,9ca646f219bc6381,344733639f56f4fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:53:16.184 INFO 2981023 --- [host-check-agent-status1,9ca646f219bc6381,344733639f56f4fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:53:16.187 INFO 2981023 --- [ocp-schedules-27,3ac4e127b5e1caa6,4de38dcd045a14b5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:53:16.188 INFO 2981023 --- [host-check-agent-status2,9ca646f219bc6381,177d553b431cd68b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:53:16.189 INFO 2981023 --- [ocp-schedules-27,3ac4e127b5e1caa6,4de38dcd045a14b5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:53:16.193 INFO 2981023 --- [ocp-schedules-8,6b8ede6d1e4dae64,f600604c3247d661] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:53:16.203 INFO 2981023 --- [ocp-schedules-29,3f24f8032ec4238b,0d17e2b39845cd39] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:53:16.204 INFO 2981023 --- [ocp-schedules-8,6b8ede6d1e4dae64,f600604c3247d661] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:53:16.205 INFO 2981023 --- [ocp-schedules-25,8fb6d771202d9a21,3c5ef237e2878d34] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:53:16.206 INFO 2981023 --- [ocp-schedules-29,3f24f8032ec4238b,0d17e2b39845cd39] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:53:16.211 INFO 2981023 --- [ocp-schedules-8,6b8ede6d1e4dae64,f600604c3247d661] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:53:16.272 INFO 2981023 --- [host-check-agent-status1,9ca646f219bc6381,344733639f56f4fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:53:16.277 INFO 2981023 --- [host-check-agent-status1,9ca646f219bc6381,344733639f56f4fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:53:16.277 INFO 2981023 --- [host-check-agent-status1,9ca646f219bc6381,344733639f56f4fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:53:16.277 INFO 2981023 --- [host-check-agent-status1,9ca646f219bc6381,344733639f56f4fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:53:16.277 INFO 2981023 --- [ocp-schedules-18,0f8e8bc08335be1a,75514df5730b0534] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:53:16.281 INFO 2981023 --- [host-check-agent-status3,9ca646f219bc6381,da9ae12816064c0d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:53:16.285 INFO 2981023 --- [host-check-agent-status3,9ca646f219bc6381,da9ae12816064c0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:53:16.285 INFO 2981023 --- [host-check-agent-status3,9ca646f219bc6381,da9ae12816064c0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:53:16.285 INFO 2981023 --- [host-check-agent-status3,9ca646f219bc6381,da9ae12816064c0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:53:16.294 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:53:16.294 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:53:16.294 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:53:16.294372433+08:00 2024-09-11 17:53:16.294 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.295 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:16.295 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:53:16.294+08:00 2024-09-11 17:53:16.295 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:53:16.296909959+08:00 2024-09-11 17:53:16.296 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:53:16.295+08:00 2024-09-11 17:53:16.297 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:53:16.294+08:00 and status AVAILABLE 2024-09-11 17:53:16.297 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:53:16.301 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:53:16.295+08:00 and status AVAILABLE 2024-09-11 17:53:16.304 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.305 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:16.305 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:53:16.305 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:53:16.309 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.310 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:16.311 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.311 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:16.317 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.317 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:16.325 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:53:16.333 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:53:16.333 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:53:16.334 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:53:16.336 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:53:16.354 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:53:16.355 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:53:16.356398994+08:00 2024-09-11 17:53:16.359 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.359 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:16.360 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:53:16.360 INFO 2981023 --- [ocp-schedules-27,3ac4e127b5e1caa6,4de38dcd045a14b5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:53:16.364 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:53:16.355+08:00 2024-09-11 17:53:16.364 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:53:16.354814+08:00, lastProcessTime=2024-09-11T17:53:06.351022+08:00 2024-09-11 17:53:16.364 INFO 2981023 --- [host-check-agent-status2,9ca646f219bc6381,177d553b431cd68b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:53:16.366 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:53:16.355+08:00 and status AVAILABLE 2024-09-11 17:53:16.367 INFO 2981023 --- [host-check-agent-status2,9ca646f219bc6381,177d553b431cd68b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:53:16.367 INFO 2981023 --- [host-check-agent-status2,9ca646f219bc6381,177d553b431cd68b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:53:16.367 INFO 2981023 --- [host-check-agent-status2,9ca646f219bc6381,177d553b431cd68b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:53:16.368 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:53:16.370 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:53:16.390 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:53:16.395 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.396 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.397 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:53:16.398 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:53:16.401 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:53:16.403 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:53:16.404 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:53:16.417 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:53:16.417 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:53:16.431 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:53:16.432 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.432 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.433 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:53:16.442 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.443 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.443 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:16.444 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:53:16.445 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.448 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:53:16.448 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:53:16.450 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:53:16.454 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:53:16.460 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:16.460 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:53:16.463 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:53:16.469 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:53:16.475 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.475 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.477 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:16.478 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:53:16.478 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.480 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:53:16.481 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:53:16.482 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:53:16.492 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:53:16.498 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:16.499 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:53:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:16.502 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:53:16.504 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:53:16.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.512 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.513 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.514 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:16.514 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:16.514 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:53:16.514 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:53:16.516 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:53:16.517 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:53:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:53:16.524 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:53:16.527 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:53:16.529 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.530 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.531 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:16.531 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:53:16.534 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:53:16.535 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:53:16.542 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:53:16.544 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:53:16.545 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:53:16.546 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:53:16.549 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:53:16.558 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:16.560 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:53:16.560 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:53:16.561 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.562 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:53:16.564 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:53:16.564 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:53:16.565 INFO 2981023 --- [host-check-host-status3,e706a7fc13b37fb3,2b545215cc89da6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:53:16.573 INFO 2981023 --- [host-check-host-status2,e706a7fc13b37fb3,07a7d5c1219ea838] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:53:16.576 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:16.576 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:53:16.583 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:53:16.603 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:16.609 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:53:16.612 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.615 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:53:16.615 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:16.616 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:53:16.618 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:53:16.619 INFO 2981023 --- [host-check-host-status1,e706a7fc13b37fb3,910c6bb4b1e6c617] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:53:16.620 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:53:16.625 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:53:16.626 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.626 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:53:16.626 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.627 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.627 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.627 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:53:16.628 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:53:16.638 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:53:16.638 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:53:16.640 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:53:16.640 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:53:16.647 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:53:16.648 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:53:16.354814+08:00 2024-09-11 17:53:16.679 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:53:16.685 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:16.685 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:53:16.691 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:53:16.691 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:53:16.691 INFO 2981023 --- [ocp-schedules-29,3f24f8032ec4238b,0d17e2b39845cd39] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:53:16.982 INFO 2981023 --- [host-check-clockdiff2,fcdface553097846,d27c5dde33d873ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048396 1 1 , error= 2024-09-11 17:53:17.189 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048320, nodeCount=463, validCount=463 2024-09-11 17:53:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,32d1e99fb82d300f,84b5ef3f5c094abf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:17.315 INFO 2981023 --- [ocp-async-2,f1523a54a738efb5,8b3df2f52548ff49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,32d1e99fb82d300f,84b5ef3f5c094abf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:17.563 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:17.564 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:17.765 INFO 2981023 --- [ocp-async-5,c62e3d5793f7005b,2bf5723926f194c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:53:19.245 INFO 2981023 --- [ocp-schedules-9,305e01de4fe31a68,b8d51d611f1719a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:53:19.289 WARN 2981023 --- [scanner-13566372-1,305e01de4fe31a68,6e3fee917973a8fd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:53:19.315 INFO 2981023 --- [ocp-async-6,52466692b01329e7,c47036d7a0696862] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,ad13f5e5f9607af9,8d092634fdc227a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,ad13f5e5f9607af9,8d092634fdc227a6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:20.564 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:20.565 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:20.841 INFO 2981023 --- [host-check-clockdiff1,fcdface553097846,1110aad47899abac] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048400 1 1 , error= 2024-09-11 17:53:21.315 INFO 2981023 --- [ocp-async-7,954d3e99710f17ec,bcbcde58a9a8111a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:22.260 INFO 2981023 --- [ocp-schedules-25,7066fd48ad5c6a6b,97c4a8d07ab9caf3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:53:22.641 INFO 2981023 --- [ocp-schedules-15,23021c8e9eead0ea,f81edc341e5c0cc2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 17:53:22.641 INFO 2981023 --- [ocp-schedules-15,23021c8e9eead0ea,f81edc341e5c0cc2] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 17:53:22.655 INFO 2981023 --- [pool-553-thread-1,23021c8e9eead0ea,ecfa7d7433309507] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:53:22.656 INFO 2981023 --- [pool-553-thread-3,23021c8e9eead0ea,a644934c9d1eb778] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:53:22.656 INFO 2981023 --- [pool-553-thread-2,23021c8e9eead0ea,8dbad9f0a88a5e92] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:53:22.672 INFO 2981023 --- [pool-553-thread-3,23021c8e9eead0ea,a644934c9d1eb778] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:22.672 INFO 2981023 --- [pool-553-thread-2,23021c8e9eead0ea,8dbad9f0a88a5e92] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:22.672 INFO 2981023 --- [pool-553-thread-1,23021c8e9eead0ea,ecfa7d7433309507] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:22.674 INFO 2981023 --- [pool-553-thread-3,23021c8e9eead0ea,a644934c9d1eb778] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726048102000000, 1726048402000000] 2024-09-11 17:53:22.675 INFO 2981023 --- [pool-553-thread-1,23021c8e9eead0ea,ecfa7d7433309507] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726048102000000, 1726048402000000] 2024-09-11 17:53:22.675 INFO 2981023 --- [pool-553-thread-2,23021c8e9eead0ea,8dbad9f0a88a5e92] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726048102000000, 1726048402000000] 2024-09-11 17:53:22.678 INFO 2981023 --- [pool-553-thread-3,23021c8e9eead0ea,a644934c9d1eb778] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:53:22.749 INFO 2981023 --- [pool-553-thread-1,23021c8e9eead0ea,ecfa7d7433309507] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:53:22.826 INFO 2981023 --- [pool-553-thread-2,23021c8e9eead0ea,8dbad9f0a88a5e92] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:53:22.826 INFO 2981023 --- [ocp-schedules-15,23021c8e9eead0ea,f81edc341e5c0cc2] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 17:53:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,5b2b0ed880c3a823,51488eaf06c49889] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:23.315 INFO 2981023 --- [ocp-async-2,e242ffb159695239,31cf550858d350d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,5b2b0ed880c3a823,51488eaf06c49889] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:23.403 INFO 2981023 --- [ocp-schedules-8,e66af8522d2470a5,7c33c120eaa822bb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:53:23.403 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:53:23.403 INFO 2981023 --- [ocp-schedules-8,e66af8522d2470a5,7c33c120eaa822bb] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:53:23.403 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:53:23.407 INFO 2981023 --- [ocp-schedules-8,e66af8522d2470a5,7c33c120eaa822bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:53:23.407 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:53:23.408 INFO 2981023 --- [ocp-schedules-8,e66af8522d2470a5,7c33c120eaa822bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:53:23.408 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:53:23.408 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:53:23.409 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:53:23.424 INFO 2981023 --- [ocp-schedules-8,e66af8522d2470a5,7c33c120eaa822bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:53:23.425 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:53:23.426 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:53:23.432 INFO 2981023 --- [ocp-schedules-8,e66af8522d2470a5,7c33c120eaa822bb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:53:23.565 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:23.565 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:24.316 INFO 2981023 --- [ocp-schedules-4,182935be62bb98b3,c2ed537124d5e712] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:53:25.315 INFO 2981023 --- [ocp-async-10,7071e77d0e693e2a,1fe6253da3174496] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,3a5644c70b0273c4,c03bab90ea2bff71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,3a5644c70b0273c4,c03bab90ea2bff71] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:53:26.336937+08:00, lastProcessTime=2024-09-11T17:53:16.354814+08:00 2024-09-11 17:53:26.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:53:26.336937+08:00 2024-09-11 17:53:26.565 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:26.565 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:27.315 INFO 2981023 --- [ocp-async-5,8caacdff5c20d8fa,4e5add0f583a92e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:27.480 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.483 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obC 2024-09-11 17:53:27.484 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.485 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.486 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.486 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.487 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.492 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.492 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:53:27.494 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.498 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 17 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.528 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.533 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.533 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:53:27.536 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.539 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 4 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.546 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.552 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.557 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 31 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.600 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 9 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.612 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 161 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.635 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.638 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.639 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.640 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.640 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.641 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.642 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.642 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:53:27.643 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.643 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.644 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:47:27.291, end=2024-09-11T17:52:27.477} 2024-09-11 17:53:27.644 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 8 dubious sqls 2024-09-11 17:53:27.765 INFO 2981023 --- [ocp-async-3,ed70d84214dd8546,b7827230f82da698] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:53:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,b51288c0f8ad121b,13dc2d4e1f0f1c37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:29.315 INFO 2981023 --- [ocp-async-8,a1edd9620b72cc72,7ebcd38779220fd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,b51288c0f8ad121b,13dc2d4e1f0f1c37] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:29.431 INFO 2981023 --- [ocp-schedules-27,61a9504a107d090a,3bd8e9705f35723e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:53:29.447 WARN 2981023 --- [scanner-13566535-1,61a9504a107d090a,bf0a948ddbd65fcc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:53:29.496 INFO 2981023 --- [alarm-detect-20,61a9504a107d090a,7467c92884c6b0f2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644929, data is empty. 2024-09-11 17:53:29.500 INFO 2981023 --- [alarm-detect-20,61a9504a107d090a,7467c92884c6b0f2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644929, data is empty. 2024-09-11 17:53:29.505 INFO 2981023 --- [alarm-detect-20,61a9504a107d090a,7467c92884c6b0f2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644929, data is empty. 2024-09-11 17:53:29.522 INFO 2981023 --- [alarm-detect-11,61a9504a107d090a,fe80ada4d6d81277] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644929, data is empty. 2024-09-11 17:53:29.522 INFO 2981023 --- [alarm-detect-16,61a9504a107d090a,6d2c1e1abb81f599] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644929, data is empty. 2024-09-11 17:53:29.527 INFO 2981023 --- [alarm-detect-11,61a9504a107d090a,fe80ada4d6d81277] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644929, data is empty. 2024-09-11 17:53:29.527 INFO 2981023 --- [alarm-detect-16,61a9504a107d090a,6d2c1e1abb81f599] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644929, data is empty. 2024-09-11 17:53:29.535 INFO 2981023 --- [alarm-detect-16,61a9504a107d090a,6d2c1e1abb81f599] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644929, data is empty. 2024-09-11 17:53:29.536 INFO 2981023 --- [alarm-detect-11,61a9504a107d090a,fe80ada4d6d81277] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644929, data is empty. 2024-09-11 17:53:29.566 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:29.567 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:29.629 INFO 2981023 --- [alarm-detect-14,61a9504a107d090a,b32fe8bcccd43f7e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644929, data is empty. 2024-09-11 17:53:29.634 INFO 2981023 --- [alarm-detect-14,61a9504a107d090a,b32fe8bcccd43f7e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644929, data is empty. 2024-09-11 17:53:29.643 INFO 2981023 --- [alarm-detect-14,61a9504a107d090a,b32fe8bcccd43f7e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644929, data is empty. 2024-09-11 17:53:31.315 INFO 2981023 --- [ocp-async-10,0792d986dc1cdebb,e6f5e30a3e69160e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:32.263 INFO 2981023 --- [ocp-schedules-3,6d20e9e5de3bb241,c06597a919eca8e0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:53:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,8c96f9e67b500fdc,a1e605fb3ba859ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,8c96f9e67b500fdc,a1e605fb3ba859ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:32.567 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:32.568 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:33.315 INFO 2981023 --- [ocp-async-5,d6f556922821172e,44020533a99b4124] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:34.593 INFO 2981023 --- [ocp-schedules-28,7af5aaf4c4688aaf,d61fd68cc5962d65] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:53:34.599 INFO 2981023 --- [ocp-schedules-28,7af5aaf4c4688aaf,d61fd68cc5962d65] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:53:34.599 INFO 2981023 --- [ocp-schedules-28,7af5aaf4c4688aaf,d61fd68cc5962d65] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:53:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,dd979cf292d54b0d,1c0684d23e852b61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:35.316 INFO 2981023 --- [ocp-async-2,4d76bdb362ce539b,41ef47011ecb0433] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,dd979cf292d54b0d,1c0684d23e852b61] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:35.568 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:35.568 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:53:36.337177+08:00, lastProcessTime=2024-09-11T17:53:26.336937+08:00 2024-09-11 17:53:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:53:36.337177+08:00 2024-09-11 17:53:37.314 INFO 2981023 --- [prometheus-cache-monitor1,1c657ac443dc619e,1b0d2cd63ca6bf31] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1806,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:53:37.315 INFO 2981023 --- [prometheus-cache-monitor1,1c657ac443dc619e,1b0d2cd63ca6bf31] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:53:37.316 INFO 2981023 --- [ocp-async-4,b9e243c595122b2b,1be0c8ddda26f071] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:37.766 INFO 2981023 --- [ocp-async-1,455757faa154e461,8a4e495e9bca7c5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:53:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,dd82d9cee815c98e,db720eda89e6279f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,dd82d9cee815c98e,db720eda89e6279f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:38.568 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:38.569 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:39.316 INFO 2981023 --- [ocp-async-9,21811856ede13372,ad435310d5334605] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:39.683 INFO 2981023 --- [ocp-schedules-7,1002bc95df05d07c,3e2e336595c13182] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:53:39.725 WARN 2981023 --- [scanner-13566898-1,1002bc95df05d07c,0ad577ad198579c6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:53:40.002 INFO 2981023 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:53:40.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:53:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,6ab15a9bf5eaffc0,7395768c60166a54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,6ab15a9bf5eaffc0,7395768c60166a54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:41.316 INFO 2981023 --- [ocp-async-2,97a313d649b3edcb,deb14efb1205bcd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:41.569 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:41.570 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:42.265 INFO 2981023 --- [ocp-schedules-17,31c9ce428575ef4f,49bfada15b382f47] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:53:42.352 INFO 2981023 --- [ocp-schedules-30,b7d774822da8f1de,d248171da0da7913] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:53:42.370 WARN 2981023 --- [ocp-schedules-30,b7d774822da8f1de,d248171da0da7913] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:53:43.316 INFO 2981023 --- [ocp-async-3,a0d0a8d222e94690,1cb595e641e38187] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,ba4369c678e6da54,2b6b2dd4def643d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,ba4369c678e6da54,2b6b2dd4def643d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:44.570 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:44.570 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:45.101 INFO 2981023 --- [prometheus-second-schedule-1,8db1c7fc57eddcf9,cb0e6b3e73885bd3] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:53:45.316 INFO 2981023 --- [ocp-async-5,b748a131a94536d9,7370f342e0ca4fb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:45.997 INFO 2981023 --- [ocp-schedules-7,dd301952d32f98e1,b11fdbd4cf32884b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:53:45.997 INFO 2981023 --- [host-check-clockdiff1,dd301952d32f98e1,7660ab7e23a41227] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:53:45.997 INFO 2981023 --- [host-check-clockdiff2,dd301952d32f98e1,43fdbb7b3908221b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:53:45.997 INFO 2981023 --- [host-check-clockdiff3,dd301952d32f98e1,8e867c112c261a8e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:53:46.007 INFO 2981023 --- [host-check-clockdiff3,dd301952d32f98e1,8e867c112c261a8e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048426 0 0 , error= 2024-09-11 17:53:46.011 INFO 2981023 --- [ocp-schedules-20,0fa77a5cba7bb87e,f1df54964f16ec4a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:53:46.011 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:53:46.012 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:53:46.012 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:53:46.013 INFO 2981023 --- [ocp-schedules-16,67b63dd5d230db6c,3d2edfc4e75fe0ff] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:53:46.013 INFO 2981023 --- [host-check-agent-status1,67b63dd5d230db6c,705c2a2a8c0f3d92] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:53:46.014 INFO 2981023 --- [host-check-agent-status2,67b63dd5d230db6c,1a469390cc058aca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:53:46.015 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:53:46.015 INFO 2981023 --- [host-check-agent-status3,67b63dd5d230db6c,ba53a16bdb98b2d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:53:46.016 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:53:46.018 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:53:46.019 INFO 2981023 --- [host-check-agent-status1,67b63dd5d230db6c,705c2a2a8c0f3d92] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:53:46.019 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:53:46.020 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:53:46.020 INFO 2981023 --- [host-check-agent-status1,67b63dd5d230db6c,705c2a2a8c0f3d92] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:53:46.021 INFO 2981023 --- [host-check-agent-status2,67b63dd5d230db6c,1a469390cc058aca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:53:46.021 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:53:46.022 INFO 2981023 --- [host-check-agent-status3,67b63dd5d230db6c,ba53a16bdb98b2d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:53:46.022 INFO 2981023 --- [host-check-agent-status2,67b63dd5d230db6c,1a469390cc058aca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:53:46.024 INFO 2981023 --- [host-check-agent-status3,67b63dd5d230db6c,ba53a16bdb98b2d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:53:46.039 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:53:46.041 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:53:46.044182773+08:00 2024-09-11 17:53:46.041 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:53:46.041+08:00 2024-09-11 17:53:46.043 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:53:46.043 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:53:46.041+08:00 and status AVAILABLE 2024-09-11 17:53:46.043 INFO 2981023 --- [host-check-agent-status1,67b63dd5d230db6c,705c2a2a8c0f3d92] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:53:46.045 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:53:46.048337293+08:00 2024-09-11 17:53:46.045 INFO 2981023 --- [host-check-agent-status3,67b63dd5d230db6c,ba53a16bdb98b2d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:53:46.045 INFO 2981023 --- [host-check-agent-status2,67b63dd5d230db6c,1a469390cc058aca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:53:46.046 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:53:46.045+08:00 2024-09-11 17:53:46.046 INFO 2981023 --- [host-check-agent-status1,67b63dd5d230db6c,705c2a2a8c0f3d92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:53:46.046 INFO 2981023 --- [host-check-agent-status1,67b63dd5d230db6c,705c2a2a8c0f3d92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:53:46.046 INFO 2981023 --- [host-check-agent-status1,67b63dd5d230db6c,705c2a2a8c0f3d92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:53:46.046 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:53:46.047 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:53:46.047215622+08:00 2024-09-11 17:53:46.047 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:53:46.045+08:00 and status AVAILABLE 2024-09-11 17:53:46.047 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:53:46.048 INFO 2981023 --- [host-check-agent-status2,67b63dd5d230db6c,1a469390cc058aca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:53:46.048 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:53:46.047+08:00 2024-09-11 17:53:46.048 INFO 2981023 --- [host-check-agent-status2,67b63dd5d230db6c,1a469390cc058aca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:53:46.048 INFO 2981023 --- [host-check-agent-status2,67b63dd5d230db6c,1a469390cc058aca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:53:46.049 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:53:46.047+08:00 and status AVAILABLE 2024-09-11 17:53:46.050 INFO 2981023 --- [host-check-agent-status3,67b63dd5d230db6c,ba53a16bdb98b2d5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:53:46.050 INFO 2981023 --- [host-check-agent-status3,67b63dd5d230db6c,ba53a16bdb98b2d5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:53:46.050 INFO 2981023 --- [host-check-agent-status3,67b63dd5d230db6c,ba53a16bdb98b2d5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:53:46.051 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:53:46.054 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:53:46.054 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:53:46.054 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:53:46.057 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:53:46.058 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:53:46.060 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:53:46.061 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:53:46.069 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:53:46.072 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:53:46.074 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:53:46.097 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:53:46.097 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:53:46.104 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:53:46.116 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:53:46.124 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:53:46.124 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:53:46.138 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:53:46.147 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:53:46.158 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:53:46.171 INFO 2981023 --- [ocp-schedules-27,3c319f630e71fdd4,8b766b81951756b8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:53:46.179 INFO 2981023 --- [ocp-schedules-27,3c319f630e71fdd4,8b766b81951756b8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:53:46.183 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:53:46.187 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:53:46.187 INFO 2981023 --- [ocp-schedules-5,88a143b69bd09f5d,92ba8363f888b9b1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:53:46.187 INFO 2981023 --- [ocp-schedules-15,9164b166bf5f02c3,a494c51bd14e9557] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:53:46.188 INFO 2981023 --- [ocp-schedules-4,9f4918335b2ba919,8fcc1c19ba381a2e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:53:46.189 INFO 2981023 --- [ocp-schedules-28,7bb1f1fc1e54c05f,f3fc8fd82fc5480b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:53:46.189 INFO 2981023 --- [ocp-schedules-15,9164b166bf5f02c3,a494c51bd14e9557] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:53:46.189 INFO 2981023 --- [ocp-schedules-5,88a143b69bd09f5d,92ba8363f888b9b1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:53:46.190 INFO 2981023 --- [ocp-schedules-4,9f4918335b2ba919,8fcc1c19ba381a2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:53:46.195 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:53:46.204 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:53:46.216 INFO 2981023 --- [ocp-schedules-10,864a5f75967e434c,77e1fc97bb27f322] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:53:46.220 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:53:46.220 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:53:46.234 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:53:46.235 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:53:46.244 INFO 2981023 --- [ocp-schedules-15,9164b166bf5f02c3,a494c51bd14e9557] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:53:46.253 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:53:46.254 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.261 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.261 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:53:46.262 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:53:46.262 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:46.263 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:53:46.265 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:53:46.265 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.273 INFO 2981023 --- [host-check-host-status2,0fa77a5cba7bb87e,78c63a4fa1ddc598] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:53:46.283 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:53:46.283 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.284 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:53:46.284 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:46.284 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:53:46.286 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.286 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:46.291 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.291 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:46.294 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:53:46.295 INFO 2981023 --- [ocp-schedules-5,88a143b69bd09f5d,92ba8363f888b9b1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:53:46.296 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.297 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:46.303 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:53:46.309 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:53:46.341 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.341 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.342 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:53:46.342 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:53:46.343 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.347 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:53:46.346916+08:00, lastProcessTime=2024-09-11T17:53:36.337177+08:00 2024-09-11 17:53:46.347 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.350 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.351 INFO 2981023 --- [host-check-host-status1,0fa77a5cba7bb87e,db83fa8654b78f14] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:53:46.351 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:53:46.359 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.360 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:46.362 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:53:46.369 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:53:46.377 INFO 2981023 --- [host-check-host-status3,0fa77a5cba7bb87e,507fed7818ca4319] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:53:46.381 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.381 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.382 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:53:46.382 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:53:46.392 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.393 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.394 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:46.394 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:53:46.395 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.397 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:53:46.399 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:53:46.404 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.404 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.405 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.405 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:53:46.407 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:46.407 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:53:46.408 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.409 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.409 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:46.410 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:53:46.410 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.411 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.412 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:53:46.412 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.413 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:53:46.413 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:53:46.414 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:53:46.414 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.415 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:53:46.415 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:53:46.415 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.420 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:46.420 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:53:46.422 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:46.422 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:53:46.429 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.429 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.429 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.430 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:53:46.430 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.430 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:46.431 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:53:46.432 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:46.433 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:53:46.434 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:53:46.437 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:53:46.441 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.441 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:53:46.441 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.442 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:53:46.442 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:46.442 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:53:46.442 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:53:46.444 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:46.444 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:53:46.446 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:53:46.452 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:53:46.454 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:53:46.459 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:53:46.459 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.460 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.460 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:53:46.461 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:53:46.461 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.464 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:53:46.469 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.470 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:53:46.472 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:53:46.476 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:53:46.476 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:53:46.479 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:53:46.486 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:53:46.488 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:53:46.497 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:53:46.502 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:53:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.520 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.521 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:53:46.523 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.524 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.524 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:53:46.524 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:53:46.525 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.528 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:53:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:53:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:53:46.549 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:53:46.558 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.558 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.559 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:53:46.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:53:46.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:53:46.573 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:53:46.576 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:53:46.346916+08:00 2024-09-11 17:53:46.588 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:53:46.590 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:53:46.595 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:53:46.595 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:46.596 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:46.596 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:53:46.599 INFO 2981023 --- [ocp-schedules-30,c1dd7b9e4cfff9fb,a107f4717ed9e2ee] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:53:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:53:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:53:46.602 INFO 2981023 --- [ocp-schedules-4,9f4918335b2ba919,8fcc1c19ba381a2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:53:46.841 INFO 2981023 --- [ocp-schedules-9,d4e9b49f5da94b1e,a4efe6830cda19cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1807,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:53:46.842 INFO 2981023 --- [ocp-schedules-9,d4e9b49f5da94b1e,a4efe6830cda19cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:53:46.942 INFO 2981023 --- [host-check-clockdiff2,dd301952d32f98e1,43fdbb7b3908221b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048426 3 3 , error= 2024-09-11 17:53:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,5338eddbb80c04bb,8e9aebb9aaa21568] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,5338eddbb80c04bb,8e9aebb9aaa21568] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:47.316 INFO 2981023 --- [ocp-async-8,0cc92d4b193ffd52,366571f4712de435] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:47.570 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:47.571 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:47.766 INFO 2981023 --- [ocp-async-5,58383d856041800a,42a2f43944992611] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:53:49.316 INFO 2981023 --- [ocp-async-6,d68160428f0a9d42,ea4ed159251b3ea6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:49.892 INFO 2981023 --- [ocp-schedules-14,9f087a8ac756da75,7abffd435b95c1a9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:53:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:53:49.911 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:53:49.935 WARN 2981023 --- [scanner-13567102-1,9f087a8ac756da75,08d0129be19595a8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:53:49.958 INFO 2981023 --- [alarm-detect-7,9f087a8ac756da75,a865b0e11c62b088] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644929, data is empty. 2024-09-11 17:53:49.962 INFO 2981023 --- [alarm-detect-7,9f087a8ac756da75,a865b0e11c62b088] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644929, data is empty. 2024-09-11 17:53:49.966 INFO 2981023 --- [alarm-detect-7,9f087a8ac756da75,a865b0e11c62b088] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644929, data is empty. 2024-09-11 17:53:49.978 INFO 2981023 --- [alarm-detect-15,9f087a8ac756da75,8e8a80960e5249fd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644929, data is empty. 2024-09-11 17:53:49.983 INFO 2981023 --- [alarm-detect-16,9f087a8ac756da75,480ddf3b53849626] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644929, data is empty. 2024-09-11 17:53:49.983 INFO 2981023 --- [alarm-detect-15,9f087a8ac756da75,8e8a80960e5249fd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644929, data is empty. 2024-09-11 17:53:49.988 INFO 2981023 --- [alarm-detect-16,9f087a8ac756da75,480ddf3b53849626] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644929, data is empty. 2024-09-11 17:53:49.990 INFO 2981023 --- [alarm-detect-15,9f087a8ac756da75,8e8a80960e5249fd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644929, data is empty. 2024-09-11 17:53:49.993 INFO 2981023 --- [alarm-detect-16,9f087a8ac756da75,480ddf3b53849626] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644929, data is empty. 2024-09-11 17:53:50.100 INFO 2981023 --- [alarm-detect-5,9f087a8ac756da75,f9a3724b9c34adcc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644929, data is empty. 2024-09-11 17:53:50.104 INFO 2981023 --- [alarm-detect-5,9f087a8ac756da75,f9a3724b9c34adcc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644929, data is empty. 2024-09-11 17:53:50.107 INFO 2981023 --- [alarm-detect-5,9f087a8ac756da75,f9a3724b9c34adcc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644929, data is empty. 2024-09-11 17:53:50.268 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:53:50.272 INFO 2981023 --- [ocp-schedules-9,289a30c3d5a6598c,91b4b7671ed176cd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.273 INFO 2981023 --- [ocp-schedules-13,ffd6266f1b5eb06a,537ec969cb638205] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:53:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,07137b49370fc576,7eb1c3c75a881cbd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,07137b49370fc576,7eb1c3c75a881cbd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:50.541 INFO 2981023 --- [host-check-clockdiff1,dd301952d32f98e1,7660ab7e23a41227] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048430 3 3 , error= 2024-09-11 17:53:50.571 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:50.571 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:51.317 INFO 2981023 --- [ocp-async-5,aebb06cf40717ff6,574de5a234a6a63f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:51.575 INFO 2981023 --- [ocp-schedules-6,b78e81849d5945ee,209132915c50edf6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:53:51.576 INFO 2981023 --- [ocp-schedules-6,b78e81849d5945ee,209132915c50edf6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:53:51.576 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:53:51.577 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:53:51.577 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:53:51.577 INFO 2981023 --- [ocp-schedules-6,b78e81849d5945ee,209132915c50edf6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:53:52.268 INFO 2981023 --- [ocp-schedules-13,5cfbab491330dabd,847ad565fef1177b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:53:53.178 INFO 2981023 --- [ocp-schedules-15,2fa35d342cda18f0,731c4e816050214c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:53:53.178 INFO 2981023 --- [ocp-schedules-15,2fa35d342cda18f0,731c4e816050214c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:53:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,2b63c8548de08d82,73432a649ff9dc9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,2b63c8548de08d82,73432a649ff9dc9e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:53.317 INFO 2981023 --- [ocp-async-8,a3fd82ac07ef20b7,a42482213511d9a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:53.572 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:53.572 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:53.662 INFO 2981023 --- [ocp-schedules-21,c674f95a5e5987bb,b7f979b11b7ff82d] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:53:53.679 WARN 2981023 --- [ocp-schedules-21,c674f95a5e5987bb,b7f979b11b7ff82d] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:53:54.318 INFO 2981023 --- [ocp-schedules-12,3874a835b66a5562,19a165d9ffb29fae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:53:54.600 INFO 2981023 --- [ocp-schedules-25,7c9dc4ff4c6359ec,277a7380964dc6f9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:53:54.606 INFO 2981023 --- [ocp-schedules-25,7c9dc4ff4c6359ec,277a7380964dc6f9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:53:54.606 INFO 2981023 --- [ocp-schedules-25,7c9dc4ff4c6359ec,277a7380964dc6f9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:53:54.736 INFO 2981023 --- [ocp-schedules-8,fe7ff009e2676394,e746e95f6beb4978] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T17:53:54.736+08:00, tenantSize=6 2024-09-11 17:53:54.738 INFO 2981023 --- [ocp-schedules-8,fe7ff009e2676394,e746e95f6beb4978] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726048434736000 2024-09-11 17:53:54.738 INFO 2981023 --- [ocp-schedules-8,fe7ff009e2676394,e746e95f6beb4978] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726048134668000 2024-09-11 17:53:54.752 INFO 2981023 --- [ocp-schedules-8,fe7ff009e2676394,e746e95f6beb4978] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 17:53:55.155 INFO 2981023 --- [ocp-schedules-22,fc577158a7b6bfa1,8c3ae5b6a62ee1c4] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:53:55.317 INFO 2981023 --- [ocp-async-10,45b28b8e3690ec53,e5dcaef5d69bdf42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:55.413 INFO 2981023 --- [ocp-schedules-18,058cb55c156e10b8,3c3f7d12175ded36] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:53:55.413+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:53:55.417 INFO 2981023 --- [ocp-schedules-18,058cb55c156e10b8,3c3f7d12175ded36] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:53:55.551 INFO 2981023 --- [ocp-schedules-18,058cb55c156e10b8,3c3f7d12175ded36] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 45 to SqlAggregateData 2024-09-11 17:53:55.553 INFO 2981023 --- [ocp-schedules-18,058cb55c156e10b8,3c3f7d12175ded36] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:53:55.774 INFO 2981023 --- [ocp-schedules-16,6d05df3d3d6850ae,5dbea3521f55344b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:53:55.774 INFO 2981023 --- [ocp-schedules-16,6d05df3d3d6850ae,5dbea3521f55344b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:53:55.774 INFO 2981023 --- [ocp-schedules-16,6d05df3d3d6850ae,5dbea3521f55344b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:53:55.774 INFO 2981023 --- [ocp-schedules-16,6d05df3d3d6850ae,5dbea3521f55344b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:53:55.774 INFO 2981023 --- [ocp-schedules-16,6d05df3d3d6850ae,5dbea3521f55344b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:53:55.774 INFO 2981023 --- [ocp-schedules-16,6d05df3d3d6850ae,5dbea3521f55344b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:53:55.908 INFO 2981023 --- [ocp-schedules-12,bef0303b41132ad1,eb710344e21dba05] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:53:55.915 INFO 2981023 --- [ocp-schedules-12,bef0303b41132ad1,eb710344e21dba05] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:53:55.915 INFO 2981023 --- [ocp-schedules-12,bef0303b41132ad1,eb710344e21dba05] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:53:56.113 INFO 2981023 --- [ocp-schedules-25,6b5f982991284b15,34414e44e716a543] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:53:56.115 INFO 2981023 --- [ocp-schedules-25,6b5f982991284b15,34414e44e716a543] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:53:56.141 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:53:56.141 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:53:56.141 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.141 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.143 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:53:56.143 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.145 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:53:56.145 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:53:56.146 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:53:56.149 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.149 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.149 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.149 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.151 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.151 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.155 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.155 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.161 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.161 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.162 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.162 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.162 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.162 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:53:56.170 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.171 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:53:56.172 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.173 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.173 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:53:56.173 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3BB1D747-0-0] 2024-09-11 17:53:56.173 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:53:56.174 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.174 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:53:56.176 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.177 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.178 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.178 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:53:56.185 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.185 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:53:56.186 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.186 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:53:56.186 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.186 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.187 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:53:56.187 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:53:56.188 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:53:56.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.189 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:53:56.199 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:53:56.204 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.205 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:53:56.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:53:56.206 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.206 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:53:56.206 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3BB1D751-0-0] 2024-09-11 17:53:56.206 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:53:56.206 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:53:56.207 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.207 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:53:56.208 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.209 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:53:56.209 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.210 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:53:56.211 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.212 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:53:56.213 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.213 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.213 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:53:56.213 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:56.214 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:53:56.214 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.215 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:53:56.215 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.215 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.215 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:53:56.215 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.215 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.216 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.216 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:53:56.216 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:53:56.216 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:53:56.217 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.217 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:53:56.218 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:53:56.218 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.219 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:53:56.219 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.220 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:53:56.220 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:53:56.223 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:53:56.229 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:53:56.231 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:56.232 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:53:56.232 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:53:56.232 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:53:56.233 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.233 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:53:56.237 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:53:56.239 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.239 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:53:56.239 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:53:56.241 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:53:56.241 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.242 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:53:56.247 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:53:56.249 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:53:56.249 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.249 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.250 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:53:56.252 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:53:56.252 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.252 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:53:56.255 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:53:56.256 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:53:56.256 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:53:56.256 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:53:56.262 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:53:56.264 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:53:56.264 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:53:56.264 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:53:56.264 INFO 2981023 --- [ocp-schedules-25,6b5f982991284b15,34414e44e716a543] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:53:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,0ec42509174d2aa4,df9e7cc032342561] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,0ec42509174d2aa4,df9e7cc032342561] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:56.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:53:56.336847+08:00, lastProcessTime=2024-09-11T17:53:46.346916+08:00 2024-09-11 17:53:56.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:53:56.336847+08:00 2024-09-11 17:53:56.572 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:56.573 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:53:56.892 INFO 2981023 --- [ocp-schedules-21,57e1cb8d9eaf4b80,630de4a172f65d7b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:53:56.892 INFO 2981023 --- [ocp-schedules-21,57e1cb8d9eaf4b80,630de4a172f65d7b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:53:56.904 INFO 2981023 --- [Collect all tenants major compactions1,57e1cb8d9eaf4b80,5b15e8950cb61622] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:53:56.905 INFO 2981023 --- [Collect all tenants major compactions2,57e1cb8d9eaf4b80,6bf7535b34b0d38f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:53:56.906 INFO 2981023 --- [Collect all tenants major compactions4,57e1cb8d9eaf4b80,1e366a574e77be88] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:53:56.906 INFO 2981023 --- [Collect all tenants major compactions3,57e1cb8d9eaf4b80,b52ec79ef712318e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:53:56.906 INFO 2981023 --- [Collect all tenants major compactions6,57e1cb8d9eaf4b80,d4a8e9213b81eaf5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:53:56.907 INFO 2981023 --- [Collect all tenants major compactions5,57e1cb8d9eaf4b80,105cadb62eb3f1e0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:53:56.918 INFO 2981023 --- [Collect all tenants major compactions2,57e1cb8d9eaf4b80,6bf7535b34b0d38f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.918 INFO 2981023 --- [Collect all tenants major compactions4,57e1cb8d9eaf4b80,1e366a574e77be88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.918 INFO 2981023 --- [Collect all tenants major compactions1,57e1cb8d9eaf4b80,5b15e8950cb61622] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.919 INFO 2981023 --- [Collect all tenants major compactions3,57e1cb8d9eaf4b80,b52ec79ef712318e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.919 INFO 2981023 --- [Collect all tenants major compactions5,57e1cb8d9eaf4b80,105cadb62eb3f1e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.920 INFO 2981023 --- [Collect all tenants major compactions4,57e1cb8d9eaf4b80,1e366a574e77be88] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:53:56.920 INFO 2981023 --- [Collect all tenants major compactions1,57e1cb8d9eaf4b80,5b15e8950cb61622] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:53:56.920 INFO 2981023 --- [Collect all tenants major compactions2,57e1cb8d9eaf4b80,6bf7535b34b0d38f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:53:56.920 INFO 2981023 --- [Collect all tenants major compactions5,57e1cb8d9eaf4b80,105cadb62eb3f1e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:53:56.921 INFO 2981023 --- [Collect all tenants major compactions3,57e1cb8d9eaf4b80,b52ec79ef712318e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:53:56.922 INFO 2981023 --- [Collect all tenants major compactions6,57e1cb8d9eaf4b80,d4a8e9213b81eaf5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:53:56.923 INFO 2981023 --- [Collect all tenants major compactions6,57e1cb8d9eaf4b80,d4a8e9213b81eaf5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:53:56.923 INFO 2981023 --- [Collect all tenants major compactions2,57e1cb8d9eaf4b80,6bf7535b34b0d38f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:53:56.923 INFO 2981023 --- [Collect all tenants major compactions3,57e1cb8d9eaf4b80,b52ec79ef712318e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:53:56.923 INFO 2981023 --- [Collect all tenants major compactions2,57e1cb8d9eaf4b80,6bf7535b34b0d38f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:53:56.923 INFO 2981023 --- [Collect all tenants major compactions3,57e1cb8d9eaf4b80,b52ec79ef712318e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:53:56.923 INFO 2981023 --- [Collect all tenants major compactions1,57e1cb8d9eaf4b80,5b15e8950cb61622] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:53:56.923 INFO 2981023 --- [Collect all tenants major compactions1,57e1cb8d9eaf4b80,5b15e8950cb61622] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:53:56.924 INFO 2981023 --- [Collect all tenants major compactions4,57e1cb8d9eaf4b80,1e366a574e77be88] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:53:56.924 INFO 2981023 --- [Collect all tenants major compactions5,57e1cb8d9eaf4b80,105cadb62eb3f1e0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:53:56.924 INFO 2981023 --- [Collect all tenants major compactions4,57e1cb8d9eaf4b80,1e366a574e77be88] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:53:56.924 INFO 2981023 --- [Collect all tenants major compactions5,57e1cb8d9eaf4b80,105cadb62eb3f1e0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:53:56.925 INFO 2981023 --- [Collect all tenants major compactions6,57e1cb8d9eaf4b80,d4a8e9213b81eaf5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:53:56.925 INFO 2981023 --- [Collect all tenants major compactions6,57e1cb8d9eaf4b80,d4a8e9213b81eaf5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:53:56.925 INFO 2981023 --- [ocp-schedules-21,57e1cb8d9eaf4b80,630de4a172f65d7b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:53:57.317 INFO 2981023 --- [ocp-async-7,ae82f376551fa15b,65dd6d216fc86b34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:57.766 INFO 2981023 --- [ocp-async-10,3e7177752ba3c742,d5d9f0ec217004d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:53:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,75b2c7a8fca3797a,940be9a6fd1ab304] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048440, interval=60 2024-09-11 17:53:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,75b2c7a8fca3797a,940be9a6fd1ab304] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:53:59.317 INFO 2981023 --- [ocp-async-2,e37a9803265ba176,87c11ea16cb7e379] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:53:59.573 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:53:59.574 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:00.149 INFO 2981023 --- [ocp-schedules-9,d22f0f6098bd1592,554365472fd3ccde] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:54:00.186 WARN 2981023 --- [scanner-13567389-1,d22f0f6098bd1592,a7d61b84951b36b0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:54:01.326 INFO 2981023 --- [ocp-async-10,2d2290f4ca43982e,65064c050b05f121] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:02.115 INFO 2981023 --- [ocp-schedules-23,49a8693f38975228,521d552813392fe2] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:54:02.115 INFO 2981023 --- [ocp-schedules-23,49a8693f38975228,521d552813392fe2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:54:02.271 INFO 2981023 --- [ocp-schedules-21,187cd0d2fa93974f,2f71d3a2bbad0ae2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:54:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,04ae92b52eca6338,85f105de3c29f808] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,04ae92b52eca6338,85f105de3c29f808] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:02.545 INFO 2981023 --- [ocp-schedules-2,181009033925cb60,4f1c70b8a590adb1] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:54:02.546 INFO 2981023 --- [ocp-schedules-2,181009033925cb60,4f1c70b8a590adb1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:54:02.564 INFO 2981023 --- [ocp-schedules-2,181009033925cb60,4f1c70b8a590adb1] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:54:02.574 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:02.575 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:03.317 INFO 2981023 --- [ocp-async-5,475141d17c5efeaa,84f1d700e9726a10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,8f6e4b3ed1edb9e8,f9f34d7d9bf2b578] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,8f6e4b3ed1edb9e8,f9f34d7d9bf2b578] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:05.320 INFO 2981023 --- [ocp-async-4,603ac95edba0e9e4,8e5ea2f3be2ffba3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:05.575 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:05.577 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:54:05.989 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:54:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:54:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:54:06.003 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:54:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:54:06.004 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:54:06.006 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:06.007 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:06.007 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:54:06.007 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:54:06.008 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:06.008 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:06.008 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:06.008 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:06.063 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:51:40, start=2024-09-11T17:47:40, end=2024-09-11T17:50:10, updates=108, granule=120000ms, delay=240000ms 2024-09-11 17:54:06.068 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:51:40, target=2024-09-11T17:50:10, granule=120000ms, delay=90000ms) 2024-09-11 17:54:06.068 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:50:10, target=2024-09-11T17:39:31, granule=600000ms, delay=639000ms) 2024-09-11 17:54:06.148 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:51:41, start=2024-09-11T17:47:41, end=2024-09-11T17:50:11, updates=334, granule=120000ms, delay=240000ms 2024-09-11 17:54:06.154 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:51:41, target=2024-09-11T17:50:11, granule=120000ms, delay=90000ms) 2024-09-11 17:54:06.154 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:50:11, target=2024-09-11T17:40:51, granule=600000ms, delay=560000ms) 2024-09-11 17:54:06.154 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=187ms, period=120000ms 2024-09-11 17:54:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:54:06.351518+08:00, lastProcessTime=2024-09-11T17:53:56.336847+08:00 2024-09-11 17:54:06.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:54:06.351518+08:00 2024-09-11 17:54:07.318 INFO 2981023 --- [ocp-async-6,bc8a1338530a9ea1,ff7a878b654c6de3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:07.589 INFO 2981023 --- [ocp-schedules-23,d207bb80d300bb9d,ce1ee83b2f7a8dfa] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:54:07.650 INFO 2981023 --- [ocp-schedules-15,7ad1548486406925,6bcc3e887a3c8923] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:54:07.650+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:54:07.651 INFO 2981023 --- [ocp-schedules-15,7ad1548486406925,6bcc3e887a3c8923] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:54:07.652 INFO 2981023 --- [ocp-schedules-15,7ad1548486406925,6bcc3e887a3c8923] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:54:07.652 INFO 2981023 --- [ocp-schedules-15,7ad1548486406925,6bcc3e887a3c8923] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:54:07.766 INFO 2981023 --- [ocp-async-1,e8ebf8732ba55647,2c9a336c060428b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:54:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,d0a02e8cf9c51683,86202f2396f52681] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,d0a02e8cf9c51683,86202f2396f52681] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:08.577 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:08.577 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:09.318 INFO 2981023 --- [ocp-async-10,28bd915cd2ab5ba4,b979591011d1c4f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:10.003 INFO 2981023 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:54:10.003 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:54:10.344 INFO 2981023 --- [ocp-schedules-30,7bd9d7e4c0b6d233,aa12a6c87c87135c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:54:10.386 WARN 2981023 --- [scanner-13567649-1,7bd9d7e4c0b6d233,ff1f5ba31422901a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:54:10.420 INFO 2981023 --- [alarm-detect-4,7bd9d7e4c0b6d233,032da6dbba44999e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644929, data is empty. 2024-09-11 17:54:10.429 INFO 2981023 --- [alarm-detect-4,7bd9d7e4c0b6d233,032da6dbba44999e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644974, data is empty. 2024-09-11 17:54:10.432 INFO 2981023 --- [alarm-detect-4,7bd9d7e4c0b6d233,032da6dbba44999e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644974, data is empty. 2024-09-11 17:54:10.443 INFO 2981023 --- [alarm-detect-8,7bd9d7e4c0b6d233,56555cbbe8eccf82] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644929, data is empty. 2024-09-11 17:54:10.446 INFO 2981023 --- [alarm-detect-4,7bd9d7e4c0b6d233,79847c9e6789fe4c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644929, data is empty. 2024-09-11 17:54:10.448 INFO 2981023 --- [alarm-detect-8,7bd9d7e4c0b6d233,56555cbbe8eccf82] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644974, data is empty. 2024-09-11 17:54:10.452 INFO 2981023 --- [alarm-detect-4,7bd9d7e4c0b6d233,79847c9e6789fe4c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644974, data is empty. 2024-09-11 17:54:10.455 INFO 2981023 --- [alarm-detect-8,7bd9d7e4c0b6d233,56555cbbe8eccf82] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644974, data is empty. 2024-09-11 17:54:10.456 INFO 2981023 --- [alarm-detect-4,7bd9d7e4c0b6d233,79847c9e6789fe4c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644974, data is empty. 2024-09-11 17:54:10.560 INFO 2981023 --- [alarm-detect-11,7bd9d7e4c0b6d233,574c0d7023dd1888] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644929, data is empty. 2024-09-11 17:54:10.564 INFO 2981023 --- [alarm-detect-11,7bd9d7e4c0b6d233,574c0d7023dd1888] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644974, data is empty. 2024-09-11 17:54:10.569 INFO 2981023 --- [alarm-detect-11,7bd9d7e4c0b6d233,574c0d7023dd1888] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644974, data is empty. 2024-09-11 17:54:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:54:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:54:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,899ae5d2582c505d,12e3bd62ef10a492] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,899ae5d2582c505d,12e3bd62ef10a492] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:11.319 INFO 2981023 --- [ocp-async-1,82419b4aef918738,70e68e43581b05fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:11.577 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:11.578 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:12.147 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048380, nodeCount=807, affectRows=807 2024-09-11 17:54:12.147 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048380, nodeCount=1000, affectRows=1000 2024-09-11 17:54:12.149 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048380, nodeCount=807, validCount=807 2024-09-11 17:54:12.150 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048380, nodeCount=999, validCount=999 2024-09-11 17:54:12.273 INFO 2981023 --- [ocp-schedules-18,de52215197e0489e,aea6291215560544] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:54:12.281 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048380, nodeCount=463, affectRows=463 2024-09-11 17:54:13.318 INFO 2981023 --- [ocp-async-6,2cb9486a64e69b5f,01c2f09d6d156063] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:13.982 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048380, nodeCount=463, validCount=463 2024-09-11 17:54:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,c586b6b624d9bafc,85770a0b380e2aa1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,06644315c034e352,9e81a81e563f24d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:54:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,c586b6b624d9bafc,85770a0b380e2aa1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:14.578 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:14.579 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:14.609 INFO 2981023 --- [ocp-schedules-7,352d4404703005f1,93e22daad7057a8c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:54:14.617 INFO 2981023 --- [ocp-schedules-7,352d4404703005f1,93e22daad7057a8c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:54:14.617 INFO 2981023 --- [ocp-schedules-7,352d4404703005f1,93e22daad7057a8c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:54:15.318 INFO 2981023 --- [ocp-async-7,3c71a572bb32d4dc,6b780278996965a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:16.017 INFO 2981023 --- [ocp-schedules-24,7980ba8fd0ac3461,f017d2265b45105b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:54:16.017 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:54:16.017 INFO 2981023 --- [ocp-schedules-19,89f69f21302f4df4,1ad82b8925055ece] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:54:16.018 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:54:16.018 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:54:16.018 INFO 2981023 --- [host-check-clockdiff1,89f69f21302f4df4,f7a742e375f69737] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:54:16.019 INFO 2981023 --- [host-check-clockdiff2,89f69f21302f4df4,8dc545b32b7e8554] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:54:16.021 INFO 2981023 --- [host-check-clockdiff3,89f69f21302f4df4,9293171b67897b87] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:54:16.042 INFO 2981023 --- [ocp-schedules-23,2153098daf0cd45a,7511586bc816e0a6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:54:16.047 INFO 2981023 --- [host-check-agent-status1,2153098daf0cd45a,90cdcdfb0e0daeba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:54:16.047 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:16.047 INFO 2981023 --- [host-check-agent-status2,2153098daf0cd45a,6ec4c00228b092ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:54:16.048 INFO 2981023 --- [host-check-agent-status3,2153098daf0cd45a,1b71d97b8556a415] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:54:16.049 INFO 2981023 --- [host-check-clockdiff3,89f69f21302f4df4,9293171b67897b87] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048456 0 0 , error= 2024-09-11 17:54:16.055 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:16.057 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:16.058 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:16.059 INFO 2981023 --- [host-check-agent-status3,2153098daf0cd45a,1b71d97b8556a415] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:16.061 INFO 2981023 --- [host-check-agent-status3,2153098daf0cd45a,1b71d97b8556a415] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:16.174 INFO 2981023 --- [host-check-agent-status2,2153098daf0cd45a,6ec4c00228b092ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:16.177 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:16.179 INFO 2981023 --- [host-check-agent-status2,2153098daf0cd45a,6ec4c00228b092ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:16.183 INFO 2981023 --- [host-check-agent-status1,2153098daf0cd45a,90cdcdfb0e0daeba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:16.184 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:16.190 INFO 2981023 --- [ocp-schedules-10,5ce43d938c675d17,b5b303ba76fb819b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:54:16.190 INFO 2981023 --- [ocp-schedules-30,030ee03baf42dd47,13f3b372e6689dc5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:54:16.190 INFO 2981023 --- [ocp-schedules-27,5d023f3d8a184477,ccb136f0d9338b13] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:54:16.191 INFO 2981023 --- [ocp-schedules-13,70bebe8d37ad0cc4,cf50a098a483874e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:54:16.194 INFO 2981023 --- [host-check-agent-status1,2153098daf0cd45a,90cdcdfb0e0daeba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:16.204 INFO 2981023 --- [ocp-schedules-30,030ee03baf42dd47,13f3b372e6689dc5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:54:16.206 INFO 2981023 --- [ocp-schedules-30,030ee03baf42dd47,13f3b372e6689dc5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:54:16.206 INFO 2981023 --- [ocp-schedules-10,5ce43d938c675d17,b5b303ba76fb819b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:54:16.207 INFO 2981023 --- [ocp-schedules-27,5d023f3d8a184477,ccb136f0d9338b13] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:54:16.277 INFO 2981023 --- [host-check-agent-status3,2153098daf0cd45a,1b71d97b8556a415] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:54:16.282 INFO 2981023 --- [host-check-agent-status3,2153098daf0cd45a,1b71d97b8556a415] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:54:16.282 INFO 2981023 --- [host-check-agent-status3,2153098daf0cd45a,1b71d97b8556a415] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:54:16.282 INFO 2981023 --- [host-check-agent-status3,2153098daf0cd45a,1b71d97b8556a415] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:54:16.286 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:54:16.288 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:54:16.291690119+08:00 2024-09-11 17:54:16.289 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:54:16.288+08:00 2024-09-11 17:54:16.289 INFO 2981023 --- [host-check-agent-status2,2153098daf0cd45a,6ec4c00228b092ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:54:16.290 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:54:16.291 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:54:16.291047345+08:00 2024-09-11 17:54:16.291 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:54:16.288+08:00 and status AVAILABLE 2024-09-11 17:54:16.292 INFO 2981023 --- [host-check-agent-status1,2153098daf0cd45a,90cdcdfb0e0daeba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:54:16.292 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:54:16.291+08:00 2024-09-11 17:54:16.293 INFO 2981023 --- [ocp-schedules-21,60a685aa5f012c6a,a4a30b70b22bc09e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:54:16.294 INFO 2981023 --- [host-check-agent-status2,2153098daf0cd45a,6ec4c00228b092ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:54:16.294 INFO 2981023 --- [host-check-agent-status2,2153098daf0cd45a,6ec4c00228b092ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:54:16.294 INFO 2981023 --- [host-check-agent-status2,2153098daf0cd45a,6ec4c00228b092ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:54:16.294 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:54:16.291+08:00 and status AVAILABLE 2024-09-11 17:54:16.294 INFO 2981023 --- [host-check-agent-status1,2153098daf0cd45a,90cdcdfb0e0daeba] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:54:16.294 INFO 2981023 --- [host-check-agent-status1,2153098daf0cd45a,90cdcdfb0e0daeba] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:54:16.294 INFO 2981023 --- [host-check-agent-status1,2153098daf0cd45a,90cdcdfb0e0daeba] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:54:16.297 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.298 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:16.299 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:54:16.301 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:54:16.302 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.302 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:16.303 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:54:16.307 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:54:16.323 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:54:16.324 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:54:16.327831140+08:00 2024-09-11 17:54:16.325 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:54:16.324+08:00 2024-09-11 17:54:16.328 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:54:16.324+08:00 and status AVAILABLE 2024-09-11 17:54:16.331 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.331 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:16.331 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:16.332 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:54:16.332 INFO 2981023 --- [ocp-schedules-10,5ce43d938c675d17,b5b303ba76fb819b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:54:16.353 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:16.353 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:54:16.354 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.355 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:16.355 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.356 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:16.358 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:16.359 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:16.360 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:54:16.358939+08:00, lastProcessTime=2024-09-11T17:54:06.351518+08:00 2024-09-11 17:54:16.368 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.369 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:16.371 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:54:16.372 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:16.375 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:16.385 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:54:16.395 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:54:16.400 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.401 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.401 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:54:16.409 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:54:16.410 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:54:16.420 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:54:16.427 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.428 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.429 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:54:16.432 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:54:16.443 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:54:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.459 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.459 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:16.460 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:54:16.460 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.463 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:54:16.464 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:54:16.467 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.470 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:16.471 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.474 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:54:16.475 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:54:16.477 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:54:16.478 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:16.478 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:54:16.481 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:54:16.483 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:16.483 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:54:16.492 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:54:16.505 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:16.507 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:54:16.511 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:16.515 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:54:16.516 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:54:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.519 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:16.519 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:54:16.524 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:54:16.524 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.525 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.525 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:16.527 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:54:16.528 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:54:16.530 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:54:16.530 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:54:16.534 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:16.534 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:16.535 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:16.541 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:54:16.541 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.542 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:54:16.546 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:54:16.549 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:54:16.566 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:54:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:54:16.575 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:54:16.577 INFO 2981023 --- [host-check-host-status1,7980ba8fd0ac3461,342b2c9dab3a30ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:54:16.583 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.583 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.584 INFO 2981023 --- [host-check-host-status3,7980ba8fd0ac3461,0766bfc37ac3a994] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:54:16.584 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:16.585 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:54:16.585 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.587 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:54:16.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:54:16.595 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:54:16.597 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:16.597 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:54:16.604 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:54:16.618 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:16.621 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:54:16.630 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.631 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.633 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:54:16.637 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.638 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.638 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:16.639 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:54:16.641 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:54:16.641 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:54:16.642 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:54:16.642 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:54:16.644 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:54:16.651 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:16.653 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:16.654 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.654 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.655 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:54:16.662 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:54:16.662 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:54:16.666 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:54:16.358939+08:00 2024-09-11 17:54:16.667 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:54:16.669 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:54:16.674 INFO 2981023 --- [host-check-host-status2,7980ba8fd0ac3461,edc9b223c60b8b5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:54:16.695 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:54:16.700 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:16.701 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:16.701 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:54:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:54:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:54:16.707 INFO 2981023 --- [ocp-schedules-27,5d023f3d8a184477,ccb136f0d9338b13] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:54:16.984 INFO 2981023 --- [host-check-clockdiff2,89f69f21302f4df4,8dc545b32b7e8554] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048456 4 4 , error= 2024-09-11 17:54:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,be560007a40f7213,8969806c0a95525e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:17.315 INFO 2981023 --- [prometheus-minute-schedule-1,be560007a40f7213,8969806c0a95525e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:17.318 INFO 2981023 --- [ocp-async-10,2ffe9762be7ac4e4,f567d91957c924db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:17.579 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:17.579 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:17.766 INFO 2981023 --- [ocp-async-7,68af8b4a9be15c8b,891da68030d9b366] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:54:19.192 INFO 2981023 --- [host-check-clockdiff1,89f69f21302f4df4,f7a742e375f69737] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048459 4 4 , error= 2024-09-11 17:54:19.318 INFO 2981023 --- [ocp-async-6,80c3a1968b85d996,1fbcc1d3bb778181] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:19.699 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:54:19.699 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000007 2024-09-11 17:54:19.699 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000009 2024-09-11 17:54:19.699 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2024-09-11 17:54:19.727 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:54:19.728 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:54:19.729 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:54:19.749 WARN 2981023 --- [ForkJoinPool.commonPool-worker-1,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:54:19.750 WARN 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:54:19.752 WARN 2981023 --- [ForkJoinPool.commonPool-worker-2,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:54:19.768 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:54:19.768 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:54:19.768 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 17:54:19.769 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2024-09-11 17:54:19.769 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:54:19.769 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:54:19.769 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 17:54:19.769 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000007 agent sql collect property 2024-09-11 17:54:19.780 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:19.782 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:19.784 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:19.785 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:54:19.785 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:54:19.786 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 17:54:19.786 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:19.786 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000009 agent sql collect property 2024-09-11 17:54:19.793 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2000009 to set sql collect property inactive 2024-09-11 17:54:19.795 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:19.797 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:19.805 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1718689631), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.5), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=rtdob), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"in 2024-09-11 17:54:19.808 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.6), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4 2024-09-11 17:54:19.817 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994437), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.10), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkprdocp), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"include 2024-09-11 17:54:19.825 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000007, do nothing 2024-09-11 17:54:19.827 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2024-09-11 17:54:19.833 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000009, do nothing 2024-09-11 17:54:19.835 INFO 2981023 --- [ocp-schedules-18,a3db0f3abc0c2c62,a46884d977e33b83] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-09-11 17:54:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,6524341e17197667,9435c5f5fbce59db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:20.315 INFO 2981023 --- [prometheus-minute-schedule-1,6524341e17197667,9435c5f5fbce59db] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:20.579 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:20.580 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:20.641 INFO 2981023 --- [ocp-schedules-4,b1ceae5c48df0ab9,152af569f3c30185] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:54:20.692 WARN 2981023 --- [scanner-13569449-1,b1ceae5c48df0ab9,56f38dc753e63cac] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:54:21.318 INFO 2981023 --- [ocp-async-9,f6f1bad05d2717f7,b7906b3db410bbb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:22.276 INFO 2981023 --- [ocp-schedules-1,3f2c5ba5507a69b5,bb3f9c15ce459aee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:54:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,c52c16861680dfb5,9976d09cf5eadadb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,c52c16861680dfb5,9976d09cf5eadadb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:23.319 INFO 2981023 --- [ocp-async-10,b4534972dfea0f76,a5531c4a66fba82f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:23.436 INFO 2981023 --- [ocp-schedules-2,ca78c4ba37bb5507,5d725b626ba8fd3d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:54:23.436 INFO 2981023 --- [ocp-schedules-2,ca78c4ba37bb5507,5d725b626ba8fd3d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:54:23.436 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:54:23.436 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:54:23.439 INFO 2981023 --- [ocp-schedules-2,ca78c4ba37bb5507,5d725b626ba8fd3d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:23.440 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:23.440 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:23.441 INFO 2981023 --- [ocp-schedules-2,ca78c4ba37bb5507,5d725b626ba8fd3d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:23.441 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:23.441 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:23.455 INFO 2981023 --- [ocp-schedules-2,ca78c4ba37bb5507,5d725b626ba8fd3d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:54:23.455 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:54:23.455 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:54:23.458 INFO 2981023 --- [ocp-schedules-2,ca78c4ba37bb5507,5d725b626ba8fd3d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:54:23.580 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:23.580 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:24.321 INFO 2981023 --- [ocp-schedules-26,9df8721710c4f1b3,dc4ee1da2213cf14] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:54:25.319 INFO 2981023 --- [ocp-async-2,5f9c0125b438da12,746c5dd7957cd779] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,aeebb04147c0c222,557d8690c60aab95] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,aeebb04147c0c222,557d8690c60aab95] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:54:26.336489+08:00, lastProcessTime=2024-09-11T17:54:16.358939+08:00 2024-09-11 17:54:26.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:54:26.336489+08:00 2024-09-11 17:54:26.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:26.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:27.319 INFO 2981023 --- [ocp-async-7,1aa3c0c44adb3a90,a4cbb410fd5be13e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:27.401 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 17:54:27.401 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 17:54:27.401 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.401 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:54:27.402 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.402 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.403 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.403 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 17:54:27.415 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.415 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.415 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.416 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 17:54:27.423 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.424 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.424 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.424 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 17:54:27.432 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.433 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.433 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.434 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 17:54:27.443 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.443 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.443 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.444 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 17:54:27.447 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.447 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.448 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.448 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 17:54:27.451 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.451 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.452 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.452 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 17:54:27.454 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.455 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.455 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.455 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 17:54:27.459 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.460 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.460 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.460 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 17:54:27.464 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.464 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.464 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.465 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 17:54:27.468 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.468 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.469 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.469 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 17:54:27.474 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.475 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.475 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.475 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 17:54:27.485 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.486 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.486 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.487 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 17:54:27.490 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.490 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.490 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.491 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 17:54:27.494 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.495 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.495 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.496 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 17:54:27.498 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.499 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.499 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.500 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 17:54:27.502 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.503 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.503 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.503 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 17:54:27.507 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.507 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.508 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.509 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 17:54:27.512 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.512 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.512 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.513 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 17:54:27.516 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.516 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.517 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.517 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 17:54:27.520 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.520 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.521 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.521 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 17:54:27.525 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.525 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.525 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.526 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 17:54:27.529 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.529 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.529 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.530 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 17:54:27.532 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.533 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.533 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.533 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 17:54:27.536 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.536 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.537 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.537 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 17:54:27.540 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.540 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.541 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.541 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 17:54:27.544 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.545 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.545 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.545 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 17:54:27.548 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.548 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.549 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.549 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 17:54:27.552 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.552 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.553 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.553 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 17:54:27.556 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.556 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.556 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.557 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 17:54:27.559 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.559 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.560 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.560 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 17:54:27.563 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.563 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.564 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.564 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 17:54:27.566 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.567 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.567 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.567 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 17:54:27.570 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.570 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.571 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.571 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 17:54:27.576 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.576 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.577 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.577 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 17:54:27.581 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.581 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.581 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.582 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 17:54:27.585 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.585 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.585 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.586 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 17:54:27.589 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.589 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.589 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.590 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 17:54:27.592 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.592 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.592 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.593 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 17:54:27.595 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.595 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.596 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.596 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 17:54:27.598 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.598 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.599 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.599 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 17:54:27.601 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.602 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.602 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.602 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 17:54:27.609 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.609 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.609 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.610 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 17:54:27.626 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.626 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.627 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.627 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 17:54:27.631 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.632 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.632 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.632 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 17:54:27.637 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.637 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.638 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.638 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 17:54:27.642 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.643 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.643 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.643 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 17:54:27.645 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.646 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.646 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.646 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 17:54:27.649 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.649 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.649 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.650 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 17:54:27.652 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 17:54:27.652 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 17:54:27.653 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.653 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:54:27.655 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.655 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.656 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.656 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 17:54:27.660 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.660 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:54:27.660 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:27.661 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 17:54:27.663 INFO 2981023 --- [ocp-schedules-7,f7a0df4f749481b1,b89fc056f3f722ff] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 17:54:27.766 INFO 2981023 --- [ocp-async-2,c031edd4b17a826b,70fd881df0b367c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:54:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,f78048e5027d35e2,07f2ed105d02fe6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,f78048e5027d35e2,07f2ed105d02fe6e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:29.319 INFO 2981023 --- [ocp-async-10,cbe4db6234c43dd2,84784997538f20a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:29.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:29.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:30.893 INFO 2981023 --- [ocp-schedules-10,ae4f32478a28f84e,41823a61337afdfa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:54:30.950 WARN 2981023 --- [scanner-13569619-1,ae4f32478a28f84e,789b4ada3d439bc2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:54:30.964 INFO 2981023 --- [alarm-detect-8,ae4f32478a28f84e,a6ef242b7119370c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644974, data is empty. 2024-09-11 17:54:30.969 INFO 2981023 --- [alarm-detect-8,ae4f32478a28f84e,a6ef242b7119370c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644974, data is empty. 2024-09-11 17:54:30.974 INFO 2981023 --- [alarm-detect-8,ae4f32478a28f84e,a6ef242b7119370c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644974, data is empty. 2024-09-11 17:54:31.005 INFO 2981023 --- [alarm-detect-18,ae4f32478a28f84e,45c9862d7aa17f6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644974, data is empty. 2024-09-11 17:54:31.012 INFO 2981023 --- [alarm-detect-18,ae4f32478a28f84e,45c9862d7aa17f6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644974, data is empty. 2024-09-11 17:54:31.015 INFO 2981023 --- [alarm-detect-3,ae4f32478a28f84e,c43ef0f40e0a71fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644974, data is empty. 2024-09-11 17:54:31.017 INFO 2981023 --- [alarm-detect-18,ae4f32478a28f84e,45c9862d7aa17f6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644974, data is empty. 2024-09-11 17:54:31.021 INFO 2981023 --- [alarm-detect-3,ae4f32478a28f84e,c43ef0f40e0a71fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644974, data is empty. 2024-09-11 17:54:31.025 INFO 2981023 --- [alarm-detect-3,ae4f32478a28f84e,c43ef0f40e0a71fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644974, data is empty. 2024-09-11 17:54:31.268 INFO 2981023 --- [alarm-detect-15,ae4f32478a28f84e,b78b45eb819332f2] c.o.o.alarm.core.detect.MetricDetector : alarmType agent_process_count_abnormal detect time range 1726048341 ~ 1726048401 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={agent_process_count=[SeriesMetricValues(metric=Metric{name='agent_process_count', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, name=ocp_mgragent, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726048380, value=2.0)])]}, metricEvaluateResults={agent_process_count != 1=[ExpressionEvaluateResult(matchValue=1.0, metric=agent_process_count, match=true, value=2.0, values={1726048380=2.0}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, name=ocp_mgragent, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:54:31.268 INFO 2981023 --- [alarm-detect-15,ae4f32478a28f84e,b78b45eb819332f2] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=agent_process_count_abnormal, target=alarm_template_id=0:host=10.31.35.10:name=ocp_mgragent-3, currentState=Pending 2024-09-11 17:54:31.319 INFO 2981023 --- [ocp-async-2,446a5f28a4a708fc,092ef575225d0f45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:31.345 INFO 2981023 --- [alarm-detect-3,ae4f32478a28f84e,b64b009af82e81a3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644974, data is empty. 2024-09-11 17:54:31.348 INFO 2981023 --- [alarm-detect-3,ae4f32478a28f84e,b64b009af82e81a3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644974, data is empty. 2024-09-11 17:54:31.352 INFO 2981023 --- [alarm-detect-3,ae4f32478a28f84e,b64b009af82e81a3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644974, data is empty. 2024-09-11 17:54:32.279 INFO 2981023 --- [ocp-schedules-9,7d69a0d63719f53c,ac4166299598a837] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:54:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,3464cb9cd1fddcfd,4e1deb90ed9c6d60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,3464cb9cd1fddcfd,4e1deb90ed9c6d60] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:32.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:32.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:33.319 INFO 2981023 --- [ocp-async-7,0715feaad89f4023,b0d224f1aa443c40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:34.619 INFO 2981023 --- [ocp-schedules-27,c05a994c0c66d3dd,38d7ebd5261c7e72] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:54:34.626 INFO 2981023 --- [ocp-schedules-27,c05a994c0c66d3dd,38d7ebd5261c7e72] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:54:34.626 INFO 2981023 --- [ocp-schedules-27,c05a994c0c66d3dd,38d7ebd5261c7e72] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:54:34.767 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:54:34.767 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:54:34.769 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:54:34.769 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:54:34.769 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:54:34.769 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:54:34.771 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:54:34.771 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:54:34.771 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:54:34.771 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:54:34.795 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:54:34.799 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:54:34.799 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:54:34.799 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:54:34.799 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:54:34.799 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:54:34.803 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:54:35.069 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:54:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,82ec37ba1999b47e,8e3481dd0a52a072] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,82ec37ba1999b47e,8e3481dd0a52a072] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:35.319 INFO 2981023 --- [ocp-async-5,15f19f8f1ac4ff50,3d6fc926a9a976a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:35.320 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.548946087S 2024-09-11 17:54:35.321 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:54:35.378 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:54:35.378 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:54:35.396 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:54:35.431 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:54:35.431 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:54:35.431 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:54:35.431 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:54:35.431 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:54:35.431 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:54:35.437 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:54:35.515 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] o.s.boot.SpringApplication : Started application in 0.745 seconds (JVM running for 84275.387) 2024-09-11 17:54:35.515 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.74597857S 2024-09-11 17:54:35.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:35.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:35.714 INFO 2981023 --- [ocp-schedules-28,67b8e442ab5f8d40,93426c1c5602fd5a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:54:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:54:36.337096+08:00, lastProcessTime=2024-09-11T17:54:26.336489+08:00 2024-09-11 17:54:36.467 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:54:36.337096+08:00 2024-09-11 17:54:37.314 INFO 2981023 --- [prometheus-cache-monitor1,6fd01ada1e3822f6,3a8bfa1a8fd02dd5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7225,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1807,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:54:37.315 INFO 2981023 --- [prometheus-cache-monitor1,6fd01ada1e3822f6,3a8bfa1a8fd02dd5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:54:37.319 INFO 2981023 --- [ocp-async-4,186a2a297dd9ecc6,840e79a0a5092e2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:37.766 INFO 2981023 --- [ocp-async-8,cd001329dd59d320,ee689925e03376b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:54:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,3ba8b90f8111d4a9,b6dcfaf04ffc57f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,3ba8b90f8111d4a9,b6dcfaf04ffc57f3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:38.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:38.584 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:39.319 INFO 2981023 --- [ocp-async-3,34bc5429f514e5e7,d0015526b593df65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:40.002 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:54:40.002 INFO 2981023 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:54:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,1a39ba82170da744,677de37f4cd0b2b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,1a39ba82170da744,677de37f4cd0b2b8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:41.326 INFO 2981023 --- [ocp-async-5,564c5e87064b2811,9e86722bdbd672ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:41.399 INFO 2981023 --- [ocp-schedules-26,3169a38b02defe6d,3834630b23fa2f2d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:54:41.441 WARN 2981023 --- [scanner-13569984-1,3169a38b02defe6d,17d6c2f7c5d53a09] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:54:41.585 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:41.586 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:42.281 INFO 2981023 --- [ocp-schedules-4,33b77d723454f0bb,7f7562f89ab2b751] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:54:42.496 INFO 2981023 --- [ocp-schedules-1,b5aebde24b599412,63b88e6689858cda] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:54:42.539 WARN 2981023 --- [ocp-schedules-1,b5aebde24b599412,63b88e6689858cda] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:54:43.321 INFO 2981023 --- [ocp-async-4,95dc238db0ec9942,c820fa744634789f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,6a445de2cefceff5,682e059210f0b661] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,6a445de2cefceff5,682e059210f0b661] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:44.586 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:44.586 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:45.101 INFO 2981023 --- [prometheus-second-schedule-1,0bb8f4a3a9d343dc,5fdd7232ed81e90e] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:54:45.321 INFO 2981023 --- [ocp-async-7,957fba65d38c4cca,7e52d0ebda69eeea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:45.997 INFO 2981023 --- [ocp-schedules-12,d78f3df25b1fdd85,1ece4739101926d9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:54:45.998 INFO 2981023 --- [host-check-clockdiff1,d78f3df25b1fdd85,03271ddd9d9dcfd0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:54:45.998 INFO 2981023 --- [host-check-clockdiff2,d78f3df25b1fdd85,752cb0f7bdd7b544] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:54:45.998 INFO 2981023 --- [host-check-clockdiff3,d78f3df25b1fdd85,bc039dc44be3383b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:54:46.005 INFO 2981023 --- [host-check-clockdiff3,d78f3df25b1fdd85,bc039dc44be3383b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048486 0 0 , error= 2024-09-11 17:54:46.007 INFO 2981023 --- [ocp-schedules-15,5bb409b7335e597a,ca21664fa1a57037] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:54:46.007 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:54:46.008 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:54:46.008 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:54:46.009 INFO 2981023 --- [ocp-schedules-30,3df0362ce18ec2d6,3f7529a83a00d777] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:54:46.009 INFO 2981023 --- [host-check-agent-status2,3df0362ce18ec2d6,33dc25c193a2a33d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:54:46.010 INFO 2981023 --- [host-check-agent-status1,3df0362ce18ec2d6,9972b349117450cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:54:46.011 INFO 2981023 --- [host-check-agent-status3,3df0362ce18ec2d6,7510e7acb59140c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:54:46.013 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:46.013 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:46.014 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:46.015 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:46.015 INFO 2981023 --- [host-check-agent-status2,3df0362ce18ec2d6,33dc25c193a2a33d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:46.016 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:46.016 INFO 2981023 --- [host-check-agent-status2,3df0362ce18ec2d6,33dc25c193a2a33d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:46.016 INFO 2981023 --- [host-check-agent-status1,3df0362ce18ec2d6,9972b349117450cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:46.016 INFO 2981023 --- [host-check-agent-status3,3df0362ce18ec2d6,7510e7acb59140c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:46.017 INFO 2981023 --- [host-check-agent-status3,3df0362ce18ec2d6,7510e7acb59140c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:46.018 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:46.018 INFO 2981023 --- [host-check-agent-status1,3df0362ce18ec2d6,9972b349117450cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:46.038 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:54:46.038 INFO 2981023 --- [host-check-agent-status3,3df0362ce18ec2d6,7510e7acb59140c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:54:46.040 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:54:46.041 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:54:46.044204639+08:00 2024-09-11 17:54:46.042 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:54:46.041+08:00 2024-09-11 17:54:46.042 INFO 2981023 --- [host-check-agent-status2,3df0362ce18ec2d6,33dc25c193a2a33d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:54:46.042 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:54:46.046214854+08:00 2024-09-11 17:54:46.043 INFO 2981023 --- [host-check-agent-status3,3df0362ce18ec2d6,7510e7acb59140c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:54:46.043 INFO 2981023 --- [host-check-agent-status3,3df0362ce18ec2d6,7510e7acb59140c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:54:46.043 INFO 2981023 --- [host-check-agent-status3,3df0362ce18ec2d6,7510e7acb59140c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:54:46.043 INFO 2981023 --- [host-check-agent-status1,3df0362ce18ec2d6,9972b349117450cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:54:46.043 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:54:46.042+08:00 2024-09-11 17:54:46.044 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:54:46.044 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:54:46.044640335+08:00 2024-09-11 17:54:46.045 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:54:46.044+08:00 2024-09-11 17:54:46.046 INFO 2981023 --- [host-check-agent-status2,3df0362ce18ec2d6,33dc25c193a2a33d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:54:46.046 INFO 2981023 --- [host-check-agent-status2,3df0362ce18ec2d6,33dc25c193a2a33d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:54:46.046 INFO 2981023 --- [host-check-agent-status2,3df0362ce18ec2d6,33dc25c193a2a33d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:54:46.046 INFO 2981023 --- [host-check-agent-status1,3df0362ce18ec2d6,9972b349117450cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:54:46.046 INFO 2981023 --- [host-check-agent-status1,3df0362ce18ec2d6,9972b349117450cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:54:46.046 INFO 2981023 --- [host-check-agent-status1,3df0362ce18ec2d6,9972b349117450cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:54:46.048 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:54:46.042+08:00 and status AVAILABLE 2024-09-11 17:54:46.049 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:54:46.041+08:00 and status AVAILABLE 2024-09-11 17:54:46.050 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:54:46.044+08:00 and status AVAILABLE 2024-09-11 17:54:46.052 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:54:46.054 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:54:46.055 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:54:46.059 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:46.060 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:46.061 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:46.061 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:46.061 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:46.062 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:46.077 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:54:46.079 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:54:46.079 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:54:46.098 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:54:46.101 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:54:46.107 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:54:46.108 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:54:46.109 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:54:46.121 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:54:46.146 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:54:46.147 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:54:46.149 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:54:46.159 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:54:46.172 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:54:46.179 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:54:46.180 INFO 2981023 --- [ocp-schedules-21,96eb3dd19e72f4d0,823f73082ab47ee4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:54:46.181 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:54:46.187 INFO 2981023 --- [ocp-schedules-5,51b4420d3bf213c2,4a9a25c858e60de4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:54:46.187 INFO 2981023 --- [ocp-schedules-16,5d6b443241020d3a,8aa79ecb46934170] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:54:46.188 INFO 2981023 --- [ocp-schedules-1,3af8f5e7e0957fc1,9a752ecab6ef8c50] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:54:46.188 INFO 2981023 --- [ocp-schedules-22,20d1ead3a76c4653,7425df7f17da21eb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:54:46.189 INFO 2981023 --- [ocp-schedules-16,5d6b443241020d3a,8aa79ecb46934170] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:54:46.189 INFO 2981023 --- [ocp-schedules-5,51b4420d3bf213c2,4a9a25c858e60de4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:54:46.190 INFO 2981023 --- [ocp-schedules-1,3af8f5e7e0957fc1,9a752ecab6ef8c50] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:54:46.190 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:54:46.202 INFO 2981023 --- [ocp-schedules-21,96eb3dd19e72f4d0,823f73082ab47ee4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:54:46.205 INFO 2981023 --- [ocp-schedules-16,5d6b443241020d3a,8aa79ecb46934170] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:54:46.214 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:54:46.236 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.237 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.238 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:46.240 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:54:46.242 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:46.247 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.248 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:46.249 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:54:46.250 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:54:46.253 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:54:46.255 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:54:46.260 INFO 2981023 --- [ocp-schedules-5,51b4420d3bf213c2,4a9a25c858e60de4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:54:46.260 INFO 2981023 --- [ocp-schedules-25,8372bf35419325c3,5ac250b7a35d013d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:54:46.267 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.267 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:46.273 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.274 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:46.278 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.278 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:46.280 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:54:46.281 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:54:46.282 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.283 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:54:46.283 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:54:46.284 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.300 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:54:46.303 INFO 2981023 --- [host-check-host-status2,5bb409b7335e597a,bf8deabd6bc057f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:54:46.303 INFO 2981023 --- [host-check-host-status1,5bb409b7335e597a,461ed83a19a56b52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:54:46.309 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:54:46.318 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.323 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.324 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:54:46.324 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:54:46.329 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.331 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.332 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:54:46.340 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:54:46.339134+08:00, lastProcessTime=2024-09-11T17:54:36.337096+08:00 2024-09-11 17:54:46.349 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.354 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:54:46.362 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.369 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:54:46.374 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.375 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.375 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:46.376 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:54:46.383 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.383 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.384 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.384 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:54:46.385 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:54:46.385 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.387 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:54:46.387 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:54:46.395 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.396 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.397 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:46.399 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.401 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.401 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.401 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:54:46.402 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:54:46.402 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:54:46.402 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.403 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:54:46.405 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:54:46.405 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:54:46.408 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:46.408 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:54:46.412 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:54:46.417 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:46.417 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:54:46.422 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:46.422 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:54:46.424 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:54:46.426 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:46.434 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:54:46.434 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.435 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.435 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:46.436 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:54:46.437 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:46.439 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:54:46.445 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:54:46.446 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:54:46.447 INFO 2981023 --- [host-check-host-status3,5bb409b7335e597a,13849d18122abd41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:54:46.447 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.448 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:54:46.448 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.448 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:46.449 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:54:46.454 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:54:46.456 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.458 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.459 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.460 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:54:46.460 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:54:46.460 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:54:46.461 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:54:46.467 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:54:46.468 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:54:46.468 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.470 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.472 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:54:46.472 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:54:46.472 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:54:46.474 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:54:46.476 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.476 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:54:46.477 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:54:46.477 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.482 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:54:46.483 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:54:46.493 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:54:46.501 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:54:46.513 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:54:46.518 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.520 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.520 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:54:46.521 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:54:46.521 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.526 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:54:46.537 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:54:46.537 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:54:46.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:54:46.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:54:46.544 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.544 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.545 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:54:46.551 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:54:46.551 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:54:46.553 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:54:46.560 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:54:46.567 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.567 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:46.568 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.568 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.568 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:54:46.568 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:54:46.569 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:54:46.569 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:46.575 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:54:46.575 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:54:46.578 INFO 2981023 --- [ocp-schedules-1,3af8f5e7e0957fc1,9a752ecab6ef8c50] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:54:46.585 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:54:46.339134+08:00 2024-09-11 17:54:46.611 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:54:46.628 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:54:46.635 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:54:46.638 INFO 2981023 --- [ocp-schedules-9,4b37f8b30b00b694,f7ffc07f3e84e77e] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:54:46.944 INFO 2981023 --- [host-check-clockdiff1,d78f3df25b1fdd85,03271ddd9d9dcfd0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048486 4 4 , error= 2024-09-11 17:54:46.951 INFO 2981023 --- [host-check-clockdiff2,d78f3df25b1fdd85,752cb0f7bdd7b544] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048486 4 4 , error= 2024-09-11 17:54:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,40074d9d501c947a,35cc5201f3590c88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,40074d9d501c947a,35cc5201f3590c88] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:47.322 INFO 2981023 --- [ocp-async-10,3ee11f5b6ae09546,b1429eff833dd90d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:47.586 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:47.587 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:47.766 INFO 2981023 --- [ocp-async-9,cca5c8450e061cdc,82e5b7db98486004] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:54:49.322 INFO 2981023 --- [ocp-async-6,b80cb9edc74473c4,1c27524988844005] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:54:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:54:50.274 INFO 2981023 --- [ocp-schedules-29,5639d57b33c810e5,c3fb8d065ae3e59a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:54:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,8823613aeae976dd,2be03b13b87249f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,8823613aeae976dd,2be03b13b87249f3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:50.587 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:50.587 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:51.322 INFO 2981023 --- [ocp-async-9,1efa444eb55ccf26,3a8ae15117140cdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:51.577 INFO 2981023 --- [ocp-schedules-27,4bdb9ff71865051e,270cd402b16c6764] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:54:51.578 INFO 2981023 --- [ocp-schedules-27,4bdb9ff71865051e,270cd402b16c6764] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:54:51.578 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:54:51.579 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:54:51.579 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:54:51.579 INFO 2981023 --- [ocp-schedules-27,4bdb9ff71865051e,270cd402b16c6764] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:54:51.587 INFO 2981023 --- [ocp-schedules-15,469c2086d005f58d,2447714f8cbe7ce7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:54:51.643 INFO 2981023 --- [alarm-detect-20,469c2086d005f58d,77cc09272a364f9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644974, data is empty. 2024-09-11 17:54:51.647 INFO 2981023 --- [alarm-detect-20,469c2086d005f58d,77cc09272a364f9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644974, data is empty. 2024-09-11 17:54:51.650 INFO 2981023 --- [alarm-detect-20,469c2086d005f58d,77cc09272a364f9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644974, data is empty. 2024-09-11 17:54:51.650 WARN 2981023 --- [scanner-13570195-1,469c2086d005f58d,ac66d5cd3f3489fc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:54:51.679 INFO 2981023 --- [alarm-detect-16,469c2086d005f58d,928a1d16654c7480] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644974, data is empty. 2024-09-11 17:54:51.688 INFO 2981023 --- [alarm-detect-15,469c2086d005f58d,a09ec532e999c5e5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644974, data is empty. 2024-09-11 17:54:51.688 INFO 2981023 --- [alarm-detect-16,469c2086d005f58d,928a1d16654c7480] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644974, data is empty. 2024-09-11 17:54:51.694 INFO 2981023 --- [alarm-detect-15,469c2086d005f58d,a09ec532e999c5e5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644974, data is empty. 2024-09-11 17:54:51.694 INFO 2981023 --- [alarm-detect-16,469c2086d005f58d,928a1d16654c7480] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644974, data is empty. 2024-09-11 17:54:51.699 INFO 2981023 --- [alarm-detect-15,469c2086d005f58d,a09ec532e999c5e5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644974, data is empty. 2024-09-11 17:54:51.788 INFO 2981023 --- [alarm-detect-11,469c2086d005f58d,7f47eab20505d60c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644974, data is empty. 2024-09-11 17:54:51.792 INFO 2981023 --- [alarm-detect-11,469c2086d005f58d,7f47eab20505d60c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644974, data is empty. 2024-09-11 17:54:51.798 INFO 2981023 --- [alarm-detect-11,469c2086d005f58d,7f47eab20505d60c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644974, data is empty. 2024-09-11 17:54:52.284 INFO 2981023 --- [ocp-schedules-1,02cbabf4133ac94e,d1f1f155e75daf80] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:54:53.181 INFO 2981023 --- [ocp-schedules-17,150732f4ccfb3239,29d7a31056d02d5b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:54:53.181 INFO 2981023 --- [ocp-schedules-17,150732f4ccfb3239,29d7a31056d02d5b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:54:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,ca504d1afc047a28,7bad4d3d66de797f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,ca504d1afc047a28,7bad4d3d66de797f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:53.322 INFO 2981023 --- [ocp-async-10,d86e74b33e71e34b,afb122d346b5e383] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:53.587 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:53.588 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:53.704 INFO 2981023 --- [ocp-schedules-1,4559b6ebb4fbdff4,20f2133be82bad3e] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:54:53.721 WARN 2981023 --- [ocp-schedules-1,4559b6ebb4fbdff4,20f2133be82bad3e] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:54:54.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ed7c44ff22967191,ca8ed2f4aba90fed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.31.33.41, traceId=ed7c44ff22967191, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2024-09-11 17:54:54.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,19d0d1b54953cc3e,f06e1bb248072bc8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.31.33.41, traceId=19d0d1b54953cc3e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:54:54.106 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,19d0d1b54953cc3e,f06e1bb248072bc8] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:54:54.145 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,19d0d1b54953cc3e,f06e1bb248072bc8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=19d0d1b54953cc3e, duration=46 ms] 2024-09-11 17:54:54.156 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ed7c44ff22967191,ca8ed2f4aba90fed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.31.33.41, traceId=ed7c44ff22967191, duration=58 ms] 2024-09-11 17:54:54.309 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3083ad5111ad9bc3,53a3043a05837c92] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.31.33.41, traceId=3083ad5111ad9bc3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2024-09-11 17:54:54.316 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3083ad5111ad9bc3,53a3043a05837c92] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.31.33.41, traceId=3083ad5111ad9bc3, duration=10 ms] 2024-09-11 17:54:54.323 INFO 2981023 --- [ocp-schedules-23,382adb8517331957,eec1626a0c8559c7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:54:54.627 INFO 2981023 --- [ocp-schedules-4,8e3f6fabe69b5e18,87bb9572f3d5794a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:54:54.632 INFO 2981023 --- [ocp-schedules-4,8e3f6fabe69b5e18,87bb9572f3d5794a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:54:54.632 INFO 2981023 --- [ocp-schedules-4,8e3f6fabe69b5e18,87bb9572f3d5794a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:54:54.914 INFO 2981023 --- [ocp-schedules-20,83394b38b7842d9f,206ea5f4e5a205a9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:54:54.914 INFO 2981023 --- [ocp-schedules-20,83394b38b7842d9f,206ea5f4e5a205a9] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:54:54.917 INFO 2981023 --- [ocp-schedules-20,83394b38b7842d9f,206ea5f4e5a205a9] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:54:54.917 INFO 2981023 --- [ocp-schedules-20,83394b38b7842d9f,206ea5f4e5a205a9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:54:54.917 INFO 2981023 --- [ocp-schedules-20,83394b38b7842d9f,206ea5f4e5a205a9] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:54:54.920 INFO 2981023 --- [ocp-schedules-20,83394b38b7842d9f,206ea5f4e5a205a9] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:54:54.920 INFO 2981023 --- [ocp-schedules-20,83394b38b7842d9f,206ea5f4e5a205a9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:54:54.920 INFO 2981023 --- [ocp-schedules-20,83394b38b7842d9f,206ea5f4e5a205a9] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:54:54.929 INFO 2981023 --- [ocp-schedules-20,83394b38b7842d9f,206ea5f4e5a205a9] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:54:55.322 INFO 2981023 --- [ocp-async-4,b18052f090f9341a,34c80cc1d913ce7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:55.566 INFO 2981023 --- [ocp-schedules-27,f30f2a961861330c,c655a362e2b47ff4] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:54:55.566+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:54:55.569 INFO 2981023 --- [ocp-schedules-27,f30f2a961861330c,c655a362e2b47ff4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:54:55.638 INFO 2981023 --- [ocp-schedules-1,fcfff193d9576411,aa2876707ce621c2] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:54:55.723 INFO 2981023 --- [ocp-schedules-27,f30f2a961861330c,c655a362e2b47ff4] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 21 to SqlAggregateData 2024-09-11 17:54:55.724 INFO 2981023 --- [ocp-schedules-27,f30f2a961861330c,c655a362e2b47ff4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:54:55.775 INFO 2981023 --- [ocp-schedules-28,80e9473eefb1ad75,2dcf65d4918c003e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:54:55.775 INFO 2981023 --- [ocp-schedules-28,80e9473eefb1ad75,2dcf65d4918c003e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:54:55.778 INFO 2981023 --- [ocp-schedules-28,80e9473eefb1ad75,2dcf65d4918c003e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:54:55.778 INFO 2981023 --- [ocp-schedules-28,80e9473eefb1ad75,2dcf65d4918c003e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:54:55.781 INFO 2981023 --- [ocp-schedules-28,80e9473eefb1ad75,2dcf65d4918c003e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:54:55.781 INFO 2981023 --- [ocp-schedules-28,80e9473eefb1ad75,2dcf65d4918c003e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:54:55.915 INFO 2981023 --- [ocp-schedules-16,f3d0e2ba01ad3982,2b7ac3193e9da2b0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:54:55.923 INFO 2981023 --- [ocp-schedules-16,f3d0e2ba01ad3982,2b7ac3193e9da2b0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:54:55.923 INFO 2981023 --- [ocp-schedules-16,f3d0e2ba01ad3982,2b7ac3193e9da2b0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:54:56.264 INFO 2981023 --- [ocp-schedules-9,d1a682795c85651d,ef43f4b728a8aaf5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:54:56.265 INFO 2981023 --- [ocp-schedules-9,d1a682795c85651d,ef43f4b728a8aaf5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:54:56.293 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:54:56.293 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.294 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:54:56.295 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.296 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:54:56.299 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:54:56.299 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:54:56.300 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.302 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.302 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.302 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:54:56.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.307 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.308 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.313 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.313 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:54:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,b0a2ead7aeb69b31,516e2e7e325985f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,b0a2ead7aeb69b31,516e2e7e325985f9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:56.317 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.317 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.317 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.317 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.324 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.324 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:54:56.329 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.329 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:54:56.332 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.332 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.333 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.333 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.334 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:54:56.334 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3CD1D6DA-0-0] 2024-09-11 17:54:56.334 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:54:56.334 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.335 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:54:56.337 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.337 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:54:56.344 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:54:56.341116+08:00, lastProcessTime=2024-09-11T17:54:46.339134+08:00 2024-09-11 17:54:56.347 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.348 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.348 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.349 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:54:56.359 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.360 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:54:56.361 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.361 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:54:56.362 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.362 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.363 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:54:56.363 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:54:56.363 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:54:56.363 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:54:56.363 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3CD1D6E4-0-0] 2024-09-11 17:54:56.363 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:54:56.370 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:54:56.373 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.373 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:54:56.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.374 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:54:56.375 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:54:56.375 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.375 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.376 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:54:56.376 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:54:56.376 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.377 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:54:56.377 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.378 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:54:56.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:54:56.378 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:54:56.379 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.380 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:54:56.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:54:56.381 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.381 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.382 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:54:56.382 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:54:56.382 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.382 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:54:56.383 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:54:56.384 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.384 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:54:56.385 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:54:56.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:54:56.385 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:56.388 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:54:56.389 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.389 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.398 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:56.404 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:54:56.405 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.406 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:54:56.407 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:54:56.407 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:54:56.407 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:54:56.411 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:54:56.412 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.412 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:54:56.419 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:54:56.419 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:54:56.421 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:54:56.421 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.421 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:54:56.422 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:54:56.422 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.423 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.428 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:54:56.429 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:54:56.430 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.430 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:54:56.434 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:54:56.436 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:54:56.436 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:54:56.436 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:54:56.438 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:54:56.441 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:54:56.441 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:54:56.441 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:54:56.441 INFO 2981023 --- [ocp-schedules-9,d1a682795c85651d,ef43f4b728a8aaf5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:54:56.514 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:54:56.341116+08:00 2024-09-11 17:54:56.588 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:56.588 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:54:56.925 INFO 2981023 --- [ocp-schedules-20,2966f4ba9a59ef3a,4745107c3a00d978] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:54:56.925 INFO 2981023 --- [ocp-schedules-20,2966f4ba9a59ef3a,4745107c3a00d978] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:54:56.938 INFO 2981023 --- [Collect all tenants major compactions1,2966f4ba9a59ef3a,9ff4fe15f64f505b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:54:56.938 INFO 2981023 --- [Collect all tenants major compactions2,2966f4ba9a59ef3a,83e2dde6d393bead] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:54:56.939 INFO 2981023 --- [Collect all tenants major compactions4,2966f4ba9a59ef3a,8aa52249b000e405] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:54:56.940 INFO 2981023 --- [Collect all tenants major compactions3,2966f4ba9a59ef3a,9c4d2a355fed3702] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:54:56.940 INFO 2981023 --- [Collect all tenants major compactions6,2966f4ba9a59ef3a,c5c428a6f9cbb82e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:54:56.942 INFO 2981023 --- [Collect all tenants major compactions5,2966f4ba9a59ef3a,6631e55493b198c4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:54:56.956 INFO 2981023 --- [Collect all tenants major compactions4,2966f4ba9a59ef3a,8aa52249b000e405] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.957 INFO 2981023 --- [Collect all tenants major compactions4,2966f4ba9a59ef3a,8aa52249b000e405] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:54:56.958 INFO 2981023 --- [Collect all tenants major compactions5,2966f4ba9a59ef3a,6631e55493b198c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.960 INFO 2981023 --- [Collect all tenants major compactions5,2966f4ba9a59ef3a,6631e55493b198c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:54:56.961 INFO 2981023 --- [Collect all tenants major compactions2,2966f4ba9a59ef3a,83e2dde6d393bead] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.961 INFO 2981023 --- [Collect all tenants major compactions3,2966f4ba9a59ef3a,9c4d2a355fed3702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.961 INFO 2981023 --- [Collect all tenants major compactions1,2966f4ba9a59ef3a,9ff4fe15f64f505b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.962 INFO 2981023 --- [Collect all tenants major compactions3,2966f4ba9a59ef3a,9c4d2a355fed3702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:54:56.962 INFO 2981023 --- [Collect all tenants major compactions2,2966f4ba9a59ef3a,83e2dde6d393bead] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:54:56.962 INFO 2981023 --- [Collect all tenants major compactions6,2966f4ba9a59ef3a,c5c428a6f9cbb82e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:54:56.964 INFO 2981023 --- [Collect all tenants major compactions6,2966f4ba9a59ef3a,c5c428a6f9cbb82e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:54:56.964 INFO 2981023 --- [Collect all tenants major compactions1,2966f4ba9a59ef3a,9ff4fe15f64f505b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:54:56.967 INFO 2981023 --- [Collect all tenants major compactions3,2966f4ba9a59ef3a,9c4d2a355fed3702] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:54:56.968 INFO 2981023 --- [Collect all tenants major compactions5,2966f4ba9a59ef3a,6631e55493b198c4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:54:56.969 INFO 2981023 --- [Collect all tenants major compactions1,2966f4ba9a59ef3a,9ff4fe15f64f505b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:54:56.969 INFO 2981023 --- [Collect all tenants major compactions2,2966f4ba9a59ef3a,83e2dde6d393bead] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:54:56.970 INFO 2981023 --- [Collect all tenants major compactions3,2966f4ba9a59ef3a,9c4d2a355fed3702] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:54:56.971 INFO 2981023 --- [Collect all tenants major compactions6,2966f4ba9a59ef3a,c5c428a6f9cbb82e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:54:56.971 INFO 2981023 --- [Collect all tenants major compactions6,2966f4ba9a59ef3a,c5c428a6f9cbb82e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:54:56.972 INFO 2981023 --- [Collect all tenants major compactions4,2966f4ba9a59ef3a,8aa52249b000e405] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:54:56.972 INFO 2981023 --- [Collect all tenants major compactions4,2966f4ba9a59ef3a,8aa52249b000e405] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:54:56.973 INFO 2981023 --- [Collect all tenants major compactions2,2966f4ba9a59ef3a,83e2dde6d393bead] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:54:56.974 INFO 2981023 --- [Collect all tenants major compactions5,2966f4ba9a59ef3a,6631e55493b198c4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:54:56.975 INFO 2981023 --- [Collect all tenants major compactions1,2966f4ba9a59ef3a,9ff4fe15f64f505b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:54:56.976 INFO 2981023 --- [ocp-schedules-20,2966f4ba9a59ef3a,4745107c3a00d978] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:54:57.322 INFO 2981023 --- [ocp-async-9,388d236b939a22ce,93e10ffc91f73e5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:57.767 INFO 2981023 --- [ocp-async-4,2dd78acf528d5593,ca936f955fb11bd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:54:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,9ea62537e7752e14,e77b2c0e0c23bffb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048500, interval=60 2024-09-11 17:54:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,9ea62537e7752e14,e77b2c0e0c23bffb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:54:59.322 INFO 2981023 --- [ocp-async-1,ba7edf232d5c1d5b,755aff103711ea83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:54:59.589 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:54:59.590 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:00.721 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:50:01.0, nextRun=2024-09-11 17:55:00.0, currentDate=Wed Sep 11 17:55:00 HKT 2024 2024-09-11 17:55:00.721 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:50:01.0, nextRunTime=2024-09-11 17:55:00.0} 2024-09-11 17:55:00.727 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 17:55:00.728 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:55:00.728 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T17:55:00.728+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:55:00.739 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:50:01.0, nextRun=2024-09-11 17:55:00.0, currentDate=Wed Sep 11 17:55:00 HKT 2024 2024-09-11 17:55:00.739 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:50:01.0, nextRunTime=2024-09-11 17:55:00.0} 2024-09-11 17:55:00.757 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2, 1000006] 2024-09-11 17:55:00.766 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 17:55:00.766 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 17:55:00.767 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:55:00.767 INFO 2981023 --- [task-coordinator3,ddd72be6e49ae268,bcdf43e36f3d60c9] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2,1000006]]}}, createTime=null, startTime=2024-09-11T17:55:00.767+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 17:55:01.167 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,d512540fccc63867] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048179, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:01.181 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,d512540fccc63867] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048180, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:01.253 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048179, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048196, task_operation=execute, latest_execution_start_time=2024-09-11T17:55:01.149+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048179}, listMap={}}, executor=10.31.35.10 2024-09-11 17:55:01.270 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048180, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048197, task_operation=execute, latest_execution_start_time=2024-09-11T17:55:01.180+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048180}, listMap={cluster_ids=[[1,2,1000006]]}}, executor=10.31.35.10 2024-09-11 17:55:01.281 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 17:55:01.297 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:55:01.326 INFO 2981023 --- [ocp-async-6,a5e32815df62aa55,ce8a0b53f7ca1fb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:01.335 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 17:55:01.355 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 17:55:01.377 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 17:55:01.378 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 17:55:01.378 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 17:55:01.412 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 17:55:01.422 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 17:55:01.422 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1000006 2024-09-11 17:55:01.439 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:55:01.442 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,817f3a425be80756] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048180, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:01.492 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048197, status:SUCCESS 2024-09-11 17:55:01.498 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048197, status:SUCCESS 2024-09-11 17:55:01.554 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:55:01.571 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:55:01.601 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:55:01.608 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 17:55:01.608 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 17:55:01.619 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:55:01.723 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:55:01.772 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 17:55:01.799 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:55:01.816 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:55:01.834 INFO 2981023 --- [ocp-schedules-29,ad7cd91ba81b6a19,e7092c5d516f1ec9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:55:01.892 WARN 2981023 --- [scanner-13570621-1,ad7cd91ba81b6a19,c9b691983787dc8f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:55:01.916 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:55:01.945 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:55:02.040 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:55:02.047 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 17:55:02.047 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 17:55:02.052 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:55:02.140 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:55:02.173 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 17:55:02.184 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:55:02.190 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:55:02.199 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:55:02.205 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:55:02.211 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:55:02.212 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 17:55:02.213 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:55:02.226 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:55:02.235 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 17:55:02.245 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,f7e7fc363f398519] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048179, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:02.287 INFO 2981023 --- [ocp-schedules-2,81f25c2ac8f5d739,7a276efa3a3f3ae7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:55:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,f50eb1ad5830b117,c43e5defe13b366c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,f50eb1ad5830b117,c43e5defe13b366c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:02.511 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048196, status:SUCCESS 2024-09-11 17:55:02.511 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048196, status:SUCCESS 2024-09-11 17:55:02.590 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:02.591 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:03.322 INFO 2981023 --- [ocp-async-7,63a808bdb8c5c512,0fa5f72118dca762] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,00e3b5a6c47861bc,80a759b453aaf7c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,00e3b5a6c47861bc,80a759b453aaf7c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:05.322 INFO 2981023 --- [ocp-async-10,69cbd40cac424dad,0bdd6ef5c56e65b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:05.591 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:05.591 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:06.351 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:51:05.949, end=2024-09-11T17:53:06.347} 2024-09-11 17:55:06.358 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:55:06.357972+08:00, lastProcessTime=2024-09-11T17:54:56.341116+08:00 2024-09-11 17:55:06.359 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obCluste 2024-09-11 17:55:06.360 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:55:06.362 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:05.949, end=2024-09-11T17:53:06.347},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:55:06.362 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:55:06.362 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:55:06.374 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:55:06.395 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106395000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.395Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.398 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:55:06.398 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.404 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.404 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.406 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.406 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.412 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.412 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.413 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:55:06.413 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.395, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.413 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:55:06.414 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:05.949, end=2024-09-11T17:53:06.347},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:55:06.414 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:55:06.414 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:55:06.415 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:55:06.418 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:05.949, end=2024-09-11T17:53:06.347},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:55:06.418 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:55:06.418 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:55:06.460 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106460000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.460Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.466 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106465000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.465Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.467 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106467000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.467Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.481 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106481000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.481Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.482 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106482000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.482Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.482 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 64 2024-09-11 17:55:06.482 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.486 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106486000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.486Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.486 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106486000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.486Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.487 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106487000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.487Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.493 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.493 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.494 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.494 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.499 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:55:06.499 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.503 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.503 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.505 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.505 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.509 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:55:06.509 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.513 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.513 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.515 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.515 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.516 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.516 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.516 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:55:06.516 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.517 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.517 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.521 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.521 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.523 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.523 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.526 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.526 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.527 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.527 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.528 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.528 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.529 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.529 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.529 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.531 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.531 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.532 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.532 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.533 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.533 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.533 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.533 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.534 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.536 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.46, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.537 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.537 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.538 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.538 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.538 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.538 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.541 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.541 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.543 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.543 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.543 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.545 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.465, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.551 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.551 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.551 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.551 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.552 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.552 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.554 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.555 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.555 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.559 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.486, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.560 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.467, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.560 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.560 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.562 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.481, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.564 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.482, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.564 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.566 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.566 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.569 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.573 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.487, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.577 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.486, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.578 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106578000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.578Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.594 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:55:06.594 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.600 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106600000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.600Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.610 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106610000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.610Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.611 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106611000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.611Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.613 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106613000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.613Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.614 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.614 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.617 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106617000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.617Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.622 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106622000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.622Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.622 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:55:06.622 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.628 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962106628000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:55:06.628Z, endTime=2024-09-11T09:53:06.347Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:55:06.632 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.632 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.636 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.636 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.637 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.637 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.639 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.639 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.640 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.640 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.645 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.645 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.647 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.647 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.652 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.652 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.654 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.654 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.654 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.658 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:55:06.658 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:55:06.658 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.658 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.662 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.578, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.662 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:55:06.357972+08:00 2024-09-11 17:55:06.662 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.662 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.662 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.662 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.665 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.665 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.665 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.665 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.665 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.665 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.666 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.666 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.667 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.667 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.671 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.672 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.672 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.673 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.673 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.673 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.673 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.677 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.677 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.677 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.677 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.679 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.6, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.680 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.680 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.680 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.681 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:55:06.681 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:55:06.683 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.685 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.611, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.685 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.685 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.685 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.685 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.686 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.687 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.688 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.622, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.688 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.688 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.690 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.690 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.613, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.692 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:55:06.692 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048386347000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:55:06.692 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.617, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.693 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:55:06.693 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.628, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.695 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:55:06.61, end=2024-09-11T17:53:06.347}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:55:06.696 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:55:06.696 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:05.949, end=2024-09-11T17:53:06.347},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:55:06.696 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:55:06.696 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:55:06.703 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:55:06.703 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:05.949, end=2024-09-11T17:53:06.347},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:55:06.703 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:55:06.703 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:55:06.704 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:55:06.706 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:05.949, end=2024-09-11T17:53:06.347},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:55:06.706 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:55:06.706 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:55:06.715 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:55:06.715 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:55:07.323 INFO 2981023 --- [ocp-async-2,6eb17230adcff9f8,aca7c067769e94a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:07.602 INFO 2981023 --- [ocp-schedules-19,94985888442a0986,68e459457c608138] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:55:07.666 INFO 2981023 --- [ocp-schedules-21,41f4bac112eba7cb,9fa1028b7d42518e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:55:07.666+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:55:07.667 INFO 2981023 --- [ocp-schedules-21,41f4bac112eba7cb,9fa1028b7d42518e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:55:07.668 INFO 2981023 --- [ocp-schedules-21,41f4bac112eba7cb,9fa1028b7d42518e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:55:07.669 INFO 2981023 --- [ocp-schedules-21,41f4bac112eba7cb,9fa1028b7d42518e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:55:07.767 INFO 2981023 --- [ocp-async-5,34650bde5641726a,e3956b38d8c6eb83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:55:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,d45ea942bd5b3dbd,a422a6f2d0fd9c11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,d45ea942bd5b3dbd,a422a6f2d0fd9c11] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:08.592 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:08.592 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:09.323 INFO 2981023 --- [ocp-async-9,b0ec9492c83415ea,59279f801d1e1a5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:10.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:55:10.003 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:55:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:55:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 15, ruleSize = 15. 2024-09-11 17:55:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,daf728ae345e11a0,c72da4cea073a0af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,daf728ae345e11a0,c72da4cea073a0af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:11.323 INFO 2981023 --- [ocp-async-8,134bd393f52bbdbb,541b8c41034fbfef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:11.592 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:11.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:12.044 INFO 2981023 --- [ocp-schedules-15,512786436f0c22a9,5b0974660c71d800] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:55:12.101 WARN 2981023 --- [scanner-13572387-1,512786436f0c22a9,6302f24cace80ec8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:55:12.118 INFO 2981023 --- [alarm-detect-5,512786436f0c22a9,68974aacbc08473b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644974, data is empty. 2024-09-11 17:55:12.122 INFO 2981023 --- [alarm-detect-5,512786436f0c22a9,68974aacbc08473b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644995, data is empty. 2024-09-11 17:55:12.126 INFO 2981023 --- [alarm-detect-5,512786436f0c22a9,68974aacbc08473b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644995, data is empty. 2024-09-11 17:55:12.144 INFO 2981023 --- [alarm-detect-12,512786436f0c22a9,5b0effdf392a75b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644974, data is empty. 2024-09-11 17:55:12.147 INFO 2981023 --- [alarm-detect-2,512786436f0c22a9,e9d21d8612d2bb43] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644974, data is empty. 2024-09-11 17:55:12.150 INFO 2981023 --- [alarm-detect-2,512786436f0c22a9,e9d21d8612d2bb43] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644995, data is empty. 2024-09-11 17:55:12.152 INFO 2981023 --- [alarm-detect-12,512786436f0c22a9,5b0effdf392a75b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644995, data is empty. 2024-09-11 17:55:12.156 INFO 2981023 --- [alarm-detect-2,512786436f0c22a9,e9d21d8612d2bb43] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644995, data is empty. 2024-09-11 17:55:12.165 INFO 2981023 --- [alarm-detect-12,512786436f0c22a9,5b0effdf392a75b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644995, data is empty. 2024-09-11 17:55:12.284 INFO 2981023 --- [alarm-detect-5,512786436f0c22a9,cc5f5f4a446550bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644974, data is empty. 2024-09-11 17:55:12.290 INFO 2981023 --- [ocp-schedules-27,885f473a32617188,cbe0d751273e8049] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:55:12.293 INFO 2981023 --- [alarm-detect-5,512786436f0c22a9,cc5f5f4a446550bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644995, data is empty. 2024-09-11 17:55:12.301 INFO 2981023 --- [alarm-detect-5,512786436f0c22a9,cc5f5f4a446550bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644995, data is empty. 2024-09-11 17:55:12.422 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048440, nodeCount=820, affectRows=820 2024-09-11 17:55:12.422 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048440, nodeCount=987, affectRows=987 2024-09-11 17:55:12.424 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048440, nodeCount=819, validCount=819 2024-09-11 17:55:12.424 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048440, nodeCount=987, validCount=987 2024-09-11 17:55:13.323 INFO 2981023 --- [ocp-async-1,d3bb48260e530849,7db820bf9e30087e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:14.074 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048440, nodeCount=463, affectRows=463 2024-09-11 17:55:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,007d86e438fa1bd0,b7ae7e932977acf0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,43580daf308571d4,bac38978c7ea7a38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:55:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,007d86e438fa1bd0,b7ae7e932977acf0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:14.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:14.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:14.636 INFO 2981023 --- [ocp-schedules-16,6263f00e4bf26fd8,66195eb007cdcba4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:55:14.647 INFO 2981023 --- [ocp-schedules-16,6263f00e4bf26fd8,66195eb007cdcba4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:55:14.647 INFO 2981023 --- [ocp-schedules-16,6263f00e4bf26fd8,66195eb007cdcba4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:55:15.323 INFO 2981023 --- [ocp-async-3,1acc0c3398b15c8f,6aa29a60f8e5ed34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:15.801 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,f1ef27cda87d539b,dd7453626daf5496] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/validate, client 10.31.33.41, traceId f1ef27cda87d539b] 2024-09-11 17:55:15.802 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,f1ef27cda87d539b,dd7453626daf5496] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/validate?null, client=10.31.33.41, traceId=f1ef27cda87d539b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.validateConnect(ValidateCredentialParam), args=ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=null, batchSshValidateProperty=BatchSshValidateProperty(addressPairs=[BatchSshValidateProperty.AddressPair(hostAddress=10.31.35.7, sshPort=22, vpcId=1)], type=null, username=null, passphrase=null, allowSudo=true, changePassphrase=false, credentialId=6)),] 2024-09-11 17:55:15.825 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048440, nodeCount=463, validCount=463 2024-09-11 17:55:15.830 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,f1ef27cda87d539b,dd7453626daf5496] c.o.o.s.c.CredentialServiceImpl : Batch validate credential SshSecret(type=PASSWORD, allowSudo=true, username=root, hosts=[Host(id=2, name=null, alias=null, description=null, operatingSystem=null, operatingSystemRelease=null, architecture=null, serialNumber=null, innerIpAddress=10.31.35.5, sshPort=22, kind=null, publishPorts=null, services=null, hostTags=null, status=null, operatingTaskInstanceId=null, statusAnalysis=null, vpcId=1, vpcName=null, idcId=0, idcName=null, idcDescription=null, typeId=0, typeName=null, typeDescription=null, regionId=0, regionName=null, regionDescription=null, createTime=null, updateTime=null, hostAgentId=0, hostAgentVersion=null, hostAgentStatus=null, checkSystemInfoStatus=null, checkSystemInfoTaskId=null, checkSystemInfoRiskyCount=0, ioPerformance=null, tags=null), Host(id=2000007, name=null, alias=null, description=null, operatingSystem=null, operatingSystemRelease=null, architecture=null, serialNumber=null, innerIpAddress=10.31.35.6, sshPort=22, kind=null, publishPorts=null, services=null, hostTags=null, status=null, operatingTaskInstanceId=null, statusAnalysis=null, vpcId=1, vpcName=null, idcId=0, idcName=null, idcDescription=null, typeId=0, typeName=null, typeDescription=null, regionId=0, regionName=null, regionDescription=null, createTime=null, updateTime=null, hostAgentId=0, hostAgentVersion=null, hostAgentStatus=null, checkSystemInfoStatus=null, checkSystemInfoTaskId=null, checkSystemInfoRiskyCount=0, ioPerformance=null, tags=null), Host(id=2000009, name=null, alias=null, description=null, operatingSystem=null, operatingSystemRelease=null, architecture=null, serialNumber=null, innerIpAddress=10.31.35.10, sshPort=22, kind=null, publishPorts=null, services=null, hostTags=null, status=null, operatingTaskInstanceId=null, statusAnalysis=null, vpcId=1, vpcName=null, idcId=0, idcName=null, idcDescription=null, typeId=0, typeName=null, typeDescription=null, regionId=0, regionName=null, regionDescription=null, createTime=null, updateTime=null, hostAgentId=0, hostAgentVersion=null, hostAgentStatus=null, checkSystemInfoSt 2024-09-11 17:55:15.831 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:15.851 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:15.892 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:15.911 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:15.952 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:15.970 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:15.970 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:16.016 INFO 2981023 --- [ocp-schedules-9,a336686bf6ca7149,c6e9b948b5e001cd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:55:16.016 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:55:16.017 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:55:16.017 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:55:16.021 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:16.025 INFO 2981023 --- [ocp-schedules-24,0bc8384682619849,64a151c451440b1f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:55:16.026 INFO 2981023 --- [host-check-agent-status1,0bc8384682619849,264a97d1232ed8a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:55:16.026 INFO 2981023 --- [host-check-agent-status2,0bc8384682619849,9ff052eda254c485] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:55:16.026 INFO 2981023 --- [host-check-agent-status3,0bc8384682619849,c52fb22ac4a7f98b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:55:16.027 INFO 2981023 --- [ocp-schedules-19,df6f8fbf9fa862af,a07b1d6712c91623] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:55:16.027 INFO 2981023 --- [host-check-clockdiff1,df6f8fbf9fa862af,acda0d03e35e7b76] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:55:16.027 INFO 2981023 --- [host-check-clockdiff2,df6f8fbf9fa862af,eab5bffdebc6dfcc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:55:16.028 INFO 2981023 --- [host-check-clockdiff3,df6f8fbf9fa862af,ab802ef6fed16a42] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:55:16.038 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:16.048 INFO 2981023 --- [host-check-clockdiff2,df6f8fbf9fa862af,eab5bffdebc6dfcc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048516 4 4 , error= 2024-09-11 17:55:16.051 INFO 2981023 --- [host-check-clockdiff3,df6f8fbf9fa862af,ab802ef6fed16a42] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048516 0 0 , error= 2024-09-11 17:55:16.059 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:55:16.059 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:55:16.060 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:55:16.064 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:55:16.064 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:55:16.064 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:55:16.083 INFO 2981023 --- [host-check-agent-status1,0bc8384682619849,264a97d1232ed8a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:55:16.084 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 10.31.35.7 2024-09-11 17:55:16.084 INFO 2981023 --- [host-check-agent-status2,0bc8384682619849,9ff052eda254c485] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:55:16.085 INFO 2981023 --- [host-check-agent-status1,0bc8384682619849,264a97d1232ed8a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:55:16.089 INFO 2981023 --- [host-check-agent-status2,0bc8384682619849,9ff052eda254c485] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:55:16.113 INFO 2981023 --- [host-check-agent-status3,0bc8384682619849,c52fb22ac4a7f98b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:55:16.119 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:16.119 INFO 2981023 --- [pool-554-thread-1,f1ef27cda87d539b,7c467ea3bd502a54] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:16.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,f1ef27cda87d539b,dd7453626daf5496] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/validate client=10.31.33.41, traceId=f1ef27cda87d539b, duration=322 ms] 2024-09-11 17:55:16.172 INFO 2981023 --- [host-check-agent-status3,0bc8384682619849,c52fb22ac4a7f98b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:55:16.193 INFO 2981023 --- [ocp-schedules-16,51582b25f2763503,4467d62e9b4db5a3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:55:16.201 INFO 2981023 --- [ocp-schedules-5,36cee1cdf8133371,3e043e45c337d63a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:55:16.203 INFO 2981023 --- [ocp-schedules-18,c7d56e6a32d5500b,f882b9bedceafb92] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:55:16.203 INFO 2981023 --- [ocp-schedules-23,b07e201128096ddc,bd92f63c208f6647] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:55:16.269 INFO 2981023 --- [ocp-schedules-16,51582b25f2763503,4467d62e9b4db5a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:55:16.280 INFO 2981023 --- [ocp-schedules-18,c7d56e6a32d5500b,f882b9bedceafb92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:55:16.281 INFO 2981023 --- [ocp-schedules-5,36cee1cdf8133371,3e043e45c337d63a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:55:16.282 INFO 2981023 --- [ocp-schedules-5,36cee1cdf8133371,3e043e45c337d63a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:55:16.370 INFO 2981023 --- [ocp-schedules-4,9b29c94895c7eb2e,7c67029d304f27a7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:55:16.374 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:55:16.372574+08:00, lastProcessTime=2024-09-11T17:55:06.357972+08:00 2024-09-11 17:55:16.374 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:55:16.376 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:55:16.376234401+08:00 2024-09-11 17:55:16.376 INFO 2981023 --- [host-check-agent-status2,0bc8384682619849,9ff052eda254c485] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:55:16.376 INFO 2981023 --- [host-check-agent-status3,0bc8384682619849,c52fb22ac4a7f98b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:55:16.377 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:55:16.376+08:00 2024-09-11 17:55:16.380 INFO 2981023 --- [host-check-agent-status3,0bc8384682619849,c52fb22ac4a7f98b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:55:16.380 INFO 2981023 --- [host-check-agent-status3,0bc8384682619849,c52fb22ac4a7f98b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:55:16.380 INFO 2981023 --- [host-check-agent-status3,0bc8384682619849,c52fb22ac4a7f98b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:55:16.381 INFO 2981023 --- [host-check-agent-status2,0bc8384682619849,9ff052eda254c485] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:55:16.381 INFO 2981023 --- [host-check-agent-status2,0bc8384682619849,9ff052eda254c485] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:55:16.381 INFO 2981023 --- [host-check-agent-status2,0bc8384682619849,9ff052eda254c485] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:55:16.382 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:55:16.376+08:00 and status AVAILABLE 2024-09-11 17:55:16.383 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:55:16.383 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:55:16.384 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:55:16.388567181+08:00 2024-09-11 17:55:16.385 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:55:16.384+08:00 2024-09-11 17:55:16.386 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:55:16.389722294+08:00 2024-09-11 17:55:16.388 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:55:16.384+08:00 and status AVAILABLE 2024-09-11 17:55:16.388 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.388 INFO 2981023 --- [host-check-agent-status1,0bc8384682619849,264a97d1232ed8a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:55:16.389 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:55:16.386+08:00 2024-09-11 17:55:16.389 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:16.390 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.390 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:16.391 INFO 2981023 --- [host-check-agent-status1,0bc8384682619849,264a97d1232ed8a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:55:16.391 INFO 2981023 --- [host-check-agent-status1,0bc8384682619849,264a97d1232ed8a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:55:16.391 INFO 2981023 --- [host-check-agent-status1,0bc8384682619849,264a97d1232ed8a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:55:16.391 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:55:16.393 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.393 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.393 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:55:16.393 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:55:16.394 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:55:16.386+08:00 and status AVAILABLE 2024-09-11 17:55:16.394 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:16.394 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:16.395 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:55:16.395 INFO 2981023 --- [ocp-schedules-16,51582b25f2763503,4467d62e9b4db5a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:55:16.400 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.401 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:16.405 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.406 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:16.416 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:55:16.417 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:55:16.421 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:55:16.456 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:55:16.458 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:55:16.458 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:55:16.463 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:55:16.464 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:55:16.475 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.476 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.477 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:55:16.480 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:55:16.481 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:55:16.491 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:55:16.492 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hosts/batchCreate, client 10.31.33.41, traceId 0fec2275d3095882] 2024-09-11 17:55:16.493 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.493 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hosts/batchCreate?null, client=10.31.33.41, traceId=0fec2275d3095882, method=IterableResponse com.oceanbase.ocp.server.common.controller.ComputeController.batchCreateHost(BatchCreateHostRequest), args=BatchCreateHostRequest(hostBasicDataList=[BatchCreateHostRequest.HostBasicData(serialNumber=null, innerIpAddress=10.31.35.7)], sshPort=22, kind=DEDICATED_ECS, publishPorts=null, idcId=1, vpcId=1, typeId=2, credentialId=6, alias=null, description=null, mgragentPort=null, monagentPort=null),] 2024-09-11 17:55:16.494 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.494 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:55:16.497 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=batch-create-host-2136169906, method=batchCreateHost 2024-09-11 17:55:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.516 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:55:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:16.518 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:55:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.519 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:55:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:55:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:55:16.527 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:55:16.532 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:55:16.536 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:55:16.536 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:16.536 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:55:16.554 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:55:16.571 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.571 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:16.580 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:55:16.585 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:16.591 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:16.592 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.593 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:55:16.595 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:55:16.595 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:55:16.596 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:55:16.599 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:55:16.602 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Prepare host, nodeCount=8, taskCount=8 2024-09-11 17:55:16.602 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:55:16.602 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Prepare host', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=null, tenantId=null, hostId=2000010, creator='admin', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, prohibit_rollback=false, host_credential_id=6, host_id=2000010}, listMap={}}, createTime=null, startTime=2024-09-11T17:55:16.602+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Create random agent credential, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Pre check for create host, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.PreCreateHostCheckTask, seriesId=6, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update host arch info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateHostSystemInfoTask, seriesId=7, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Uninstall legacy obagent, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obagent.UninstallLegacyObAgentTask, seriesId=2, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update install agent info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateInstallAgentInfoTask, seriesId=0, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Install ocp agent, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask, seriesId=4, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update host status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateHostStatusTask, seriesId=3, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Uninstall legacy ocp agent, state= 2024-09-11 17:55:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:55:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:55:16.615 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:16.615 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:55:16.622 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:55:16.625 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:55:16.625 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:55:16.637 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:55:16.638 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:55:16.642 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:16.655 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:55:16.664 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:55:16.664 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.665 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.665 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:16.666 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:55:16.666 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=4048198, taskName=Prepare host, argument=Argument{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, prohibit_rollback=false, host_credential_id=6, host_id=2000010}, listMap={}} 2024-09-11 17:55:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:55:16.673 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:55:16.676 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.o.c.host.service.HostServiceImpl : Batch creating host: [2000010:10.31.35.7], current status [NEW] 2024-09-11 17:55:16.681 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:55:16.684 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:55:16.689 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:55:16.690 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:55:16.694 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.695 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:16.697 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:55:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.700 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:55:16.702 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:55:16.706 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:55:16.708 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=ADD_HOSTS, eventType=COMPUTE_HOST_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ComputeController.batchCreateHost, resourceType=COMPUTE_HOST, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=10.31.35.7, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=4048198, eventInfo={"hostRequest":{"credentialId":6,"hostBasicDataList":[{"innerIpAddress":"10.31.35.7"}],"idcId":1,"kind":"DEDICATED_ECS","sshPort":22,"typeId":2,"vpcId":1}}, executionType=NEW, requestUrl=http://pms-cas.linkreit.com:15002/api/v2/compute/hosts/batchCreate, clientIp=10.31.33.41, serverIp=10.31.35.10, startTime=2024-09-11T17:55:16.494+08:00, endTime=null, role=null, aliUid=null) 2024-09-11 17:55:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:55:16.719 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:55:16.724 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:55:16.729 INFO 2981023 --- [host-check-host-status3,a336686bf6ca7149,6fc3891338436abf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:55:16.731 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:55:16.732 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:55:16.733 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.733 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.734 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:55:16.734 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:55:16.736 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:16.741 INFO 2981023 --- [host-check-host-status1,a336686bf6ca7149,4112cf9e07da4d97] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:55:16.743 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:55:16.744 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:55:16.744 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.745 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:55:16.748 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0fec2275d3095882,997b8df64cf2f77a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hosts/batchCreate client=10.31.33.41, traceId=0fec2275d3095882, duration=261 ms] 2024-09-11 17:55:16.750 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:55:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:55:16.754 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:55:16.768 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:55:16.772 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:55:16.372574+08:00 2024-09-11 17:55:16.786 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:55:16.791 INFO 2981023 --- [host-check-host-status2,a336686bf6ca7149,aca817f9e0d6075f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:55:16.798 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:55:16.799 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:55:16.805 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.805 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.806 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:55:16.807 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:16.807 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:16.808 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:55:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:55:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:55:16.814 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:55:16.814 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:55:16.814 INFO 2981023 --- [ocp-schedules-18,c7d56e6a32d5500b,f882b9bedceafb92] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:55:16.982 INFO 2981023 --- [host-check-clockdiff1,df6f8fbf9fa862af,acda0d03e35e7b76] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048516 4 4 , error= 2024-09-11 17:55:17.258 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,01fabe1d4281a5bb,eaed0ace6f988c54] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=01fabe1d4281a5bb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:55:17.259 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,01fabe1d4281a5bb,eaed0ace6f988c54] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:55:17.296 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,01fabe1d4281a5bb,eaed0ace6f988c54] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=01fabe1d4281a5bb, duration=41 ms] 2024-09-11 17:55:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,d2715f8d29d3f930,a4fb29ff3af45fa5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,d2715f8d29d3f930,a4fb29ff3af45fa5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:17.322 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,13af4b93d6ad5f4f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048182, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:17.323 INFO 2981023 --- [ocp-async-10,4d47ddeb99e3c031,2959557418882154] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:17.364 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048182, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, task_instance_id=4048198, task_operation=execute, host_credential_id=6, latest_execution_start_time=2024-09-11T17:55:17.318+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=4048182, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:55:17.387 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:17.408 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:17.414 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=10.31.35.7 2024-09-11 17:55:17.450 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:17.472 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:17.513 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 10.31.35.7 2024-09-11 17:55:17.539 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:17.543 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:17.546 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2024-09-11 17:55:17.550 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=10.31.35.7, port=22, loopCount=1, minSuccessCount=1 2024-09-11 17:55:17.581 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:17.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:17.595 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:17.604 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:17.646 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 10.31.35.7 2024-09-11 17:55:17.674 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:17.678 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:17.682 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[126] 2024-09-11 17:55:17.686 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=126, maxAcceptableDelayMillis=3000 2024-09-11 17:55:17.691 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:55:17.767 INFO 2981023 --- [ocp-async-6,444273d8202b7625,c2dfaeb580711ccb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:55:19.323 INFO 2981023 --- [ocp-async-4,2acd6602c6fd4bdc,20cd0ce691e09cb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,6e4ee91970c2e7f1,917d2f042ce211eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,6e4ee91970c2e7f1,917d2f042ce211eb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:20.595 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:20.597 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:21.324 INFO 2981023 --- [ocp-async-6,f69ef1f9908aebb7,debf1017852909f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:22.293 INFO 2981023 --- [ocp-schedules-9,8617d59d3d1fcec2,b33e5c5d30b690ec] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:55:22.349 INFO 2981023 --- [ocp-schedules-22,ac296af8937ece5a,4b3309431a6c4014] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:55:22.401 WARN 2981023 --- [scanner-13572719-1,ac296af8937ece5a,8774f8585c1cd98f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:55:22.861 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,80fba455aeebadb9,f152855580685209] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=80fba455aeebadb9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:55:22.862 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,80fba455aeebadb9,f152855580685209] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:55:22.904 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,80fba455aeebadb9,f152855580685209] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=80fba455aeebadb9, duration=48 ms] 2024-09-11 17:55:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,acf694c4f2c273b1,cef955e0c727364d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,acf694c4f2c273b1,cef955e0c727364d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:23.324 INFO 2981023 --- [ocp-async-5,6d3fd8ddfa997810,c3e7f54658cafda3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:23.468 INFO 2981023 --- [ocp-schedules-16,36f7fb4f11cffff5,4678fcce47ac5d80] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:55:23.468 INFO 2981023 --- [ocp-schedules-16,36f7fb4f11cffff5,4678fcce47ac5d80] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:55:23.468 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:55:23.468 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:55:23.468 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 17:55:23.493 INFO 2981023 --- [ocp-schedules-16,36f7fb4f11cffff5,4678fcce47ac5d80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:55:23.493 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:55:23.494 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:55:23.496 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:55:23.497 INFO 2981023 --- [ocp-schedules-16,36f7fb4f11cffff5,4678fcce47ac5d80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:55:23.498 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:55:23.556 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:23.557 WARN 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2000010 2024-09-11 17:55:23.558 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:23.578 WARN 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2000010 2024-09-11 17:55:23.597 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:23.667 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:23.675 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048523 1 1 , error= 2024-09-11 17:55:23.687 INFO 2981023 --- [ocp-schedules-16,36f7fb4f11cffff5,4678fcce47ac5d80] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:55:23.683 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=1, maxDiff=50 2024-09-11 17:55:23.691 ERROR 2981023 --- [ocp-schedules-16,36f7fb4f11cffff5,4678fcce47ac5d80] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:64) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:118) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor2097.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:62) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2000010 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1672) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1553) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:376) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1082) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$27aa8d6b.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:99) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:55:23.695 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:55:23.698 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:55:23.698 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,4ebb095aa5395bcd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048182, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:23.860 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048183] 2024-09-11 17:55:24.326 INFO 2981023 --- [ocp-schedules-22,9afa8da6a3c0e9fa,b4c5dd3244b703ba] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:55:24.363 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,467af47b730d6f21] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048183, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:24.412 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048183, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, task_instance_id=4048198, task_operation=execute, host_credential_id=6, latest_execution_start_time=2024-09-11T17:55:24.361+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=4048183, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:55:24.416 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2024-09-11 17:55:24.445 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:24.468 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:24.510 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:24.531 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:24.573 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 10.31.35.7 2024-09-11 17:55:24.608 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "vm-eas-prd-rtd-mid05", "release": "4.18.0-425.3.1.el8.x86_64", "version": "#1 SMP Fri Sep 30 11:45:06 EDT 2022", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "rhel", "linux_distribution_release": "8.7"}, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:24.611 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "vm-eas-prd-rtd-mid05", "release": "4.18.0-425.3.1.el8.x86_64", "version": "#1 SMP Fri Sep 30 11:45:06 EDT 2022", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "rhel", "linux_distribution_release": "8.7"}, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:24.645 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2024-09-11 17:55:24.656 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,a54f3f495d7e0c4c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048183, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:24.876 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048184] 2024-09-11 17:55:25.324 INFO 2981023 --- [ocp-async-2,2a07a870c88a5b6d,abfc334d91372ef2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:25.391 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,2b3eb42cde65dbe3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048184, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:25.424 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048184, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, task_instance_id=4048198, task_operation=execute, host_credential_id=6, latest_execution_start_time=2024-09-11T17:55:25.389+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=4048184, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:55:25.429 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2024-09-11 17:55:25.442 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:25.465 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:25.507 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:25.529 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:25.571 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:25.605 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:25.609 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:25.647 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:25.676 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:25.718 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.31.35.7 2024-09-11 17:55:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,92406ec5cfb99cbe,2db3b1e0f36356e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,92406ec5cfb99cbe,2db3b1e0f36356e9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:26.364 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:55:26.363722+08:00, lastProcessTime=2024-09-11T17:55:16.372574+08:00 2024-09-11 17:55:26.508 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:26.512 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:26.517 WARN 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2024-09-11 17:55:26.521 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2024-09-11 17:55:26.535 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:55:26.363722+08:00 2024-09-11 17:55:26.550 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:26.572 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:26.614 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.31.35.7 2024-09-11 17:55:26.667 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:26.667 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:27.324 INFO 2981023 --- [ocp-async-7,1a0cc2d6ca2dc083,f2904cb8932750cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:27.402 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:27.406 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:27.410 WARN 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2024-09-11 17:55:27.413 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2024-09-11 17:55:27.417 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2024-09-11 17:55:27.426 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,027773ae7e8c808b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048184, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:27.767 INFO 2981023 --- [ocp-async-2,debc3a9c39aa4c7c,2e4a249d58e0ab6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:55:27.827 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3212d538552d3ac4,4ba0e796c382f0a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=3212d538552d3ac4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:55:27.828 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3212d538552d3ac4,4ba0e796c382f0a0] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:55:27.871 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3212d538552d3ac4,4ba0e796c382f0a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=3212d538552d3ac4, duration=47 ms] 2024-09-11 17:55:27.918 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048188] 2024-09-11 17:55:28.420 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,3d86ad417e30bd6f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048188, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:28.461 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048188, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, task_instance_id=4048198, task_operation=execute, host_credential_id=6, latest_execution_start_time=2024-09-11T17:55:28.419+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=4048188, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:55:28.464 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2024-09-11 17:55:28.512 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:28.535 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.577 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:28.599 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.641 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:28.671 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.675 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.713 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:28.735 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.777 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:28.807 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.811 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.849 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:28.870 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.913 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:28.944 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.947 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:28.986 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:29.008 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.050 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:29.080 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.084 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.122 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:29.143 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.185 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:29.214 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.222 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.256 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:29.279 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,2f6d85758b881188,9966c041a37630f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:29.318 INFO 2981023 --- [prometheus-minute-schedule-1,2f6d85758b881188,9966c041a37630f5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:29.324 INFO 2981023 --- [ocp-async-5,ea7e483b5607912a,412ae31c270fe9a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:29.321 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:29.364 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.369 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.406 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:29.428 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.471 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:29.501 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.506 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.543 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:29.564 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.606 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:29.635 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.640 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.667 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:29.669 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:29.678 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:29.699 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.741 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:29.771 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.775 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.813 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:29.836 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:29.878 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7 2024-09-11 17:55:30.664 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:30.668 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:30.671 WARN 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-09-11 17:55:30.675 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-09-11 17:55:30.706 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:30.728 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:30.770 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:30.800 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:30.804 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:30.842 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:30.863 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:30.906 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:30.936 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:30.941 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:30.978 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:31.000 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:31.043 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:31.072 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:31.076 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:31.080 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2024-09-11 17:55:31.114 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:31.151 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:31.193 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7 2024-09-11 17:55:31.324 INFO 2981023 --- [ocp-async-2,6efa4890dd983286,275146cd55f3519c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:31.982 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:31.986 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:31.989 WARN 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2024-09-11 17:55:31.993 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2024-09-11 17:55:31.996 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2024-09-11 17:55:32.024 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:32.046 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:32.088 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.31.35.7 2024-09-11 17:55:32.295 INFO 2981023 --- [ocp-schedules-7,d703afe0c7cd3464,74a3f0d009d6a079] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:55:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,bd59170a1167f13b,ba74b55bca87522a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:32.315 INFO 2981023 --- [prometheus-minute-schedule-1,bd59170a1167f13b,ba74b55bca87522a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:32.552 INFO 2981023 --- [ocp-schedules-20,5cf97df95f97b867,486d4b41b6c8df01] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:55:32.605 WARN 2981023 --- [scanner-13572932-1,5cf97df95f97b867,a71ba1d4d0ea19d7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:55:32.637 INFO 2981023 --- [alarm-detect-16,5cf97df95f97b867,a5303d0c8c2eecc3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644995, data is empty. 2024-09-11 17:55:32.644 INFO 2981023 --- [alarm-detect-16,5cf97df95f97b867,a5303d0c8c2eecc3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644995, data is empty. 2024-09-11 17:55:32.648 INFO 2981023 --- [alarm-detect-16,5cf97df95f97b867,a5303d0c8c2eecc3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644995, data is empty. 2024-09-11 17:55:32.657 INFO 2981023 --- [alarm-detect-7,5cf97df95f97b867,dede7c82df9960a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644995, data is empty. 2024-09-11 17:55:32.657 INFO 2981023 --- [alarm-detect-3,5cf97df95f97b867,219e835483242ce2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644995, data is empty. 2024-09-11 17:55:32.661 INFO 2981023 --- [alarm-detect-3,5cf97df95f97b867,219e835483242ce2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644995, data is empty. 2024-09-11 17:55:32.664 INFO 2981023 --- [alarm-detect-7,5cf97df95f97b867,dede7c82df9960a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644995, data is empty. 2024-09-11 17:55:32.665 INFO 2981023 --- [alarm-detect-3,5cf97df95f97b867,219e835483242ce2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644995, data is empty. 2024-09-11 17:55:32.669 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:32.669 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:32.671 INFO 2981023 --- [alarm-detect-7,5cf97df95f97b867,dede7c82df9960a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644995, data is empty. 2024-09-11 17:55:32.765 INFO 2981023 --- [alarm-detect-6,5cf97df95f97b867,ad8b10847469cbf9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644995, data is empty. 2024-09-11 17:55:32.772 INFO 2981023 --- [alarm-detect-6,5cf97df95f97b867,ad8b10847469cbf9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644995, data is empty. 2024-09-11 17:55:32.776 INFO 2981023 --- [alarm-detect-6,5cf97df95f97b867,ad8b10847469cbf9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644995, data is empty. 2024-09-11 17:55:32.844 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,310612d69462ab90,992564e9d1abfa69] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=310612d69462ab90, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:55:32.845 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-8,310612d69462ab90,992564e9d1abfa69] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:55:32.875 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:32.885 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,310612d69462ab90,992564e9d1abfa69] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=310612d69462ab90, duration=43 ms] 2024-09-11 17:55:32.879 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:32.886 WARN 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2024-09-11 17:55:32.891 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2024-09-11 17:55:32.918 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:32.940 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:32.982 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.31.35.7 2024-09-11 17:55:33.020 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:33.024 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent/*'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:33.028 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2024-09-11 17:55:33.037 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,2dff897ea481a3a5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048188, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:33.324 INFO 2981023 --- [ocp-async-4,e665cb814345505f,06d9f4cd40a11560] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:33.990 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048181] 2024-09-11 17:55:34.460 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,2dd2bb66ab896d31] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048181, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:34.470 INFO 2981023 --- [ocp-schedules-22,0060ef3fa0020583,ed388a969af09db6] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY,C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkprdocp-1719994437/H_10.31.35.10,C_linkrtd-1719994443/H_10.31.35.6,C_rtdob-1718689631/H_10.31.35.5,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6, current targets ,,,,,C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY,C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkprdocp-1719994437/H_10.31.35.10,C_linkrtd-1719994443/H_10.31.35.6,C_rtdob-1718689631/H_10.31.35.5,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:34.495 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,b67a5675a7194c46] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048181, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, task_instance_id=4048198, task_operation=execute, host_credential_id=6, latest_execution_start_time=2024-09-11T17:55:34.458+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=4048181, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:55:34.499 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,b67a5675a7194c46] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=690c237d-d13d-4b1e-b96c-4a6b58614263, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=2000010, authType=OCP-HMACSHA1)) 2024-09-11 17:55:34.515 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,b67a5675a7194c46] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=690c237d-d13d-4b1e-b96c-4a6b58614263, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=2000010, authType=OCP-HMACSHA1)) 2024-09-11 17:55:34.526 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,b67a5675a7194c46] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:2000035 2024-09-11 17:55:34.535 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,b67a5675a7194c46] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048181, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:34.650 INFO 2981023 --- [ocp-schedules-3,974caa989207c374,c7b8cbad1a4357e0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:55:34.659 INFO 2981023 --- [ocp-schedules-3,974caa989207c374,c7b8cbad1a4357e0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:55:34.659 INFO 2981023 --- [ocp-schedules-3,974caa989207c374,c7b8cbad1a4357e0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:55:35.007 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048186] 2024-09-11 17:55:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,6021149ca1c2634d,262ea7b33d334dd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:35.318 INFO 2981023 --- [prometheus-minute-schedule-1,6021149ca1c2634d,262ea7b33d334dd7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:35.324 INFO 2981023 --- [ocp-async-3,07000030d5d3920e,c64cc4d4ff6daf61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:35.482 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,62b406f91ddb56ef] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048186, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:35.512 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048186, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, task_instance_id=4048198, task_operation=execute, host_credential_id=6, latest_execution_start_time=2024-09-11T17:55:35.480+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=4048186, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:55:35.516 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2024-09-11 17:55:35.520 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:35.527 WARN 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2000010 2024-09-11 17:55:35.531 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:35.536 WARN 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2000010 2024-09-11 17:55:35.540 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2024-09-11 17:55:35.543 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2024-09-11 17:55:35.589 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:35.613 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:35.656 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:35.670 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:35.671 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:35.678 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:35.721 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.31.35.7 2024-09-11 17:55:35.763 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:35.767 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:35.806 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:35.828 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:35.870 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.31.35.7 2024-09-11 17:55:35.913 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:35.918 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:35.932 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2024-09-11 17:55:35.955 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:35.977 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.019 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' 'http://10.31.35.10:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' on 10.31.35.7 2024-09-11 17:55:36.059 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d918c72ac71edb2c,cecb693c6443f461] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm?null, client=10.31.35.7, traceId=d918c72ac71edb2c, method=ResponseEntity com.oceanbase.ocp.server.common.controller.ObjectStorageController.downloadFile(HttpServletRequest,String,String), args=sys-package,ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm,] 2024-09-11 17:55:36.065 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d918c72ac71edb2c,cecb693c6443f461] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm, objectId=4000021 2024-09-11 17:55:36.065 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d918c72ac71edb2c,cecb693c6443f461] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm, sha1=96a52e728d75f568dbadf8cc6545c9b49fbd9cac 2024-09-11 17:55:36.065 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d918c72ac71edb2c,cecb693c6443f461] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/object-storage/sys-package/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm client=10.31.35.7, traceId=d918c72ac71edb2c, duration=7 ms] 2024-09-11 17:55:36.181 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' 'http://10.31.35.10:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' 'http://10.31.35.10:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.186 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' 'http://10.31.35.10:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' 'http://10.31.35.10:8080/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.224 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:36.252 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.294 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' | awk '{ print $1 }' on 10.31.35.7 2024-09-11 17:55:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:55:36.352541+08:00, lastProcessTime=2024-09-11T17:55:26.363722+08:00 2024-09-11 17:55:36.385 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' | awk '{ print $1 }' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' | awk '{ print $1 }', out=96a52e728d75f568dbadf8cc6545c9b49fbd9cac, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.389 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' | awk '{ print $1 }' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo sha1sum '/tmp/'/'ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm' | awk '{ print $1 }', out=96a52e728d75f568dbadf8cc6545c9b49fbd9cac, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.393 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.ocp.executor.executor.SshExecutor : download file complete, path=/tmp/ fileName=ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm 2024-09-11 17:55:36.428 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:36.451 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:55:36.352541+08:00 2024-09-11 17:55:36.493 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/44e46de520d8488e on 10.31.35.7 2024-09-11 17:55:36.517 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/44e46de520d8488e on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=mkdir -p /tmp/44e46de520d8488e, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.526 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/44e46de520d8488e on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=mkdir -p /tmp/44e46de520d8488e, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.559 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:36.589 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.631 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/44e46de520d8488e/install_package.sh && chmod +x /tmp/44e46de520d8488e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'' > /tmp/44e46de520d8488e/install_package.sh on 10.31.35.7 2024-09-11 17:55:36.655 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/44e46de520d8488e/install_package.sh && chmod +x /tmp/44e46de520d8488e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'' > /tmp/44e46de520d8488e/install_package.sh on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=touch /tmp/44e46de520d8488e/install_package.sh && chmod +x /tmp/44e46de520d8488e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'' > /tmp/44e46de520d8488e/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.661 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/44e46de520d8488e/install_package.sh && chmod +x /tmp/44e46de520d8488e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'' > /tmp/44e46de520d8488e/install_package.sh on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=touch /tmp/44e46de520d8488e/install_package.sh && chmod +x /tmp/44e46de520d8488e/install_package.sh && echo '#!/bin/bash ... install_package '/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'' > /tmp/44e46de520d8488e/install_package.sh, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.697 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:36.720 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:36.762 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/44e46de520d8488e/install_package.sh on 10.31.35.7 2024-09-11 17:55:37.314 INFO 2981023 --- [prometheus-cache-monitor1,4df1cb7b2ca11c8b,7b8d558426a1d4e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7226,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1808,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:55:37.315 INFO 2981023 --- [prometheus-cache-monitor1,4df1cb7b2ca11c8b,7b8d558426a1d4e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:55:37.325 INFO 2981023 --- [ocp-async-5,0f0b5d852d64666b,71dffab720df5d24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:37.767 INFO 2981023 --- [ocp-async-9,88ca73a4e579cb60,a2df0ed301a3f3fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:55:37.854 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a36dacf2b146f6c8,9a3eaf4f8a10b9ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=a36dacf2b146f6c8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:55:37.855 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a36dacf2b146f6c8,9a3eaf4f8a10b9ff] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:55:37.894 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a36dacf2b146f6c8,9a3eaf4f8a10b9ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=a36dacf2b146f6c8, duration=43 ms] 2024-09-11 17:55:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,aec093163f3bdb9a,e4c36206ce4c9c3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,aec093163f3bdb9a,e4c36206ce4c9c3e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:38.671 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:38.673 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:39.325 INFO 2981023 --- [ocp-async-7,c9fcab4b574fe2e5,bb9edb7cbe7a06fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:40.003 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:55:40.003 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:55:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,c61379bfce134d5c,a8b5d7852b97303a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,c61379bfce134d5c,a8b5d7852b97303a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:41.325 INFO 2981023 --- [ocp-async-9,91bce84ec971a352,3197e7de3446618b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:41.673 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:41.673 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:42.152 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/44e46de520d8488e/install_package.sh on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=/tmp/44e46de520d8488e/install_package.sh, out=Verifying... ######################################## Preparing... ######################################## Updating / installing... ocp-agent-ce-4.3.1-20240805192406.el7 ########################################, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.161 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/44e46de520d8488e/install_package.sh on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=/tmp/44e46de520d8488e/install_package.sh, out=Verifying... ######################################## Preparing... ######################################## Updating / installing... ocp-agent-ce-4.3.1-20240805192406.el7 ########################################, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.193 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:42.217 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.259 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: rm -rf /tmp/44e46de520d8488e on 10.31.35.7 2024-09-11 17:55:42.280 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: rm -rf /tmp/44e46de520d8488e on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=rm -rf /tmp/44e46de520d8488e, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.285 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: rm -rf /tmp/44e46de520d8488e on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=rm -rf /tmp/44e46de520d8488e, out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.288 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.ocp.executor.executor.SshExecutor : install complete, rpmName=/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm, result=Verifying... ######################################## Preparing... ######################################## Updating / installing... ocp-agent-ce-4.3.1-20240805192406.el7 ######################################## 2024-09-11 17:55:42.298 INFO 2981023 --- [ocp-schedules-13,0ddf3cb522538ecf,3ae434578cef1126] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:55:42.323 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:42.344 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.386 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'' on 10.31.35.7 2024-09-11 17:55:42.502 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.508 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c 'cp -r '"'"'/tmp//ocp-agent-ce-4.3.1-20240805192406.el7.x86_64.rpm'"'"' '"'"'/home/admin/ocp_agent/pkg_store/'"'"'', out=, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.515 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent success 2024-09-11 17:55:42.546 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:42.568 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.610 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx on 10.31.35.7 2024-09-11 17:55:42.681 INFO 2981023 --- [ocp-schedules-16,2d1561c0e3bc8580,63cbe88188179a8d] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:55:42.699 WARN 2981023 --- [ocp-schedules-16,2d1561c0e3bc8580,63cbe88188179a8d] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:55:42.757 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx, out={"successful":true,"message":"success","error":""}, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.761 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl -c '"'"'/home/admin/ocp_agent'"'"'/conf/agentctl.yaml config --update '"'"'agent.http.basic.auth.username=ocp_agent,agent.http.basic.auth.password=xxx, out={"successful":true,"message":"success","error":""}, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.765 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.s.c.AgentInstallationTaskService : start ocp-agent... 2024-09-11 17:55:42.799 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:42.811 INFO 2981023 --- [ocp-schedules-28,cec34eb4bf3fd583,5961704295f68ea7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:55:42.821 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:42.838 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,fab8276b9334c23e,68a500f956b4c379] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=fab8276b9334c23e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:55:42.841 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,fab8276b9334c23e,68a500f956b4c379] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:55:42.861 WARN 2981023 --- [scanner-13573218-1,cec34eb4bf3fd583,043af61d63905a81] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:55:42.864 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start' on 10.31.35.7 2024-09-11 17:55:42.913 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,fab8276b9334c23e,68a500f956b4c379] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=fab8276b9334c23e, duration=81 ms] 2024-09-11 17:55:43.325 INFO 2981023 --- [ocp-async-5,d1ca5e8f31ad0af5,dcf7e5be291d7e6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,c88b4aa52b32de38,d104684f8b5e5588] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,c88b4aa52b32de38,d104684f8b5e5588] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:44.673 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:44.674 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:45.101 INFO 2981023 --- [prometheus-second-schedule-1,a168e92df3501f3b,424767ae5ae55ef9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=7, collectTaskCount=7 2024-09-11 17:55:45.325 INFO 2981023 --- [ocp-async-4,f87a65618a2037a1,3464413b0b1445c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:46.021 INFO 2981023 --- [ocp-schedules-2,5153ce80e78d2b0f,cdb011dbc57d5a41] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:55:46.021 INFO 2981023 --- [ocp-schedules-10,f20954ef4e849197,13d3fb0387a28f03] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:55:46.021 INFO 2981023 --- [host-check-clockdiff1,5153ce80e78d2b0f,1bfeb3743f2d263d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:55:46.022 INFO 2981023 --- [host-check-clockdiff3,5153ce80e78d2b0f,367cd1eafcbe1975] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:55:46.022 INFO 2981023 --- [host-check-agent-status1,f20954ef4e849197,af1b1b44d44d2fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:55:46.022 INFO 2981023 --- [host-check-agent-status2,f20954ef4e849197,a2c1a58e2ed1d9df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:55:46.023 INFO 2981023 --- [host-check-agent-status3,f20954ef4e849197,c8771b2420a81169] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:55:46.023 INFO 2981023 --- [ocp-schedules-27,e8b3d334054befb5,d0d5db1f636c4747] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10] 2024-09-11 17:55:46.024 INFO 2981023 --- [host-check-clockdiff2,5153ce80e78d2b0f,ba48ae8b21d1c113] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:55:46.024 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:55:46.026 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:55:46.026 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:55:46.030 INFO 2981023 --- [host-check-agent-status2,f20954ef4e849197,a2c1a58e2ed1d9df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:55:46.036 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:55:46.036 INFO 2981023 --- [host-check-agent-status3,f20954ef4e849197,c8771b2420a81169] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:55:46.039 INFO 2981023 --- [host-check-clockdiff1,5153ce80e78d2b0f,1bfeb3743f2d263d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048546 6 6 , error= 2024-09-11 17:55:46.039 INFO 2981023 --- [host-check-agent-status1,f20954ef4e849197,af1b1b44d44d2fb4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:55:46.041 INFO 2981023 --- [host-check-agent-status1,f20954ef4e849197,af1b1b44d44d2fb4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:55:46.041 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:55:46.041 INFO 2981023 --- [host-check-clockdiff3,5153ce80e78d2b0f,367cd1eafcbe1975] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048546 0 0 , error= 2024-09-11 17:55:46.041 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:55:46.041 INFO 2981023 --- [host-check-agent-status2,f20954ef4e849197,a2c1a58e2ed1d9df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:55:46.042 INFO 2981023 --- [host-check-agent-status3,f20954ef4e849197,c8771b2420a81169] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:55:46.042 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:55:46.043 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:55:46.044 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:55:46.043 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start' on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start', out={"successful":true,"message":"ok","error":""}, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.172 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:55:46.183 INFO 2981023 --- [ocp-schedules-9,f107de8fa015c04b,06192ceb82ca3db2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:55:46.187 INFO 2981023 --- [ocp-schedules-16,22bec2ad7e195c25,4fa0a1e1a1088116] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:55:46.187 INFO 2981023 --- [ocp-schedules-15,1754e1096d6145db,8d29ea2dc84b80ce] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:55:46.190 INFO 2981023 --- [ocp-schedules-19,c5c584dc419261cb,ccf0002a7875535b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:55:46.190 INFO 2981023 --- [ocp-schedules-3,85ab3a3d3b33a207,6c2289b050e3cb1e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:55:46.257 INFO 2981023 --- [ocp-schedules-19,c5c584dc419261cb,ccf0002a7875535b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:55:46.268 INFO 2981023 --- [ocp-schedules-15,1754e1096d6145db,8d29ea2dc84b80ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:55:46.268 INFO 2981023 --- [ocp-schedules-16,22bec2ad7e195c25,4fa0a1e1a1088116] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:55:46.274 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:55:46.276 INFO 2981023 --- [ocp-schedules-9,f107de8fa015c04b,06192ceb82ca3db2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:55:46.071 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start' on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=sudo bash -c ''"'"'/home/admin/ocp_agent'"'"'/bin/ocp_agentctl start', out={"successful":true,"message":"ok","error":""}, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.284 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:55:46.284112673+08:00 2024-09-11 17:55:46.296 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:55:46.284+08:00 2024-09-11 17:55:46.299 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:55:46.284+08:00 and status AVAILABLE 2024-09-11 17:55:46.281 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:46.306 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:55:46.335 INFO 2981023 --- [ocp-schedules-15,1754e1096d6145db,8d29ea2dc84b80ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:55:46.355 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:55:46.365 INFO 2981023 --- [ocp-schedules-1,d366b87abe51c709,880293ef1e8e92bd] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:55:46.321 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.396 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:55:46.397 INFO 2981023 --- [host-check-agent-status1,f20954ef4e849197,af1b1b44d44d2fb4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:55:46.399 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:55:46.404024972+08:00 2024-09-11 17:55:46.399 INFO 2981023 --- [host-check-agent-status2,f20954ef4e849197,a2c1a58e2ed1d9df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:55:46.400 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:55:46.400 INFO 2981023 --- [host-check-agent-status1,f20954ef4e849197,af1b1b44d44d2fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:55:46.400 INFO 2981023 --- [host-check-agent-status1,f20954ef4e849197,af1b1b44d44d2fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:55:46.400 INFO 2981023 --- [host-check-agent-status1,f20954ef4e849197,af1b1b44d44d2fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:55:46.401 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:55:46.399+08:00 2024-09-11 17:55:46.401 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:55:46.402 INFO 2981023 --- [host-check-agent-status2,f20954ef4e849197,a2c1a58e2ed1d9df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:55:46.402 INFO 2981023 --- [host-check-agent-status2,f20954ef4e849197,a2c1a58e2ed1d9df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:55:46.402 INFO 2981023 --- [host-check-agent-status2,f20954ef4e849197,a2c1a58e2ed1d9df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:55:46.402 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:55:46.407939255+08:00 2024-09-11 17:55:46.402 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:55:46.403 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:55:46.402+08:00 2024-09-11 17:55:46.390 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:46.457 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:55:46.399+08:00 and status AVAILABLE 2024-09-11 17:55:46.459 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:55:46.402+08:00 and status AVAILABLE 2024-09-11 17:55:46.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:55:46.459177+08:00, lastProcessTime=2024-09-11T17:55:36.352541+08:00 2024-09-11 17:55:46.464 INFO 2981023 --- [host-check-agent-status3,f20954ef4e849197,c8771b2420a81169] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:55:46.467 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:55:46.468 INFO 2981023 --- [host-check-agent-status3,f20954ef4e849197,c8771b2420a81169] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:55:46.468 INFO 2981023 --- [host-check-agent-status3,f20954ef4e849197,c8771b2420a81169] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:55:46.468 INFO 2981023 --- [host-check-agent-status3,f20954ef4e849197,c8771b2420a81169] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:55:46.472 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:55:46.473 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.474 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:46.474 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:55:46.475 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.477 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:46.478 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:55:46.480 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:55:46.480 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:55:46.482 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:55:46.493 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.493 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:46.494 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:55:46.498 INFO 2981023 --- [ocp-schedules-16,22bec2ad7e195c25,4fa0a1e1a1088116] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:55:46.499 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:55:46.500 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:55:46.503 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.491 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.505 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:46.506 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.506 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.506 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:46.506 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:55:46.507 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:55:46.507 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.516 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:55:46.503 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.521 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:55:46.522 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.522 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:46.516 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.s.c.AgentInstallationTaskService : ocp_agentd running 2024-09-11 17:55:46.524 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:55:46.529 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:55:46.540 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.531 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:46.541 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.541 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:55:46.547 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:55:46.550 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:55:46.554 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:55:46.564 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:55:46.573 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:55:46.558 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.601 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:55:46.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.602 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.603 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:55:46.604 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.604 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:46.605 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:55:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.606 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:55:46.607 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.608 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:55:46.608 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:55:46.609 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:55:46.612 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.613 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.613 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:55:46.614 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:55:46.599 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:46.615 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:46.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:55:46.629 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:55:46.635 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:55:46.639 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:55:46.640 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:55:46.652 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:55:46.662 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:55:46.662 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:55:46.642 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.668 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:46.669 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:55:46.674 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.667 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.675 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.675 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:46.676 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.679 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:55:46.680 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:55:46.674 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.s.c.AgentInstallationTaskService : ocp_mgragent running 2024-09-11 17:55:46.684 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.684 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.685 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:46.686 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:55:46.690 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:55:46.690 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:55:46.691 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:55:46.694 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:55:46.685 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 10.31.35.7 2024-09-11 17:55:46.699 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:46.699 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:55:46.700 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:55:46.701 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:55:46.702 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.718 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.720 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:55:46.722 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.722 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:55:46.722 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:55:46.723 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:55:46.723 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:55:46.724 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.724 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:55:46.725 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:55:46.725 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.726 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:46.718 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.730 INFO 2981023 --- [host-check-host-status2,e8b3d334054befb5,da1349b82c3fe371] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:55:46.731 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:55:46.734 INFO 2981023 --- [host-check-host-status1,e8b3d334054befb5,a5e596ba7bf6452f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:55:46.735 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:55:46.737 INFO 2981023 --- [host-check-host-status3,e8b3d334054befb5,f0cf4e48c6c30c57] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:55:46.742 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.743 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.744 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:46.744 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.745 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.745 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:55:46.746 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:55:46.756 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:55:46.760 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:55:46.763 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:55:46.764 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:55:46.760 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7 2024-09-11 17:55:46.782 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.783 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.783 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:55:46.784 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:55:46.785 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.787 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:55:46.789 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.789 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.790 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:55:46.794 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:55:46.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:55:46.797 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:55:46.797 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.799 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7, result: SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:46.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:55:46.811 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:55:46.806 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 10.31.35.7,result:SshResult(host=10.31.35.7, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2024-09-11 17:55:46.817 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:55:46.811 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.s.c.AgentInstallationTaskService : ocp_monagent running 2024-09-11 17:55:46.822 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:55:46.825 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.826 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.826 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:55:46.837 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:55:46.833 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=10.31.35.7, httpPort=62888, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:10 2024-09-11 17:55:46.838 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.839 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.840 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:55:46.840 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:55:46.841 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:55:46.841 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:55:46.842 INFO 2981023 --- [ocp-schedules-13,0336ba99099d16c3,3ee13e23df1f514e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7226,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1808,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:55:46.842 INFO 2981023 --- [ocp-schedules-13,0336ba99099d16c3,3ee13e23df1f514e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:55:46.838 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:55:46.845 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:55:46.459177+08:00 2024-09-11 17:55:46.848 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:55:46.861 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:55:46.867 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.868 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.869 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.869 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:55:46.870 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.870 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:55:46.870 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.870 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:55:46.877 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:55:46.878 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:55:46.877 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] completed 2024-09-11 17:55:46.887 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:55:46.887 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:55:46.893 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,c7d987c1e35eded0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048186, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:55:46.914 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:55:46.922 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:55:46.928 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:46.929 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:46.929 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:55:46.931 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:55:46.935 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:55:46.935 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:55:46.935 INFO 2981023 --- [ocp-schedules-19,c5c584dc419261cb,ccf0002a7875535b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:55:46.938 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:55:46.940 INFO 2981023 --- [ocp-schedules-7,9efea05d9a0e4dc6,e3ef2ebb28de635c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:55:46.970 INFO 2981023 --- [host-check-clockdiff2,5153ce80e78d2b0f,ba48ae8b21d1c113] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048546 5 5 , error= 2024-09-11 17:55:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,5a2d23150263c7f2,8619612590e57f09] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,5a2d23150263c7f2,8619612590e57f09] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=8, collectTaskCount=8 2024-09-11 17:55:47.325 INFO 2981023 --- [ocp-async-4,f0bbfd13fcdcc2bf,6d9aa0cf3de479e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:47.408 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048185] 2024-09-11 17:55:47.533 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,66e655070fcb8865] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048185, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:47.579 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048185, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, task_instance_id=4048198, task_operation=execute, host_credential_id=6, latest_execution_start_time=2024-09-11T17:55:47.531+08:00, sub_task_instance_name=Update install agent info, sub_task_instance_id=4048185, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:55:47.583 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.s.t.b.h.UpdateInstallAgentInfoTask : [UpdateInstallAgentInfoTask] started 2024-09-11 17:55:47.599 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:47.605 WARN 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2000010 2024-09-11 17:55:47.609 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:47.613 WARN 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2000010 2024-09-11 17:55:47.618 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.s.c.AgentInstallationTaskService : Saving host agent records: hostId=2000010, version=4.3.1-20240805192406, isReinstall=false 2024-09-11 17:55:47.633 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Installed agent 2000010 on host 2000010 2024-09-11 17:55:47.638 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Creating OCP agent process on host 2000010 of start mode ALWAYS 2024-09-11 17:55:47.646 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Creating OCP agent process on host 2000010 of meta 1 2024-09-11 17:55:47.651 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Creating OCP agent process on host 2000010 of meta 2 2024-09-11 17:55:47.657 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Creating OCP agent process on host 2000010 of meta 3 2024-09-11 17:55:47.666 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing running info of processes on host 2000010 2024-09-11 17:55:47.674 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:47.674 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:47.674 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:47.678 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:55:47.687 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:47.695 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:55:47.708 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:55:47.732 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 17:55:47.737 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:47.742 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:55:47.749 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:55:47.767 INFO 2981023 --- [ocp-async-3,4fcfe4e51d2cca94,17cdce515bcbf1f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:55:47.773 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 17:55:47.777 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:47.782 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:55:47.789 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:55:47.819 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 17:55:47.829 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Updating OCP agent last install time, hostId=2000010, lastInstallTime=2024-09-11T17:55:47.829+08:00 2024-09-11 17:55:47.834 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Updated agent 2000010 last install time 2024-09-11T17:55:47.829+08:00 2024-09-11 17:55:47.845 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d05cd0a30e6b63e4,139661e1206f1681] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=d05cd0a30e6b63e4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:55:47.846 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d05cd0a30e6b63e4,139661e1206f1681] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:55:47.842 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Offlining all exporters: hostId=2000010 2024-09-11 17:55:47.852 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Registering all exporters: hostId=2000010 2024-09-11 17:55:47.856 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : register host performance exporter, host id:2000010 2024-09-11 17:55:47.862 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d05cd0a30e6b63e4,139661e1206f1681] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=d05cd0a30e6b63e4, duration=19 ms] 2024-09-11 17:55:47.862 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:47.867 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:55:47.879 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:55:47.889 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:55:47.896 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.agent.HostAgentServiceImpl : get hostServiceType OCP_AGENT, exporters /metrics/node/host 2024-09-11 17:55:47.914 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.s.t.b.h.UpdateInstallAgentInfoTask : [UpdateInstallAgentInfoTask] completed 2024-09-11 17:55:47.926 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,9dd258438e159dc0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048185, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:48.433 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048187] 2024-09-11 17:55:48.565 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,e895573e4230c8af] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048187, operation:EXECUTE, state: RUNNING 2024-09-11 17:55:48.596 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048187, context=Context{parallelIdx=-1, stringMap={host_ip=10.31.35.7, ssh_port=22, task_instance_id=4048198, task_operation=execute, host_credential_id=6, latest_execution_start_time=2024-09-11T17:55:48.564+08:00, sub_task_instance_name=Update host status, sub_task_instance_id=4048187, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:55:48.609 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.o.s.t.b.host.UpdateHostStatusTask : Begin to update host status and credential relations, host ids :[2000010] 2024-09-11 17:55:48.614 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.o.s.t.b.host.UpdateHostStatusTask : Begin to update host 2000010 status 2024-09-11 17:55:48.621 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.o.s.c.CredentialServiceImpl : Binding credential 6 to hosts [2000010] 2024-09-11 17:55:48.627 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.o.s.c.CredentialServiceImpl : Success to bind credential 6 to hosts [2000010] 2024-09-11 17:55:48.635 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.ocp.service.compute.HostDaoManager : Updated host status: host [2000010:10.31.35.7], from status [NEW] to [AVAILABLE] 2024-09-11 17:55:48.639 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE SERVER STATE, property:UpdateServerStateProperty(serialNumber=null, hostStatus=AVAILABLE) 2024-09-11 17:55:48.650 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.o.s.t.b.host.UpdateHostStatusTask : Success to update host 2000010 status 2024-09-11 17:55:48.659 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.o.s.t.b.host.UpdateHostStatusTask : Success to update host status and credential relations 2024-09-11 17:55:48.669 INFO 2981023 --- [manual-subtask-executor16,0fec2275d3095882,8594e97d69ad9752] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048187, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:55:49.325 INFO 2981023 --- [ocp-async-2,260ee8311ff1aae2,43016fe84b55f002] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:49.490 INFO 2981023 --- [task-coordinator2,,] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007104 2024-09-11 17:55:49.499 INFO 2981023 --- [task-coordinator2,,] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"activeAt":"2024-09-11T09:55:49.484Z","alarmType":"host_operation","appType":"OCP","id":6007104,"labels":{"alarm_name":"伺服器運維操作提醒","alarm_status_zh_tw":"警示中","app_type":"OCP","svr_ip":"10.31.35.7","alarm_status_native":"Active","task_id":"4048198","alarm_level":"5","alarm_level_zh_tw":"提醒","operator":"admin","alarm_summary_zh_tw":"alarm_template_id=0:host=10.31.35.7:operation_name=Prepare host SUCCESS","alarm_level_native":"5","alarm_scope_en_us":"Host","alarm_summary_en_us":"alarm_template_id=0:host=10.31.35.7:operation_name=Prepare host SUCCESS","alarm_level_en_us":"Info","alarm_type":"host_operation","auto_resolved":"1","alarm_status":"Active","alarm_group_zh_tw":"提示","host":"10.31.35.7","alarm_scope":"Host","alarm_active_at":"2024-09-11T17:55:49.484+08:00","task_result":"success","alarm_target":"alarm_template_id=0:host=10.31.35.7:operation_name=Prepare host","alarm_scope_native":"Host","alarm_description_zh_tw":"伺服器：10.31.35.7，警示：Prepare host SUCCESS, 操作者 admin，運行結果：success。","alarm_name_zh_tw":"伺服器運維操作提醒","app":"OCP","host_ip":"10.31.35.7","alarm_name_en_us":"Host operation info","alarm_name_zh_cn":"主机运维操作提醒","operation_result":"SUCCESS","alarm_status_en_us":"Active","message_type":"Alarm","alarm_status_zh_cn":"告警中","alarm_scope_zh_tw":"主機","alarm_template_id":"0","alarm_group_en_us":"info","operation_name":"Prepare host","alarm_description_en_us":"Host: 10.31.35.7, alert: Prepare host SUCCESS, operator was admin, result was success.","__host.id":"2000010","alarm_scope_zh_cn":"主机","alarm_level_zh_cn":"提醒","alarm_description_zh_cn":"服务器：10.31.35.7，告警：Prepare host SUCCESS, 操作者 admin，运行结果：success。","alarm_summary_zh_cn":"alarm_template_id=0:host=10.31.35.7:operation_name=Prepare host SUCCESS","alarm_updated_at":"2024-09-11T17:55:49.484+08:00","ocp_ip":"10.31.35.10","host_name":"vm-eas-prd-rtd-mid05","alarm_group_zh_cn":"提示"},"level":5,"scope":"Host","status":"Active","statusUpdatedAt":"2024-09-11T09:55:49.486Z","target":"alarm_template_id=0:host=10.31.35.7:operation_name=Prepare 2024-09-11 17:55:49.499 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048198, status:SUCCESS 2024-09-11 17:55:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:55:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:55:50.277 INFO 2981023 --- [ocp-schedules-16,854143e307f81edf,6db85a21e6a418ff] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:55:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,c78e9ca9d5d05f32,f9e72ef2e23fd10f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:50.317 ERROR 2981023 --- [prometheus-minute-schedule-1,c78e9ca9d5d05f32,f9e72ef2e23fd10f] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:55:50.317 ERROR 2981023 --- [prometheus-minute-schedule-1,c78e9ca9d5d05f32,f9e72ef2e23fd10f] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:55:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,c78e9ca9d5d05f32,f9e72ef2e23fd10f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:55:50.674 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:50.676 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:51.325 INFO 2981023 --- [ocp-async-7,50e0ea310c35dfc6,04ac0baa6787eb2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:51.579 INFO 2981023 --- [ocp-schedules-18,91af1982ef752811,85a7257ba433fafc] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:55:51.580 INFO 2981023 --- [ocp-schedules-18,91af1982ef752811,85a7257ba433fafc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:55:51.580 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:55:51.580 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:55:51.581 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:55:51.581 INFO 2981023 --- [ocp-schedules-18,91af1982ef752811,85a7257ba433fafc] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:55:52.300 INFO 2981023 --- [ocp-schedules-21,36bc178f73327d70,1c8d183adac7382d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:55:53.027 INFO 2981023 --- [ocp-schedules-19,01e6e2f871ac8ba9,b3f381693ed60a0d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:55:53.074 WARN 2981023 --- [scanner-13573445-1,01e6e2f871ac8ba9,c14c3fa363fb6a85] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:55:53.098 INFO 2981023 --- [alarm-detect-16,01e6e2f871ac8ba9,b6e47290c59b7bb8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644995, data is empty. 2024-09-11 17:55:53.103 INFO 2981023 --- [alarm-detect-16,01e6e2f871ac8ba9,b6e47290c59b7bb8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644995, data is empty. 2024-09-11 17:55:53.108 INFO 2981023 --- [alarm-detect-16,01e6e2f871ac8ba9,b6e47290c59b7bb8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644995, data is empty. 2024-09-11 17:55:53.117 INFO 2981023 --- [alarm-detect-13,01e6e2f871ac8ba9,15ed06f9974e4ead] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644995, data is empty. 2024-09-11 17:55:53.118 INFO 2981023 --- [alarm-detect-16,01e6e2f871ac8ba9,784d2a28cd4e996c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644995, data is empty. 2024-09-11 17:55:53.124 INFO 2981023 --- [alarm-detect-16,01e6e2f871ac8ba9,784d2a28cd4e996c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644995, data is empty. 2024-09-11 17:55:53.125 INFO 2981023 --- [alarm-detect-13,01e6e2f871ac8ba9,15ed06f9974e4ead] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644995, data is empty. 2024-09-11 17:55:53.128 INFO 2981023 --- [alarm-detect-16,01e6e2f871ac8ba9,784d2a28cd4e996c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644995, data is empty. 2024-09-11 17:55:53.130 INFO 2981023 --- [alarm-detect-13,01e6e2f871ac8ba9,15ed06f9974e4ead] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644995, data is empty. 2024-09-11 17:55:53.186 INFO 2981023 --- [ocp-schedules-18,60715fb3ada0ec9b,6dfa5d71ae1b5bfa] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:55:53.186 INFO 2981023 --- [ocp-schedules-18,60715fb3ada0ec9b,6dfa5d71ae1b5bfa] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:55:53.224 INFO 2981023 --- [alarm-detect-8,01e6e2f871ac8ba9,b5688a25b5481e22] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644995, data is empty. 2024-09-11 17:55:53.228 INFO 2981023 --- [alarm-detect-8,01e6e2f871ac8ba9,b5688a25b5481e22] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644995, data is empty. 2024-09-11 17:55:53.233 INFO 2981023 --- [alarm-detect-8,01e6e2f871ac8ba9,b5688a25b5481e22] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644995, data is empty. 2024-09-11 17:55:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,20875738aaa59c5a,b0a5713e369df660] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:53.316 ERROR 2981023 --- [prometheus-minute-schedule-1,20875738aaa59c5a,b0a5713e369df660] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:55:53.316 ERROR 2981023 --- [prometheus-minute-schedule-1,20875738aaa59c5a,b0a5713e369df660] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:55:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,20875738aaa59c5a,b0a5713e369df660] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:55:53.326 INFO 2981023 --- [ocp-async-4,b4f81b5db4ea3ec7,9a0dadb661fa070e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:53.676 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:53.677 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:53.758 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:55:53.776 WARN 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:55:53.792 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2024-09-11 17:55:53.792 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.792 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2024-09-11 17:55:53.792 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2024-09-11 17:55:53.792 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2024-09-11 17:55:53.792 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.792 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2024-09-11 17:55:53.792 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2024-09-11 17:55:53.792 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2024-09-11 17:55:53.793 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2024-09-11 17:55:53.794 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2024-09-11 17:55:53.795 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2024-09-11 17:55:53.796 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2024-09-11 17:55:53.797 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2024-09-11 17:55:53.798 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2024-09-11 17:55:53.799 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2024-09-11 17:55:53.800 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2024-09-11 17:55:53.801 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 3 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2024-09-11 17:55:53.802 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2024-09-11 17:55:53.803 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2024-09-11 17:55:53.804 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2024-09-11 17:55:53.805 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2024-09-11 17:55:53.806 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2024-09-11 17:55:53.807 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2024-09-11 17:55:53.808 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2024-09-11 17:55:53.809 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2024-09-11 17:55:53.810 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 17:55:53.810 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2024-09-11 17:55:53.810 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2024-09-11 17:55:53.810 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2024-09-11 17:55:53.810 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 17:55:53.810 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2024-09-11 17:55:53.810 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2024-09-11 17:55:53.810 INFO 2981023 --- [ocp-schedules-6,6e2899f16fa56ab5,b318928953a6f402] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=126, updatedCount=126, failedToUpdateCount=0 2024-09-11 17:55:54.329 INFO 2981023 --- [ocp-schedules-30,540af8b7dafcd7ab,ac8fd9025f3dc5d4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:55:54.662 INFO 2981023 --- [ocp-schedules-5,de134da6c0e0b449,74c3200a5236ee2a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:55:54.671 INFO 2981023 --- [ocp-schedules-5,de134da6c0e0b449,74c3200a5236ee2a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:55:54.671 INFO 2981023 --- [ocp-schedules-5,de134da6c0e0b449,74c3200a5236ee2a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:55:55.326 INFO 2981023 --- [ocp-async-5,e1925f5becae7e36,b3ed451e8be486b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:55.754 INFO 2981023 --- [ocp-schedules-13,900055428ed2e0d7,5aa14150ddf34014] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:55:55.754+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:55:55.757 INFO 2981023 --- [ocp-schedules-13,900055428ed2e0d7,5aa14150ddf34014] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:55:55.783 INFO 2981023 --- [ocp-schedules-2,84141febb5b939cc,e0a155b531e02ccb] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:55:55.783 INFO 2981023 --- [ocp-schedules-2,84141febb5b939cc,e0a155b531e02ccb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:55:55.803 INFO 2981023 --- [ocp-schedules-2,84141febb5b939cc,e0a155b531e02ccb] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:55:55.803 INFO 2981023 --- [ocp-schedules-2,84141febb5b939cc,e0a155b531e02ccb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:55:55.804 INFO 2981023 --- [ocp-schedules-2,84141febb5b939cc,e0a155b531e02ccb] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:55:55.804 INFO 2981023 --- [ocp-schedules-2,84141febb5b939cc,e0a155b531e02ccb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:55:55.893 INFO 2981023 --- [ocp-schedules-13,900055428ed2e0d7,5aa14150ddf34014] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 41 to SqlAggregateData 2024-09-11 17:55:55.894 INFO 2981023 --- [ocp-schedules-13,900055428ed2e0d7,5aa14150ddf34014] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:55:55.923 INFO 2981023 --- [ocp-schedules-30,82d2f2fd5c7647c0,efe565e1327519d4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:55:55.929 INFO 2981023 --- [ocp-schedules-30,82d2f2fd5c7647c0,efe565e1327519d4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:55:55.929 INFO 2981023 --- [ocp-schedules-30,82d2f2fd5c7647c0,efe565e1327519d4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:55:56.091 INFO 2981023 --- [ocp-schedules-16,1d4f686eb20a385b,3a9c0b7242b57f3a] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:55:56.320 INFO 2981023 --- [prometheus-minute-schedule-1,f5841fb9fe3f3a7c,6ebfb427bf661352] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:56.325 ERROR 2981023 --- [prometheus-minute-schedule-1,f5841fb9fe3f3a7c,6ebfb427bf661352] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:55:56.325 ERROR 2981023 --- [prometheus-minute-schedule-1,f5841fb9fe3f3a7c,6ebfb427bf661352] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:55:56.325 INFO 2981023 --- [prometheus-minute-schedule-1,f5841fb9fe3f3a7c,6ebfb427bf661352] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:55:56.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:55:56.354983+08:00, lastProcessTime=2024-09-11T17:55:46.459177+08:00 2024-09-11 17:55:56.442 INFO 2981023 --- [ocp-schedules-22,76e03edc1050040f,f2456227f94f9535] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:55:56.444 INFO 2981023 --- [ocp-schedules-22,76e03edc1050040f,f2456227f94f9535] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:55:56.499 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:55:56.499 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:55:56.499 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.502 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.503 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:55:56.506 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:55:56.507 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:55:56.507 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.508 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.508 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.511 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.511 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.523 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.523 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.523 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:55:56.524 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.524 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.527 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.528 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.534 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.534 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:55:56.537 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.538 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.538 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:55:56.538 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3DE1D6E9-0-0] 2024-09-11 17:55:56.538 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:55:56.539 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.539 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:55:56.543 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.543 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:55:56.546 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.546 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.550 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.550 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:55:56.552 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.552 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:55:56.562 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.562 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.571 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.571 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:55:56.572 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:55:56.572 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:55:56.572 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3DE1D6F3-0-0] 2024-09-11 17:55:56.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:55:56.574 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.574 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.574 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:55:56.575 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:55:56.575 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.576 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:55:56.577 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.577 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:55:56.578 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:55:56.581 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.581 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:55:56.582 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.583 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:55:56.583 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.583 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:55:56.584 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:55:56.585 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.586 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:55:56.587 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.588 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:55:56.588 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.589 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:55:56.589 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:55:56.354983+08:00 2024-09-11 17:55:56.589 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:55:56.590 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.590 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:55:56.591 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.592 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:55:56.593 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:55:56.593 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.594 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:55:56.595 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.596 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:55:56.596 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.597 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:55:56.598 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.599 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:55:56.599 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.600 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:55:56.600 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.600 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:56.601 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:55:56.601 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.602 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:55:56.602 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:55:56.604 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.604 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:55:56.605 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.614 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:56.616 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.616 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:55:56.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:55:56.618 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:55:56.620 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.620 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:55:56.627 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:55:56.627 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:55:56.627 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:55:56.628 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:55:56.630 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:55:56.631 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:55:56.632 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.632 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.633 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:55:56.633 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.634 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:55:56.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:55:56.642 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:55:56.642 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:55:56.642 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:56.642 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:55:56.644 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:55:56.644 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:55:56.644 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:55:56.647 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:55:56.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:55:56.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:55:56.648 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:55:56.649 INFO 2981023 --- [ocp-schedules-22,76e03edc1050040f,f2456227f94f9535] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:55:56.677 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:56.678 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:55:56.976 INFO 2981023 --- [ocp-schedules-11,267db494f9932eaa,f1067e5f190803cc] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:55:56.976 INFO 2981023 --- [ocp-schedules-11,267db494f9932eaa,f1067e5f190803cc] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:55:56.989 INFO 2981023 --- [Collect all tenants major compactions1,267db494f9932eaa,4e7fc4b2a14ddcaf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:55:56.989 INFO 2981023 --- [Collect all tenants major compactions2,267db494f9932eaa,93546c272baf9e55] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:55:56.990 INFO 2981023 --- [Collect all tenants major compactions3,267db494f9932eaa,faa4ba6bf148886a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:55:56.991 INFO 2981023 --- [Collect all tenants major compactions4,267db494f9932eaa,ab8e4aac9b8fed03] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:55:56.992 INFO 2981023 --- [Collect all tenants major compactions6,267db494f9932eaa,366d04b94dd3ddec] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:55:56.993 INFO 2981023 --- [Collect all tenants major compactions5,267db494f9932eaa,8997d87db0e26cda] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:55:57.002 INFO 2981023 --- [Collect all tenants major compactions6,267db494f9932eaa,366d04b94dd3ddec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:57.003 INFO 2981023 --- [Collect all tenants major compactions2,267db494f9932eaa,93546c272baf9e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:57.003 INFO 2981023 --- [Collect all tenants major compactions3,267db494f9932eaa,faa4ba6bf148886a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:57.003 INFO 2981023 --- [Collect all tenants major compactions4,267db494f9932eaa,ab8e4aac9b8fed03] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:57.003 INFO 2981023 --- [Collect all tenants major compactions6,267db494f9932eaa,366d04b94dd3ddec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:55:57.004 INFO 2981023 --- [Collect all tenants major compactions2,267db494f9932eaa,93546c272baf9e55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:55:57.004 INFO 2981023 --- [Collect all tenants major compactions4,267db494f9932eaa,ab8e4aac9b8fed03] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:55:57.005 INFO 2981023 --- [Collect all tenants major compactions1,267db494f9932eaa,4e7fc4b2a14ddcaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:57.006 INFO 2981023 --- [Collect all tenants major compactions3,267db494f9932eaa,faa4ba6bf148886a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:55:57.006 INFO 2981023 --- [Collect all tenants major compactions1,267db494f9932eaa,4e7fc4b2a14ddcaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:55:57.008 INFO 2981023 --- [Collect all tenants major compactions6,267db494f9932eaa,366d04b94dd3ddec] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:55:57.008 INFO 2981023 --- [Collect all tenants major compactions6,267db494f9932eaa,366d04b94dd3ddec] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:55:57.008 INFO 2981023 --- [Collect all tenants major compactions5,267db494f9932eaa,8997d87db0e26cda] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:55:57.008 INFO 2981023 --- [Collect all tenants major compactions2,267db494f9932eaa,93546c272baf9e55] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:55:57.009 INFO 2981023 --- [Collect all tenants major compactions2,267db494f9932eaa,93546c272baf9e55] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:55:57.009 INFO 2981023 --- [Collect all tenants major compactions4,267db494f9932eaa,ab8e4aac9b8fed03] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:55:57.009 INFO 2981023 --- [Collect all tenants major compactions4,267db494f9932eaa,ab8e4aac9b8fed03] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:55:57.009 INFO 2981023 --- [Collect all tenants major compactions3,267db494f9932eaa,faa4ba6bf148886a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:55:57.009 INFO 2981023 --- [Collect all tenants major compactions3,267db494f9932eaa,faa4ba6bf148886a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:55:57.010 INFO 2981023 --- [Collect all tenants major compactions5,267db494f9932eaa,8997d87db0e26cda] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:55:57.010 INFO 2981023 --- [Collect all tenants major compactions1,267db494f9932eaa,4e7fc4b2a14ddcaf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:55:57.010 INFO 2981023 --- [Collect all tenants major compactions1,267db494f9932eaa,4e7fc4b2a14ddcaf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:55:57.011 INFO 2981023 --- [Collect all tenants major compactions5,267db494f9932eaa,8997d87db0e26cda] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:55:57.011 INFO 2981023 --- [Collect all tenants major compactions5,267db494f9932eaa,8997d87db0e26cda] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:55:57.012 INFO 2981023 --- [ocp-schedules-11,267db494f9932eaa,f1067e5f190803cc] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:55:57.326 INFO 2981023 --- [ocp-async-7,3f66afde5b99afab,52f917ae9890f7a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:57.767 INFO 2981023 --- [ocp-async-5,9a992142d8d3860d,525b9c29cd1ba292] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:55:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,c6a4fcb87340469a,5a494cfa66b1052a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048560, interval=60 2024-09-11 17:55:59.318 ERROR 2981023 --- [prometheus-minute-schedule-1,c6a4fcb87340469a,5a494cfa66b1052a] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:55:59.318 ERROR 2981023 --- [prometheus-minute-schedule-1,c6a4fcb87340469a,5a494cfa66b1052a] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:55:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,c6a4fcb87340469a,5a494cfa66b1052a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:55:59.326 INFO 2981023 --- [ocp-async-4,d1a3bd2441d00f14,636c4a4b48711a30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:55:59.678 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:55:59.680 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:00.101 ERROR 2981023 --- [prometheus-second-schedule-1,2a5237e6c7436afd,a483aafeb7f7b9ca] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/node/host java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collect(OcpMetricCollectServiceImpl.java:130) at com.oceanbase.ocp.monitor.OcpMonitorManager$CollectSecondTask.run(OcpMonitorManager.java:225) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:01.334 INFO 2981023 --- [ocp-async-5,82d3a600d68a8e39,4e872a80e77c8667] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:02.303 INFO 2981023 --- [ocp-schedules-27,9cb742c4b2a47fdb,7c962eede2edc9d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:56:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,54f77948cbac8dce,27caa5d61045587f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:02.316 ERROR 2981023 --- [prometheus-minute-schedule-1,54f77948cbac8dce,27caa5d61045587f] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:02.316 ERROR 2981023 --- [prometheus-minute-schedule-1,54f77948cbac8dce,27caa5d61045587f] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,54f77948cbac8dce,27caa5d61045587f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:02.564 INFO 2981023 --- [ocp-schedules-30,b65928974e02f154,4562eea2e9745d42] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:56:02.565 INFO 2981023 --- [ocp-schedules-30,b65928974e02f154,4562eea2e9745d42] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:02.583 INFO 2981023 --- [ocp-schedules-30,b65928974e02f154,4562eea2e9745d42] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:56:02.680 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:02.681 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:03.268 INFO 2981023 --- [ocp-schedules-2,cdd3a395721ff43b,3fae3ac1a808de2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:56:03.322 WARN 2981023 --- [scanner-13573774-1,cdd3a395721ff43b,e29f7a5b04439987] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:56:03.326 INFO 2981023 --- [ocp-async-7,0b0d4ec8a7a52ee7,a1f43adb6dea665d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:03.354 INFO 2981023 --- [alarm-detect-15,cdd3a395721ff43b,26feed4cc6fee094] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8644995, data is empty. 2024-09-11 17:56:03.360 INFO 2981023 --- [alarm-detect-15,cdd3a395721ff43b,26feed4cc6fee094] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645036, data is empty. 2024-09-11 17:56:03.363 INFO 2981023 --- [alarm-detect-15,cdd3a395721ff43b,26feed4cc6fee094] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645036, data is empty. 2024-09-11 17:56:03.374 INFO 2981023 --- [alarm-detect-10,cdd3a395721ff43b,5a51718fbc625687] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8644995, data is empty. 2024-09-11 17:56:03.375 INFO 2981023 --- [alarm-detect-17,cdd3a395721ff43b,64dc7a278330f0d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8644995, data is empty. 2024-09-11 17:56:03.377 INFO 2981023 --- [alarm-detect-10,cdd3a395721ff43b,5a51718fbc625687] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645036, data is empty. 2024-09-11 17:56:03.379 INFO 2981023 --- [alarm-detect-17,cdd3a395721ff43b,64dc7a278330f0d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645036, data is empty. 2024-09-11 17:56:03.382 INFO 2981023 --- [alarm-detect-10,cdd3a395721ff43b,5a51718fbc625687] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645036, data is empty. 2024-09-11 17:56:03.383 INFO 2981023 --- [alarm-detect-17,cdd3a395721ff43b,64dc7a278330f0d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645036, data is empty. 2024-09-11 17:56:03.418 INFO 2981023 --- [alarm-detect-1,cdd3a395721ff43b,6f7a15d70a829c87] c.o.o.alarm.core.detect.MetricDetector : alarmType agent_process_count_abnormal detect time range 1726048373 ~ 1726048433 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={agent_process_count=[SeriesMetricValues(metric=Metric{name='agent_process_count', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, name=ocp_mgragent, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726048380, value=2.0)])]}, metricEvaluateResults={agent_process_count != 1=[ExpressionEvaluateResult(matchValue=1.0, metric=agent_process_count, match=true, value=2.0, values={1726048380=2.0}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, name=ocp_mgragent, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:56:03.419 INFO 2981023 --- [alarm-detect-1,cdd3a395721ff43b,6f7a15d70a829c87] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=agent_process_count_abnormal, target=alarm_template_id=0:host=10.31.35.10:name=ocp_mgragent-3, currentState=Pending 2024-09-11 17:56:03.474 INFO 2981023 --- [alarm-detect-11,cdd3a395721ff43b,3f5ee29b6785206d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8644995, data is empty. 2024-09-11 17:56:03.477 INFO 2981023 --- [alarm-detect-11,cdd3a395721ff43b,3f5ee29b6785206d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645036, data is empty. 2024-09-11 17:56:03.482 INFO 2981023 --- [alarm-detect-11,cdd3a395721ff43b,3f5ee29b6785206d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645036, data is empty. 2024-09-11 17:56:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,781e44ade415e7d2,f3eeeb6771036ad5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:05.316 ERROR 2981023 --- [prometheus-minute-schedule-1,781e44ade415e7d2,f3eeeb6771036ad5] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:05.316 ERROR 2981023 --- [prometheus-minute-schedule-1,781e44ade415e7d2,f3eeeb6771036ad5] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,781e44ade415e7d2,f3eeeb6771036ad5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:05.326 INFO 2981023 --- [ocp-async-8,87908cda7da59d08,5ff0a14437ba2f81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:05.681 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:05.681 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:56:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:56:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:56:05.983 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:56:05.988 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:56:05.993 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:56:05.998 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:56:05.998 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:56:05.998 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:56:06.001 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:06.001 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:06.002 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:56:06.002 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:56:06.002 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:06.003 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:06.003 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:56:06.003 INFO 2981023 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:56:06.061 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:53:41, start=2024-09-11T17:50:10, end=2024-09-11T17:52:40, updates=143, granule=120000ms, delay=211000ms 2024-09-11 17:56:06.066 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:53:41, target=2024-09-11T17:52:40, granule=120000ms, delay=61000ms) 2024-09-11 17:56:06.066 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:52:40, target=2024-09-11T17:39:31, granule=600000ms, delay=789000ms) 2024-09-11 17:56:06.147 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:53:41, start=2024-09-11T17:50:11, end=2024-09-11T17:52:41, updates=289, granule=120000ms, delay=210000ms 2024-09-11 17:56:06.160 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:53:41, target=2024-09-11T17:52:41, granule=120000ms, delay=60000ms) 2024-09-11 17:56:06.161 INFO 2981023 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:52:41, target=2024-09-11T17:40:51, granule=600000ms, delay=710000ms) 2024-09-11 17:56:06.161 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=191ms, period=120000ms 2024-09-11 17:56:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:56:06.352464+08:00, lastProcessTime=2024-09-11T17:55:56.354983+08:00 2024-09-11 17:56:06.484 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:56:06.352464+08:00 2024-09-11 17:56:07.326 INFO 2981023 --- [ocp-async-1,293276c2d6484183,90f62cbcbd06dba7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:07.616 INFO 2981023 --- [ocp-schedules-10,fe6fb8924cdd2032,5e0f5a05f8c7443a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:56:07.681 INFO 2981023 --- [ocp-schedules-25,859f98d8d91db2ec,37b2d83d8e0a0d22] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:56:07.681+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:56:07.682 INFO 2981023 --- [ocp-schedules-25,859f98d8d91db2ec,37b2d83d8e0a0d22] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:56:07.684 INFO 2981023 --- [ocp-schedules-25,859f98d8d91db2ec,37b2d83d8e0a0d22] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:56:07.685 INFO 2981023 --- [ocp-schedules-25,859f98d8d91db2ec,37b2d83d8e0a0d22] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:56:07.767 INFO 2981023 --- [ocp-async-10,16d1af33c5e49e89,5b8c0d6979a57432] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:56:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,ac4705511a2fbc86,3a7afe57c20eb6ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:08.316 ERROR 2981023 --- [prometheus-minute-schedule-1,ac4705511a2fbc86,3a7afe57c20eb6ff] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:08.316 ERROR 2981023 --- [prometheus-minute-schedule-1,ac4705511a2fbc86,3a7afe57c20eb6ff] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,ac4705511a2fbc86,3a7afe57c20eb6ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:08.681 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:08.682 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:09.326 INFO 2981023 --- [ocp-async-3,08fb00912150696e,b24d4810ae88a2b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:10.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:56:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:56:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:56:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:56:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,8b031e44562b4f7b,bcf5b99ee19c4a65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:11.316 ERROR 2981023 --- [prometheus-minute-schedule-1,8b031e44562b4f7b,bcf5b99ee19c4a65] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:11.316 ERROR 2981023 --- [prometheus-minute-schedule-1,8b031e44562b4f7b,bcf5b99ee19c4a65] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,8b031e44562b4f7b,bcf5b99ee19c4a65] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:11.327 INFO 2981023 --- [ocp-async-8,2bcdaeaf0807754e,aa4bd76acc662d38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:11.682 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:11.682 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:12.306 INFO 2981023 --- [ocp-schedules-13,f567b0c06fd7f995,7c949490133a077e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:56:12.669 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048500, nodeCount=994, affectRows=994 2024-09-11 17:56:12.669 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048500, nodeCount=814, affectRows=814 2024-09-11 17:56:12.671 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048500, nodeCount=813, validCount=813 2024-09-11 17:56:12.672 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048500, nodeCount=994, validCount=994 2024-09-11 17:56:13.327 INFO 2981023 --- [ocp-async-1,9ce0cf6c887be02a,f948a6c80b75362e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:13.520 INFO 2981023 --- [ocp-schedules-26,7276e6cd2add10a3,d871b608679493df] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:56:13.566 WARN 2981023 --- [scanner-13575599-1,7276e6cd2add10a3,33bb05532892f62e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:56:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,94c769c83e4f4371,0f68b0b79eaa71b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,4c318bd7c761e9b6,19b99925ac9f7702] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:56:14.316 ERROR 2981023 --- [prometheus-minute-schedule-1,94c769c83e4f4371,0f68b0b79eaa71b1] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:14.319 ERROR 2981023 --- [prometheus-minute-schedule-1,94c769c83e4f4371,0f68b0b79eaa71b1] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:14.319 INFO 2981023 --- [prometheus-minute-schedule-1,94c769c83e4f4371,0f68b0b79eaa71b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:14.674 INFO 2981023 --- [ocp-schedules-28,ef1d398db07a9831,f986d03f461c13d9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:56:14.682 INFO 2981023 --- [ocp-schedules-28,ef1d398db07a9831,f986d03f461c13d9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:56:14.682 INFO 2981023 --- [ocp-schedules-28,ef1d398db07a9831,f986d03f461c13d9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:56:14.683 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:14.683 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:15.101 ERROR 2981023 --- [prometheus-second-schedule-1,9c02fd025de7cdad,16dfbfa19631b578] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/node/host java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collect(OcpMetricCollectServiceImpl.java:130) at com.oceanbase.ocp.monitor.OcpMonitorManager$CollectSecondTask.run(OcpMonitorManager.java:225) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:15.311 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048500, nodeCount=463, affectRows=463 2024-09-11 17:56:15.327 INFO 2981023 --- [ocp-async-5,f4da6e025f028758,218eb518add173a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:16.020 INFO 2981023 --- [ocp-schedules-16,4b9fe61e4a098bb4,533dce8ff304f1fe] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:56:16.021 INFO 2981023 --- [host-check-clockdiff1,4b9fe61e4a098bb4,10e30cef83fd63c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:56:16.021 INFO 2981023 --- [host-check-clockdiff2,4b9fe61e4a098bb4,da37aa325b0606c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:56:16.021 INFO 2981023 --- [host-check-clockdiff3,4b9fe61e4a098bb4,e9707aaf68104554] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:56:16.023 INFO 2981023 --- [ocp-schedules-14,45233c442a4c04db,9835fec18eaa327d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:56:16.023 INFO 2981023 --- [host-check-clockdiff4,4b9fe61e4a098bb4,1d124c3808feb33e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:56:16.024 INFO 2981023 --- [host-check-agent-status1,45233c442a4c04db,9dfa9f7375fce311] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:56:16.024 INFO 2981023 --- [host-check-agent-status2,45233c442a4c04db,33ca3c229dc3b9b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:56:16.025 INFO 2981023 --- [host-check-agent-status3,45233c442a4c04db,e16a8218a3e72372] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:56:16.025 INFO 2981023 --- [host-check-agent-status4,45233c442a4c04db,2e4cb56d07fdfd79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:56:16.025 INFO 2981023 --- [ocp-schedules-8,8032c8990ee014fd,5149ba27d0ae8e7d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:56:16.045 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:56:16.045 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:56:16.058 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:56:16.061 INFO 2981023 --- [host-check-agent-status3,45233c442a4c04db,e16a8218a3e72372] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:56:16.062 INFO 2981023 --- [host-check-agent-status2,45233c442a4c04db,33ca3c229dc3b9b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:56:16.063 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:56:16.068 INFO 2981023 --- [host-check-clockdiff3,4b9fe61e4a098bb4,e9707aaf68104554] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048576 0 0 , error= 2024-09-11 17:56:16.069 INFO 2981023 --- [host-check-agent-status2,45233c442a4c04db,33ca3c229dc3b9b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:56:16.071 INFO 2981023 --- [host-check-agent-status1,45233c442a4c04db,9dfa9f7375fce311] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:56:16.072 INFO 2981023 --- [host-check-agent-status1,45233c442a4c04db,9dfa9f7375fce311] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:56:16.073 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:56:16.073 INFO 2981023 --- [host-check-agent-status4,45233c442a4c04db,2e4cb56d07fdfd79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:56:16.074 INFO 2981023 --- [host-check-agent-status4,45233c442a4c04db,2e4cb56d07fdfd79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:56:16.075 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:56:16.075 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:56:16.078 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:56:16.078 INFO 2981023 --- [host-check-agent-status3,45233c442a4c04db,e16a8218a3e72372] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:56:16.079 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:56:16.080 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:56:16.187 INFO 2981023 --- [ocp-schedules-3,78421227e937451b,76d7ae327e26e14f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:56:16.190 INFO 2981023 --- [ocp-schedules-21,5340b4455fc1ddff,0b051fd3ececbbb1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:56:16.192 INFO 2981023 --- [ocp-schedules-18,b3e4319aee3c658b,a947cdd51c754269] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:56:16.195 INFO 2981023 --- [ocp-schedules-26,d7e57ea73f2b98be,9c965def3ec1ae8f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:56:16.258 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:56:16.259 INFO 2981023 --- [ocp-schedules-18,b3e4319aee3c658b,a947cdd51c754269] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:16.260 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:56:16.268 INFO 2981023 --- [ocp-schedules-3,78421227e937451b,76d7ae327e26e14f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:16.274 INFO 2981023 --- [ocp-schedules-21,5340b4455fc1ddff,0b051fd3ececbbb1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:16.297 INFO 2981023 --- [ocp-schedules-21,5340b4455fc1ddff,0b051fd3ececbbb1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:56:16.324 INFO 2981023 --- [ocp-schedules-12,1053e73ce02f6a68,9c6df69caf0d1b5d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:56:16.325 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:56:16.327 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:56:16.326769824+08:00 2024-09-11 17:56:16.329 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:56:16.327+08:00 2024-09-11 17:56:16.329 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:56:16.333 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T17:56:16.335018511+08:00 2024-09-11 17:56:16.333 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:56:16.335 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:56:16.341010389+08:00 2024-09-11 17:56:16.355 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:56:16.335+08:00 2024-09-11 17:56:16.355 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T17:56:16.333+08:00 2024-09-11 17:56:16.356 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:56:16.327+08:00 and status AVAILABLE 2024-09-11 17:56:16.358 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:56:16.335+08:00 and status AVAILABLE 2024-09-11 17:56:16.359 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T17:56:16.333+08:00 and status AVAILABLE 2024-09-11 17:56:16.361 INFO 2981023 --- [host-check-agent-status4,45233c442a4c04db,2e4cb56d07fdfd79] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:56:16.364 INFO 2981023 --- [host-check-agent-status1,45233c442a4c04db,9dfa9f7375fce311] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:56:16.365 INFO 2981023 --- [host-check-agent-status4,45233c442a4c04db,2e4cb56d07fdfd79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:56:16.365 INFO 2981023 --- [host-check-agent-status4,45233c442a4c04db,2e4cb56d07fdfd79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:56:16.365 INFO 2981023 --- [host-check-agent-status4,45233c442a4c04db,2e4cb56d07fdfd79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:56:16.366 INFO 2981023 --- [host-check-agent-status2,45233c442a4c04db,33ca3c229dc3b9b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:56:16.368 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:56:16.369 INFO 2981023 --- [host-check-agent-status1,45233c442a4c04db,9dfa9f7375fce311] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:56:16.369 INFO 2981023 --- [host-check-agent-status1,45233c442a4c04db,9dfa9f7375fce311] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:56:16.369 INFO 2981023 --- [host-check-agent-status1,45233c442a4c04db,9dfa9f7375fce311] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:56:16.369 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.369 INFO 2981023 --- [host-check-agent-status2,45233c442a4c04db,33ca3c229dc3b9b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:56:16.369 INFO 2981023 --- [host-check-agent-status2,45233c442a4c04db,33ca3c229dc3b9b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:56:16.369 INFO 2981023 --- [host-check-agent-status2,45233c442a4c04db,33ca3c229dc3b9b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:56:16.371 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:56:16.370474+08:00, lastProcessTime=2024-09-11T17:56:06.352464+08:00 2024-09-11 17:56:16.372 INFO 2981023 --- [host-check-agent-status3,45233c442a4c04db,e16a8218a3e72372] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:56:16.372 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 17:56:16.373 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:56:16.373 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.373 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:16.374 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:56:16.374 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:56:16.380600397+08:00 2024-09-11 17:56:16.374 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:16.375 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:56:16.375 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:56:16.379 INFO 2981023 --- [host-check-agent-status3,45233c442a4c04db,e16a8218a3e72372] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:56:16.379 INFO 2981023 --- [host-check-agent-status3,45233c442a4c04db,e16a8218a3e72372] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:56:16.379 INFO 2981023 --- [host-check-agent-status3,45233c442a4c04db,e16a8218a3e72372] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:56:16.384 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:56:16.374+08:00 2024-09-11 17:56:16.392 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:56:16.374+08:00 and status AVAILABLE 2024-09-11 17:56:16.398 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:56:16.400 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:56:16.401 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.401 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:16.401 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:56:16.402 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:56:16.404 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:56:16.405 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:56:16.405 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.405 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:16.406 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:56:16.406 INFO 2981023 --- [ocp-schedules-3,78421227e937451b,76d7ae327e26e14f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:56:16.407 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:56:16.457 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.458 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:16.458 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.459 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:16.459 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:56:16.466 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:56:16.469 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:56:16.474 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:56:16.475 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:56:16.491 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:56:16.497 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:56:16.497 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 17:56:16.509 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:56:16.515 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:56:16.518 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 17:56:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.520 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:56:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:56:16.523 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:56:16.535 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.536 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.536 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:56:16.538 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:56:16.564 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:56:16.568 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:56:16.569 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:56:16.573 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.574 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.575 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:16.577 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.579 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.579 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:16.580 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:56:16.580 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:56:16.580 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.581 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:56:16.583 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:56:16.583 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:56:16.589 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 17:56:16.591 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:16.591 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:56:16.599 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:56:16.602 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:56:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:16.603 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 17:56:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:56:16.611 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:56:16.613 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:16.613 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:56:16.614 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.615 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.616 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:56:16.616 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:56:16.623 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.624 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.625 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:16.626 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:56:16.630 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:56:16.631 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:56:16.631 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:16.635 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:56:16.636 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:56:16.639 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:56:16.639 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:56:16.651 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 17:56:16.656 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:56:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.661 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:56:16.663 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.664 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:16.665 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:56:16.666 INFO 2981023 --- [host-check-host-status4,8032c8990ee014fd,9070ed58ec6a03b1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 17:56:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.669 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:56:16.670 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.671 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:16.671 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:56:16.672 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:56:16.672 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:56:16.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:56:16.676 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:56:16.677 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:56:16.682 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:56:16.684 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:56:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:56:16.691 INFO 2981023 --- [host-check-host-status2,8032c8990ee014fd,82dad97483f40453] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:56:16.699 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:56:16.700 INFO 2981023 --- [host-check-host-status3,8032c8990ee014fd,93ffe623cfa05462] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:56:16.703 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:56:16.745 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:56:16.746 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:56:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:16.755 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.757 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.758 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:56:16.759 INFO 2981023 --- [host-check-host-status1,8032c8990ee014fd,e4c8217465f38d44] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:56:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.761 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:16.761 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:56:16.766 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:56:16.768 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:56:16.768 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:56:16.770 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:56:16.772 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:56:16.790 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:56:16.796 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:56:16.801 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.802 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.802 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:56:16.803 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:56:16.370474+08:00 2024-09-11 17:56:16.808 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:56:16.808 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:56:16.822 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:56:16.828 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:16.828 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:16.829 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:56:16.835 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:56:16.835 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:56:16.838 INFO 2981023 --- [ocp-schedules-18,b3e4319aee3c658b,a947cdd51c754269] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:56:16.968 INFO 2981023 --- [host-check-clockdiff2,4b9fe61e4a098bb4,da37aa325b0606c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048576 6 6 , error= 2024-09-11 17:56:16.972 INFO 2981023 --- [host-check-clockdiff1,4b9fe61e4a098bb4,10e30cef83fd63c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048576 6 6 , error= 2024-09-11 17:56:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,99477b5a509d9548,ab4ed0dc09a34d4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:17.316 ERROR 2981023 --- [prometheus-minute-schedule-1,99477b5a509d9548,ab4ed0dc09a34d4b] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:17.316 ERROR 2981023 --- [prometheus-minute-schedule-1,99477b5a509d9548,ab4ed0dc09a34d4b] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,99477b5a509d9548,ab4ed0dc09a34d4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:17.327 INFO 2981023 --- [ocp-async-3,b1a103f7439549f2,f84ab5f5512b5c32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:17.490 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048500, nodeCount=463, validCount=463 2024-09-11 17:56:17.683 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:17.684 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:17.768 INFO 2981023 --- [ocp-async-6,10d02abc0bbdc03c,f889438a93266a1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:56:19.327 INFO 2981023 --- [ocp-async-4,d984f1d0b8504ab2,e6c2edaa7248d873] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,9d66f68fb7d15278,0618b72f3fa4f777] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:20.317 ERROR 2981023 --- [prometheus-minute-schedule-1,9d66f68fb7d15278,0618b72f3fa4f777] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:20.317 ERROR 2981023 --- [prometheus-minute-schedule-1,9d66f68fb7d15278,0618b72f3fa4f777] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,9d66f68fb7d15278,0618b72f3fa4f777] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:20.684 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:20.685 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:21.327 INFO 2981023 --- [ocp-async-6,562d062d4a8f5083,2ffb1a8665fb65ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:22.000 INFO 2981023 --- [host-check-clockdiff4,4b9fe61e4a098bb4,1d124c3808feb33e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048581 2 2 , error= 2024-09-11 17:56:22.309 INFO 2981023 --- [ocp-schedules-24,a87992b559433acf,21d00d30237883df] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:56:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,4ad10bcd3da52e6b,6b9898b9055c7b48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:23.316 ERROR 2981023 --- [prometheus-minute-schedule-1,4ad10bcd3da52e6b,6b9898b9055c7b48] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:23.316 ERROR 2981023 --- [prometheus-minute-schedule-1,4ad10bcd3da52e6b,6b9898b9055c7b48] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,4ad10bcd3da52e6b,6b9898b9055c7b48] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:23.327 INFO 2981023 --- [ocp-async-3,8d66ba06aa0fd6ef,73b7c4fff9ee1951] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:23.685 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:23.687 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:23.701 INFO 2981023 --- [ocp-schedules-3,720780b49fabe2a5,94c4affcb96c31f9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:56:23.702 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:56:23.702 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 17:56:23.702 INFO 2981023 --- [ocp-schedules-3,720780b49fabe2a5,94c4affcb96c31f9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:56:23.702 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:56:23.704 INFO 2981023 --- [ocp-schedules-26,07a4ceaa5135369a,fe5a0c663590acec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:56:23.732 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:56:23.732 INFO 2981023 --- [ocp-schedules-3,720780b49fabe2a5,94c4affcb96c31f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:56:23.733 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:56:23.736 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:56:23.736 INFO 2981023 --- [ocp-schedules-3,720780b49fabe2a5,94c4affcb96c31f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:56:23.736 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:56:23.737 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:56:23.741 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:56:23.802 WARN 2981023 --- [scanner-13575800-1,07a4ceaa5135369a,a231d62f3f2d62ed] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:56:23.900 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:56:23.905 INFO 2981023 --- [alarm-detect-18,07a4ceaa5135369a,c3d964ea575bfe65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645036, data is empty. 2024-09-11 17:56:23.906 INFO 2981023 --- [ocp-schedules-3,720780b49fabe2a5,94c4affcb96c31f9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:56:23.912 INFO 2981023 --- [alarm-detect-18,07a4ceaa5135369a,c3d964ea575bfe65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645036, data is empty. 2024-09-11 17:56:23.913 INFO 2981023 --- [alarm-detect-4,07a4ceaa5135369a,fd435aa6a46f5348] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645036, data is empty. 2024-09-11 17:56:23.915 INFO 2981023 --- [alarm-detect-17,07a4ceaa5135369a,fbc5cf41ed4d5063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645036, data is empty. 2024-09-11 17:56:23.917 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:56:23.917 INFO 2981023 --- [alarm-detect-17,07a4ceaa5135369a,fbc5cf41ed4d5063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645036, data is empty. 2024-09-11 17:56:23.919 INFO 2981023 --- [alarm-detect-18,07a4ceaa5135369a,c3d964ea575bfe65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645036, data is empty. 2024-09-11 17:56:23.920 INFO 2981023 --- [alarm-detect-17,07a4ceaa5135369a,fbc5cf41ed4d5063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645036, data is empty. 2024-09-11 17:56:23.922 INFO 2981023 --- [alarm-detect-4,07a4ceaa5135369a,fd435aa6a46f5348] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645036, data is empty. 2024-09-11 17:56:23.925 INFO 2981023 --- [alarm-detect-4,07a4ceaa5135369a,fd435aa6a46f5348] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645036, data is empty. 2024-09-11 17:56:23.943 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=10.31.35.7, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:11 2024-09-11 17:56:23.943 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:56:23.986 INFO 2981023 --- [ocp-schedules-3,720780b49fabe2a5,94c4affcb96c31f9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:56:24.064 INFO 2981023 --- [alarm-detect-2,07a4ceaa5135369a,b065baed5268f715] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645036, data is empty. 2024-09-11 17:56:24.068 INFO 2981023 --- [alarm-detect-2,07a4ceaa5135369a,b065baed5268f715] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645036, data is empty. 2024-09-11 17:56:24.073 INFO 2981023 --- [alarm-detect-2,07a4ceaa5135369a,b065baed5268f715] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645036, data is empty. 2024-09-11 17:56:24.332 INFO 2981023 --- [ocp-schedules-19,3daee0ad7f0bc6d6,0b5524d352f63cf3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:56:25.327 INFO 2981023 --- [ocp-async-2,715116b1c4bacbd1,b65b76292d8f0a75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,ce52978e68014e72,476c5b22f0ccbef5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:26.317 ERROR 2981023 --- [prometheus-minute-schedule-1,ce52978e68014e72,476c5b22f0ccbef5] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:26.317 ERROR 2981023 --- [prometheus-minute-schedule-1,ce52978e68014e72,476c5b22f0ccbef5] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,ce52978e68014e72,476c5b22f0ccbef5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:56:26.351962+08:00, lastProcessTime=2024-09-11T17:56:16.370474+08:00 2024-09-11 17:56:26.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:56:26.351962+08:00 2024-09-11 17:56:26.687 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:26.688 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:27.328 INFO 2981023 --- [ocp-async-4,9122f12eaa28efc5,cc218e8999bb48ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:27.768 INFO 2981023 --- [ocp-async-2,eb306c80d3875cc0,899dc12c81ba0919] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:56:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,b5a489094236448c,bd43612eb1e66288] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:29.318 ERROR 2981023 --- [prometheus-minute-schedule-1,b5a489094236448c,bd43612eb1e66288] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:29.318 ERROR 2981023 --- [prometheus-minute-schedule-1,b5a489094236448c,bd43612eb1e66288] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:29.318 INFO 2981023 --- [prometheus-minute-schedule-1,b5a489094236448c,bd43612eb1e66288] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:29.328 INFO 2981023 --- [ocp-async-3,1cd8ac3f6c2780e8,bff78d44985ab889] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:29.688 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:29.689 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:30.101 ERROR 2981023 --- [prometheus-second-schedule-1,e9572dfae82ef6a2,454ba87b260175e0] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/node/host java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collect(OcpMetricCollectServiceImpl.java:130) at com.oceanbase.ocp.monitor.OcpMonitorManager$CollectSecondTask.run(OcpMonitorManager.java:225) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:31.328 INFO 2981023 --- [ocp-async-8,299dc2ab4b429328,0953ad423f267384] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:32.312 INFO 2981023 --- [ocp-schedules-4,4ed9fbe7bcd5e94b,856609e0f313cd26] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:56:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,c2d8762c8fec7ccc,145c7fa3dd63bc67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:32.316 ERROR 2981023 --- [prometheus-minute-schedule-1,c2d8762c8fec7ccc,145c7fa3dd63bc67] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:32.316 ERROR 2981023 --- [prometheus-minute-schedule-1,c2d8762c8fec7ccc,145c7fa3dd63bc67] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,c2d8762c8fec7ccc,145c7fa3dd63bc67] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:32.690 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:32.690 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:33.328 INFO 2981023 --- [ocp-async-4,958197d7870d2761,854eb64a94ac7815] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:34.119 INFO 2981023 --- [ocp-schedules-17,d5f39208d9e4b303,1d0d0f1189529c91] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:56:34.166 WARN 2981023 --- [scanner-13576127-1,d5f39208d9e4b303,71f92d366de6d655] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:56:34.685 INFO 2981023 --- [ocp-schedules-27,d25b6d856f6b9876,2d54c1c76d928cc0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:56:34.693 INFO 2981023 --- [ocp-schedules-27,d25b6d856f6b9876,2d54c1c76d928cc0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:56:34.693 INFO 2981023 --- [ocp-schedules-27,d25b6d856f6b9876,2d54c1c76d928cc0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:56:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,c8a8cf5e0ecc187d,97bb530c69aa89eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:35.316 ERROR 2981023 --- [prometheus-minute-schedule-1,c8a8cf5e0ecc187d,97bb530c69aa89eb] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:35.316 ERROR 2981023 --- [prometheus-minute-schedule-1,c8a8cf5e0ecc187d,97bb530c69aa89eb] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,c8a8cf5e0ecc187d,97bb530c69aa89eb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:35.328 INFO 2981023 --- [ocp-async-5,3801e34ab8ebbd11,359469111646cbfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:35.690 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:35.691 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:56:36.353174+08:00, lastProcessTime=2024-09-11T17:56:26.351962+08:00 2024-09-11 17:56:36.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:56:36.353174+08:00 2024-09-11 17:56:37.314 INFO 2981023 --- [prometheus-cache-monitor1,eb4b012abce4fdd5,21f1b83b2546007d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7228,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":1813,"seriesIdKeySize":11634,"secondWriteQueue":0,"minuteWriteCacheSize":4530} 2024-09-11 17:56:37.315 INFO 2981023 --- [prometheus-cache-monitor1,eb4b012abce4fdd5,21f1b83b2546007d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:56:37.328 INFO 2981023 --- [ocp-async-10,fd790565aea64c2b,7633ba4753ecefb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:37.768 INFO 2981023 --- [ocp-async-9,d98c289dcc2a735d,2759d861a6f64583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:56:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,a1744d5a78a94886,6bb7d1a37fb829c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:38.316 ERROR 2981023 --- [prometheus-minute-schedule-1,a1744d5a78a94886,6bb7d1a37fb829c1] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:38.316 ERROR 2981023 --- [prometheus-minute-schedule-1,a1744d5a78a94886,6bb7d1a37fb829c1] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,a1744d5a78a94886,6bb7d1a37fb829c1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:38.691 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:38.691 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:39.328 INFO 2981023 --- [ocp-async-7,fb87ecfdfa3df0a0,d012f0615eb509ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:40.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:56:40.003 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:56:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,faf14a80761907a4,0816ac3344bbdd3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:41.316 ERROR 2981023 --- [prometheus-minute-schedule-1,faf14a80761907a4,0816ac3344bbdd3f] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:41.316 ERROR 2981023 --- [prometheus-minute-schedule-1,faf14a80761907a4,0816ac3344bbdd3f] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,faf14a80761907a4,0816ac3344bbdd3f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:41.328 INFO 2981023 --- [ocp-async-9,51ced75862b2a387,7e10d2c1f893de60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:41.691 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:41.692 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:42.315 INFO 2981023 --- [ocp-schedules-7,d1d06a65ca15802f,16b7c84830d870cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:56:42.837 INFO 2981023 --- [ocp-schedules-2,4536a212ddd42f8b,5bbf81fef5d72066] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:56:42.861 WARN 2981023 --- [ocp-schedules-2,4536a212ddd42f8b,5bbf81fef5d72066] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:56:43.328 INFO 2981023 --- [ocp-async-2,578adaeb8ca4545a,f618c659ff93901b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:44.308 INFO 2981023 --- [ocp-schedules-16,9794741e6c4d48a4,777a19bd504de4c3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:56:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,753be98cd75ad6cf,d1b394b697b24bac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:44.320 ERROR 2981023 --- [prometheus-minute-schedule-1,753be98cd75ad6cf,d1b394b697b24bac] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:44.320 ERROR 2981023 --- [prometheus-minute-schedule-1,753be98cd75ad6cf,d1b394b697b24bac] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:44.320 INFO 2981023 --- [prometheus-minute-schedule-1,753be98cd75ad6cf,d1b394b697b24bac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:44.363 WARN 2981023 --- [scanner-13576340-1,9794741e6c4d48a4,5b4bee7050f9a078] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:56:44.380 INFO 2981023 --- [alarm-detect-14,9794741e6c4d48a4,b4902807df8af3e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645036, data is empty. 2024-09-11 17:56:44.389 INFO 2981023 --- [alarm-detect-14,9794741e6c4d48a4,b4902807df8af3e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645036, data is empty. 2024-09-11 17:56:44.394 INFO 2981023 --- [alarm-detect-14,9794741e6c4d48a4,b4902807df8af3e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645036, data is empty. 2024-09-11 17:56:44.399 INFO 2981023 --- [alarm-detect-11,9794741e6c4d48a4,c744063095ab80d5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645036, data is empty. 2024-09-11 17:56:44.402 INFO 2981023 --- [alarm-detect-11,9794741e6c4d48a4,c744063095ab80d5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645036, data is empty. 2024-09-11 17:56:44.406 INFO 2981023 --- [alarm-detect-16,9794741e6c4d48a4,a7e9053d11fdb71d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645036, data is empty. 2024-09-11 17:56:44.409 INFO 2981023 --- [alarm-detect-11,9794741e6c4d48a4,c744063095ab80d5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645036, data is empty. 2024-09-11 17:56:44.412 INFO 2981023 --- [alarm-detect-16,9794741e6c4d48a4,a7e9053d11fdb71d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645036, data is empty. 2024-09-11 17:56:44.417 INFO 2981023 --- [alarm-detect-16,9794741e6c4d48a4,a7e9053d11fdb71d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645036, data is empty. 2024-09-11 17:56:44.523 INFO 2981023 --- [alarm-detect-3,9794741e6c4d48a4,ebbff8fe9b9ea726] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645036, data is empty. 2024-09-11 17:56:44.527 INFO 2981023 --- [alarm-detect-3,9794741e6c4d48a4,ebbff8fe9b9ea726] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645036, data is empty. 2024-09-11 17:56:44.530 INFO 2981023 --- [alarm-detect-3,9794741e6c4d48a4,ebbff8fe9b9ea726] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645036, data is empty. 2024-09-11 17:56:44.692 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:44.692 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:45.101 ERROR 2981023 --- [prometheus-second-schedule-1,0b25d140ebb6cc12,22caef6f384418a9] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/node/host java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collect(OcpMetricCollectServiceImpl.java:130) at com.oceanbase.ocp.monitor.OcpMonitorManager$CollectSecondTask.run(OcpMonitorManager.java:225) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:45.101 INFO 2981023 --- [prometheus-second-schedule-1,0b25d140ebb6cc12,22caef6f384418a9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:56:45.329 INFO 2981023 --- [ocp-async-4,1c693e96bf0c436e,97d292dee8e65013] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:46.018 INFO 2981023 --- [ocp-schedules-27,6533d4bb617dd062,c7100f1e339659b3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:56:46.018 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:56:46.018 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:56:46.019 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:56:46.019 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:56:46.021 INFO 2981023 --- [ocp-schedules-16,8707a2f0f078b212,fdecf56921e1a6cf] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:56:46.025 INFO 2981023 --- [host-check-clockdiff1,8707a2f0f078b212,3fa161c55b3d477b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:56:46.026 INFO 2981023 --- [host-check-clockdiff2,8707a2f0f078b212,10d8566797ca546f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:56:46.026 INFO 2981023 --- [host-check-clockdiff3,8707a2f0f078b212,1404a8dbc38c4a7b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:56:46.027 INFO 2981023 --- [host-check-clockdiff4,8707a2f0f078b212,d4384cca098f18ed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:56:46.032 INFO 2981023 --- [ocp-schedules-28,d1a9370f6fd7c137,f671ce00bd376f40] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:56:46.044 INFO 2981023 --- [host-check-agent-status1,d1a9370f6fd7c137,052eb8f00b50d728] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:56:46.052 INFO 2981023 --- [host-check-agent-status2,d1a9370f6fd7c137,e851d0add4928384] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:56:46.052 INFO 2981023 --- [host-check-agent-status4,d1a9370f6fd7c137,0601bfdd1ab26e5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:56:46.057 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:56:46.061 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:56:46.062 INFO 2981023 --- [host-check-agent-status4,d1a9370f6fd7c137,0601bfdd1ab26e5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:56:46.063 INFO 2981023 --- [host-check-agent-status4,d1a9370f6fd7c137,0601bfdd1ab26e5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:56:46.064 INFO 2981023 --- [host-check-agent-status3,d1a9370f6fd7c137,db48ede1299c2b73] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:56:46.068 INFO 2981023 --- [host-check-clockdiff2,8707a2f0f078b212,10d8566797ca546f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048606 6 6 , error= 2024-09-11 17:56:46.068 INFO 2981023 --- [host-check-clockdiff3,8707a2f0f078b212,1404a8dbc38c4a7b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048606 0 0 , error= 2024-09-11 17:56:46.071 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:56:46.072 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:56:46.168 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:56:46.190 INFO 2981023 --- [ocp-schedules-4,0e669037a2cd8cfc,e7eb67514b58880a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:56:46.190 INFO 2981023 --- [ocp-schedules-26,8294f77509ebcb42,2bb810f1137222de] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:56:46.190 INFO 2981023 --- [ocp-schedules-14,0cbd683824d96c0e,15a01c742e2f2b08] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:56:46.190 INFO 2981023 --- [ocp-schedules-5,f5bf6c663fa9e048,f863a0ee67e6df6c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:56:46.191 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:56:46.201 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:56:46.201 INFO 2981023 --- [host-check-agent-status1,d1a9370f6fd7c137,052eb8f00b50d728] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:56:46.234 INFO 2981023 --- [ocp-schedules-13,2806de47d372b8ac,f9060bb75920829b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 17:56:46.234 INFO 2981023 --- [ocp-schedules-13,2806de47d372b8ac,f9060bb75920829b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 17:56:46.260 INFO 2981023 --- [ocp-schedules-26,8294f77509ebcb42,2bb810f1137222de] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:46.260 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:56:46.261 INFO 2981023 --- [ocp-schedules-4,0e669037a2cd8cfc,e7eb67514b58880a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:46.261 INFO 2981023 --- [ocp-schedules-14,0cbd683824d96c0e,15a01c742e2f2b08] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:46.261 INFO 2981023 --- [ocp-schedules-11,e840af57bf526ae7,fc7f69dc7535dacb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:56:46.262 INFO 2981023 --- [host-check-agent-status1,d1a9370f6fd7c137,052eb8f00b50d728] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:56:46.263 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:56:46.268 INFO 2981023 --- [host-check-agent-status3,d1a9370f6fd7c137,db48ede1299c2b73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:56:46.268 INFO 2981023 --- [host-check-agent-status2,d1a9370f6fd7c137,e851d0add4928384] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:56:46.271 INFO 2981023 --- [host-check-agent-status3,d1a9370f6fd7c137,db48ede1299c2b73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:56:46.275 INFO 2981023 --- [host-check-agent-status2,d1a9370f6fd7c137,e851d0add4928384] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:56:46.293 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:56:46.295 INFO 2981023 --- [ocp-schedules-19,5ed9f42373737a5b,44adeb7176c6cac7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:56:46.295 INFO 2981023 --- [ocp-schedules-11,e840af57bf526ae7,fc7f69dc7535dacb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:56:46.298 INFO 2981023 --- [ocp-schedules-26,8294f77509ebcb42,2bb810f1137222de] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:56:46.302 INFO 2981023 --- [ocp-schedules-6,ca3c3d82572c9f3f,e2dc31228162ac88] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 17:56:46.303 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:56:46.305 INFO 2981023 --- [ocp-schedules-6,ca3c3d82572c9f3f,e2dc31228162ac88] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:46.306 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:56:46.312409602+08:00 2024-09-11 17:56:46.306 INFO 2981023 --- [host-check-agent-status4,d1a9370f6fd7c137,0601bfdd1ab26e5d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:56:46.307 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:56:46.306+08:00 2024-09-11 17:56:46.308 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:56:46.310 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:56:46.316664332+08:00 2024-09-11 17:56:46.311 INFO 2981023 --- [host-check-agent-status4,d1a9370f6fd7c137,0601bfdd1ab26e5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:56:46.311 INFO 2981023 --- [host-check-agent-status4,d1a9370f6fd7c137,0601bfdd1ab26e5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:56:46.311 INFO 2981023 --- [host-check-agent-status4,d1a9370f6fd7c137,0601bfdd1ab26e5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:56:46.311 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:56:46.306+08:00 and status AVAILABLE 2024-09-11 17:56:46.314 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:56:46.310+08:00 2024-09-11 17:56:46.361 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:56:46.310+08:00 and status AVAILABLE 2024-09-11 17:56:46.365 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:56:46.394 INFO 2981023 --- [host-check-agent-status1,d1a9370f6fd7c137,052eb8f00b50d728] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:56:46.394 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:56:46.397 INFO 2981023 --- [host-check-agent-status1,d1a9370f6fd7c137,052eb8f00b50d728] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:56:46.397 INFO 2981023 --- [host-check-agent-status1,d1a9370f6fd7c137,052eb8f00b50d728] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:56:46.397 INFO 2981023 --- [host-check-agent-status1,d1a9370f6fd7c137,052eb8f00b50d728] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:56:46.399 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:56:46.401 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:56:46.401 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:56:46.398564+08:00, lastProcessTime=2024-09-11T17:56:36.353174+08:00 2024-09-11 17:56:46.402 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T17:56:46.404068386+08:00 2024-09-11 17:56:46.404 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T17:56:46.402+08:00 2024-09-11 17:56:46.406 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:56:46.405833693+08:00 2024-09-11 17:56:46.408 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:56:46.406+08:00 2024-09-11 17:56:46.409 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T17:56:46.402+08:00 and status AVAILABLE 2024-09-11 17:56:46.410 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:56:46.406+08:00 and status AVAILABLE 2024-09-11 17:56:46.454 INFO 2981023 --- [host-check-agent-status3,d1a9370f6fd7c137,db48ede1299c2b73] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:56:46.458 INFO 2981023 --- [host-check-agent-status3,d1a9370f6fd7c137,db48ede1299c2b73] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:56:46.458 INFO 2981023 --- [host-check-agent-status3,d1a9370f6fd7c137,db48ede1299c2b73] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:56:46.458 INFO 2981023 --- [host-check-agent-status3,d1a9370f6fd7c137,db48ede1299c2b73] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:56:46.466 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:56:46.469 INFO 2981023 --- [host-check-agent-status2,d1a9370f6fd7c137,e851d0add4928384] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:56:46.476 INFO 2981023 --- [host-check-agent-status2,d1a9370f6fd7c137,e851d0add4928384] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:56:46.476 INFO 2981023 --- [host-check-agent-status2,d1a9370f6fd7c137,e851d0add4928384] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:56:46.476 INFO 2981023 --- [host-check-agent-status2,d1a9370f6fd7c137,e851d0add4928384] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:56:46.478 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.479 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:56:46.487 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:56:46.487 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,244baeee7fd56744,3781cb0514caef4b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=244baeee7fd56744, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 17:56:46.487 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,244baeee7fd56744,3781cb0514caef4b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=244baeee7fd56744, duration=1 ms] 2024-09-11 17:56:46.489 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.490 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 17:56:46.490 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:56:46.491 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:56:46.491 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:56:46.491 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:56:46.491 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.491 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.492 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:56:46.493 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.493 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:56:46.494 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:56:46.494 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:56:46.496 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.500 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:46.500 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.500 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.504 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:46.504 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,facfff9105548ded,9c8702ed0061f780] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&ObClusterId=1719994443&version=2, client=10.31.35.10, traceId=facfff9105548ded, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkrtd,1719994443,false,2,] 2024-09-11 17:56:46.505 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,facfff9105548ded,9c8702ed0061f780] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=facfff9105548ded, duration=1 ms] 2024-09-11 17:56:46.505 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:56:46.506 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.507 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:56:46.507 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:46.509 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,c8073c930cf45f83,57b2648929200c10] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=c8073c930cf45f83, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 17:56:46.509 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,c8073c930cf45f83,57b2648929200c10] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=c8073c930cf45f83, duration=0 ms] 2024-09-11 17:56:46.510 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.510 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.511 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:46.511 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 17:56:46.512 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.513 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:56:46.513 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.513 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 17:56:46.514 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:46.514 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:56:46.514 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.515 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:46.516 INFO 2981023 --- [ocp-schedules-4,0e669037a2cd8cfc,e7eb67514b58880a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:56:46.516 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.518 INFO 2981023 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 17:56:46.521 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 17:56:46.522 INFO 2981023 --- [ocp-schedules-6,ca3c3d82572c9f3f,e2dc31228162ac88] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 17:56:46.536 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:56:46.554 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:56:46.558 INFO 2981023 --- [pool-10-thread-7,f1d80b229ef02d40,47421cedf14b5b7f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 17:56:46.559 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.560 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:56:46.560 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:56:46.561 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.567 INFO 2981023 --- [ForkJoinPool-6-worker-6,f1d80b229ef02d40,bd6e67574a1b3665] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 17:56:46.567 INFO 2981023 --- [ForkJoinPool-6-worker-6,f1d80b229ef02d40,bd6e67574a1b3665] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 17:56:46.567 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 17:56:46.567 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 17:56:46.567 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 17:56:46.567 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 17:56:46.568 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:56:46.574 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:56:46.581 INFO 2981023 --- [pool-10-thread-8,f1d80b229ef02d40,36102f409d82beed] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 17:56:46.582 INFO 2981023 --- [pool-10-thread-8,f1d80b229ef02d40,36102f409d82beed] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 17:56:46.582 INFO 2981023 --- [pool-10-thread-5,f1d80b229ef02d40,e6db3d7f4b0858a7] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 17:56:46.593 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:56:46.596 INFO 2981023 --- [pool-10-thread-6,f1d80b229ef02d40,5d122c92a7f1df63] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 17:56:46.612 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.613 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:56:46.614 INFO 2981023 --- [pool-10-thread-5,f1d80b229ef02d40,e6db3d7f4b0858a7] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 17:56:46.614 INFO 2981023 --- [pool-10-thread-5,f1d80b229ef02d40,e6db3d7f4b0858a7] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 17:56:46.615 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.615 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:56:46.620 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:56:46.623 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:56:46.623 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.625 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:56:46.625 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.634 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:56:46.640 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:56:46.647 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.649 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:56:46.650 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.650 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:56:46.651 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 17:56:46.656 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:56:46.659 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 17:56:46.661 WARN 2981023 --- [ForkJoinPool-6-worker-6,f1d80b229ef02d40,bd6e67574a1b3665] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 17:56:46.667 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:56:46.677 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:56:46.686 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:56:46.687 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:56:46.687 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 17:56:46.688 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.689 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.692 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:56:46.694 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.697 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:56:46, 2024-09-11 17:56:46] 2024-09-11 17:56:46.699 INFO 2981023 --- [ForkJoinPool-6-worker-10,f1d80b229ef02d40,91b50decf0dcadb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.700 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 17:56:46.700 INFO 2981023 --- [ForkJoinPool-6-worker-15,f1d80b229ef02d40,302b930d9b71ae04] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 17:56:46.701 INFO 2981023 --- [ForkJoinPool-6-worker-10,f1d80b229ef02d40,91b50decf0dcadb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:56:46.702 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.706 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 17:56:46.721 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.721 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.722 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:56:46.724 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.724 INFO 2981023 --- [ForkJoinPool-6-worker-10,f1d80b229ef02d40,91b50decf0dcadb9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 17:56:46.724 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.724 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:46.725 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 17:56:46.725 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:56:46.725 INFO 2981023 --- [ForkJoinPool-6-worker-10,f1d80b229ef02d40,91b50decf0dcadb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.726 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.726 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 17:56:46.726 INFO 2981023 --- [ForkJoinPool-6-worker-10,f1d80b229ef02d40,91b50decf0dcadb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:56:46.726 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.727 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 17:56:46.729 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:56:46.729 INFO 2981023 --- [ForkJoinPool-6-worker-10,f1d80b229ef02d40,91b50decf0dcadb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.730 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.730 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:56:46.730 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-10,f1d80b229ef02d40,91b50decf0dcadb9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:56:46.731 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:56:46.732 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:56:46.732 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.753 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:56:46.759 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:56:46.766 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:56:46.771 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:56:46.774 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:56:46.774 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.775 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.776 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:46.777 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.780 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:56:46.780 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:56:46.780 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:56:46.781 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.782 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:56:46.783 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.784 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:56:46.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:46.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:56:46.786 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:56:46.790 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 17:56:46.791 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.792 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:56:46, 2024-09-11 17:56:46] 2024-09-11 17:56:46.803 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:56:46.805 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:46.805 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:56:46.809 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:56:46.809 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:56:46.816 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.823 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:46.825 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:56:46.826 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.829 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:56:46.833 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 17:56:46.835 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 17:56:46.838 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:56:46.839 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.840 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 17:56:46, 2024-09-11 17:56:46] 2024-09-11 17:56:46.841 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 17:56:46.841 INFO 2981023 --- [ForkJoinPool-6-worker-13,f1d80b229ef02d40,0d85750e01014797] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 17:56:46.841 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.842 INFO 2981023 --- [pool-10-thread-7,f1d80b229ef02d40,47421cedf14b5b7f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 17:56:46.848 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:56:46.848 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 17:56:46.851 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.853 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 17:56:46.854 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.855 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.857 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:56:46.858 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:56:46.858 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.859 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:46.859 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:56:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:56:46.860 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:56:46.860 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.863 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.863 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.864 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:46.864 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:56:46.869 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.870 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.870 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:46.870 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:56:46.871 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:56:46.879 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:56:46.881 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:56:46.883 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:56:46.885 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 17:56:46.886 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.887 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 17:56:46.888 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.891 INFO 2981023 --- [ForkJoinPool-6-worker-8,f1d80b229ef02d40,ac6642aafe2309cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 17:56:46.891 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:56:46.891 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:56:46.891 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:56:46.892 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:56:46.893 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:56:46.893 INFO 2981023 --- [pool-10-thread-6,f1d80b229ef02d40,5d122c92a7f1df63] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 17:56:46.894 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:56:46.895 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:56:46.906 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.907 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.908 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:56:46.909 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:56:46.909 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.910 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:56:46.913 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:56:46.914 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:56:46.914 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 17:56:46.916 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:56:46.929 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:56:46.937 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:46.937 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:56:46.937 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:56:46.939 INFO 2981023 --- [host-check-host-status1,6533d4bb617dd062,1f39229d35324b3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:56:46.939 INFO 2981023 --- [host-check-host-status4,6533d4bb617dd062,7bff4007129a1b33] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 17:56:46.945 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:56:46.950 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.951 INFO 2981023 --- [host-check-host-status2,6533d4bb617dd062,f68917fbd29e3e79] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:56:46.953 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.954 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:56:46.954 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:56:46.954 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.956 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:56:46.969 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:56:46.970 INFO 2981023 --- [host-check-host-status3,6533d4bb617dd062,731fef10ed2dbfe9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:56:46.975 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:56:46.987 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:46.988 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:46.988 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:56:46.989 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:56:46.993 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:56:46.994 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:56:46.997 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:56:46.997 INFO 2981023 --- [pool-10-thread-8,f1d80b229ef02d40,36102f409d82beed] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 17:56:47.000 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:56:47.001 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:56:47.008 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:47.008 INFO 2981023 --- [pool-10-thread-8,f1d80b229ef02d40,36102f409d82beed] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 17:56:47.009 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:47.009 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:56:47.021 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:56:47.021 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:56:47.021 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:47.022 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:47.023 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:56:47.023 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:56:47.024 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:47.026 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:56:47.041 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:56:47.049 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:47.049 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:47.049 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:56:47.057 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:56:47.057 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:56:47.059 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:56:47.073 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:47.074 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:47.075 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:56:47.076 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:56:46.398564+08:00 2024-09-11 17:56:47.079 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:56:47.080 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:56:47.080 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:56:47.080 INFO 2981023 --- [ocp-schedules-14,0cbd683824d96c0e,15a01c742e2f2b08] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:56:47.094 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:56:47.100 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:56:47.102 INFO 2981023 --- [ocp-schedules-25,9c98ac87729b131f,b416ba907c13156f] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:56:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,beefb52748e0949e,b9824ece20a749c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:47.316 ERROR 2981023 --- [prometheus-minute-schedule-1,beefb52748e0949e,b9824ece20a749c9] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:47.316 ERROR 2981023 --- [prometheus-minute-schedule-1,beefb52748e0949e,b9824ece20a749c9] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,beefb52748e0949e,b9824ece20a749c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:47.329 INFO 2981023 --- [ocp-async-9,6adb5ba6d2b21f25,a2e45deeae0333f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:47.692 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:47.692 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:47.768 INFO 2981023 --- [ocp-async-7,a00c032c1b703bbf,ddfb0a40c7f97f45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:56:48.569 INFO 2981023 --- [host-check-clockdiff1,8707a2f0f078b212,3fa161c55b3d477b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048608 6 6 , error= 2024-09-11 17:56:49.329 INFO 2981023 --- [ocp-async-8,b9bb382037d966ac,8dafa6f270660253] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:56:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:56:50.276 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:56:50.280 INFO 2981023 --- [ocp-schedules-1,33bbf02faaedd5a7,870d59d9d2573da7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.281 INFO 2981023 --- [ocp-schedules-29,88fd1428c5346a78,6cad864f7af90ca6] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:56:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,22f3556b03aede02,65d17537ee847ffb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:50.317 ERROR 2981023 --- [prometheus-minute-schedule-1,22f3556b03aede02,65d17537ee847ffb] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:50.317 ERROR 2981023 --- [prometheus-minute-schedule-1,22f3556b03aede02,65d17537ee847ffb] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,22f3556b03aede02,65d17537ee847ffb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:50.692 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:50.694 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:51.329 INFO 2981023 --- [ocp-async-7,903f8fb2ddafb4fa,fb76850feb44eb40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:51.581 INFO 2981023 --- [ocp-schedules-6,d71153b9c9efcea0,d6b90606b01d1346] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:56:51.582 INFO 2981023 --- [ocp-schedules-6,d71153b9c9efcea0,d6b90606b01d1346] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:51.583 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:56:51.583 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:56:51.583 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:56:51.583 INFO 2981023 --- [ocp-schedules-6,d71153b9c9efcea0,d6b90606b01d1346] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:56:52.002 INFO 2981023 --- [host-check-clockdiff4,8707a2f0f078b212,d4384cca098f18ed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048611 2 2 , error= 2024-09-11 17:56:52.318 INFO 2981023 --- [ocp-schedules-22,2ad8e4be6afb1d4a,0bf054f133fe4323] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:56:53.195 INFO 2981023 --- [ocp-schedules-8,50963db940f15132,8f8991e7c9d9f805] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:56:53.195 INFO 2981023 --- [ocp-schedules-8,50963db940f15132,8f8991e7c9d9f805] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:56:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,62862e76374db740,04d72c4c94b37fd3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:53.316 ERROR 2981023 --- [prometheus-minute-schedule-1,62862e76374db740,04d72c4c94b37fd3] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:53.316 ERROR 2981023 --- [prometheus-minute-schedule-1,62862e76374db740,04d72c4c94b37fd3] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,62862e76374db740,04d72c4c94b37fd3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:53.329 INFO 2981023 --- [ocp-async-3,67bb64a71e8cdc6c,f4ef3a28d826a0c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:53.694 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:53.695 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:53.822 INFO 2981023 --- [ocp-schedules-9,ad42601af5ac4de0,b01e53731b3df23d] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:56:53.841 WARN 2981023 --- [ocp-schedules-9,ad42601af5ac4de0,b01e53731b3df23d] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:56:54.334 INFO 2981023 --- [ocp-schedules-13,00a8043da7a9aae7,f699dc2177c23cb6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:56:54.564 INFO 2981023 --- [ocp-schedules-23,2556749925235dc4,50c36b6df0ae9b67] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:56:54.630 WARN 2981023 --- [scanner-13576667-1,2556749925235dc4,e83761ca8b52bcd9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:56:54.700 INFO 2981023 --- [ocp-schedules-6,c10fd5a5940da8a7,7480a912f8e3b874] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:56:54.727 INFO 2981023 --- [ocp-schedules-6,c10fd5a5940da8a7,7480a912f8e3b874] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:56:54.727 INFO 2981023 --- [ocp-schedules-6,c10fd5a5940da8a7,7480a912f8e3b874] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:56:55.329 INFO 2981023 --- [ocp-async-2,04d808e275aab9ad,e7885920377d5763] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:55.805 INFO 2981023 --- [ocp-schedules-25,95f8fe10887017cd,aa347d9659374d64] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:56:55.805 INFO 2981023 --- [ocp-schedules-25,95f8fe10887017cd,aa347d9659374d64] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:56:55.806 INFO 2981023 --- [ocp-schedules-25,95f8fe10887017cd,aa347d9659374d64] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:56:55.806 INFO 2981023 --- [ocp-schedules-25,95f8fe10887017cd,aa347d9659374d64] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:56:55.807 INFO 2981023 --- [ocp-schedules-25,95f8fe10887017cd,aa347d9659374d64] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:56:55.807 INFO 2981023 --- [ocp-schedules-25,95f8fe10887017cd,aa347d9659374d64] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:56:55.914 INFO 2981023 --- [ocp-schedules-22,7da3a2e7901d3940,63500ec2059b3de6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:56:55.914+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:56:55.920 INFO 2981023 --- [ocp-schedules-22,7da3a2e7901d3940,63500ec2059b3de6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:56:55.929 INFO 2981023 --- [ocp-schedules-12,f97955581711b902,b081491eacf73c86] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:56:55.937 INFO 2981023 --- [ocp-schedules-12,f97955581711b902,b081491eacf73c86] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:56:55.937 INFO 2981023 --- [ocp-schedules-12,f97955581711b902,b081491eacf73c86] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:56:56.036 INFO 2981023 --- [ocp-schedules-22,7da3a2e7901d3940,63500ec2059b3de6] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 40 to SqlAggregateData 2024-09-11 17:56:56.037 INFO 2981023 --- [ocp-schedules-22,7da3a2e7901d3940,63500ec2059b3de6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:56:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,b3e48a63044056c5,6cee41a0986eb934] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:56.316 ERROR 2981023 --- [prometheus-minute-schedule-1,b3e48a63044056c5,6cee41a0986eb934] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:56.316 ERROR 2981023 --- [prometheus-minute-schedule-1,b3e48a63044056c5,6cee41a0986eb934] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,b3e48a63044056c5,6cee41a0986eb934] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:56:56.351877+08:00, lastProcessTime=2024-09-11T17:56:46.398564+08:00 2024-09-11 17:56:56.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:56:56.351877+08:00 2024-09-11 17:56:56.649 INFO 2981023 --- [ocp-schedules-8,6561ddf5b03bbf73,084941237491753c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:56:56.650 INFO 2981023 --- [ocp-schedules-8,6561ddf5b03bbf73,084941237491753c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:56:56.684 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:56:56.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:56:56.684 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.684 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:56:56.684 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.687 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:56:56.688 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:56:56.689 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:56:56.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.695 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:56.697 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:56:56.697 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.697 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.699 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.699 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.704 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.705 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.706 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.706 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.712 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.712 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.713 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:56:56.713 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.719 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.720 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.720 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:56:56.720 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3F21D6DA-0-0] 2024-09-11 17:56:56.721 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:56:56.721 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.721 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:56:56.726 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.726 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:56:56.729 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.729 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.734 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.734 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:56:56.734 INFO 2981023 --- [ocp-schedules-16,bdcc2ef4976b39c7,de63d3701b6b457c] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:56:56.735 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:56:56.740 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.741 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:56:56.741 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.742 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:56:56.743 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.744 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:56:56.744 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:56:56.746 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.746 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:56:56.747 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:56:56.747 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE3F21D6E5-0-0] 2024-09-11 17:56:56.747 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:56:56.751 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:56:56.754 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.755 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:56:56.756 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.756 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:56:56.757 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.757 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:56:56.758 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:56:56.759 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.760 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:56:56.760 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.761 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:56:56.762 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.762 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:56:56.762 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:56:56.770 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:56:56.774 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:56.774 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.775 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:56:56.776 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.776 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:56:56.776 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.777 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.777 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.777 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:56:56.778 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.778 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:56:56.780 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.792 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:56.792 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:56:56.793 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:56:56.794 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.795 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.795 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:56:56.795 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:56:56.796 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.796 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:56:56.797 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:56:56.797 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:56:56.797 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:56:56.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:56:56.798 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:56:56.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:56:56.801 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:56:56.806 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:56:56.809 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:56:56.809 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.809 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.815 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:56:56.817 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.817 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:56:56.820 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:56:56.823 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:56:56.825 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:56:56.825 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:56:56.825 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:56:56.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:56:56.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.826 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:56:56.832 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:56:56.833 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:56:56.834 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:56.834 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:56:56.840 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:56:56.842 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:56:56.842 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:56:56.842 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:56:56.842 INFO 2981023 --- [ocp-schedules-8,6561ddf5b03bbf73,084941237491753c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:56:57.012 INFO 2981023 --- [ocp-schedules-19,4a66c3515468136f,f9395f6eb9634978] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:56:57.012 INFO 2981023 --- [ocp-schedules-19,4a66c3515468136f,f9395f6eb9634978] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:56:57.027 INFO 2981023 --- [Collect all tenants major compactions1,4a66c3515468136f,aa2b8295e623875a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:56:57.027 INFO 2981023 --- [Collect all tenants major compactions2,4a66c3515468136f,178bff4c1cb34c3a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:56:57.029 INFO 2981023 --- [Collect all tenants major compactions3,4a66c3515468136f,c203888b3b480f23] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:56:57.030 INFO 2981023 --- [Collect all tenants major compactions5,4a66c3515468136f,e6d7d4fde08d7204] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:56:57.032 INFO 2981023 --- [Collect all tenants major compactions4,4a66c3515468136f,e31f28d9b854ed6b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:56:57.033 INFO 2981023 --- [Collect all tenants major compactions6,4a66c3515468136f,7a242ff298964f44] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:56:57.041 INFO 2981023 --- [Collect all tenants major compactions1,4a66c3515468136f,aa2b8295e623875a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:57.041 INFO 2981023 --- [Collect all tenants major compactions3,4a66c3515468136f,c203888b3b480f23] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:57.041 INFO 2981023 --- [Collect all tenants major compactions2,4a66c3515468136f,178bff4c1cb34c3a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:57.042 INFO 2981023 --- [Collect all tenants major compactions4,4a66c3515468136f,e31f28d9b854ed6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:57.042 INFO 2981023 --- [Collect all tenants major compactions3,4a66c3515468136f,c203888b3b480f23] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:56:57.042 INFO 2981023 --- [Collect all tenants major compactions1,4a66c3515468136f,aa2b8295e623875a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:56:57.043 INFO 2981023 --- [Collect all tenants major compactions4,4a66c3515468136f,e31f28d9b854ed6b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:56:57.044 INFO 2981023 --- [Collect all tenants major compactions2,4a66c3515468136f,178bff4c1cb34c3a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:56:57.046 INFO 2981023 --- [Collect all tenants major compactions5,4a66c3515468136f,e6d7d4fde08d7204] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:57.047 INFO 2981023 --- [Collect all tenants major compactions5,4a66c3515468136f,e6d7d4fde08d7204] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:56:57.047 INFO 2981023 --- [Collect all tenants major compactions1,4a66c3515468136f,aa2b8295e623875a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:56:57.047 INFO 2981023 --- [Collect all tenants major compactions1,4a66c3515468136f,aa2b8295e623875a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:56:57.047 INFO 2981023 --- [Collect all tenants major compactions3,4a66c3515468136f,c203888b3b480f23] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:56:57.047 INFO 2981023 --- [Collect all tenants major compactions3,4a66c3515468136f,c203888b3b480f23] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:56:57.048 INFO 2981023 --- [Collect all tenants major compactions4,4a66c3515468136f,e31f28d9b854ed6b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:56:57.048 INFO 2981023 --- [Collect all tenants major compactions2,4a66c3515468136f,178bff4c1cb34c3a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:56:57.048 INFO 2981023 --- [Collect all tenants major compactions4,4a66c3515468136f,e31f28d9b854ed6b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:56:57.048 INFO 2981023 --- [Collect all tenants major compactions2,4a66c3515468136f,178bff4c1cb34c3a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:56:57.049 INFO 2981023 --- [Collect all tenants major compactions5,4a66c3515468136f,e6d7d4fde08d7204] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:56:57.049 INFO 2981023 --- [Collect all tenants major compactions5,4a66c3515468136f,e6d7d4fde08d7204] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:56:57.049 INFO 2981023 --- [Collect all tenants major compactions6,4a66c3515468136f,7a242ff298964f44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:56:57.051 INFO 2981023 --- [Collect all tenants major compactions6,4a66c3515468136f,7a242ff298964f44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:56:57.053 INFO 2981023 --- [Collect all tenants major compactions6,4a66c3515468136f,7a242ff298964f44] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:56:57.053 INFO 2981023 --- [Collect all tenants major compactions6,4a66c3515468136f,7a242ff298964f44] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:56:57.054 INFO 2981023 --- [ocp-schedules-19,4a66c3515468136f,f9395f6eb9634978] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:56:57.329 INFO 2981023 --- [ocp-async-6,9294e758ff1b57f7,8e7bbaa9b828359c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:57.768 INFO 2981023 --- [ocp-async-2,ce32e87a2e9a00db,d0fe542ba34b4b61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:56:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,01ce478bbc528b91,5683cc672269e9a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048620, interval=60 2024-09-11 17:56:59.316 ERROR 2981023 --- [prometheus-minute-schedule-1,01ce478bbc528b91,5683cc672269e9a0] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:59.316 ERROR 2981023 --- [prometheus-minute-schedule-1,01ce478bbc528b91,5683cc672269e9a0] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:56:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,01ce478bbc528b91,5683cc672269e9a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:56:59.329 INFO 2981023 --- [ocp-async-9,63965b86fa2eef2b,bb8ab014c55c2831] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:56:59.697 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:56:59.697 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:00.101 ERROR 2981023 --- [prometheus-second-schedule-1,6c2897035ebe3c14,85a1b37875bf361d] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/node/host java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collect(OcpMetricCollectServiceImpl.java:130) at com.oceanbase.ocp.monitor.OcpMonitorManager$CollectSecondTask.run(OcpMonitorManager.java:225) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:01.333 INFO 2981023 --- [ocp-async-2,26877bf1820934f7,b9afe6380fd43cca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,9ae4c615e7c5bead,b035c526a90c477e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:02.316 ERROR 2981023 --- [prometheus-minute-schedule-1,9ae4c615e7c5bead,b035c526a90c477e] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:02.316 ERROR 2981023 --- [prometheus-minute-schedule-1,9ae4c615e7c5bead,b035c526a90c477e] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,9ae4c615e7c5bead,b035c526a90c477e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:02.321 INFO 2981023 --- [ocp-schedules-16,4e0229e64e233647,145e4ea561c51593] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:57:02.697 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:02.697 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:03.330 INFO 2981023 --- [ocp-async-5,ce8e7995dfa359b8,0e8e8e3090b5e287] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:04.779 INFO 2981023 --- [ocp-schedules-28,595db31366b67ad4,e9517509df278a1f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:57:04.837 WARN 2981023 --- [scanner-13576873-1,595db31366b67ad4,7deca06c278f7ce6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:57:04.855 INFO 2981023 --- [alarm-detect-8,595db31366b67ad4,6849c866a9d3e561] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645036, data is empty. 2024-09-11 17:57:04.858 INFO 2981023 --- [alarm-detect-8,595db31366b67ad4,6849c866a9d3e561] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645076, data is empty. 2024-09-11 17:57:04.862 INFO 2981023 --- [alarm-detect-8,595db31366b67ad4,6849c866a9d3e561] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645076, data is empty. 2024-09-11 17:57:04.884 INFO 2981023 --- [alarm-detect-3,595db31366b67ad4,c965e57a27912f37] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645036, data is empty. 2024-09-11 17:57:04.887 INFO 2981023 --- [alarm-detect-3,595db31366b67ad4,c965e57a27912f37] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645076, data is empty. 2024-09-11 17:57:04.889 INFO 2981023 --- [alarm-detect-13,595db31366b67ad4,817d1efb621f13bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645036, data is empty. 2024-09-11 17:57:04.891 INFO 2981023 --- [alarm-detect-3,595db31366b67ad4,c965e57a27912f37] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645076, data is empty. 2024-09-11 17:57:04.893 INFO 2981023 --- [alarm-detect-13,595db31366b67ad4,817d1efb621f13bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645076, data is empty. 2024-09-11 17:57:04.897 INFO 2981023 --- [alarm-detect-13,595db31366b67ad4,817d1efb621f13bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645076, data is empty. 2024-09-11 17:57:05.008 INFO 2981023 --- [alarm-detect-12,595db31366b67ad4,aa49edc7b3be9555] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645036, data is empty. 2024-09-11 17:57:05.013 INFO 2981023 --- [alarm-detect-12,595db31366b67ad4,aa49edc7b3be9555] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645076, data is empty. 2024-09-11 17:57:05.017 INFO 2981023 --- [alarm-detect-12,595db31366b67ad4,aa49edc7b3be9555] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645076, data is empty. 2024-09-11 17:57:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,8f2ebd465cb677c8,7d1f3dd1ed748979] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:05.316 ERROR 2981023 --- [prometheus-minute-schedule-1,8f2ebd465cb677c8,7d1f3dd1ed748979] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:05.318 ERROR 2981023 --- [prometheus-minute-schedule-1,8f2ebd465cb677c8,7d1f3dd1ed748979] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:05.318 INFO 2981023 --- [prometheus-minute-schedule-1,8f2ebd465cb677c8,7d1f3dd1ed748979] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:05.330 INFO 2981023 --- [ocp-async-3,260d1ea2450b6cd8,da071d3e20232bc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:05.698 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:05.698 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:57:06.337608+08:00, lastProcessTime=2024-09-11T17:56:56.351877+08:00 2024-09-11 17:57:06.467 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:57:06.337608+08:00 2024-09-11 17:57:06.762 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:53:06.347, end=2024-09-11T17:55:06.757} 2024-09-11 17:57:06.765 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obCluste 2024-09-11 17:57:06.765 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:57:06.766 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:53:06.347, end=2024-09-11T17:55:06.757},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:57:06.766 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:57:06.766 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:57:06.767 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:57:06.769 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:53:06.347, end=2024-09-11T17:55:06.757},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:57:06.772 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:57:06.772 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:57:06.816 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226815000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.815Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.821 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226821000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.821Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.822 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226822000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.822Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.822 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226822000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.822Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.823 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226823000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.823Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.825 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226825000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.825Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.831 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 64 2024-09-11 17:57:06.831 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.832 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226832000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.832Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.833 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226833000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.833Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.839 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.839 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.862 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:57:06.862 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.862 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:57:06.862 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.865 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:57:06.865 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.866 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:57:06.866 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.873 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:57:06.873 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.874 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:57:06.874 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.877 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.877 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.878 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:57:06.878 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.881 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.881 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.881 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.881 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.882 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.882 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.882 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.882 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.883 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.883 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.885 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.885 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.885 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.885 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.885 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.885 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.885 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.885 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.889 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.889 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.890 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.890 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.891 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.891 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.894 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.894 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.894 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.895 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.895 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.896 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.896 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.896 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.899 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.899 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.901 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.901 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.901 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.823, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.903 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.903 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.903 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.903 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.903 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.905 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.905 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.906 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.906 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.907 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.908 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.822, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.912 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.815, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.915 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.915 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.822, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.916 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.916 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.916 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.821, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.917 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.825, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.919 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.921 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.833, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.926 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.832, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.936 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226936000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.936Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.941 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226941000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.941Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.947 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:57:06.947 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.948 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 92 2024-09-11 17:57:06.948 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.952 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226952000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.952Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.958 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.958 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.958 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226958000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=215632585E2CA5F0C915A1E0D150020F, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.958Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.959 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.959 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.961 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226961000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.961Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.963 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:57:06.963 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.966 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.966 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.968 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226968000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.968Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.969 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226969000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.969Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.970 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 92 2024-09-11 17:57:06.970 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.971 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.971 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.973 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 17:57:06.973 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.974 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962226974000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:06.974Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:06.980 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.980 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.982 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:57:06.982 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.984 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.985 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.985 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.987 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:57:06.988 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.988 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:57:06.988 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:06.989 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.989 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:06.990 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.990 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.991 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:06.995 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.936, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.996 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:06.996 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:06.997 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.941, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:06.998 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:06.998 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:07.001 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.001 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.001 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.001 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.001 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.001 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=215632585E2CA5F0C915A1E0D150020F, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.004 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:07.004 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:07.004 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:07.004 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:07.005 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.005 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.005 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:07.005 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:07.006 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.006 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.008 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:07.008 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:07.008 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:07.012 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.012 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.014 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:07.015 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.015 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.016 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.961, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:07.017 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.017 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=215632585E2CA5F0C915A1E0D150020F, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.018 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:07.019 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.019 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.019 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.952, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:07.019 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:07.022 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:07.023 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.968, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:07.023 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.023 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.025 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:07.025 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.958, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=215632585E2CA5F0C915A1E0D150020F, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:07.025 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962227025000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:07.025Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:07.026 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.969, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:07.028 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:06.974, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:07.033 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:57:07.033 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:07.038 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.038 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.039 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:07.039 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:07.045 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.045 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.046 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:57:07.048 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:07.025, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:07.049 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:57:07.049 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:53:06.347, end=2024-09-11T17:55:06.757},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:57:07.049 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:57:07.049 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:57:07.050 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:57:07.051 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:53:06.347, end=2024-09-11T17:55:06.757},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:57:07.051 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:57:07.051 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:57:07.061 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:57:07.075 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962227074000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:57:07.074Z, endTime=2024-09-11T09:55:06.757Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:57:07.077 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:57:07.077 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:57:07.082 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.082 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.083 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:57:07.083 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:57:07.088 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:57:07.088 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048506757000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:57:07.089 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:57:07.089 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:57:07.074, end=2024-09-11T17:55:06.757}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:57:07.090 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:57:07.090 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:53:06.347, end=2024-09-11T17:55:06.757},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:57:07.090 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:57:07.090 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:57:07.091 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:57:07.093 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:53:06.347, end=2024-09-11T17:55:06.757},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:57:07.093 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:57:07.093 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:57:07.103 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:57:07.103 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:57:07.330 INFO 2981023 --- [ocp-async-8,42289ab1d993d431,5e41142183cd69f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:07.628 INFO 2981023 --- [ocp-schedules-7,4df4aa81345c9f8a,a4223a4b05d1710b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:57:07.698 INFO 2981023 --- [ocp-schedules-1,f5aa5907f7f7a74c,ffb139cc4b13e2c7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:57:07.698+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:57:07.698 INFO 2981023 --- [ocp-schedules-1,f5aa5907f7f7a74c,ffb139cc4b13e2c7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:57:07.700 INFO 2981023 --- [ocp-schedules-1,f5aa5907f7f7a74c,ffb139cc4b13e2c7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:57:07.700 INFO 2981023 --- [ocp-schedules-1,f5aa5907f7f7a74c,ffb139cc4b13e2c7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:57:07.768 INFO 2981023 --- [ocp-async-3,3ccdcf701d68c83c,e3615ab54bdf9f21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:57:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,d3e8dc9aba4937f1,c6e65f629e51fab3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:08.316 ERROR 2981023 --- [prometheus-minute-schedule-1,d3e8dc9aba4937f1,c6e65f629e51fab3] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:08.316 ERROR 2981023 --- [prometheus-minute-schedule-1,d3e8dc9aba4937f1,c6e65f629e51fab3] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,d3e8dc9aba4937f1,c6e65f629e51fab3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:08.698 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:08.700 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:09.330 INFO 2981023 --- [ocp-async-5,67132ef7ab72c9e4,9b0ed3251f3dd83a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:10.003 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:57:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:57:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:57:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:57:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,ba96ca7d7a16f485,e2e25cfcddc8d242] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:11.318 ERROR 2981023 --- [prometheus-minute-schedule-1,ba96ca7d7a16f485,e2e25cfcddc8d242] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62889/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:11.318 ERROR 2981023 --- [prometheus-minute-schedule-1,ba96ca7d7a16f485,e2e25cfcddc8d242] c.o.ocp.monitor.OcpMonitorManager : Collect single exporter failed, exporter address = http://10.31.35.7:62888/metrics/stat java.lang.RuntimeException: Agent cache does not contain host id, hostId=2000010 at com.oceanbase.ocp.monitor.helper.ExporterRequestHelper.get(ExporterRequestHelper.java:159) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.asyncCollect(OcpMetricCollectServiceImpl.java:152) at com.oceanbase.ocp.monitor.service.OcpMetricCollectServiceImpl.collectIntervalMetricData(OcpMetricCollectServiceImpl.java:135) at com.oceanbase.ocp.monitor.OcpMonitorManager$IntervalCollectScheduleTask.run(OcpMonitorManager.java:284) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,ba96ca7d7a16f485,e2e25cfcddc8d242] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:11.330 INFO 2981023 --- [ocp-async-3,b0ebf37508f51c96,55b9d988990264ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:11.700 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:11.700 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:12.325 INFO 2981023 --- [ocp-schedules-5,21b98b3e9d0f18ef,795100a49378a53c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:57:12.583 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048560, nodeCount=463, affectRows=463 2024-09-11 17:57:12.910 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048560, nodeCount=1000, affectRows=1000 2024-09-11 17:57:12.912 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048560, nodeCount=814, affectRows=814 2024-09-11 17:57:12.912 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048560, nodeCount=999, validCount=999 2024-09-11 17:57:12.914 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048560, nodeCount=814, validCount=814 2024-09-11 17:57:12.974 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 17:57:12.987 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:12.990 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:13.047 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:13.048 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:13.060 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:13.060 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:13.072 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:13.073 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:13.086 INFO 2981023 --- [ocp-schedules-11,d07e792b5123012b,f64e03d468a37ad7] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2024-09-11 17:57:13.330 INFO 2981023 --- [ocp-async-10,11222065be7c50c8,1368dc6d7713f402] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,9476e0414835d833,f140150bee5cdd41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,4a80a3c4f026a2a3,054c75eb7a78d6ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:57:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,9476e0414835d833,f140150bee5cdd41] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:14.446 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048560, nodeCount=463, validCount=463 2024-09-11 17:57:14.700 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:14.701 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:14.730 INFO 2981023 --- [ocp-schedules-17,83cee554b650007d,5acee1d75c4f3e15] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:57:14.738 INFO 2981023 --- [ocp-schedules-17,83cee554b650007d,5acee1d75c4f3e15] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:57:14.738 INFO 2981023 --- [ocp-schedules-17,83cee554b650007d,5acee1d75c4f3e15] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:57:15.056 INFO 2981023 --- [ocp-schedules-9,81910e329d9dcd34,867cc2853570f4d0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:57:15.114 WARN 2981023 --- [scanner-13578728-1,81910e329d9dcd34,c3e693e3041cb2ff] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:57:15.202 INFO 2981023 --- [alarm-detect-2,81910e329d9dcd34,72f14c280b0813cd] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726048605 ~ 1726048625 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726048620, value=1.743)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.743, values={1726048620=1.743}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:57:15.202 INFO 2981023 --- [alarm-detect-2,81910e329d9dcd34,72f14c280b0813cd] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Pending 2024-09-11 17:57:15.330 INFO 2981023 --- [ocp-async-7,05f0c2d0aac8d51b,673d5395588d65bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:16.020 INFO 2981023 --- [ocp-schedules-4,e209f3851713bcb0,b89279390a0108cf] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:57:16.021 INFO 2981023 --- [host-check-clockdiff1,e209f3851713bcb0,b67dc0efb50912ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:57:16.025 INFO 2981023 --- [host-check-clockdiff2,e209f3851713bcb0,f7b8da5b858b90eb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:57:16.026 INFO 2981023 --- [host-check-clockdiff3,e209f3851713bcb0,431e0ccf38090246] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:57:16.026 INFO 2981023 --- [host-check-clockdiff4,e209f3851713bcb0,e839cc91676b9491] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:57:16.030 INFO 2981023 --- [ocp-schedules-19,64f04adac65299cf,6634e391b9f4389f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:57:16.031 INFO 2981023 --- [host-check-clockdiff1,e209f3851713bcb0,b67dc0efb50912ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048636 7 7 , error= 2024-09-11 17:57:16.036 INFO 2981023 --- [host-check-clockdiff3,e209f3851713bcb0,431e0ccf38090246] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048636 0 0 , error= 2024-09-11 17:57:16.036 INFO 2981023 --- [host-check-agent-status1,64f04adac65299cf,175d3c85c70fdf93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:57:16.036 INFO 2981023 --- [host-check-agent-status2,64f04adac65299cf,fc78160a7fd8bf7b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:57:16.036 INFO 2981023 --- [host-check-agent-status3,64f04adac65299cf,b14b29a8066808f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:57:16.036 INFO 2981023 --- [host-check-agent-status4,64f04adac65299cf,e6c0153ea56d7515] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:57:16.034 INFO 2981023 --- [ocp-schedules-3,30d5669661a4f8fe,0df645554cfc8407] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:57:16.038 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:57:16.039 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:57:16.039 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:57:16.039 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:57:16.065 INFO 2981023 --- [host-check-agent-status1,64f04adac65299cf,175d3c85c70fdf93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:16.066 INFO 2981023 --- [host-check-agent-status4,64f04adac65299cf,e6c0153ea56d7515] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:16.067 INFO 2981023 --- [host-check-agent-status2,64f04adac65299cf,fc78160a7fd8bf7b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:16.067 INFO 2981023 --- [host-check-agent-status1,64f04adac65299cf,175d3c85c70fdf93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:16.067 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:16.067 INFO 2981023 --- [host-check-agent-status4,64f04adac65299cf,e6c0153ea56d7515] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:16.068 INFO 2981023 --- [host-check-agent-status3,64f04adac65299cf,b14b29a8066808f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:16.068 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:16.068 INFO 2981023 --- [host-check-agent-status2,64f04adac65299cf,fc78160a7fd8bf7b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:16.069 INFO 2981023 --- [host-check-agent-status3,64f04adac65299cf,b14b29a8066808f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:16.070 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:16.070 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:16.071 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:16.072 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:16.080 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:16.083 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:16.102 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:57:16.104 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:57:16.110759540+08:00 2024-09-11 17:57:16.105 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:57:16.104+08:00 2024-09-11 17:57:16.109 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:57:16.104+08:00 and status AVAILABLE 2024-09-11 17:57:16.118 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:57:16.121 INFO 2981023 --- [host-check-agent-status1,64f04adac65299cf,175d3c85c70fdf93] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:57:16.123 INFO 2981023 --- [host-check-agent-status1,64f04adac65299cf,175d3c85c70fdf93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:57:16.123 INFO 2981023 --- [host-check-agent-status1,64f04adac65299cf,175d3c85c70fdf93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:57:16.123 INFO 2981023 --- [host-check-agent-status1,64f04adac65299cf,175d3c85c70fdf93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:57:16.137 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:57:16.139 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:57:16.140 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T17:57:16.141650249+08:00 2024-09-11 17:57:16.141 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T17:57:16.140+08:00 2024-09-11 17:57:16.144 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T17:57:16.140+08:00 and status AVAILABLE 2024-09-11 17:57:16.144 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:57:16.145 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:57:16.144489843+08:00 2024-09-11 17:57:16.146 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:16.147 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:57:16.145+08:00 2024-09-11 17:57:16.147 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:16.147 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:57:16.153562271+08:00 2024-09-11 17:57:16.148 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:57:16.147+08:00 2024-09-11 17:57:16.149 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:57:16.145+08:00 and status AVAILABLE 2024-09-11 17:57:16.152 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:57:16.147+08:00 and status AVAILABLE 2024-09-11 17:57:16.167 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:57:16.168 INFO 2981023 --- [host-check-agent-status3,64f04adac65299cf,b14b29a8066808f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:57:16.168 INFO 2981023 --- [host-check-agent-status4,64f04adac65299cf,e6c0153ea56d7515] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:57:16.170 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 17:57:16.170 INFO 2981023 --- [host-check-agent-status3,64f04adac65299cf,b14b29a8066808f0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:57:16.170 INFO 2981023 --- [host-check-agent-status3,64f04adac65299cf,b14b29a8066808f0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:57:16.170 INFO 2981023 --- [host-check-agent-status3,64f04adac65299cf,b14b29a8066808f0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:57:16.171 INFO 2981023 --- [host-check-agent-status4,64f04adac65299cf,e6c0153ea56d7515] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:57:16.171 INFO 2981023 --- [host-check-agent-status4,64f04adac65299cf,e6c0153ea56d7515] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:57:16.171 INFO 2981023 --- [host-check-agent-status4,64f04adac65299cf,e6c0153ea56d7515] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:57:16.173 INFO 2981023 --- [host-check-agent-status2,64f04adac65299cf,fc78160a7fd8bf7b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:57:16.175 INFO 2981023 --- [host-check-agent-status2,64f04adac65299cf,fc78160a7fd8bf7b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:57:16.175 INFO 2981023 --- [host-check-agent-status2,64f04adac65299cf,fc78160a7fd8bf7b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:57:16.175 INFO 2981023 --- [host-check-agent-status2,64f04adac65299cf,fc78160a7fd8bf7b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:57:16.176 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:16.177 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:16.181 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:57:16.183 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:57:16.187 INFO 2981023 --- [ocp-schedules-20,b0be95456ee2a4f3,27f5e0707031c2a9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:57:16.187 INFO 2981023 --- [ocp-schedules-5,8bc6c203a0a103f5,88d83062007c354d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:57:16.189 INFO 2981023 --- [ocp-schedules-5,8bc6c203a0a103f5,88d83062007c354d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:57:16.190 INFO 2981023 --- [ocp-schedules-20,b0be95456ee2a4f3,27f5e0707031c2a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:57:16.195 INFO 2981023 --- [ocp-schedules-26,ff8989e4b281848d,a2bf10a7bca72520] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:57:16.195 INFO 2981023 --- [ocp-schedules-18,0a408dcbd42fdb3f,bc7f381031bb3fb8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:57:16.195 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:16.196 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:16.197 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:16.197 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:16.199 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:57:16.199 INFO 2981023 --- [ocp-schedules-18,0a408dcbd42fdb3f,bc7f381031bb3fb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:57:16.200 INFO 2981023 --- [ocp-schedules-17,c9dc505e5331c732,219ef6a5d1e97418] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:57:16.222 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:57:16.226 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:57:16.246 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 17:57:16.266 INFO 2981023 --- [ocp-schedules-5,8bc6c203a0a103f5,88d83062007c354d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:57:16.267 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:57:16.291 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:57:16.294 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 17:57:16.298 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:57:16.299 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.300 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:16.302 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.302 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:57:16.303 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:16.304 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:57:16.307 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.311 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:16.313 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:57:16.314 INFO 2981023 --- [ocp-schedules-20,b0be95456ee2a4f3,27f5e0707031c2a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:57:16.319 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:57:16.322 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.322 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:16.324 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:57:16.330 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.330 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:16.334 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:57:16.339 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:57:16.343 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.343 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:16.353 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:57:16.362 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:57:16.368 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:57:16.367575+08:00, lastProcessTime=2024-09-11T17:57:06.337608+08:00 2024-09-11 17:57:16.373 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:57:16.374 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.375 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.376 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:57:16.379 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:57:16.386 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:57:16.395 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 17:57:16.403 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:57:16.410 INFO 2981023 --- [host-check-host-status1,30d5669661a4f8fe,760b44aed55556bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:57:16.412 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:57:16.415 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 17:57:16.417 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.418 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:16.418 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:57:16.418 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.421 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:57:16.422 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:57:16.434 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.434 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:16.434 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:57:16.435 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.435 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:57:16.438 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:57:16.449 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:57:16.450 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:57:16.452 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:16.453 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:57:16.462 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.463 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.463 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:16.464 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:57:16.464 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:57:16.466 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.467 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.468 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:16.468 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:57:16.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.470 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 17:57:16.471 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:57:16.472 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:57:16.474 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:16.475 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:16.477 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.478 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.478 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:57:16.479 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:57:16.483 INFO 2981023 --- [host-check-host-status4,30d5669661a4f8fe,5e8caf52d89d5ffe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 17:57:16.492 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:57:16.492 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:16.493 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:57:16.496 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:57:16.509 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.510 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.510 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:16.511 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:57:16.511 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.513 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:16.513 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:57:16.516 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:57:16.518 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:57:16.521 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:57:16.525 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.526 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.527 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:16.527 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:57:16.529 INFO 2981023 --- [host-check-host-status3,30d5669661a4f8fe,6ac6555f4a084e90] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:57:16.529 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:16.529 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:57:16.532 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:57:16.533 INFO 2981023 --- [host-check-host-status2,30d5669661a4f8fe,cc24e1731684e219] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:57:16.539 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:16.541 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:16.546 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:16.558 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:57:16.566 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:57:16.570 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:16.570 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:57:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:16.580 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:57:16.586 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:57:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:57:16.593 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:57:16.593 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:57:16.619 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:57:16.621 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:57:16.367575+08:00 2024-09-11 17:57:16.624 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.625 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.625 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:57:16.629 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:57:16.632 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:57:16.632 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:57:16.637 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:16.638 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:16.638 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:57:16.644 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:57:16.644 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:57:16.644 INFO 2981023 --- [ocp-schedules-18,0a408dcbd42fdb3f,bc7f381031bb3fb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:57:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,1f578b441281fb0d,f0dd446bd03f1602] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,1f578b441281fb0d,f0dd446bd03f1602] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:17.330 INFO 2981023 --- [ocp-async-6,8baf240582fb9a51,206670641f316fd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:17.701 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:17.701 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:17.768 INFO 2981023 --- [ocp-async-2,6ed2eb36219b4dd1,4ddbd1708103d780] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:57:17.812 INFO 2981023 --- [host-check-clockdiff2,e209f3851713bcb0,f7b8da5b858b90eb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048637 6 6 , error= 2024-09-11 17:57:19.330 INFO 2981023 --- [ocp-async-8,744bcb6b53853d23,daf89479b1e5f605] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,76fff3d7c470a246,53b7c4ff35456a0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,76fff3d7c470a246,53b7c4ff35456a0a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:20.701 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:20.701 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:21.331 INFO 2981023 --- [ocp-async-1,ab9b2895c61900ed,c4569e90dfc98f3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:22.021 INFO 2981023 --- [host-check-clockdiff4,e209f3851713bcb0,e839cc91676b9491] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048642 2 2 , error= 2024-09-11 17:57:22.328 INFO 2981023 --- [ocp-schedules-12,aacd17a2b6ec93f1,06f2e42bc8d53145] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:57:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,e64a38719b361b01,d7ad8f92d05d1ba4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,e64a38719b361b01,d7ad8f92d05d1ba4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:23.331 INFO 2981023 --- [ocp-async-4,cff13ac02c5e2111,ae9e40f5f62b4a9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:23.702 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:23.702 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:23.991 INFO 2981023 --- [ocp-schedules-2,7ad04f437ae0d557,67ff662b1344bcbe] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:57:23.991 INFO 2981023 --- [ocp-schedules-2,7ad04f437ae0d557,67ff662b1344bcbe] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:57:23.991 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 17:57:23.991 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:57:23.991 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:57:23.996 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:23.997 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:23.997 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:23.998 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:23.999 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:23.999 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:24.000 INFO 2981023 --- [ocp-schedules-2,7ad04f437ae0d557,67ff662b1344bcbe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:24.001 INFO 2981023 --- [ocp-schedules-2,7ad04f437ae0d557,67ff662b1344bcbe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:24.014 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:57:24.015 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:57:24.018 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:57:24.018 INFO 2981023 --- [ocp-schedules-2,7ad04f437ae0d557,67ff662b1344bcbe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:57:24.020 INFO 2981023 --- [ocp-schedules-2,7ad04f437ae0d557,67ff662b1344bcbe] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:57:24.337 INFO 2981023 --- [ocp-schedules-10,06b41ffb3ec636d1,0c1b3e8df947ded0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:57:25.291 INFO 2981023 --- [ocp-schedules-20,8a3073ffd3500826,4d8891e1bf9bcd37] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:57:25.331 INFO 2981023 --- [ocp-async-9,8f4b5efd48722681,ce6fa06784719078] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:25.337 WARN 2981023 --- [scanner-13578929-1,8a3073ffd3500826,fb61a88206a91d97] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:57:25.357 INFO 2981023 --- [alarm-detect-12,8a3073ffd3500826,ae3a60b2f52b7dfb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645076, data is empty. 2024-09-11 17:57:25.362 INFO 2981023 --- [alarm-detect-12,8a3073ffd3500826,ae3a60b2f52b7dfb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645076, data is empty. 2024-09-11 17:57:25.370 INFO 2981023 --- [alarm-detect-12,8a3073ffd3500826,ae3a60b2f52b7dfb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645076, data is empty. 2024-09-11 17:57:25.383 INFO 2981023 --- [alarm-detect-9,8a3073ffd3500826,fb1e6666373fb723] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645076, data is empty. 2024-09-11 17:57:25.384 INFO 2981023 --- [alarm-detect-8,8a3073ffd3500826,baa441f926bcf29e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645076, data is empty. 2024-09-11 17:57:25.387 INFO 2981023 --- [alarm-detect-9,8a3073ffd3500826,fb1e6666373fb723] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645076, data is empty. 2024-09-11 17:57:25.390 INFO 2981023 --- [alarm-detect-8,8a3073ffd3500826,baa441f926bcf29e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645076, data is empty. 2024-09-11 17:57:25.394 INFO 2981023 --- [alarm-detect-9,8a3073ffd3500826,fb1e6666373fb723] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645076, data is empty. 2024-09-11 17:57:25.394 INFO 2981023 --- [alarm-detect-8,8a3073ffd3500826,baa441f926bcf29e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645076, data is empty. 2024-09-11 17:57:25.505 INFO 2981023 --- [alarm-detect-7,8a3073ffd3500826,19704b04e84cc496] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645076, data is empty. 2024-09-11 17:57:25.513 INFO 2981023 --- [alarm-detect-7,8a3073ffd3500826,19704b04e84cc496] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645076, data is empty. 2024-09-11 17:57:25.521 INFO 2981023 --- [alarm-detect-7,8a3073ffd3500826,19704b04e84cc496] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645076, data is empty. 2024-09-11 17:57:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,2e699889f57266b4,508062982af8c65b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,2e699889f57266b4,508062982af8c65b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:57:26.336735+08:00, lastProcessTime=2024-09-11T17:57:16.367575+08:00 2024-09-11 17:57:26.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:57:26.336735+08:00 2024-09-11 17:57:26.680 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,282ef8c8f3198d16,a23cd66bd2610043] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=282ef8c8f3198d16, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:57:26.681 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,282ef8c8f3198d16,a23cd66bd2610043] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:57:26.702 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:26.702 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:26.711 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,282ef8c8f3198d16,a23cd66bd2610043] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=282ef8c8f3198d16, duration=36 ms] 2024-09-11 17:57:27.331 INFO 2981023 --- [ocp-async-8,de055b3ab776bf44,91cd16091ee6ffe0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:27.768 INFO 2981023 --- [ocp-async-3,0124aa9b43172c6e,c7a88db4bf597fad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:57:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,a2539d90111e2f98,1a5f4c577109096e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,a2539d90111e2f98,1a5f4c577109096e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:29.331 INFO 2981023 --- [ocp-async-6,778a4c789a681278,4eea4a68e5e798d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:29.703 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:29.703 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:31.331 INFO 2981023 --- [ocp-async-3,9a3d5a5ece9b6a30,d168ae6950ed46ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,9fcbc1e851eb5c01,889702eaf1031ec5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,9fcbc1e851eb5c01,889702eaf1031ec5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:32.330 INFO 2981023 --- [ocp-schedules-2,61cd2bb4da483f50,521b7b176ec46d10] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:57:32.703 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:32.703 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:33.331 INFO 2981023 --- [ocp-async-2,ab9cc2c3c701e002,f409ac7662603987] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:34.740 INFO 2981023 --- [ocp-schedules-28,c115ceb080f6d3b1,799b9b80445372b4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:57:34.749 INFO 2981023 --- [ocp-schedules-28,c115ceb080f6d3b1,799b9b80445372b4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:57:34.749 INFO 2981023 --- [ocp-schedules-28,c115ceb080f6d3b1,799b9b80445372b4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:57:35.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,007c5ffdbf023577,feab07705ff01ca4] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hosts/2000010/checkAndRepairSystemInfo, client 10.31.33.41, traceId 007c5ffdbf023577] 2024-09-11 17:57:35.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,007c5ffdbf023577,feab07705ff01ca4] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hosts/2000010/checkAndRepairSystemInfo?id=2000010, client=10.31.33.41, traceId=007c5ffdbf023577, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.checkAndRepairSystemInfo(Long), args=2000010,] 2024-09-11 17:57:35.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,007c5ffdbf023577,feab07705ff01ca4] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=check system and generate repair plan, nodeCount=2, taskCount=2 2024-09-11 17:57:35.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,007c5ffdbf023577,feab07705ff01ca4] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:57:35.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,007c5ffdbf023577,feab07705ff01ca4] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='check system and generate repair plan', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=null, tenantId=null, hostId=2000010, creator='admin', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false, system_checker_operation=CHECK_AND_REPAIR, host_id=2000010}, listMap={}}, createTime=null, startTime=2024-09-11T17:57:35.120+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Generate repair plan task, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.GenerateRepairPlanInstancesTask, seriesId=0, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Check system task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.CheckSystemInfoTask, seriesId=1, startTime=null, endTime=null}], template=null, traceId='007c5ffdbf023577', operation=EXECUTE} 2024-09-11 17:57:35.138 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,007c5ffdbf023577,feab07705ff01ca4] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=4048199, taskName=check system and generate repair plan, argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false, system_checker_operation=CHECK_AND_REPAIR, host_id=2000010}, listMap={}} 2024-09-11 17:57:35.142 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,007c5ffdbf023577,feab07705ff01ca4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hosts/2000010/checkAndRepairSystemInfo client=10.31.33.41, traceId=007c5ffdbf023577, duration=26 ms] 2024-09-11 17:57:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,ddca32bbee74311e,9534dd8b57a1eb51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,ddca32bbee74311e,9534dd8b57a1eb51] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:35.331 INFO 2981023 --- [ocp-async-1,9970ee14842c7f32,c01bcbcdcaa54f80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:35.560 INFO 2981023 --- [ocp-schedules-11,597c9fc02f790c2b,43bfc5af56cc17d9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:57:35.591 WARN 2981023 --- [scanner-13579235-1,597c9fc02f790c2b,61a224698b557cb4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:57:35.704 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:35.705 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:35.715 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 17:57:35.715 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 17:57:35.718 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:57:35.718 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:57:35.718 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:57:35.718 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:57:35.721 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 17:57:35.721 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 17:57:35.721 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 17:57:35.721 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 17:57:35.775 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:57:35.779 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:57:35.779 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:57:35.779 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:57:35.779 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:57:35.779 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:57:35.783 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:57:35.854 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,07d6e713377da962] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048190, operation:EXECUTE, state: RUNNING 2024-09-11 17:57:35.884 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048190, context=Context{parallelIdx=-1, stringMap={system_checker_operation=CHECK_AND_REPAIR, task_instance_id=4048199, task_operation=execute, latest_execution_start_time=2024-09-11T17:57:35.849+08:00, sub_task_instance_name=Check system task, sub_task_instance_id=4048190, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:57:36.013 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Clock Offset] will apply to target 10.31.35.7 2024-09-11 17:57:36.018 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [libaio DLL] will apply to target 10.31.35.7 2024-09-11 17:57:36.022 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [crond Service] will apply to target 10.31.35.7 2024-09-11 17:57:36.025 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [sshd Service] will apply to target 10.31.35.7 2024-09-11 17:57:36.029 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [firewalld Service] will apply to target 10.31.35.7 2024-09-11 17:57:36.032 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Transparent Huge Pages] will apply to target 10.31.35.7 2024-09-11 17:57:36.036 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Python Version] will apply to target 10.31.35.7 2024-09-11 17:57:36.042 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Swap Information] will apply to target 10.31.35.7 2024-09-11 17:57:36.045 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.conf.default.rp_filter] will apply to target 10.31.35.7 2024-09-11 17:57:36.049 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.conf.default.accept_source_route] will apply to target 10.31.35.7 2024-09-11 17:57:36.054 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_syncookies] will apply to target 10.31.35.7 2024-09-11 17:57:36.065 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.swappiness] will apply to target 10.31.35.7 2024-09-11 17:57:36.069 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_tw_reuse] will apply to target 10.31.35.7 2024-09-11 17:57:36.073 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_slow_start_after_idle] will apply to target 10.31.35.7 2024-09-11 17:57:36.076 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.overcommit_memory] will apply to target 10.31.35.7 2024-09-11 17:57:36.083 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 17:57:36.080 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.nr_hugepages] will apply to target 10.31.35.7 2024-09-11 17:57:36.088 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.somaxconn] will apply to target 10.31.35.7 2024-09-11 17:57:36.091 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.netdev_max_backlog] will apply to target 10.31.35.7 2024-09-11 17:57:36.094 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.rmem_default] will apply to target 10.31.35.7 2024-09-11 17:57:36.098 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.wmem_default] will apply to target 10.31.35.7 2024-09-11 17:57:36.101 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.rmem_max] will apply to target 10.31.35.7 2024-09-11 17:57:36.105 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.wmem_max] will apply to target 10.31.35.7 2024-09-11 17:57:36.113 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ min of net.ipv4.tcp_rmem] will apply to target 10.31.35.7 2024-09-11 17:57:36.116 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ default of net.ipv4.tcp_rmem] will apply to target 10.31.35.7 2024-09-11 17:57:36.120 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ max of net.ipv4.tcp_rmem] will apply to target 10.31.35.7 2024-09-11 17:57:36.128 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ min of net.ipv4.tcp_wmem] will apply to target 10.31.35.7 2024-09-11 17:57:36.132 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ default of net.ipv4.tcp_wmem] will apply to target 10.31.35.7 2024-09-11 17:57:36.136 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ max of net.ipv4.tcp_wmem] will apply to target 10.31.35.7 2024-09-11 17:57:36.140 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_max_syn_backlog] will apply to target 10.31.35.7 2024-09-11 17:57:36.143 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_fin_timeout] will apply to target 10.31.35.7 2024-09-11 17:57:36.148 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.min_free_kbytes] will apply to target 10.31.35.7 2024-09-11 17:57:36.151 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.max_map_count] will apply to target 10.31.35.7 2024-09-11 17:57:36.155 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [fs.aio-max-nr] will apply to target 10.31.35.7 2024-09-11 17:57:36.158 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [kernel.numa_balancing] will apply to target 10.31.35.7 2024-09-11 17:57:36.162 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.zone_reclaim_mode] will apply to target 10.31.35.7 2024-09-11 17:57:36.166 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [fs.file-max] will apply to target 10.31.35.7 2024-09-11 17:57:36.170 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [fs.pipe-user-pages-soft] will apply to target 10.31.35.7 2024-09-11 17:57:36.173 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [mtr Command] will apply to target 10.31.35.7 2024-09-11 17:57:36.177 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [tar Command] will apply to target 10.31.35.7 2024-09-11 17:57:36.180 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [curl Command] will apply to target 10.31.35.7 2024-09-11 17:57:36.208 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [nc Command] will apply to target 10.31.35.7 2024-09-11 17:57:36.212 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [binutils Toolkit] will apply to target 10.31.35.7 2024-09-11 17:57:36.216 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [bind-utils Toolkit] will apply to target 10.31.35.7 2024-09-11 17:57:36.219 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Network Management Toolkit] will apply to target 10.31.35.7 2024-09-11 17:57:36.223 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [libatomic DLL] will apply to target 10.31.35.7 2024-09-11 17:57:36.227 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [os user(ocp.operation.default.os.user in System Parameters) existence] will apply to target 10.31.35.7 2024-09-11 17:57:36.231 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of openFiles of root user] will apply to target 10.31.35.7 2024-09-11 17:57:36.234 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of openFiles of os user(ocp.operation.default.os.user in System Parameters)] will apply to target 10.31.35.7 2024-09-11 17:57:36.237 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of stackSize of os user(ocp.operation.default.os.user in System Parameters)] will apply to target 10.31.35.7 2024-09-11 17:57:36.241 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of maxUserProcesses of os user(ocp.operation.default.os.user in System Parameters)] will apply to target 10.31.35.7 2024-09-11 17:57:36.245 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of coreFileSize of os user(ocp.operation.default.os.user in System Parameters)] will apply to target 10.31.35.7 2024-09-11 17:57:36.249 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [SELinux config] will apply to target 10.31.35.7 2024-09-11 17:57:36.252 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [File Type of OceanBase Data Disk] will apply to target 10.31.35.7 2024-09-11 17:57:36.256 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [OS version check] will apply to target 10.31.35.7 2024-09-11 17:57:36.259 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [fio Command] will apply to target 10.31.35.7 2024-09-11 17:57:36.263 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ clock source checks] will apply to target 10.31.35.7 2024-09-11 17:57:36.295 INFO 2981023 --- [inspection_task3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:36.299 INFO 2981023 --- [inspection_task3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:57:36.338491+08:00, lastProcessTime=2024-09-11T17:57:26.336735+08:00 2024-09-11 17:57:36.357 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.357 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/swapInfo, request body:null, params:null 2024-09-11 17:57:36.357 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/service/info, request body:[object Object], params:null 2024-09-11 17:57:36.359 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/pythonVersion, request body:null, params:null 2024-09-11 17:57:36.359 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/transparentHugePageInfo, request body:null, params:null 2024-09-11 17:57:36.359 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/service/info, request body:[object Object], params:null 2024-09-11 17:57:36.366 INFO 2981023 --- [inspection_task4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:36.367 INFO 2981023 --- [inspection_task4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:36.419 INFO 2981023 --- [inspection_task4,,] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:57:36.420 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/libExists, request body:[object Object], params:null 2024-09-11 17:57:36.422 INFO 2981023 --- [inspection_task6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.424 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.424 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.424 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/service/info, request body:[object Object], params:null 2024-09-11 17:57:36.424 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.429 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.707994203S 2024-09-11 17:57:36.430 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 17:57:36.439 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.451 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.457 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.466 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.472 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.504 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 17:57:36.504 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 17:57:36.514 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.527 INFO 2981023 --- [inspection_task6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.536 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 17:57:36.594 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 17:57:36.594 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 17:57:36.594 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 17:57:36.594 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 17:57:36.594 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 17:57:36.594 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 17:57:36.600 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.605 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 17:57:36.619 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.635 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.640 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.662 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.670 INFO 2981023 --- [inspection_task6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.687 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.703 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.711 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] o.s.boot.SpringApplication : Started application in 0.993 seconds (JVM running for 84456.584) 2024-09-11 17:57:36.711 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.993455937S 2024-09-11 17:57:36.711 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.742 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.754 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.762 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.772 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.780 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:57:36.786 INFO 2981023 --- [inspection_task6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:57:36.792 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:57:36.808 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:57:36.815 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:57:36.835 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:57:36.840 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:57:36.848 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:57:36.853 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/libExists, request body:[object Object], params:null 2024-09-11 17:57:36.862 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/userExists, request body:[object Object], params:null 2024-09-11 17:57:36.876 INFO 2981023 --- [inspection_task6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:57:36.884 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:57:36.890 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:57:36.897 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:57:36.913 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:57:36.921 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/selinuxInfo, request body:null, params:null 2024-09-11 17:57:36.927 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/disk/batchInfos, request body:null, params:null 2024-09-11 17:57:36.934 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/hostInfo, request body:null, params:null 2024-09-11 17:57:36.939 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:57:36.945 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/clockSourceInfo, request body:null, params:null 2024-09-11 17:57:36.948 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/hostInfo, request body:null, params:null 2024-09-11 17:57:36.970 INFO 2981023 --- [ocp-schedules-25,84a8e14681627c40,5ea207feb0513253] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 17:57:37.314 INFO 2981023 --- [prometheus-cache-monitor1,66ff6374a26d92b4,b9e58aa997c2cb4c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7236,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2045,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":4692} 2024-09-11 17:57:37.315 INFO 2981023 --- [prometheus-cache-monitor1,66ff6374a26d92b4,b9e58aa997c2cb4c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:57:37.332 INFO 2981023 --- [ocp-async-10,81112cb0453eadf6,d0f86f0724b3171b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:37.768 INFO 2981023 --- [ocp-async-1,740b7425272117e6,608b2bc269cd4113] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:57:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,79dd176cad532ee4,38e33cc087ac6ebb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,79dd176cad532ee4,38e33cc087ac6ebb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:38.705 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:38.706 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:38.863 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,5f0c032c3b72809f,84d460d52c452fd4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=5f0c032c3b72809f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:57:38.864 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-6,5f0c032c3b72809f,84d460d52c452fd4] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:57:38.895 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,5f0c032c3b72809f,84d460d52c452fd4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=5f0c032c3b72809f, duration=37 ms] 2024-09-11 17:57:39.332 INFO 2981023 --- [ocp-async-7,310389c0f1b15434,ca7b8249a0da1af0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:40.004 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:57:40.004 INFO 2981023 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:57:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,8f8e15422615666e,aa837c727e637e17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,8f8e15422615666e,aa837c727e637e17] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:41.332 INFO 2981023 --- [ocp-async-1,12aa342293ef9fc0,f379fbf4e080699a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:41.707 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:41.707 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:42.333 INFO 2981023 --- [ocp-schedules-10,cc405de27798d401,598e788c2069a119] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:57:42.420 INFO 2981023 --- [inspection_task4,,] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048662 1 1 , error= 2024-09-11 17:57:42.651 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,4ac72482018d5ed9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048190, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:57:42.793 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048189] 2024-09-11 17:57:42.895 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,d8b96e4e0ddaefc6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048189, operation:EXECUTE, state: RUNNING 2024-09-11 17:57:42.922 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048189, context=Context{parallelIdx=-1, stringMap={system_checker_operation=CHECK_AND_REPAIR, task_instance_id=4048199, task_operation=execute, latest_execution_start_time=2024-09-11T17:57:42.893+08:00, sub_task_instance_name=Generate repair plan task, sub_task_instance_id=4048189, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:57:43.040 INFO 2981023 --- [ocp-schedules-12,78cfc178cde7e877,4324fc941d46cb73] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:57:43.065 WARN 2981023 --- [ocp-schedules-12,78cfc178cde7e877,4324fc941d46cb73] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:57:43.116 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=210, repairable=false, repairHint=null, description=检查主机时钟偏移，单位(ms)。OBServer 和 OCP 的时钟偏移应保持在 50ms 内，来满足 OceanBase 集群的时钟偏移要求|Checks the clock offset of the host, in milliseconds. In an OceanBase cluster, the clock offset between OBServers and OCP must be within 50 ms., repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6920bd18 2024-09-11 17:57:43.121 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=211, repairable=true, repairHint=null, description=libaio 是 OceanBase 运行依赖的动态链接库，必须安装|libaio dynamic-link library (DLL) is a dependency of OceanBase Database and must be installed., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"libaio"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@290193a 2024-09-11 17:57:43.125 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=212, repairable=true, repairHint=null, description=crond 用于设置定时任务，OceanBase 依赖的基础能力|The crond service is a basic dependency of OceanBase Database used for setting up scheduled tasks., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, parameters={"name":"crond","opType":"start"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, rollbackParameters={"name":"crond","opType":"stop"}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@194c6257 2024-09-11 17:57:43.129 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=213, repairable=true, repairHint=null, description=sshd 用于加密登录或者数据传输，OceanBase 依赖的基础能力|The sshd service is a basic dependency of OceanBase Database for encrypted logon or data transmission., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, parameters={"name":"sshd","opType":"start"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, rollbackParameters={"name":"sshd","opType":"stop"}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@1e0746fd 2024-09-11 17:57:43.132 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=214, repairable=true, repairHint=null, description=建议用户禁用 firewalld 或为 OceanBase 集群各服务添加允许规则。避免防火墙拦截|We recommend that you disable the firewalld service or add whitelist rules for services of the OceanBase cluster as needed. Otherwise, services will be blocked by the firewall., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, parameters={"name":"firewalld","opType":"stop"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, rollbackParameters={"name":"sshd","opType":"start"}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@72a3e058 2024-09-11 17:57:43.136 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=215, repairable=true, repairHint=null, description=检查操作系统 /sys/kernel/mm/transparent_hugepage/enabled 配置。需要设置为 never，保障系统稳定性|Checks the settings of the enabled file in the /sys/kernel/mm/transparent_hugepage/ directory of the operating system. You must set this parameter to `never` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairTHP@21dccea5, parameters={"newOption":"never"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairTHP@21dccea5, rollbackParameters={"newOption":"never","isReset":true}, rollbackReplaceField=newOption), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@70f86eff 2024-09-11 17:57:43.140 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=216, repairable=false, repairHint=null, description=检查主机安装的 Python 版本是否 2.7.x，确保相关 OceanBase 脚本能够正常运行。建议用户自行安装|Checks whether the version of Python installed on the host is 2.7.x to ensure the normal execution of relevant OceanBase scripts. We recommend that you install it by yourself., repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@adcf46d 2024-09-11 17:57:43.143 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=217, repairable=true, repairHint=null, description=检查操作系统 swap 信息。要求移除 swap 分区|Checks the swap information of the operating system. Swap partitions must be removed., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSwap@44955c9b, parameters={"newOption":"off"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSwap@44955c9b, rollbackParameters={"newOption":"off"}, rollbackReplaceField=newOption), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2e487b65 2024-09-11 17:57:43.146 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=218, repairable=true, repairHint=null, description=net.ipv4.conf.default.rp_filter 检查网卡对接收到的数据包进行反向路由验证的规则。需要设置为 1，开启严格的反向路由校验，保障系统安全性|An operating system parameter. It checks the NIC rules for reverse routing verification on received packets. To ensure system security, you must set this parameter to `1` to enable strict reverse routing verification., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.conf.default.rp_filter","paramValue":"1"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.conf.default.rp_filter","paramValue":"1"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4304ffd4 2024-09-11 17:57:43.151 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=219, repairable=true, repairHint=null, description=net.ipv4.conf.default.accept_source_route 允许数据包的发送者指定数据包的发送路径，以及返回给发送者的数据包所走的路径。需设置为 0，禁用该特性，保障系统安全性|An operating system parameter. It controls whether to allow the sender to specify the path for sending packets and the path by which packets are returned to the sender. To ensure system security, you must set this parameter to `0` to disable this feature., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.conf.default.accept_source_route","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.conf.default.accept_source_route","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@70b8fedd 2024-09-11 17:57:43.154 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=220, repairable=true, repairHint=null, description=net.ipv4.tcp_syncookies 当出现 syn 等候队列出现溢出时向对方发送 syncookies。需要设置为1，目的是为了防止 syn flood 攻击|An operating system parameter. It controls whether to send syncookies to the peer end when the SYN queue overflows. To prevent an SYN flood, you must set this parameter to `1`., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_syncookies","paramValue":"1"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_syncookies","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@1daa033b 2024-09-11 17:57:43.158 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=221, repairable=true, repairHint=null, description=vm.swappiness 换出运行时内存的相对权重。需要设置为 0，保障系统稳定|An operating system parameter. It controls the relative weight given to swapping out of runtime memory. You must set this parameter to `0` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.swappiness","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.swappiness","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7054d0b8 2024-09-11 17:57:43.161 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=222, repairable=true, repairHint=null, description=net.ipv4.tcp_tw_reuse 检查是否允许将处于 TIME-WAIT 状态的 socket（TIME-WAIT 的端口）用于新的 TCP 连接。需要设置为 1，保障系统性能|An operating system parameter. It checks whether sockets in the TIME_WAIT state can be reused for new TCP connections. You must set this parameter to `1` to ensure system performance., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_tw_reuse","paramValue":"1"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_tw_reuse","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@144a6b39 2024-09-11 17:57:43.165 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=223, repairable=true, repairHint=null, description=net.ipv4.tcp_slow_start_after_idle 检查 TCP 空闲后是否慢启动。需要设置为 0，保障系统性能|An operating system parameter. It checks whether a TCP connection starts slowly after an idle period. You must set this parameter to `0` to ensure system performance., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_slow_start_after_idle","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_slow_start_after_idle","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2333317b 2024-09-11 17:57:43.168 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=224, repairable=true, repairHint=null, description=vm.overcommit_memory 检查是否接受大型内存请求或拒绝的条件。需要设置为 0，保障系统稳定性|An operating system parameter. It checks the conditions for receiving large memory requests are met. You must set this parameter to `0` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.overcommit_memory","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.overcommit_memory","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6e4d5dfd 2024-09-11 17:57:43.172 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=225, repairable=true, repairHint=null, description=vm.nr_hugepages 检查是否开启 HugePages。需要设置为 0，保障系统稳定性|An operating system parameter. It checks whether the HugePages feature is enabled. You must set this parameter to `0` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.nr_hugepages","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.nr_hugepages","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@38ca819c 2024-09-11 17:57:43.175 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=238, repairable=true, repairHint=null, description=net.core.somaxconn 检查系统中每一个端口最大的监听队列的长度。建议设置在范围 [2048,16384]|check net.core.somaxconn value.Recommended value: [2048,16384], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.somaxconn","paramValue":"2048"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.somaxconn","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7f9e261f 2024-09-11 17:57:43.178 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=239, repairable=true, repairHint=null, description=net.core.netdev_max_backlog 检查在每个网络接口接收数据包的速率比内核处理这些包的速率快时，允许送到队列的数据包的最大数目。建议设置在范围 [500,10000]|check net.core.netdev_max_backlog value.Recommended value: [500,10000], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.netdev_max_backlog","paramValue":"10000"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.netdev_max_backlog","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@459c687e 2024-09-11 17:57:43.182 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=240, repairable=true, repairHint=null, description=net.core.rmem_default 检查主机默认的 TCP 数据接收窗口大小（字节）。建议设置在范围 [65536,16777216]|check net.core.rmem_default value.Recommended value: [65536,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.rmem_default","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.rmem_default","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@41ba7785 2024-09-11 17:57:43.186 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=241, repairable=true, repairHint=null, description=net.core.wmem_default 检查主机默认的 TCP 数据发送窗口大小（字节）。建议设置在范围 [65536,16777216]|check net.core.wmem_default value.Recommended value: [65536,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.wmem_default","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.wmem_default","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2d90c84f 2024-09-11 17:57:43.189 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=242, repairable=true, repairHint=null, description=net.core.rmem_max 检查主机最大的 TCP 数据接收窗口（字节）。建议设置在范围 [8388608,16777216]|check net.core.rmem_max value.Recommended value: [8388608,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.rmem_max","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.rmem_max","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2330bd3c 2024-09-11 17:57:43.193 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=243, repairable=true, repairHint=null, description=net.core.wmem_max 检查主机最大的 TCP 数据发送窗口（字节）。建议设置在范围 [8388608,16777216]|check net.core.wmem_max value.Recommended value: [8388608,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.wmem_max","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.wmem_max","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@549febc 2024-09-11 17:57:43.197 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=245, repairable=true, repairHint=null, description=net.ipv4.tcp_rmem 检查 socket 接收缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_rmem value.Recommended value: 4096 87380 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"4096 87380 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@175db7cf 2024-09-11 17:57:43.200 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=245, repairable=true, repairHint=null, description=net.ipv4.tcp_rmem 检查 socket 接收缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_rmem value.Recommended value: 4096 87380 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"4096 87380 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@3c24c7db 2024-09-11 17:57:43.203 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=245, repairable=true, repairHint=null, description=net.ipv4.tcp_rmem 检查 socket 接收缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_rmem value.Recommended value: 4096 87380 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"4096 87380 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@1a5b443e 2024-09-11 17:57:43.207 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=246, repairable=true, repairHint=null, description=net.ipv4.tcp_wmem 检查 socket 发送缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_wmem value.Recommended value: 4096 65536 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"4096 65536 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@502df5d7 2024-09-11 17:57:43.210 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=246, repairable=true, repairHint=null, description=net.ipv4.tcp_wmem 检查 socket 发送缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_wmem value.Recommended value: 4096 65536 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"4096 65536 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2ab4d3d2 2024-09-11 17:57:43.214 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=246, repairable=true, repairHint=null, description=net.ipv4.tcp_wmem 检查 socket 发送缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_wmem value.Recommended value: 4096 65536 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"4096 65536 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@30b4d340 2024-09-11 17:57:43.217 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=256, repairable=true, repairHint=null, description=net.ipv4.tcp_max_syn_backlog 检查可保存在队列中未获得对方确认的最大连接请求数目。建议设置在范围：[1024,16384]|check net.ipv4.tcp_max_syn_backlog value.Recommended value: [1024,16384], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_max_syn_backlog","paramValue":"1024"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_max_syn_backlog","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@708e1bdc 2024-09-11 17:57:43.221 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=257, repairable=true, repairHint=null, description=net.ipv4.tcp_fin_timeout 检查 TCP 保持在 FIN-WAIT-2 状态的时间（秒）。建议设置在范围：[15,60]|check net.ipv4.tcp_fin_timeout value.Recommended value: [15,60], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_fin_timeout","paramValue":"60"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_fin_timeout","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4f32db57 2024-09-11 17:57:43.226 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=258, repairable=true, repairHint=null, description=vm.min_free_kbytes 检查保留的空闲页面池的大小。建议设置在范围：[32768,2097152]|check vm.min_free_kbytes value.Recommended value: [32768,2097152], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.min_free_kbytes","paramValue":"2097152"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.min_free_kbytes","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4394bb20 2024-09-11 17:57:43.229 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=259, repairable=true, repairHint=null, description=vm.max_map_count 检查进程可以使用的最大内存映射区域值。建议设置为 655360|check vm.max_map_count value.Recommended value: 655360, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.max_map_count","paramValue":"655360"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.max_map_count","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@552c5ec1 2024-09-11 17:57:43.233 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=260, repairable=true, repairHint=null, description=fs.aio-max-nr 检查同时可支持的异步 IO 数目。建议设置为 >= 1048576|check fs.aio-max-nr value.Recommended value >= 1048576, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"fs.aio-max-nr","paramValue":"1048576"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"fs.aio-max-nr","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@557cff43 2024-09-11 17:57:43.236 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=261, repairable=true, repairHint=null, description=kernel.numa_balancing 检查自动 numa 平衡是否禁用，为避免内存节点数据迁移可能造成的性能抖动，建议禁用|check kernel.numa_balancing value.Recommended value: 0, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"kernel.numa_balancing","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"kernel.numa_balancing","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@c4d7ed8 2024-09-11 17:57:43.240 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=262, repairable=true, repairHint=null, description=vm.zone_reclaim_mode 检查控制内存 zone 回收模式是否禁用，为避免因回收导致的性能抖动，建议禁用|check vm.zone_reclaim_mode value.Recommended value: 0, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.zone_reclaim_mode","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.zone_reclaim_mode","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6a6699ca 2024-09-11 17:57:43.243 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=263, repairable=true, repairHint=null, description=fs.file-max 检查整个系统能够打开的最大文件数量，建议设置为 >= 6573688|check fs.file-max value.Recommended value >= 6573688, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"fs.file-max","paramValue":"6573688"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"fs.file-max","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5d8b0811 2024-09-11 17:57:43.248 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=264, repairable=true, repairHint=null, description=fs.pipe-user-pages-soft 检查由单个非特权用户创建或设置的所有管道的总大小(以页为单位)的软限制，建议设置为默认值|check fs.pipe-user-pages-soft value.Recommended default value, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"fs.pipe-user-pages-soft","paramValue":"16384"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"fs.pipe-user-pages-soft","paramValue":"16384"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6fbf7204 2024-09-11 17:57:43.252 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=227, repairable=true, repairHint=null, description=检查主机 mtr 命令是否存在，网络测试工具，为运维或检查提供便利。建议安装|Checks whether the mtr command is available on the host. mtr is a network test tool. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"mtr"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@8e37a95 2024-09-11 17:57:43.256 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=228, repairable=true, repairHint=null, description=检查主机 tar 命令是否存在，Linux 打包工具，OCP 依赖的重要的运维工具。建议安装|Checks whether the tar command is available on the host. tar is a packaging tool in Linux. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"tar"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@56950a8a 2024-09-11 17:57:43.259 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=229, repairable=true, repairHint=null, description=检查主机是否安装 curl 命令，命令行文本传输工具，为运维或检查提供便利。建议安装|Checks whether the curl command is available on the host. curl is a command-line tool for transferring data. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"curl"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@53db4bc9 2024-09-11 17:57:43.263 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=230, repairable=true, repairHint=null, description=检查主机是否安装 nc 命令，netcat 网络工具，OCP 依赖的重要的运维工具。建议安装|Checks whether the NetCat (nc) command is available on the host. nc is a networking tool. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"nc"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@47602ce4 2024-09-11 17:57:43.266 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=231, repairable=true, repairHint=null, description=检查主机是否安装 binutils 二进制工具集，为运维或检查提供便利。建议安装|Checks whether the binutils binary toolkit is installed on the host. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"binutils"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@54c4c14f 2024-09-11 17:57:43.270 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=232, repairable=true, repairHint=null, description=检查主机是否安装 bind-utils DNS 工具集，为运维或检查提供便利。建议安装|Checks whether the bind-utils toolkit is installed on the host. The bind-utils toolkit contains a collection of utilities for querying DNS name servers. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"bind-utils"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2afb4007 2024-09-11 17:57:43.275 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=233, repairable=true, repairHint=null, description=检查主机是否安装 iproute 网络管理工具包，为运维或检查提供便利。建议安装|Checks whether the iproute network management toolkit is installed on the host. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"iproute"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2377c392 2024-09-11 17:57:43.278 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=244, repairable=true, repairHint=null, description=libatomic 是 OceanBase 运行依赖的动态链接库，必须安装|libatomic dynamic-link library (DLL) is a dependency of OceanBase Database and must be installed., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"libatomic"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6536cc14 2024-09-11 17:57:43.282 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=247, repairable=false, repairHint=null, description=检查指定的用户（系统参数 ocp.operation.default.os.user）是否存在|check if user(ocp.operation.default.os.user in System Parameters) exists, repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@170ab744 2024-09-11 17:57:43.285 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=248, repairable=true, repairHint=null, description=检查 root 用户 openFiles 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 655350| check if (user root) limit of openFiles is set. Recommend value: 655350, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"openFiles","userName":"root","expectedValue":"655350"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@670126da 2024-09-11 17:57:43.288 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=249, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）openFiles 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 655350| check if (ocp.operation.default.os.user in System Parameters) limit of openFiles is set. Recommend value: 655350, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"openFiles","userName":"admin","expectedValue":"655350"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7af712fb 2024-09-11 17:57:43.292 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=250, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）stackSize 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 10240| check if (ocp.operation.default.os.user in System Parameters) limit of stackSize is set. Recommend value: 10240, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"stackSize","userName":"admin","expectedValue":"10240"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@297237dd 2024-09-11 17:57:43.296 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=251, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）maxUserProcesses 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 655360| check if (ocp.operation.default.os.user in System Parameters) limit of maxUserProcesses is set. Recommend value: 655360, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"maxUserProcesses","userName":"admin","expectedValue":"655360"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6b001520 2024-09-11 17:57:43.299 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=252, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）coreFileSize 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 unlimited| check if (ocp.operation.default.os.user in System Parameters) limit of coreFileSize is set. Recommend value: unlimited, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"coreFileSize","userName":"admin","expectedValue":"unlimited"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@559c96c 2024-09-11 17:57:43.302 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=253, repairable=true, repairHint=null, description=检查 SELinux 是否启用，建议用户禁用 selinux，SELinux 使用所谓的委任式访问控制 (Mandatory Access Control, MAC) ，可以针对特定的程序与特定的档案资源来进行权限的控管| check if selinux is disabled., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSelinux@1c78e5d6, parameters={"state":"disabled"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSelinux@1c78e5d6, rollbackParameters={"state":"disabled"}, rollbackReplaceField=state), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@477b73f6 2024-09-11 17:57:43.306 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=254, repairable=false, repairHint=null, description=OceanBase 数据盘文件类型，磁盘容量 > 16T 时应为 xfs 类型|File Type of OceanBase Data Disk. File system type should be xfs if disk size > 16T, repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7f0f781b 2024-09-11 17:57:43.310 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=265, repairable=false, repairHint=null, description=检查操作系统版本，一般推荐使用标准化主机文档中列举的操作系统版本 |check OS version. Recommend using OS versions listed in document, repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@144dcbcf 2024-09-11 17:57:43.313 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=266, repairable=true, repairHint=null, description=检查主机是否安装 fio 命令，磁盘压测工具，OBServer 依赖的重要的运维工具。建议安装|Checks whether the fio command is available on the host. fio is a disk pressure test tool. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"fio"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5f146283 2024-09-11 17:57:43.317 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=277, repairable=true, repairHint=null, description=检查主机 clock source 配置。建议设置为 tsc 类型（aarch64 推荐 arch_sys_conter）|Checks host OS clock source config. We recommend that you set it to tsc type(arch_sys_conter on aarch64)., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairClockSource@43f6ef2d, parameters={"sourceType":"tsc"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairClockSource@43f6ef2d, rollbackParameters={"isReset":"true", "sourceType":""}, rollbackReplaceField=sourceType), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@43ec5e54 2024-09-11 17:57:43.332 INFO 2981023 --- [ocp-async-6,ef10592392e7d5ab,4acf1369ee6daff2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:43.329 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=repair system, nodeCount=4, taskCount=4 2024-09-11 17:57:43.333 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:57:43.338 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='repair system', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=null, tenantId=null, hostId=2000010, creator='admin', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false, system_checker_operation=CHECK_AND_REPAIR, system_checker_record_id=46, host_id=2000010}, listMap={}}, createTime=null, startTime=2024-09-11T17:57:43.333+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Before repair system, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.BeforeRepairSystemInfoTask, seriesId=1, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name= clock source checks, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.RepairCheckItemTask, seriesId=0, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=SELinux config, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.RepairCheckItemTask, seriesId=2, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=After repair system, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.AfterRepairSystemInfoTask, seriesId=3, startTime=null, endTime=null}], template=null, traceId='007c5ffdbf023577', operation=EXECUTE} 2024-09-11 17:57:43.363 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=4048200, taskName=repair system, argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false, system_checker_operation=CHECK_AND_REPAIR, system_checker_record_id=46, host_id=2000010}, listMap={}} 2024-09-11 17:57:43.379 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,585c2631dc40a460] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048189, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:57:43.839 INFO 2981023 --- [task-coordinator2,,] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007105 2024-09-11 17:57:43.849 INFO 2981023 --- [task-coordinator2,,] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"activeAt":"2024-09-11T09:57:43.832Z","alarmType":"host_operation","appType":"OCP","id":6007105,"labels":{"alarm_name":"伺服器運維操作提醒","alarm_status_zh_tw":"警示中","app_type":"OCP","svr_ip":"10.31.35.7","alarm_status_native":"Active","task_id":"4048199","alarm_level":"5","alarm_level_zh_tw":"提醒","operator":"admin","alarm_summary_zh_tw":"alarm_template_id=0:host=10.31.35.7:operation_name=check system and generate repair plan SUCCESS","alarm_level_native":"5","alarm_scope_en_us":"Host","alarm_summary_en_us":"alarm_template_id=0:host=10.31.35.7:operation_name=check system and generate repair plan SUCCESS","alarm_level_en_us":"Info","alarm_type":"host_operation","auto_resolved":"1","alarm_status":"Active","alarm_group_zh_tw":"提示","host":"10.31.35.7","alarm_scope":"Host","alarm_active_at":"2024-09-11T17:57:43.832+08:00","task_result":"success","alarm_target":"alarm_template_id=0:host=10.31.35.7:operation_name=check system and generate repair plan","alarm_scope_native":"Host","alarm_description_zh_tw":"伺服器：10.31.35.7，警示：check system and generate repair plan SUCCESS, 操作者 admin，運行結果：success。","alarm_name_zh_tw":"伺服器運維操作提醒","app":"OCP","host_ip":"10.31.35.7","alarm_name_en_us":"Host operation info","alarm_name_zh_cn":"主机运维操作提醒","operation_result":"SUCCESS","alarm_status_en_us":"Active","message_type":"Alarm","alarm_status_zh_cn":"告警中","alarm_scope_zh_tw":"主機","alarm_template_id":"0","alarm_group_en_us":"info","operation_name":"check system and generate repair plan","alarm_description_en_us":"Host: 10.31.35.7, alert: check system and generate repair plan SUCCESS, operator was admin, result was success.","__host.id":"2000010","alarm_scope_zh_cn":"主机","alarm_level_zh_cn":"提醒","alarm_description_zh_cn":"服务器：10.31.35.7，告警：check system and generate repair plan SUCCESS, 操作者 admin，运行结果：success。","alarm_summary_zh_cn":"alarm_template_id=0:host=10.31.35.7:operation_name=check system and generate repair plan SUCCESS","alarm_updated_at":"2024-09-11T17:57:43.832+08:00","ocp_ip":"10.31.35.10","host_name":"vm-ea 2024-09-11 17:57:43.849 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048199, status:SUCCESS 2024-09-11 17:57:43.852 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048199, status:SUCCESS 2024-09-11 17:57:43.915 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,960b815427634697] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048191, operation:EXECUTE, state: RUNNING 2024-09-11 17:57:43.943 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,c75d7acdbc4f735b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048191, context=Context{parallelIdx=-1, stringMap={system_checker_operation=CHECK_AND_REPAIR, task_instance_id=4048200, task_operation=execute, latest_execution_start_time=2024-09-11T17:57:43.914+08:00, sub_task_instance_name=Before repair system, system_checker_record_id=46, sub_task_instance_id=4048191, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:57:43.958 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,c75d7acdbc4f735b] c.o.o.c.c.i.t.BeforeRepairSystemInfoTask : update system checker status successfully, hostId: 2000010, recordId: 46 2024-09-11 17:57:43.983 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,c75d7acdbc4f735b] c.o.o.c.c.i.t.BeforeRepairSystemInfoTask : update system checker record successfully, hostId: 2000010, recordId: 46 2024-09-11 17:57:43.992 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,c75d7acdbc4f735b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048191, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:57:44.265 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,acc27e5e389dd295,10926ecc68bef823] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=acc27e5e389dd295, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:57:44.266 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-1,acc27e5e389dd295,10926ecc68bef823] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:57:44.279 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,acc27e5e389dd295,10926ecc68bef823] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=acc27e5e389dd295, duration=17 ms] 2024-09-11 17:57:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,7ee1c4846c741614,2c1a0c8c479b84f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,7ee1c4846c741614,2c1a0c8c479b84f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:44.707 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:44.708 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:44.858 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048193] 2024-09-11 17:57:44.932 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,dedda2a74cbfc6b5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048193, operation:EXECUTE, state: RUNNING 2024-09-11 17:57:44.957 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,b7b940a489b97783] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048193, context=Context{parallelIdx=-1, stringMap={system_checker_operation=CHECK_AND_REPAIR, task_instance_id=4048200, task_operation=execute, latest_execution_start_time=2024-09-11T17:57:44.930+08:00, sub_task_instance_name=SELinux config, system_checker_record_id=46, sub_task_instance_id=4048193, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:57:44.965 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,b7b940a489b97783] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:44.972 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,b7b940a489b97783] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:44.994 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,b7b940a489b97783] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/setSelinux, request body:SetSelinuxRequest(state=disabled), params:null 2024-09-11 17:57:45.035 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,b7b940a489b97783] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048193, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:57:45.101 INFO 2981023 --- [prometheus-second-schedule-1,3e6a05c4f716b5f8,abe52b9ffc7ad858] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:57:45.332 INFO 2981023 --- [ocp-async-3,7c35129a29681554,13018c67193eae02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:45.859 INFO 2981023 --- [ocp-schedules-22,699c177541f43100,6924997362a80981] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:57:45.875 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048192] 2024-09-11 17:57:45.950 WARN 2981023 --- [scanner-13579475-1,699c177541f43100,023171b7d1e21167] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:57:45.952 INFO 2981023 --- [alarm-detect-16,699c177541f43100,5ffaa9b82f508f92] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645076, data is empty. 2024-09-11 17:57:45.956 INFO 2981023 --- [alarm-detect-16,699c177541f43100,5ffaa9b82f508f92] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645076, data is empty. 2024-09-11 17:57:45.961 INFO 2981023 --- [alarm-detect-16,699c177541f43100,5ffaa9b82f508f92] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645076, data is empty. 2024-09-11 17:57:45.962 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,8b7a752e43234494] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048192, operation:EXECUTE, state: RUNNING 2024-09-11 17:57:45.969 INFO 2981023 --- [alarm-detect-16,699c177541f43100,967d96dd9edafb0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645076, data is empty. 2024-09-11 17:57:45.974 INFO 2981023 --- [alarm-detect-17,699c177541f43100,76fc294088277cea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645076, data is empty. 2024-09-11 17:57:45.977 INFO 2981023 --- [alarm-detect-17,699c177541f43100,76fc294088277cea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645076, data is empty. 2024-09-11 17:57:45.980 INFO 2981023 --- [alarm-detect-17,699c177541f43100,76fc294088277cea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645076, data is empty. 2024-09-11 17:57:45.983 INFO 2981023 --- [alarm-detect-16,699c177541f43100,967d96dd9edafb0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645076, data is empty. 2024-09-11 17:57:45.987 INFO 2981023 --- [alarm-detect-16,699c177541f43100,967d96dd9edafb0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645076, data is empty. 2024-09-11 17:57:46.024 INFO 2981023 --- [ocp-schedules-9,b6d4f617cfefe486,c7c9991ca5eb668d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:57:46.013 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,7231233824818f06] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048192, context=Context{parallelIdx=-1, stringMap={system_checker_operation=CHECK_AND_REPAIR, task_instance_id=4048200, task_operation=execute, latest_execution_start_time=2024-09-11T17:57:45.958+08:00, sub_task_instance_name= clock source checks, system_checker_record_id=46, sub_task_instance_id=4048192, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:57:46.048 INFO 2981023 --- [ocp-schedules-15,6f6ffb12457a459b,ecf5e14895410184] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:57:46.054 INFO 2981023 --- [ocp-schedules-10,2d289cff38382b87,9b8ee80fe306d5d5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:57:46.064 INFO 2981023 --- [host-check-clockdiff1,b6d4f617cfefe486,6615917c355366b0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:57:46.064 INFO 2981023 --- [host-check-clockdiff2,b6d4f617cfefe486,93141a0f92b0c942] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:57:46.065 INFO 2981023 --- [host-check-clockdiff3,b6d4f617cfefe486,cb47895b3b2d83d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:57:46.067 INFO 2981023 --- [host-check-clockdiff4,b6d4f617cfefe486,f02d9a17cbfd83c0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:57:46.066 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,7231233824818f06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:46.079 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,7231233824818f06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:46.103 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,7231233824818f06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/setClockSource, request body:SetClockSourceRequest(sourceType=tsc), params:null 2024-09-11 17:57:46.113 ERROR 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,7231233824818f06] c.o.o.c.c.i.r.methods.RepairClockSource : set clock source to tsc failed: [AgentClient]:http request is failed, response:Unexpected error: symlink /usr/lib/systemd/system/set_clocksource.service /etc/systemd/system/multi-user.target.wants/set_clocksource.service: file exists 2024-09-11 17:57:46.129 ERROR 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,7231233824818f06] c.o.o.c.c.i.h.SystemCheckerHelperImpl : Failed to repair 277. Please see the log for details 2024-09-11 17:57:46.134 ERROR 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,7231233824818f06] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.common.unexpected, and args [2000010] 2024-09-11 17:57:46.160 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:57:46.164 INFO 2981023 --- [ocp-schedules-29,a1578ce3a57a95c7,c8b460d19ad4a9af] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:57:46.171 INFO 2981023 --- [ocp-schedules-29,a1578ce3a57a95c7,c8b460d19ad4a9af] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:57:46.187 INFO 2981023 --- [ocp-schedules-27,f3771affd05fd7c5,300f3731f18a982b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:57:46.187 INFO 2981023 --- [ocp-schedules-12,dfa346fd3ec117c2,cfbd6e9e2c8ff9cb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:57:46.187 INFO 2981023 --- [ocp-schedules-6,fefd08949c11d5c0,ce95b506a9313f7a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:57:46.187 INFO 2981023 --- [ocp-schedules-17,206d0741a0817df9,9f357c16ae95c36e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:57:46.189 INFO 2981023 --- [ocp-schedules-12,dfa346fd3ec117c2,cfbd6e9e2c8ff9cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:57:46.190 INFO 2981023 --- [ocp-schedules-27,f3771affd05fd7c5,300f3731f18a982b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:57:46.190 INFO 2981023 --- [ocp-schedules-6,fefd08949c11d5c0,ce95b506a9313f7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:57:46.198 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:57:46.210 INFO 2981023 --- [ocp-schedules-11,6de469e2be70d917,cf5649ad0a3ed88c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:57:46.264 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:46.268 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:46.268 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:57:46.269 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:57:46.269 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:46.280 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:57:46.291 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:57:46.291 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:57:46.297 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:46.297 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:46.298 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:57:46.298 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:57:46.300 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:46.301 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:46.301 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:46.303 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:46.331 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:46.332 INFO 2981023 --- [host-check-agent-status1,6f6ffb12457a459b,b8622c10b0cd8930] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:57:46.332 INFO 2981023 --- [host-check-agent-status2,6f6ffb12457a459b,bbe882849c6f18c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:57:46.332 INFO 2981023 --- [host-check-agent-status3,6f6ffb12457a459b,ad351135f38ea5a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:57:46.333 INFO 2981023 --- [host-check-agent-status4,6f6ffb12457a459b,70f5ec443b509f77] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:57:46.333 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:46.337 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:46.338 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:46.338 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:57:46.339 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:57:46.339 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:46.343 INFO 2981023 --- [host-check-agent-status2,6f6ffb12457a459b,bbe882849c6f18c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:46.344 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:57:46.345 INFO 2981023 --- [host-check-agent-status2,6f6ffb12457a459b,bbe882849c6f18c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:46.346 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:57:46.351622489+08:00 2024-09-11 17:57:46.347 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:57:46.346+08:00 2024-09-11 17:57:46.353 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:57:46.346+08:00 and status AVAILABLE 2024-09-11 17:57:46.358 INFO 2981023 --- [host-check-agent-status4,6f6ffb12457a459b,70f5ec443b509f77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:46.358 INFO 2981023 --- [host-check-agent-status1,6f6ffb12457a459b,b8622c10b0cd8930] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:46.360 INFO 2981023 --- [host-check-agent-status3,6f6ffb12457a459b,ad351135f38ea5a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:46.360 INFO 2981023 --- [host-check-agent-status4,6f6ffb12457a459b,70f5ec443b509f77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:46.361 INFO 2981023 --- [host-check-agent-status3,6f6ffb12457a459b,ad351135f38ea5a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:46.362 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:57:46.362 INFO 2981023 --- [host-check-agent-status1,6f6ffb12457a459b,b8622c10b0cd8930] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:46.368 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:57:46.370 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:57:46.368649+08:00, lastProcessTime=2024-09-11T17:57:36.338491+08:00 2024-09-11 17:57:46.372 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:57:46.373 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:57:46.374 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:57:46.379720400+08:00 2024-09-11 17:57:46.376 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:57:46.374+08:00 2024-09-11 17:57:46.379 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:57:46.374+08:00 and status AVAILABLE 2024-09-11 17:57:46.382 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:57:46.384 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T17:57:46.384635537+08:00 2024-09-11 17:57:46.387 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T17:57:46.384+08:00 2024-09-11 17:57:46.387 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:57:46.389 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T17:57:46.384+08:00 and status AVAILABLE 2024-09-11 17:57:46.405 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 17:57:46.422 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:57:46.428 INFO 2981023 --- [host-check-clockdiff3,b6d4f617cfefe486,cb47895b3b2d83d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048666 0 0 , error= 2024-09-11 17:57:46.429 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:57:46.429467549+08:00 2024-09-11 17:57:46.430 INFO 2981023 --- [ocp-schedules-12,dfa346fd3ec117c2,cfbd6e9e2c8ff9cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:57:46.460 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:57:46.429+08:00 2024-09-11 17:57:46.461 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:46.464 INFO 2981023 --- [host-check-agent-status2,6f6ffb12457a459b,bbe882849c6f18c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:57:46.466 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:57:46.429+08:00 and status AVAILABLE 2024-09-11 17:57:46.469 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:46.473 INFO 2981023 --- [host-check-agent-status2,6f6ffb12457a459b,bbe882849c6f18c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:57:46.473 INFO 2981023 --- [host-check-agent-status2,6f6ffb12457a459b,bbe882849c6f18c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:57:46.473 INFO 2981023 --- [host-check-agent-status2,6f6ffb12457a459b,bbe882849c6f18c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:57:46.475 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:46.427 ERROR 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,7231233824818f06] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : An unknown error has occurred. Cause: 2,000,010. Error message: {1}. Contact the administrator. com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=COMMON_UNEXPECTED, args=2000010 at sun.reflect.GeneratedConstructorAccessor732.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:71) at com.oceanbase.ocp.compute.checker.internal.task.RepairCheckItemTask.run(RepairCheckItemTask.java:59) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:57:46.479 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:57:46.479 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:46.482 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:57:46.484 INFO 2981023 --- [host-check-agent-status1,6f6ffb12457a459b,b8622c10b0cd8930] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:57:46.485 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:57:46.487 INFO 2981023 --- [host-check-agent-status1,6f6ffb12457a459b,b8622c10b0cd8930] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:57:46.487 INFO 2981023 --- [host-check-agent-status1,6f6ffb12457a459b,b8622c10b0cd8930] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:57:46.487 INFO 2981023 --- [host-check-agent-status1,6f6ffb12457a459b,b8622c10b0cd8930] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:57:46.488 INFO 2981023 --- [host-check-agent-status3,6f6ffb12457a459b,ad351135f38ea5a6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:57:46.492 INFO 2981023 --- [host-check-agent-status3,6f6ffb12457a459b,ad351135f38ea5a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:57:46.492 INFO 2981023 --- [host-check-agent-status3,6f6ffb12457a459b,ad351135f38ea5a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:57:46.492 INFO 2981023 --- [host-check-agent-status3,6f6ffb12457a459b,ad351135f38ea5a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:57:46.495 INFO 2981023 --- [host-check-agent-status4,6f6ffb12457a459b,70f5ec443b509f77] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:57:46.500 INFO 2981023 --- [host-check-agent-status4,6f6ffb12457a459b,70f5ec443b509f77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:57:46.500 INFO 2981023 --- [host-check-agent-status4,6f6ffb12457a459b,70f5ec443b509f77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:57:46.500 INFO 2981023 --- [host-check-agent-status4,6f6ffb12457a459b,70f5ec443b509f77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:57:46.506 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,7231233824818f06] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048192, operation:EXECUTE, state: DISREGARDED 2024-09-11 17:57:46.668 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:46.718 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:46.720 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:46.721 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:57:46.722 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:57:46.754 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:46.755 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:57:46.755 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:57:46.756 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:46.763 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:57:46.843 INFO 2981023 --- [ocp-schedules-3,30f3e8425b57cf07,419557d9547fb0df] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7236,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":4757} 2024-09-11 17:57:46.843 INFO 2981023 --- [ocp-schedules-3,30f3e8425b57cf07,419557d9547fb0df] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:57:46.975 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:57:46.978 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:57:46.997 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 17:57:46.997 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:57:47.000 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:47.004 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.004 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:47.005 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:47.007 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:57:47.011 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:57:47.012 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.014 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:47.014 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:57:47.019 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:57:47.022 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.023 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:47.025 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.027 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.028 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:47.028 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:57:47.028 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:47.032 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 17:57:47.033 INFO 2981023 --- [alarm-detect-4,699c177541f43100,87579754ad391b34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645076, data is empty. 2024-09-11 17:57:47.046 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:57:47.047 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:57:47.049 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:57:47.049 INFO 2981023 --- [alarm-detect-4,699c177541f43100,87579754ad391b34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645076, data is empty. 2024-09-11 17:57:47.051 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.052 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:47.060 INFO 2981023 --- [alarm-detect-4,699c177541f43100,87579754ad391b34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645076, data is empty. 2024-09-11 17:57:47.066 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:57:47.067 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.068 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.070 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:57:47.072 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:57:47.072 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.089 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:57:47.091 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:57:47.121 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:57:47.131 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:57:46.368649+08:00 2024-09-11 17:57:47.144 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.152 INFO 2981023 --- [ocp-schedules-27,f3771affd05fd7c5,300f3731f18a982b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:57:47.187 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.187 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:57:47.192 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.193 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.194 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:57:47.202 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:57:47.204 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:57:47.205 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:57:47.208 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:57:47.219 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:57:47.223 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 17:57:47.235 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:57:47.238 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.239 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.243 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 17:57:47.243 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.243 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:57:47.243 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.244 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:47.244 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:57:47.248 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:57:47.248 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:57:47.248 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.249 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.251 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:57:47.252 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:57:47.254 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.254 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.254 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:47.255 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.258 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:57:47.259 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:57:47.270 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:57:47.277 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:47.277 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:57:47.282 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:57:47.289 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:47.289 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:57:47.299 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:57:47.309 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:57:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,bf5aa882805e621c,70e2b425d5b9d6cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:47.313 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:57:47.317 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:57:47.323 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:47.332 INFO 2981023 --- [ocp-async-4,02e02ba5d244e886,f64155212d0b7b2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:47.334 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 17:57:47.361 INFO 2981023 --- [prometheus-minute-schedule-1,bf5aa882805e621c,70e2b425d5b9d6cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:47.365 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:47.365 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:57:47.374 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.374 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:57:47.375 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.376 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:57:47.377 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.377 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.378 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:47.379 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:57:47.380 INFO 2981023 --- [host-check-host-status4,2d289cff38382b87,913ae61de51fa6ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 17:57:47.386 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.386 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.387 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:47.389 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:57:47.395 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:57:47.395 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:57:47.397 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:57:47.400 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:57:47.402 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:57:47.405 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:57:47.409 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:57:47.413 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.414 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.414 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:57:47.415 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:57:47.415 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.415 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:57:47.416 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:57:47.417 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:57:47.418 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:57:47.426 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:57:47.459 INFO 2981023 --- [host-check-host-status2,2d289cff38382b87,2906245926a0951a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:57:47.460 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:57:47.466 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:57:47.467 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:57:47.473 INFO 2981023 --- [host-check-host-status1,2d289cff38382b87,365bc5f8c6e9df54] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:57:47.474 INFO 2981023 --- [ocp-schedules-7,39661214448c11ba,f0479edeb8e75e3e] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:57:47.478 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:47.478 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:57:47.487 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:57:47.505 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:57:47.510 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:57:47.522 INFO 2981023 --- [host-check-host-status3,2d289cff38382b87,20e49c746f1686b7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:57:47.524 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.527 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.528 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:57:47.528 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:57:47.552 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:57:47.552 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:57:47.556 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:57:47.560 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.561 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:57:47.562 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:57:47.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.564 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:57:47.565 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.565 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:57:47.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:57:47.573 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:57:47.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:57:47.573 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:57:47.576 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:57:47.605 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:57:47.610 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:47.611 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:47.611 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:57:47.616 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:57:47.616 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:57:47.616 INFO 2981023 --- [ocp-schedules-6,fefd08949c11d5c0,ce95b506a9313f7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:57:47.708 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:47.708 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:47.769 INFO 2981023 --- [ocp-async-7,b83df0d499c084b6,75631220e99defbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:57:47.931 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048194] 2024-09-11 17:57:48.009 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,8958df3c059ae55c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048194, operation:EXECUTE, state: RUNNING 2024-09-11 17:57:48.028 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,72f4462a6fbf357b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048194, context=Context{parallelIdx=-1, stringMap={system_checker_operation=CHECK_AND_REPAIR, task_instance_id=4048200, task_operation=execute, latest_execution_start_time=2024-09-11T17:57:48.008+08:00, sub_task_instance_name=After repair system, system_checker_record_id=46, sub_task_instance_id=4048194, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:57:48.034 WARN 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,72f4462a6fbf357b] c.o.o.c.c.i.t.AfterRepairSystemInfoTask : Some check items failed to be repaired on host(hostId: 2000010). Please see the log for details 2024-09-11 17:57:48.048 INFO 2981023 --- [manual-subtask-executor16,007c5ffdbf023577,72f4462a6fbf357b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048194, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:57:49.002 INFO 2981023 --- [task-coordinator2,,] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007106 2024-09-11 17:57:49.010 INFO 2981023 --- [task-coordinator2,,] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"activeAt":"2024-09-11T09:57:48.998Z","alarmType":"host_operation","appType":"OCP","id":6007106,"labels":{"alarm_name":"伺服器運維操作提醒","alarm_status_zh_tw":"警示中","app_type":"OCP","svr_ip":"10.31.35.7","alarm_status_native":"Active","task_id":"4048200","alarm_level":"5","alarm_level_zh_tw":"提醒","operator":"admin","alarm_summary_zh_tw":"alarm_template_id=0:host=10.31.35.7:operation_name=repair system SUCCESS","alarm_level_native":"5","alarm_scope_en_us":"Host","alarm_summary_en_us":"alarm_template_id=0:host=10.31.35.7:operation_name=repair system SUCCESS","alarm_level_en_us":"Info","alarm_type":"host_operation","auto_resolved":"1","alarm_status":"Active","alarm_group_zh_tw":"提示","host":"10.31.35.7","alarm_scope":"Host","alarm_active_at":"2024-09-11T17:57:48.998+08:00","task_result":"success","alarm_target":"alarm_template_id=0:host=10.31.35.7:operation_name=repair system","alarm_scope_native":"Host","alarm_description_zh_tw":"伺服器：10.31.35.7，警示：repair system SUCCESS, 操作者 admin，運行結果：success。","alarm_name_zh_tw":"伺服器運維操作提醒","app":"OCP","host_ip":"10.31.35.7","alarm_name_en_us":"Host operation info","alarm_name_zh_cn":"主机运维操作提醒","operation_result":"SUCCESS","alarm_status_en_us":"Active","message_type":"Alarm","alarm_status_zh_cn":"告警中","alarm_scope_zh_tw":"主機","alarm_template_id":"0","alarm_group_en_us":"info","operation_name":"repair system","alarm_description_en_us":"Host: 10.31.35.7, alert: repair system SUCCESS, operator was admin, result was success.","__host.id":"2000010","alarm_scope_zh_cn":"主机","alarm_level_zh_cn":"提醒","alarm_description_zh_cn":"服务器：10.31.35.7，告警：repair system SUCCESS, 操作者 admin，运行结果：success。","alarm_summary_zh_cn":"alarm_template_id=0:host=10.31.35.7:operation_name=repair system SUCCESS","alarm_updated_at":"2024-09-11T17:57:48.998+08:00","ocp_ip":"10.31.35.10","host_name":"vm-eas-prd-rtd-mid05","alarm_group_zh_cn":"提示"},"level":5,"scope":"Host","status":"Active","statusUpdatedAt":"2024-09-11T09:57:49Z","target":"alarm_template_id=0:host=10.31.35.7:operation_name=repa 2024-09-11 17:57:49.010 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048200, status:SUCCESS 2024-09-11 17:57:49.013 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048200, status:SUCCESS 2024-09-11 17:57:49.228 INFO 2981023 --- [host-check-clockdiff2,b6d4f617cfefe486,93141a0f92b0c942] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048669 5 5 , error= 2024-09-11 17:57:49.258 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e612cfe51e1afca3,a19ae1943a29248a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=e612cfe51e1afca3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:57:49.259 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e612cfe51e1afca3,a19ae1943a29248a] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:57:49.297 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e612cfe51e1afca3,a19ae1943a29248a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=e612cfe51e1afca3, duration=44 ms] 2024-09-11 17:57:49.332 INFO 2981023 --- [ocp-async-9,9a60355d20c4aeee,535eda3be6786841] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:57:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:57:50.283 INFO 2981023 --- [ocp-schedules-5,a0c5f25f128f0102,c14e3521ad288b7f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:57:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,41d60f57422c251e,3befee4ec1621caf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,41d60f57422c251e,3befee4ec1621caf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:50.579 INFO 2981023 --- [host-check-clockdiff1,b6d4f617cfefe486,6615917c355366b0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048670 6 6 , error= 2024-09-11 17:57:50.708 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:50.710 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:51.332 INFO 2981023 --- [ocp-async-7,099627b0a386681b,0b6e8f958c0ccd4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:51.583 INFO 2981023 --- [ocp-schedules-2,8cabe0c92b5987ce,2dd0d7a0eb12c03d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:57:51.584 INFO 2981023 --- [ocp-schedules-2,8cabe0c92b5987ce,2dd0d7a0eb12c03d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:57:51.584 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:57:51.584 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:57:51.584 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:57:51.585 INFO 2981023 --- [ocp-schedules-2,8cabe0c92b5987ce,2dd0d7a0eb12c03d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:57:52.089 INFO 2981023 --- [host-check-clockdiff4,b6d4f617cfefe486,f02d9a17cbfd83c0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048672 1 1 , error= 2024-09-11 17:57:52.337 INFO 2981023 --- [ocp-schedules-11,e7ed18d43eb8a26d,29ec905f95067617] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:57:53.201 INFO 2981023 --- [ocp-schedules-1,af72148f5246460f,e9a6f6c8bf7da1ba] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:57:53.201 INFO 2981023 --- [ocp-schedules-1,af72148f5246460f,e9a6f6c8bf7da1ba] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:57:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,ba794f56d93b182f,3eb79d8597cecae4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:53.317 INFO 2981023 --- [prometheus-minute-schedule-1,ba794f56d93b182f,3eb79d8597cecae4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:53.332 INFO 2981023 --- [ocp-async-10,dc461d7a99821c1c,b68491aea4e72a35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:53.710 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:53.712 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:53.878 INFO 2981023 --- [ocp-schedules-10,e740fcac49d577bb,f4376e320d56fb46] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:57:53.898 WARN 2981023 --- [ocp-schedules-10,e740fcac49d577bb,f4376e320d56fb46] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:57:54.340 INFO 2981023 --- [ocp-schedules-30,c014265215416c22,d81a61e3ea06a38b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:57:54.752 INFO 2981023 --- [ocp-schedules-25,a903dd83d569f177,a329d825375f3162] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:57:54.760 INFO 2981023 --- [ocp-schedules-25,a903dd83d569f177,a329d825375f3162] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:57:54.760 INFO 2981023 --- [ocp-schedules-25,a903dd83d569f177,a329d825375f3162] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:57:54.929 INFO 2981023 --- [ocp-schedules-16,e4ef9077d2590f94,a0cda4273697d2f5] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 17:57:54.929 INFO 2981023 --- [ocp-schedules-16,e4ef9077d2590f94,a0cda4273697d2f5] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 17:57:54.934 INFO 2981023 --- [ocp-schedules-16,e4ef9077d2590f94,a0cda4273697d2f5] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 17:57:54.934 INFO 2981023 --- [ocp-schedules-16,e4ef9077d2590f94,a0cda4273697d2f5] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 17:57:54.934 INFO 2981023 --- [ocp-schedules-16,e4ef9077d2590f94,a0cda4273697d2f5] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 17:57:54.944 INFO 2981023 --- [ocp-schedules-16,e4ef9077d2590f94,a0cda4273697d2f5] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 17:57:54.944 INFO 2981023 --- [ocp-schedules-16,e4ef9077d2590f94,a0cda4273697d2f5] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 17:57:54.944 INFO 2981023 --- [ocp-schedules-16,e4ef9077d2590f94,a0cda4273697d2f5] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 17:57:54.955 INFO 2981023 --- [ocp-schedules-16,e4ef9077d2590f94,a0cda4273697d2f5] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 17:57:55.333 INFO 2981023 --- [ocp-async-6,cca99d88a200d0cd,beaf1ba2c1a1ea37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:55.807 INFO 2981023 --- [ocp-schedules-3,fbedc01d2ccdaefa,1d880206720c9476] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:57:55.807 INFO 2981023 --- [ocp-schedules-3,fbedc01d2ccdaefa,1d880206720c9476] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403658901516288 2024-09-11 17:57:55.811 INFO 2981023 --- [ocp-schedules-3,fbedc01d2ccdaefa,1d880206720c9476] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=54, maxSyncedRecordId=690420679617835008 2024-09-11 17:57:55.812 INFO 2981023 --- [ocp-schedules-3,fbedc01d2ccdaefa,1d880206720c9476] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:57:55.812 INFO 2981023 --- [ocp-schedules-3,fbedc01d2ccdaefa,1d880206720c9476] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:57:55.814 INFO 2981023 --- [ocp-schedules-3,fbedc01d2ccdaefa,1d880206720c9476] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:57:55.814 INFO 2981023 --- [ocp-schedules-3,fbedc01d2ccdaefa,1d880206720c9476] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:57:55.937 INFO 2981023 --- [ocp-schedules-27,414fcded368727a2,b067ad6ca1b088bb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:57:55.943 INFO 2981023 --- [ocp-schedules-27,414fcded368727a2,b067ad6ca1b088bb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:57:55.943 INFO 2981023 --- [ocp-schedules-27,414fcded368727a2,b067ad6ca1b088bb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:57:56.069 INFO 2981023 --- [ocp-schedules-30,bcd8a5791ce1ce24,3cfbf6d2186a3bd6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:57:56.069+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:57:56.075 INFO 2981023 --- [ocp-schedules-30,bcd8a5791ce1ce24,3cfbf6d2186a3bd6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:57:56.201 INFO 2981023 --- [ocp-schedules-30,bcd8a5791ce1ce24,3cfbf6d2186a3bd6] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 18 to SqlAggregateData 2024-09-11 17:57:56.202 INFO 2981023 --- [ocp-schedules-30,bcd8a5791ce1ce24,3cfbf6d2186a3bd6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:57:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,002c317f83d745ae,e7753526358b438d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,002c317f83d745ae,e7753526358b438d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:57:56.352506+08:00, lastProcessTime=2024-09-11T17:57:46.368649+08:00 2024-09-11 17:57:56.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:57:56.488 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:57:56.352506+08:00 2024-09-11 17:57:56.712 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:56.713 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:56.842 INFO 2981023 --- [ocp-schedules-8,d7895153fffa013a,d5f836a8f0ffafe0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:57:56.843 INFO 2981023 --- [ocp-schedules-8,d7895153fffa013a,d5f836a8f0ffafe0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:57:56.885 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:57:56.885 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.887 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:57:56.887 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.888 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:57:56.888 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:57:56.889 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:57:56.891 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:57:56.895 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.895 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.895 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:56.895 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:56.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:56.904 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.904 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:56.908 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.908 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:56.909 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.909 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:57:56.911 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,abe21d0b8779da19,de08d0342cad12d1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=abe21d0b8779da19, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:57:56.912 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.912 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:56.916 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.916 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:57:56.917 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,abe21d0b8779da19,de08d0342cad12d1] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:57:56.922 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.922 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:57:56.925 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.925 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:56.936 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.936 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:57:56.937 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.937 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:56.937 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:57:56.938 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4051D6D1-0-0] 2024-09-11 17:57:56.938 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:57:56.938 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.938 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:57:56.944 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.945 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:57:56.945 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.946 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:57:56.946 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:57:56.947 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:57:56.953 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.953 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:57:56.954 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:57:56.964 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.964 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.965 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:57:56.965 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:57:56.965 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:57:56.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4051D6DB-0-0] 2024-09-11 17:57:56.966 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:57:56.966 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:57:56.967 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.968 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:57:56.968 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.969 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:57:56.969 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.970 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:57:56.971 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.972 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:57:56.972 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.973 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:56.973 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:57:56.974 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.974 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.974 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:57:56.974 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:57:56.974 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.975 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.975 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.975 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:56.975 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:57:56.976 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:57:56.976 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:57:56.976 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.977 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:57:56.978 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.978 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:57:56.979 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:57:56.979 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.980 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:57:56.980 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,abe21d0b8779da19,de08d0342cad12d1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=abe21d0b8779da19, duration=80 ms] 2024-09-11 17:57:56.980 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.981 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:57:56.981 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:57:56.987 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:57:56.988 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:57:56.992 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:56.993 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:57:56.994 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:57:56.994 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:57:56.994 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:56.994 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:57:57.000 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:57:57.000 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:57:57.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.002 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:57:57.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:57:57.002 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.002 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:57:57.009 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:57:57.010 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:57:57.010 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.011 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:57.013 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:57:57.015 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:57:57.015 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.015 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:57:57.017 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:57:57.018 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:57:57.018 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:57:57.018 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:57:57.026 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:57:57.028 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:57:57.028 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:57:57.028 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:57:57.028 INFO 2981023 --- [ocp-schedules-8,d7895153fffa013a,d5f836a8f0ffafe0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:57:57.054 INFO 2981023 --- [ocp-schedules-20,7bd6b6044717f738,75ef17766a1fb4d9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:57:57.054 INFO 2981023 --- [ocp-schedules-20,7bd6b6044717f738,75ef17766a1fb4d9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:57:57.069 INFO 2981023 --- [Collect all tenants major compactions2,7bd6b6044717f738,ff7aaa6cd9aa0a44] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:57:57.070 INFO 2981023 --- [Collect all tenants major compactions4,7bd6b6044717f738,ea1d140522e8451b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:57:57.070 INFO 2981023 --- [Collect all tenants major compactions5,7bd6b6044717f738,a23674e05723b28f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:57:57.071 INFO 2981023 --- [Collect all tenants major compactions3,7bd6b6044717f738,278b612415ca2c03] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:57:57.072 INFO 2981023 --- [Collect all tenants major compactions1,7bd6b6044717f738,880f538b541f53bf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:57:57.073 INFO 2981023 --- [Collect all tenants major compactions6,7bd6b6044717f738,4ca6f7de8e0f9384] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:57:57.083 INFO 2981023 --- [Collect all tenants major compactions5,7bd6b6044717f738,a23674e05723b28f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.084 INFO 2981023 --- [Collect all tenants major compactions5,7bd6b6044717f738,a23674e05723b28f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:57:57.085 INFO 2981023 --- [Collect all tenants major compactions2,7bd6b6044717f738,ff7aaa6cd9aa0a44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.085 INFO 2981023 --- [Collect all tenants major compactions1,7bd6b6044717f738,880f538b541f53bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.086 INFO 2981023 --- [Collect all tenants major compactions2,7bd6b6044717f738,ff7aaa6cd9aa0a44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:57:57.089 INFO 2981023 --- [Collect all tenants major compactions1,7bd6b6044717f738,880f538b541f53bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:57:57.090 INFO 2981023 --- [Collect all tenants major compactions5,7bd6b6044717f738,a23674e05723b28f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:57:57.091 INFO 2981023 --- [Collect all tenants major compactions2,7bd6b6044717f738,ff7aaa6cd9aa0a44] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:57:57.094 INFO 2981023 --- [Collect all tenants major compactions4,7bd6b6044717f738,ea1d140522e8451b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.094 INFO 2981023 --- [Collect all tenants major compactions2,7bd6b6044717f738,ff7aaa6cd9aa0a44] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:57:57.094 INFO 2981023 --- [Collect all tenants major compactions1,7bd6b6044717f738,880f538b541f53bf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:57:57.094 INFO 2981023 --- [Collect all tenants major compactions1,7bd6b6044717f738,880f538b541f53bf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:57:57.095 INFO 2981023 --- [Collect all tenants major compactions4,7bd6b6044717f738,ea1d140522e8451b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:57:57.096 INFO 2981023 --- [Collect all tenants major compactions5,7bd6b6044717f738,a23674e05723b28f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:57:57.097 INFO 2981023 --- [Collect all tenants major compactions4,7bd6b6044717f738,ea1d140522e8451b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:57:57.097 INFO 2981023 --- [Collect all tenants major compactions4,7bd6b6044717f738,ea1d140522e8451b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:57:57.100 INFO 2981023 --- [Collect all tenants major compactions3,7bd6b6044717f738,278b612415ca2c03] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.100 INFO 2981023 --- [Collect all tenants major compactions3,7bd6b6044717f738,278b612415ca2c03] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:57:57.102 INFO 2981023 --- [Collect all tenants major compactions3,7bd6b6044717f738,278b612415ca2c03] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:57:57.102 INFO 2981023 --- [Collect all tenants major compactions3,7bd6b6044717f738,278b612415ca2c03] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:57:57.103 INFO 2981023 --- [Collect all tenants major compactions6,7bd6b6044717f738,4ca6f7de8e0f9384] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:57:57.104 INFO 2981023 --- [Collect all tenants major compactions6,7bd6b6044717f738,4ca6f7de8e0f9384] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:57:57.106 INFO 2981023 --- [Collect all tenants major compactions6,7bd6b6044717f738,4ca6f7de8e0f9384] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:57:57.106 INFO 2981023 --- [Collect all tenants major compactions6,7bd6b6044717f738,4ca6f7de8e0f9384] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:57:57.111 INFO 2981023 --- [ocp-schedules-20,7bd6b6044717f738,75ef17766a1fb4d9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:57:57.235 INFO 2981023 --- [ocp-schedules-28,7cef511201de62a0,e615fdb655202d8d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:57:57.255 INFO 2981023 --- [ocp-schedules-24,e62c361cf06f6294,152933f832446733] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:57:57.322 WARN 2981023 --- [scanner-13579776-1,7cef511201de62a0,5fb31c6a85c6e248] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:57:57.333 INFO 2981023 --- [ocp-async-3,8f9c436d6a0b7fee,88f8c15bc25d4075] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:57.769 INFO 2981023 --- [ocp-async-6,cde579efb946c4d9,71eda06850410668] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:57:58.006 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e246d14e8f440525,88da0248e04f78a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=e246d14e8f440525, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:57:58.007 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e246d14e8f440525,88da0248e04f78a9] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:57:58.020 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e246d14e8f440525,88da0248e04f78a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=e246d14e8f440525, duration=16 ms] 2024-09-11 17:57:58.702 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.706 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obC 2024-09-11 17:57:58.710 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.713 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.716 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.719 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.721 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.724 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.726 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.728 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.729 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.730 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.730 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.731 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.735 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.737 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.740 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.743 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.744 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.745 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.745 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.746 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.746 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.747 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.748 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.748 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:51:58.611, end=2024-09-11T17:56:58.698} 2024-09-11 17:57:58.748 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 17:57:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,0f9396bc6e07e9bb,ea46b104d4759f3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048680, interval=60 2024-09-11 17:57:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,0f9396bc6e07e9bb,ea46b104d4759f3a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:57:59.333 INFO 2981023 --- [ocp-async-10,2d182391bc2ce511,c8ff166dc00bbef6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:57:59.713 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:57:59.714 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:57:59.955 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,efacc0da9c733c74,ec0797f87e5083d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts/2000010/repairSystemInfoReport?id=2000010, client=10.31.33.41, traceId=efacc0da9c733c74, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.getRepairSystemInfoReport(Long), args=2000010,] 2024-09-11 17:57:59.959 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,efacc0da9c733c74,ec0797f87e5083d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts/2000010/repairSystemInfoReport client=10.31.33.41, traceId=efacc0da9c733c74, duration=7 ms] 2024-09-11 17:58:01.333 INFO 2981023 --- [ocp-async-6,4e28a1f1ac9e1427,aa11093ddeb8f457] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:02.138 INFO 2981023 --- [ocp-schedules-30,e4efd2d157eafaef,f71b48ad6da1c31a] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:58:02.139 INFO 2981023 --- [ocp-schedules-30,e4efd2d157eafaef,f71b48ad6da1c31a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 17:58:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,c240b25188799857,2d13c59874178aad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,c240b25188799857,2d13c59874178aad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:02.341 INFO 2981023 --- [ocp-schedules-28,0ce55373fb247378,df6e9182b0f5c0c0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:58:02.583 INFO 2981023 --- [ocp-schedules-14,8c75c352932a65c4,6ded8c1d9ce54bef] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 17:58:02.584 INFO 2981023 --- [ocp-schedules-14,8c75c352932a65c4,6ded8c1d9ce54bef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:58:02.594 INFO 2981023 --- [ocp-schedules-14,8c75c352932a65c4,6ded8c1d9ce54bef] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 17:58:02.714 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:02.714 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:03.333 INFO 2981023 --- [ocp-async-7,ef6b964d5d56c3a3,ec1743876368ec76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:04.676 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ebf6a7a38728ea81,d35b63ce8b5f3730] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hosts/2000010/checkSystemInfo, client 10.31.33.41, traceId ebf6a7a38728ea81] 2024-09-11 17:58:04.678 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ebf6a7a38728ea81,d35b63ce8b5f3730] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hosts/2000010/checkSystemInfo?id=2000010, client=10.31.33.41, traceId=ebf6a7a38728ea81, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.checkSystemInfo(Long), args=2000010,] 2024-09-11 17:58:04.679 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ebf6a7a38728ea81,d35b63ce8b5f3730] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=check system info, nodeCount=1, taskCount=1 2024-09-11 17:58:04.679 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ebf6a7a38728ea81,d35b63ce8b5f3730] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:58:04.679 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ebf6a7a38728ea81,d35b63ce8b5f3730] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='check system info', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=null, tenantId=null, hostId=2000010, creator='admin', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false, system_checker_operation=CHECK, host_id=2000010}, listMap={}}, createTime=null, startTime=2024-09-11T17:58:04.679+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Check system task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.CheckSystemInfoTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ebf6a7a38728ea81', operation=EXECUTE} 2024-09-11 17:58:04.688 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ebf6a7a38728ea81,d35b63ce8b5f3730] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=4048201, taskName=check system info, argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false, system_checker_operation=CHECK, host_id=2000010}, listMap={}} 2024-09-11 17:58:04.693 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ebf6a7a38728ea81,d35b63ce8b5f3730] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hosts/2000010/checkSystemInfo client=10.31.33.41, traceId=ebf6a7a38728ea81, duration=19 ms] 2024-09-11 17:58:05.063 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,a43b3184ab80a08a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048195, operation:EXECUTE, state: RUNNING 2024-09-11 17:58:05.082 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048195, context=Context{parallelIdx=-1, stringMap={system_checker_operation=CHECK, task_instance_id=4048201, task_operation=execute, latest_execution_start_time=2024-09-11T17:58:05.061+08:00, sub_task_instance_name=Check system task, sub_task_instance_id=4048195, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:58:05.186 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Clock Offset] will apply to target 10.31.35.7 2024-09-11 17:58:05.192 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [libaio DLL] will apply to target 10.31.35.7 2024-09-11 17:58:05.196 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [crond Service] will apply to target 10.31.35.7 2024-09-11 17:58:05.200 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [sshd Service] will apply to target 10.31.35.7 2024-09-11 17:58:05.203 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [firewalld Service] will apply to target 10.31.35.7 2024-09-11 17:58:05.207 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Transparent Huge Pages] will apply to target 10.31.35.7 2024-09-11 17:58:05.211 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Python Version] will apply to target 10.31.35.7 2024-09-11 17:58:05.214 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Swap Information] will apply to target 10.31.35.7 2024-09-11 17:58:05.218 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.conf.default.rp_filter] will apply to target 10.31.35.7 2024-09-11 17:58:05.222 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.conf.default.accept_source_route] will apply to target 10.31.35.7 2024-09-11 17:58:05.226 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_syncookies] will apply to target 10.31.35.7 2024-09-11 17:58:05.230 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.swappiness] will apply to target 10.31.35.7 2024-09-11 17:58:05.234 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_tw_reuse] will apply to target 10.31.35.7 2024-09-11 17:58:05.238 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_slow_start_after_idle] will apply to target 10.31.35.7 2024-09-11 17:58:05.243 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.overcommit_memory] will apply to target 10.31.35.7 2024-09-11 17:58:05.247 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.nr_hugepages] will apply to target 10.31.35.7 2024-09-11 17:58:05.251 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.somaxconn] will apply to target 10.31.35.7 2024-09-11 17:58:05.255 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.netdev_max_backlog] will apply to target 10.31.35.7 2024-09-11 17:58:05.259 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.rmem_default] will apply to target 10.31.35.7 2024-09-11 17:58:05.262 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.wmem_default] will apply to target 10.31.35.7 2024-09-11 17:58:05.273 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.rmem_max] will apply to target 10.31.35.7 2024-09-11 17:58:05.277 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.core.wmem_max] will apply to target 10.31.35.7 2024-09-11 17:58:05.280 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ min of net.ipv4.tcp_rmem] will apply to target 10.31.35.7 2024-09-11 17:58:05.284 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ default of net.ipv4.tcp_rmem] will apply to target 10.31.35.7 2024-09-11 17:58:05.287 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ max of net.ipv4.tcp_rmem] will apply to target 10.31.35.7 2024-09-11 17:58:05.291 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ min of net.ipv4.tcp_wmem] will apply to target 10.31.35.7 2024-09-11 17:58:05.294 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ default of net.ipv4.tcp_wmem] will apply to target 10.31.35.7 2024-09-11 17:58:05.298 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ max of net.ipv4.tcp_wmem] will apply to target 10.31.35.7 2024-09-11 17:58:05.301 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_max_syn_backlog] will apply to target 10.31.35.7 2024-09-11 17:58:05.305 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [net.ipv4.tcp_fin_timeout] will apply to target 10.31.35.7 2024-09-11 17:58:05.308 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.min_free_kbytes] will apply to target 10.31.35.7 2024-09-11 17:58:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,084c004556ac060e,239476ca03de4442] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,084c004556ac060e,239476ca03de4442] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:05.312 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.max_map_count] will apply to target 10.31.35.7 2024-09-11 17:58:05.316 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [fs.aio-max-nr] will apply to target 10.31.35.7 2024-09-11 17:58:05.326 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [kernel.numa_balancing] will apply to target 10.31.35.7 2024-09-11 17:58:05.330 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [vm.zone_reclaim_mode] will apply to target 10.31.35.7 2024-09-11 17:58:05.333 INFO 2981023 --- [ocp-async-1,20a901668d7285a6,a0e5f0bd770c8df7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:05.333 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [fs.file-max] will apply to target 10.31.35.7 2024-09-11 17:58:05.337 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [fs.pipe-user-pages-soft] will apply to target 10.31.35.7 2024-09-11 17:58:05.343 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [mtr Command] will apply to target 10.31.35.7 2024-09-11 17:58:05.348 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [tar Command] will apply to target 10.31.35.7 2024-09-11 17:58:05.353 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [curl Command] will apply to target 10.31.35.7 2024-09-11 17:58:05.357 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [nc Command] will apply to target 10.31.35.7 2024-09-11 17:58:05.360 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [binutils Toolkit] will apply to target 10.31.35.7 2024-09-11 17:58:05.364 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [bind-utils Toolkit] will apply to target 10.31.35.7 2024-09-11 17:58:05.367 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [Network Management Toolkit] will apply to target 10.31.35.7 2024-09-11 17:58:05.372 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [libatomic DLL] will apply to target 10.31.35.7 2024-09-11 17:58:05.375 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [os user(ocp.operation.default.os.user in System Parameters) existence] will apply to target 10.31.35.7 2024-09-11 17:58:05.382 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of openFiles of root user] will apply to target 10.31.35.7 2024-09-11 17:58:05.385 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of openFiles of os user(ocp.operation.default.os.user in System Parameters)] will apply to target 10.31.35.7 2024-09-11 17:58:05.389 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of stackSize of os user(ocp.operation.default.os.user in System Parameters)] will apply to target 10.31.35.7 2024-09-11 17:58:05.393 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of maxUserProcesses of os user(ocp.operation.default.os.user in System Parameters)] will apply to target 10.31.35.7 2024-09-11 17:58:05.397 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [limit of coreFileSize of os user(ocp.operation.default.os.user in System Parameters)] will apply to target 10.31.35.7 2024-09-11 17:58:05.400 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [SELinux config] will apply to target 10.31.35.7 2024-09-11 17:58:05.404 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [File Type of OceanBase Data Disk] will apply to target 10.31.35.7 2024-09-11 17:58:05.407 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [OS version check] will apply to target 10.31.35.7 2024-09-11 17:58:05.411 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [fio Command] will apply to target 10.31.35.7 2024-09-11 17:58:05.414 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.i.i.InspectionItemManagerImpl : Inspection item [ clock source checks] will apply to target 10.31.35.7 2024-09-11 17:58:05.443 INFO 2981023 --- [inspection_task6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:05.448 INFO 2981023 --- [inspection_task6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:05.600 INFO 2981023 --- [inspection_task6,,] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:58:05.605 INFO 2981023 --- [inspection_task10,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:05.606 INFO 2981023 --- [inspection_task10,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:05.715 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:05.715 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:05.744 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/transparentHugePageInfo, request body:null, params:null 2024-09-11 17:58:05.744 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/swapInfo, request body:null, params:null 2024-09-11 17:58:05.744 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/pythonVersion, request body:null, params:null 2024-09-11 17:58:05.745 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/service/info, request body:[object Object], params:null 2024-09-11 17:58:05.745 INFO 2981023 --- [inspection_task4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.745 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.745 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/service/info, request body:[object Object], params:null 2024-09-11 17:58:05.745 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/libExists, request body:[object Object], params:null 2024-09-11 17:58:05.746 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/service/info, request body:[object Object], params:null 2024-09-11 17:58:05.758 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.765 INFO 2981023 --- [inspection_task4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.773 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.781 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.802 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.818 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.831 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.842 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.853 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.862 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.871 INFO 2981023 --- [inspection_task4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.878 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.888 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.899 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.906 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.912 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.921 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.929 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.937 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.949 INFO 2981023 --- [inspection_task4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.956 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.963 INFO 2981023 --- [inspection_task4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:58:05.969 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:58:05.971 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 17:58:05.977 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.985 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 17:58:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 17:58:05.991 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=RUNNING} 2024-09-11 17:58:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 17:58:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkrtd:1719994443-1) 2024-09-11 17:58:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 17:58:06.004 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:06.005 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:06.006 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:06.006 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:58:06.007 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:06.007 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:58:06.009 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 17:58:06.010 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:06.011 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 17:58:06.013 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/kernelParam, request body:[object Object], params:null 2024-09-11 17:58:06.025 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:58:06.035 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:58:06.040 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=0->1, c=0) - source=2024-09-11T17:55:40, start=2024-09-11T17:52:40, end=2024-09-11T17:55:10, updates=112, granule=120000ms, delay=180000ms 2024-09-11 17:58:06.044 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:58:06.047 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:55:40, target=2024-09-11T17:55:10, granule=120000ms, delay=30000ms) 2024-09-11 17:58:06.063 INFO 2981023 --- [inspection_task4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:58:06.085 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:58:06.088 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=linkrtd:1719994443, server=1, level=1->2, c=0) - source=2024-09-11T17:55:10, start=2024-09-11T17:39:31, end=2024-09-11T17:49:31, updates=135, granule=600000ms, delay=939000ms 2024-09-11 17:58:06.099 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=linkrtd:1719994443, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:55:10, target=2024-09-11T17:49:31, granule=600000ms, delay=339000ms) 2024-09-11 17:58:06.102 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:58:06.112 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:58:06.120 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/libExists, request body:[object Object], params:null 2024-09-11 17:58:06.131 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/userExists, request body:[object Object], params:null 2024-09-11 17:58:06.170 INFO 2981023 --- [inspection_task10,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:58:06.179 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:55:41, start=2024-09-11T17:52:41, end=2024-09-11T17:55:11, updates=299, granule=120000ms, delay=180000ms 2024-09-11 17:58:06.179 INFO 2981023 --- [inspection_task8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:58:06.185 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:55:41, target=2024-09-11T17:55:11, granule=120000ms, delay=30000ms) 2024-09-11 17:58:06.188 INFO 2981023 --- [inspection_task1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:58:06.196 INFO 2981023 --- [inspection_task4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:58:06.207 INFO 2981023 --- [inspection_task7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/limitInfo, request body:[object Object], params:null 2024-09-11 17:58:06.215 INFO 2981023 --- [inspection_task9,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/selinuxInfo, request body:null, params:null 2024-09-11 17:58:06.220 INFO 2981023 --- [inspection_task2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/disk/batchInfos, request body:null, params:null 2024-09-11 17:58:06.226 INFO 2981023 --- [inspection_task5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/hostInfo, request body:null, params:null 2024-09-11 17:58:06.243 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:[object Object], params:null 2024-09-11 17:58:06.250 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/clockSourceInfo, request body:null, params:null 2024-09-11 17:58:06.253 INFO 2981023 --- [inspection_task3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/hostInfo, request body:null, params:null 2024-09-11 17:58:06.284 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T17:55:11, start=2024-09-11T17:40:51, end=2024-09-11T17:50:51, updates=432, granule=600000ms, delay=860000ms 2024-09-11 17:58:06.288 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T17:55:11, target=2024-09-11T17:50:51, granule=600000ms, delay=260000ms) 2024-09-11 17:58:06.288 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=317ms, period=120000ms 2024-09-11 17:58:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:58:06.336242+08:00, lastProcessTime=2024-09-11T17:57:56.352506+08:00 2024-09-11 17:58:06.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:58:06.336242+08:00 2024-09-11 17:58:07.333 INFO 2981023 --- [ocp-async-10,1cc5db83ac087bda,583d65ad43bd1185] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:07.495 INFO 2981023 --- [ocp-schedules-26,f04795dabba63dce,1cea6ce1a120152a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:58:07.560 WARN 2981023 --- [scanner-13580007-1,f04795dabba63dce,15a179e660d75d29] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:58:07.577 INFO 2981023 --- [alarm-detect-10,f04795dabba63dce,79e92146e6b4ee01] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645076, data is empty. 2024-09-11 17:58:07.580 INFO 2981023 --- [alarm-detect-10,f04795dabba63dce,79e92146e6b4ee01] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645094, data is empty. 2024-09-11 17:58:07.583 INFO 2981023 --- [alarm-detect-10,f04795dabba63dce,79e92146e6b4ee01] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645094, data is empty. 2024-09-11 17:58:07.609 INFO 2981023 --- [alarm-detect-3,f04795dabba63dce,e5eccc7bb3d17728] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645076, data is empty. 2024-09-11 17:58:07.611 INFO 2981023 --- [alarm-detect-16,f04795dabba63dce,6636d0e3f4ef6cfd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645076, data is empty. 2024-09-11 17:58:07.616 INFO 2981023 --- [alarm-detect-3,f04795dabba63dce,e5eccc7bb3d17728] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645094, data is empty. 2024-09-11 17:58:07.618 INFO 2981023 --- [alarm-detect-16,f04795dabba63dce,6636d0e3f4ef6cfd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645094, data is empty. 2024-09-11 17:58:07.621 INFO 2981023 --- [alarm-detect-3,f04795dabba63dce,e5eccc7bb3d17728] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645094, data is empty. 2024-09-11 17:58:07.627 INFO 2981023 --- [alarm-detect-16,f04795dabba63dce,6636d0e3f4ef6cfd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645094, data is empty. 2024-09-11 17:58:07.655 INFO 2981023 --- [ocp-schedules-17,ea5d6da56abb1418,72aa1ec59a5ca1c1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:58:07.676 INFO 2981023 --- [alarm-detect-16,f04795dabba63dce,40f95debf686a1a0] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726048650 ~ 1726048677 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726048665, value=1.582)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.582, values={1726048665=1.582}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:58:07.676 INFO 2981023 --- [alarm-detect-16,f04795dabba63dce,40f95debf686a1a0] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.10-3, currentState=Pending 2024-09-11 17:58:07.728 INFO 2981023 --- [ocp-schedules-10,3d3fc9e906d7d829,c2bebe31ec5a3730] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:58:07.728+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:58:07.729 INFO 2981023 --- [ocp-schedules-10,3d3fc9e906d7d829,c2bebe31ec5a3730] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:58:07.732 INFO 2981023 --- [ocp-schedules-10,3d3fc9e906d7d829,c2bebe31ec5a3730] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:58:07.732 INFO 2981023 --- [ocp-schedules-10,3d3fc9e906d7d829,c2bebe31ec5a3730] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:58:07.754 INFO 2981023 --- [alarm-detect-7,f04795dabba63dce,727e2a4a27d7edf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645076, data is empty. 2024-09-11 17:58:07.757 INFO 2981023 --- [alarm-detect-7,f04795dabba63dce,727e2a4a27d7edf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645094, data is empty. 2024-09-11 17:58:07.764 INFO 2981023 --- [alarm-detect-7,f04795dabba63dce,727e2a4a27d7edf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645094, data is empty. 2024-09-11 17:58:07.769 INFO 2981023 --- [ocp-async-5,008ea53969c77f9d,8b4a9ee74ed40882] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:58:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,5f60f58597d70d04,2cb09ac2f86ce3a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,5f60f58597d70d04,2cb09ac2f86ce3a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:08.715 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:08.717 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:09.333 INFO 2981023 --- [ocp-async-8,ba9d4cbbe52177a9,0ce21eb09a425c30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:10.003 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:58:10.003 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:58:10.918 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:58:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:58:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,2e87a85616f0b18b,c7c15adfea0d2614] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,2e87a85616f0b18b,c7c15adfea0d2614] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:11.334 INFO 2981023 --- [ocp-async-5,9f2c297d8331033c,d95dd749c725a2c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:11.568 INFO 2981023 --- [inspection_task6,,] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048691 0 0 , error= 2024-09-11 17:58:11.717 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:11.718 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:11.930 INFO 2981023 --- [manual-subtask-executor16,ebf6a7a38728ea81,7e981d86345f3cdd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048195, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:58:12.124 INFO 2981023 --- [task-coordinator2,,] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007107 2024-09-11 17:58:12.135 INFO 2981023 --- [task-coordinator2,,] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"activeAt":"2024-09-11T09:58:12.119Z","alarmType":"host_operation","appType":"OCP","id":6007107,"labels":{"alarm_name":"伺服器運維操作提醒","alarm_status_zh_tw":"警示中","app_type":"OCP","svr_ip":"10.31.35.7","alarm_status_native":"Active","task_id":"4048201","alarm_level":"5","alarm_level_zh_tw":"提醒","operator":"admin","alarm_summary_zh_tw":"alarm_template_id=0:host=10.31.35.7:operation_name=check system info SUCCESS","alarm_level_native":"5","alarm_scope_en_us":"Host","alarm_summary_en_us":"alarm_template_id=0:host=10.31.35.7:operation_name=check system info SUCCESS","alarm_level_en_us":"Info","alarm_type":"host_operation","auto_resolved":"1","alarm_status":"Active","alarm_group_zh_tw":"提示","host":"10.31.35.7","alarm_scope":"Host","alarm_active_at":"2024-09-11T17:58:12.119+08:00","task_result":"success","alarm_target":"alarm_template_id=0:host=10.31.35.7:operation_name=check system info","alarm_scope_native":"Host","alarm_description_zh_tw":"伺服器：10.31.35.7，警示：check system info SUCCESS, 操作者 admin，運行結果：success。","alarm_name_zh_tw":"伺服器運維操作提醒","app":"OCP","host_ip":"10.31.35.7","alarm_name_en_us":"Host operation info","alarm_name_zh_cn":"主机运维操作提醒","operation_result":"SUCCESS","alarm_status_en_us":"Active","message_type":"Alarm","alarm_status_zh_cn":"告警中","alarm_scope_zh_tw":"主機","alarm_template_id":"0","alarm_group_en_us":"info","operation_name":"check system info","alarm_description_en_us":"Host: 10.31.35.7, alert: check system info SUCCESS, operator was admin, result was success.","__host.id":"2000010","alarm_scope_zh_cn":"主机","alarm_level_zh_cn":"提醒","alarm_description_zh_cn":"服务器：10.31.35.7，告警：check system info SUCCESS, 操作者 admin，运行结果：success。","alarm_summary_zh_cn":"alarm_template_id=0:host=10.31.35.7:operation_name=check system info SUCCESS","alarm_updated_at":"2024-09-11T17:58:12.119+08:00","ocp_ip":"10.31.35.10","host_name":"vm-eas-prd-rtd-mid05","alarm_group_zh_cn":"提示"},"level":5,"scope":"Host","status":"Active","statusUpdatedAt":"2024-09-11T09:58:12.122Z","target":"alarm_template_id=0 2024-09-11 17:58:12.136 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048201, status:SUCCESS 2024-09-11 17:58:12.139 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048201, status:SUCCESS 2024-09-11 17:58:12.344 INFO 2981023 --- [ocp-schedules-19,468487ffd3f626a6,17ea6f1a1a33018d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:58:13.083 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048620, nodeCount=1000, affectRows=1000 2024-09-11 17:58:13.086 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048620, nodeCount=999, validCount=999 2024-09-11 17:58:13.089 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048620, nodeCount=1000, affectRows=1000 2024-09-11 17:58:13.091 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048620, nodeCount=1000, validCount=1000 2024-09-11 17:58:13.094 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048620, nodeCount=49, affectRows=49 2024-09-11 17:58:13.094 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048620, nodeCount=49, validCount=49 2024-09-11 17:58:13.334 INFO 2981023 --- [ocp-async-2,f6c9688c41b450c4,8d915639508c945f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,28b8d46f58e00f48,f2ffaf7184c15c0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,95e1f169d5fdb9a6,c982e4ee1612dc4d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:58:14.318 INFO 2981023 --- [prometheus-minute-schedule-1,28b8d46f58e00f48,f2ffaf7184c15c0b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:14.540 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048620, nodeCount=480, affectRows=480 2024-09-11 17:58:14.718 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:14.720 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:14.763 INFO 2981023 --- [ocp-schedules-21,bf5394abdaeff7ca,83cd10173dbd6e96] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:58:14.772 INFO 2981023 --- [ocp-schedules-21,bf5394abdaeff7ca,83cd10173dbd6e96] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:58:14.772 INFO 2981023 --- [ocp-schedules-21,bf5394abdaeff7ca,83cd10173dbd6e96] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:58:15.334 INFO 2981023 --- [ocp-async-6,038f650d3a43e9d4,c74c706bea58c0e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:16.021 INFO 2981023 --- [ocp-schedules-17,a2766454c3e1bb52,c156f31b215cd7e4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:58:16.024 INFO 2981023 --- [host-check-clockdiff1,a2766454c3e1bb52,97682be8bfba1127] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:58:16.024 INFO 2981023 --- [ocp-schedules-19,fb919fc56603ff6e,73f33e7046141e42] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:58:16.024 INFO 2981023 --- [host-check-clockdiff2,a2766454c3e1bb52,6faebbd0589901fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:58:16.026 INFO 2981023 --- [host-check-clockdiff3,a2766454c3e1bb52,087e86d0b48ff078] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:58:16.026 INFO 2981023 --- [host-check-clockdiff4,a2766454c3e1bb52,907ae97ea79d860e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:58:16.028 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:58:16.028 INFO 2981023 --- [ocp-schedules-29,b85ea4f7e407ca61,8271e0af33834105] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:58:16.031 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:58:16.032 INFO 2981023 --- [host-check-agent-status1,b85ea4f7e407ca61,6fb48839b761218d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:58:16.033 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:58:16.034 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:58:16.037 INFO 2981023 --- [host-check-clockdiff2,a2766454c3e1bb52,6faebbd0589901fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048696 5 5 , error= 2024-09-11 17:58:16.037 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:58:16.038 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:58:16.040 INFO 2981023 --- [host-check-agent-status2,b85ea4f7e407ca61,e1090c052ab6f51e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:58:16.040 INFO 2981023 --- [host-check-agent-status3,b85ea4f7e407ca61,9b6a2d29a62d73a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:58:16.042 INFO 2981023 --- [host-check-agent-status1,b85ea4f7e407ca61,6fb48839b761218d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:58:16.043 INFO 2981023 --- [host-check-agent-status4,b85ea4f7e407ca61,376eb96ba5d0beb9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:58:16.047 INFO 2981023 --- [host-check-agent-status1,b85ea4f7e407ca61,6fb48839b761218d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:58:16.047 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:58:16.048 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:16.049 INFO 2981023 --- [host-check-clockdiff3,a2766454c3e1bb52,087e86d0b48ff078] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048696 0 0 , error= 2024-09-11 17:58:16.049 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:16.049 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:58:16.069 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:58:16.168 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:58:16.187 INFO 2981023 --- [ocp-schedules-6,d4c552f96e3c7997,872b08b8340cedd0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:58:16.195 INFO 2981023 --- [ocp-schedules-21,dbe5888757b20483,fea7df39ae02450a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:58:16.196 INFO 2981023 --- [ocp-schedules-3,21ab84c687f6e0ba,4c2f659f04e1d191] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:58:16.196 INFO 2981023 --- [ocp-schedules-22,2d9b25f75127ad93,93835c5fcb9d110d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:58:16.204 INFO 2981023 --- [ocp-schedules-21,dbe5888757b20483,fea7df39ae02450a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:58:16.205 INFO 2981023 --- [host-check-agent-status3,b85ea4f7e407ca61,9b6a2d29a62d73a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:58:16.207 INFO 2981023 --- [host-check-agent-status3,b85ea4f7e407ca61,9b6a2d29a62d73a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:58:16.208 INFO 2981023 --- [ocp-schedules-6,d4c552f96e3c7997,872b08b8340cedd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:58:16.208 INFO 2981023 --- [host-check-agent-status2,b85ea4f7e407ca61,e1090c052ab6f51e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:58:16.208 INFO 2981023 --- [ocp-schedules-21,dbe5888757b20483,fea7df39ae02450a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:58:16.219 INFO 2981023 --- [ocp-schedules-3,21ab84c687f6e0ba,4c2f659f04e1d191] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:58:16.219 INFO 2981023 --- [host-check-agent-status2,b85ea4f7e407ca61,e1090c052ab6f51e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:58:16.223 INFO 2981023 --- [host-check-agent-status4,b85ea4f7e407ca61,376eb96ba5d0beb9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:16.261 INFO 2981023 --- [host-check-agent-status4,b85ea4f7e407ca61,376eb96ba5d0beb9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:16.270 INFO 2981023 --- [ocp-schedules-20,d7287ccca4b14d6f,19a9ae404b76603c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:58:16.319 INFO 2981023 --- [host-check-agent-status1,b85ea4f7e407ca61,6fb48839b761218d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:58:16.324 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:58:16.324 INFO 2981023 --- [host-check-agent-status1,b85ea4f7e407ca61,6fb48839b761218d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:58:16.324 INFO 2981023 --- [host-check-agent-status1,b85ea4f7e407ca61,6fb48839b761218d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:58:16.324 INFO 2981023 --- [host-check-agent-status1,b85ea4f7e407ca61,6fb48839b761218d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:58:16.326 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:58:16.331204306+08:00 2024-09-11 17:58:16.358 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:58:16.326+08:00 2024-09-11 17:58:16.361 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:58:16.326+08:00 and status AVAILABLE 2024-09-11 17:58:16.368 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:58:16.368 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:58:16.374122356+08:00 2024-09-11 17:58:16.368 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:58:16.370 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:58:16.368+08:00 2024-09-11 17:58:16.370 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:58:16.374 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T17:58:16.371170528+08:00 2024-09-11 17:58:16.375 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:58:16.368+08:00 and status AVAILABLE 2024-09-11 17:58:16.376 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T17:58:16.374+08:00 2024-09-11 17:58:16.377 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:58:16.378 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T17:58:16.374+08:00 and status AVAILABLE 2024-09-11 17:58:16.379 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:58:16.378865329+08:00 2024-09-11 17:58:16.381 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:58:16.379+08:00 2024-09-11 17:58:16.383 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:58:16.379+08:00 and status AVAILABLE 2024-09-11 17:58:16.390 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.391 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:16.396 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:58:16.397 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:58:16.396753+08:00, lastProcessTime=2024-09-11T17:58:06.336242+08:00 2024-09-11 17:58:16.400 INFO 2981023 --- [host-check-agent-status3,b85ea4f7e407ca61,9b6a2d29a62d73a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:58:16.403 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:58:16.403 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 17:58:16.404 INFO 2981023 --- [host-check-agent-status2,b85ea4f7e407ca61,e1090c052ab6f51e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:58:16.404 INFO 2981023 --- [host-check-agent-status4,b85ea4f7e407ca61,376eb96ba5d0beb9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:58:16.407 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:58:16.408 INFO 2981023 --- [host-check-agent-status4,b85ea4f7e407ca61,376eb96ba5d0beb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:58:16.408 INFO 2981023 --- [host-check-agent-status4,b85ea4f7e407ca61,376eb96ba5d0beb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:58:16.408 INFO 2981023 --- [host-check-agent-status4,b85ea4f7e407ca61,376eb96ba5d0beb9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:58:16.410 INFO 2981023 --- [host-check-agent-status3,b85ea4f7e407ca61,9b6a2d29a62d73a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:58:16.410 INFO 2981023 --- [host-check-agent-status3,b85ea4f7e407ca61,9b6a2d29a62d73a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:58:16.410 INFO 2981023 --- [host-check-agent-status3,b85ea4f7e407ca61,9b6a2d29a62d73a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:58:16.410 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:58:16.411 INFO 2981023 --- [host-check-agent-status2,b85ea4f7e407ca61,e1090c052ab6f51e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:58:16.411 INFO 2981023 --- [host-check-agent-status2,b85ea4f7e407ca61,e1090c052ab6f51e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:58:16.411 INFO 2981023 --- [host-check-agent-status2,b85ea4f7e407ca61,e1090c052ab6f51e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:58:16.460 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:58:16.461 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:58:16.462 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:16.462 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.463 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:16.463 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:16.463 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.464 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:16.464 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:58:16.464 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:58:16.466 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:58:16.467 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:58:16.467 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.468 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:16.470 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:58:16.470 INFO 2981023 --- [ocp-schedules-6,d4c552f96e3c7997,872b08b8340cedd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:58:16.476 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.476 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:16.478 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.478 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:16.487 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:58:16.487 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:58:16.490 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:58:16.492 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:58:16.511 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 17:58:16.515 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:58:16.515 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:58:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.519 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:58:16.524 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:58:16.528 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 17:58:16.528 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:58:16.534 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:58:16.548 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:58:16.578 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:58:16.579 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:58:16.580 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.581 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:58:16.582 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.582 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:58:16.583 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:58:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:16.586 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:58:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:58:16.590 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:58:16.599 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 17:58:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:58:16.606 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:58:16.610 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048620, nodeCount=480, validCount=480 2024-09-11 17:58:16.611 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:58:16.613 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:58:16.613 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 17:58:16.616 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.618 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.618 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:58:16.620 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:58:16.625 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:58:16.625 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:58:16.625 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:16.626 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.627 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.628 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:16.630 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.634 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:58:16.635 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:58:16.638 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.639 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.639 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:16.639 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:58:16.642 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:58:16.654 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:58:16.659 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:16.660 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:58:16.660 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:58:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:58:16.661 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:58:16.662 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:58:16.663 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 17:58:16.663 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:58:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:16.667 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:58:16.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.670 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:58:16.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:58:16.673 INFO 2981023 --- [host-check-host-status4,fb919fc56603ff6e,7483da734252e291] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 17:58:16.679 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:58:16.685 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:58:16.691 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:58:16.693 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.693 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.694 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:58:16.694 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:16.695 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:58:16.697 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:58:16.703 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:58:16.705 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:58:16.708 INFO 2981023 --- [host-check-host-status2,fb919fc56603ff6e,ee35e68229d30117] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:58:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:16.713 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:58:16.714 INFO 2981023 --- [host-check-host-status1,fb919fc56603ff6e,246202df63cfea4b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:58:16.724 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.725 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.725 INFO 2981023 --- [host-check-host-status3,fb919fc56603ff6e,c2cc918f8bce5f08] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:58:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:16.727 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:58:16.731 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:58:16.738 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:58:16.740 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:58:16.741 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:58:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:58:16.773 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:58:16.779 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.779 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.780 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:58:16.785 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:58:16.785 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:58:16.789 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:58:16.791 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:58:16.795 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.795 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.795 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:58:16.396753+08:00 2024-09-11 17:58:16.796 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:58:16.798 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:58:16.801 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:58:16.801 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:58:16.803 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:16.804 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:16.804 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:58:16.810 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:58:16.810 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:58:16.810 INFO 2981023 --- [ocp-schedules-3,21ab84c687f6e0ba,4c2f659f04e1d191] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:58:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,82c90363d0d8c255,fd9311149ce45eb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,82c90363d0d8c255,fd9311149ce45eb3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:17.334 INFO 2981023 --- [ocp-async-8,206c389432984502,97f09cabb06e943d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:17.720 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:17.721 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:17.769 INFO 2981023 --- [ocp-async-6,c1d3c2281437da8d,a4a2d3b7f213342a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:58:17.808 INFO 2981023 --- [ocp-schedules-18,2c4548ae67d8d150,a5853fdb50f03608] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:58:17.822 INFO 2981023 --- [host-check-clockdiff1,a2766454c3e1bb52,97682be8bfba1127] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048697 6 6 , error= 2024-09-11 17:58:17.864 WARN 2981023 --- [scanner-13581806-1,2c4548ae67d8d150,bcc449c383478657] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:58:19.334 INFO 2981023 --- [ocp-async-9,660eac8ca3b828b9,bc11fc9c6297fc25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,95b19adcf48dfcb9,4e4791ea9dfdd669] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,95b19adcf48dfcb9,4e4791ea9dfdd669] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:20.721 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:20.722 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:21.334 INFO 2981023 --- [ocp-async-3,5338cb34c578a129,3ed8ec55d3e5e0c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:22.014 INFO 2981023 --- [host-check-clockdiff4,a2766454c3e1bb52,907ae97ea79d860e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048702 0 0 , error= 2024-09-11 17:58:22.346 INFO 2981023 --- [ocp-schedules-11,24ababf3ee9e2a21,4b4680f82bddf9ac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:58:22.827 INFO 2981023 --- [ocp-schedules-30,4cb4d7d4c85d2e8b,bcba850715dd772f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 17:58:22.827 INFO 2981023 --- [ocp-schedules-30,4cb4d7d4c85d2e8b,bcba850715dd772f] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 17:58:22.842 INFO 2981023 --- [pool-555-thread-1,4cb4d7d4c85d2e8b,8d763087efa90ee8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:58:22.842 INFO 2981023 --- [pool-555-thread-2,4cb4d7d4c85d2e8b,ce0e81abdfc712ca] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:58:22.842 INFO 2981023 --- [pool-555-thread-3,4cb4d7d4c85d2e8b,54c409e39e652dd4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 17:58:22.860 INFO 2981023 --- [pool-555-thread-3,4cb4d7d4c85d2e8b,54c409e39e652dd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:22.861 INFO 2981023 --- [pool-555-thread-2,4cb4d7d4c85d2e8b,ce0e81abdfc712ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:22.861 INFO 2981023 --- [pool-555-thread-1,4cb4d7d4c85d2e8b,8d763087efa90ee8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:22.863 INFO 2981023 --- [pool-555-thread-3,4cb4d7d4c85d2e8b,54c409e39e652dd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726048102000000, 1726048702000000] 2024-09-11 17:58:22.864 INFO 2981023 --- [pool-555-thread-1,4cb4d7d4c85d2e8b,8d763087efa90ee8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726048102000000, 1726048702000000] 2024-09-11 17:58:22.864 INFO 2981023 --- [pool-555-thread-2,4cb4d7d4c85d2e8b,ce0e81abdfc712ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726048102000000, 1726048702000000] 2024-09-11 17:58:22.868 INFO 2981023 --- [pool-555-thread-3,4cb4d7d4c85d2e8b,54c409e39e652dd4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:58:22.937 INFO 2981023 --- [pool-555-thread-1,4cb4d7d4c85d2e8b,8d763087efa90ee8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:58:23.006 INFO 2981023 --- [pool-555-thread-2,4cb4d7d4c85d2e8b,ce0e81abdfc712ca] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 17:58:23.007 INFO 2981023 --- [ocp-schedules-30,4cb4d7d4c85d2e8b,bcba850715dd772f] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 17:58:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,44f2bb9f3092a195,05e858a495cfa68e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,44f2bb9f3092a195,05e858a495cfa68e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:23.334 INFO 2981023 --- [ocp-async-8,f7a84ad1ee8f3c6d,9a1dff9245fe5a1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:23.722 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:23.722 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:24.024 INFO 2981023 --- [ocp-schedules-9,32c39ebc27bbc59b,e55c6f9932ee8849] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:58:24.024 INFO 2981023 --- [ocp-schedules-9,32c39ebc27bbc59b,e55c6f9932ee8849] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:58:24.024 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:58:24.024 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:58:24.024 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 17:58:24.028 INFO 2981023 --- [ocp-schedules-9,32c39ebc27bbc59b,e55c6f9932ee8849] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:58:24.029 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:58:24.030 INFO 2981023 --- [ocp-schedules-9,32c39ebc27bbc59b,e55c6f9932ee8849] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:58:24.030 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:58:24.030 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:24.031 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:24.032 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:58:24.033 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:58:24.050 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:58:24.050 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:58:24.051 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:58:24.059 INFO 2981023 --- [ocp-schedules-9,32c39ebc27bbc59b,e55c6f9932ee8849] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:58:24.061 INFO 2981023 --- [ocp-schedules-9,32c39ebc27bbc59b,e55c6f9932ee8849] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:58:24.343 INFO 2981023 --- [ocp-schedules-14,a485bfc320bd9928,449699af50bf94ee] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:58:25.334 INFO 2981023 --- [ocp-async-5,06ac2f34b0e3ab7e,fefe361ef0e5d8f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,d05358dc1b9726be,aea31dc510e7d430] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,d05358dc1b9726be,aea31dc510e7d430] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:58:26.338004+08:00, lastProcessTime=2024-09-11T17:58:16.396753+08:00 2024-09-11 17:58:26.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:58:26.338004+08:00 2024-09-11 17:58:26.722 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:26.723 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:27.335 INFO 2981023 --- [ocp-async-10,1d30a920b4a4e797,67d58ccd9a733554] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:27.668 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.671 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obCl 2024-09-11 17:58:27.675 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.676 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:58:27.679 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.687 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.695 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.700 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.704 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 29 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.752 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 9 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.760 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 152 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.769 INFO 2981023 --- [ocp-async-2,0c0972f848bc3dfa,d91f36099345edc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:58:27.791 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.797 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.797 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:58:27.799 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.803 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 4 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.824 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.825 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.826 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.827 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.828 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.828 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.829 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.829 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 17:58:27.830 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.831 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.831 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.832 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.833 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.833 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.834 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.835 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:52:27.477, end=2024-09-11T17:57:27.661} 2024-09-11 17:58:27.835 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 4 dubious sqls 2024-09-11 17:58:28.048 INFO 2981023 --- [ocp-schedules-28,81e430e4e8b722f8,5e3f785fe1ff6407] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:58:28.088 WARN 2981023 --- [scanner-13581995-1,81e430e4e8b722f8,5d71c0b5456dc29c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:58:28.117 INFO 2981023 --- [alarm-detect-12,81e430e4e8b722f8,ef524ce2bb4edaf0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645094, data is empty. 2024-09-11 17:58:28.121 INFO 2981023 --- [alarm-detect-12,81e430e4e8b722f8,ef524ce2bb4edaf0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645094, data is empty. 2024-09-11 17:58:28.125 INFO 2981023 --- [alarm-detect-12,81e430e4e8b722f8,ef524ce2bb4edaf0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645094, data is empty. 2024-09-11 17:58:28.145 INFO 2981023 --- [alarm-detect-9,81e430e4e8b722f8,a3329127eb4cb9c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645094, data is empty. 2024-09-11 17:58:28.146 INFO 2981023 --- [alarm-detect-14,81e430e4e8b722f8,542d8ad6f6af2084] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645094, data is empty. 2024-09-11 17:58:28.154 INFO 2981023 --- [alarm-detect-14,81e430e4e8b722f8,542d8ad6f6af2084] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645094, data is empty. 2024-09-11 17:58:28.155 INFO 2981023 --- [alarm-detect-9,81e430e4e8b722f8,a3329127eb4cb9c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645094, data is empty. 2024-09-11 17:58:28.158 INFO 2981023 --- [alarm-detect-9,81e430e4e8b722f8,a3329127eb4cb9c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645094, data is empty. 2024-09-11 17:58:28.163 INFO 2981023 --- [alarm-detect-14,81e430e4e8b722f8,542d8ad6f6af2084] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645094, data is empty. 2024-09-11 17:58:28.269 INFO 2981023 --- [alarm-detect-14,81e430e4e8b722f8,71b0cbb0affe9f97] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645094, data is empty. 2024-09-11 17:58:28.278 INFO 2981023 --- [alarm-detect-14,81e430e4e8b722f8,71b0cbb0affe9f97] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645094, data is empty. 2024-09-11 17:58:28.284 INFO 2981023 --- [alarm-detect-14,81e430e4e8b722f8,71b0cbb0affe9f97] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645094, data is empty. 2024-09-11 17:58:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,66f9f7dba3c42fd4,f99891834347ce67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,66f9f7dba3c42fd4,f99891834347ce67] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:29.335 INFO 2981023 --- [ocp-async-8,0f794ead759f8156,a4983ed881f2f02e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:29.723 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:29.724 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:31.335 INFO 2981023 --- [ocp-async-2,5e67779797ca1fb2,732ea700da0763d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,55b1b0be826c0f5e,d8ee0abea51bbd51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,55b1b0be826c0f5e,d8ee0abea51bbd51] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:32.349 INFO 2981023 --- [ocp-schedules-12,a8f508f8ce40975f,7aa2171773469f80] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:58:32.724 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:32.725 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:33.335 INFO 2981023 --- [ocp-async-10,53f3d6c8c616f68e,7e78aee384056159] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:34.773 INFO 2981023 --- [ocp-schedules-29,3e19da58dc1c4680,b7eb269f175eb6f9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:58:34.778 INFO 2981023 --- [ocp-schedules-29,3e19da58dc1c4680,b7eb269f175eb6f9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:58:34.778 INFO 2981023 --- [ocp-schedules-29,3e19da58dc1c4680,b7eb269f175eb6f9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:58:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,6b88fc946559ccae,41d89a2242e89076] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,6b88fc946559ccae,41d89a2242e89076] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:35.335 INFO 2981023 --- [ocp-async-4,95491a414f87c428,8ca0c9b642743e17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:35.720 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0a54513fb6f82226,f8c4796accc429db] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=0a54513fb6f82226, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:58:35.722 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0a54513fb6f82226,f8c4796accc429db] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:58:35.725 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:35.726 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:35.754 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0a54513fb6f82226,f8c4796accc429db] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=0a54513fb6f82226, duration=38 ms] 2024-09-11 17:58:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:58:36.337611+08:00, lastProcessTime=2024-09-11T17:58:26.338004+08:00 2024-09-11 17:58:36.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:58:36.337611+08:00 2024-09-11 17:58:37.314 INFO 2981023 --- [prometheus-cache-monitor1,1c1d10dee5ced74b,1d49f9de9c2bbde6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7478,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":4997} 2024-09-11 17:58:37.315 INFO 2981023 --- [prometheus-cache-monitor1,1c1d10dee5ced74b,1d49f9de9c2bbde6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:58:37.335 INFO 2981023 --- [ocp-async-5,69404840578a6217,2652f41e479264a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:37.769 INFO 2981023 --- [ocp-async-7,1bd0801b1b5cead3,a9f1b0e6bb1dabff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:58:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,e5b0a27a9cd7c2e8,741ad2a6488720ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,e5b0a27a9cd7c2e8,741ad2a6488720ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:38.364 INFO 2981023 --- [ocp-schedules-29,ddd4cae3ffca6a13,05e68dfd8fcb872f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:58:38.377 WARN 2981023 --- [scanner-13582309-1,ddd4cae3ffca6a13,950fecacccc24351] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:58:38.726 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:38.728 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:39.335 INFO 2981023 --- [ocp-async-6,c0565f94c6896bad,081cda2799f3bc7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:39.567 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts/2000010/checkSystemInfoReport?id=2000010, client=10.31.33.41, traceId=c7953309c70d20c6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.getCheckSystemInfoReport(Long), args=2000010,] 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=210, repairable=false, repairHint=null, description=检查主机时钟偏移，单位(ms)。OBServer 和 OCP 的时钟偏移应保持在 50ms 内，来满足 OceanBase 集群的时钟偏移要求|Checks the clock offset of the host, in milliseconds. In an OceanBase cluster, the clock offset between OBServers and OCP must be within 50 ms., repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6cf267c8 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=211, repairable=true, repairHint=null, description=libaio 是 OceanBase 运行依赖的动态链接库，必须安装|libaio dynamic-link library (DLL) is a dependency of OceanBase Database and must be installed., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"libaio"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@33af1c1f 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=212, repairable=true, repairHint=null, description=crond 用于设置定时任务，OceanBase 依赖的基础能力|The crond service is a basic dependency of OceanBase Database used for setting up scheduled tasks., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, parameters={"name":"crond","opType":"start"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, rollbackParameters={"name":"crond","opType":"stop"}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@143c5f34 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=213, repairable=true, repairHint=null, description=sshd 用于加密登录或者数据传输，OceanBase 依赖的基础能力|The sshd service is a basic dependency of OceanBase Database for encrypted logon or data transmission., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, parameters={"name":"sshd","opType":"start"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, rollbackParameters={"name":"sshd","opType":"stop"}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2b76d10f 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=214, repairable=true, repairHint=null, description=建议用户禁用 firewalld 或为 OceanBase 集群各服务添加允许规则。避免防火墙拦截|We recommend that you disable the firewalld service or add whitelist rules for services of the OceanBase cluster as needed. Otherwise, services will be blocked by the firewall., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, parameters={"name":"firewalld","opType":"stop"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, rollbackParameters={"name":"sshd","opType":"start"}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@54c53f8e 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=215, repairable=true, repairHint=null, description=检查操作系统 /sys/kernel/mm/transparent_hugepage/enabled 配置。需要设置为 never，保障系统稳定性|Checks the settings of the enabled file in the /sys/kernel/mm/transparent_hugepage/ directory of the operating system. You must set this parameter to `never` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairTHP@21dccea5, parameters={"newOption":"never"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairTHP@21dccea5, rollbackParameters={"newOption":"never","isReset":true}, rollbackReplaceField=newOption), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@1fc19eed 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=216, repairable=false, repairHint=null, description=检查主机安装的 Python 版本是否 2.7.x，确保相关 OceanBase 脚本能够正常运行。建议用户自行安装|Checks whether the version of Python installed on the host is 2.7.x to ensure the normal execution of relevant OceanBase scripts. We recommend that you install it by yourself., repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@69f2a7de 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=217, repairable=true, repairHint=null, description=检查操作系统 swap 信息。要求移除 swap 分区|Checks the swap information of the operating system. Swap partitions must be removed., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSwap@44955c9b, parameters={"newOption":"off"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSwap@44955c9b, rollbackParameters={"newOption":"off"}, rollbackReplaceField=newOption), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7491ed46 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=218, repairable=true, repairHint=null, description=net.ipv4.conf.default.rp_filter 检查网卡对接收到的数据包进行反向路由验证的规则。需要设置为 1，开启严格的反向路由校验，保障系统安全性|An operating system parameter. It checks the NIC rules for reverse routing verification on received packets. To ensure system security, you must set this parameter to `1` to enable strict reverse routing verification., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.conf.default.rp_filter","paramValue":"1"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.conf.default.rp_filter","paramValue":"1"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@9d59515 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=219, repairable=true, repairHint=null, description=net.ipv4.conf.default.accept_source_route 允许数据包的发送者指定数据包的发送路径，以及返回给发送者的数据包所走的路径。需设置为 0，禁用该特性，保障系统安全性|An operating system parameter. It controls whether to allow the sender to specify the path for sending packets and the path by which packets are returned to the sender. To ensure system security, you must set this parameter to `0` to disable this feature., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.conf.default.accept_source_route","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.conf.default.accept_source_route","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@1af05434 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=220, repairable=true, repairHint=null, description=net.ipv4.tcp_syncookies 当出现 syn 等候队列出现溢出时向对方发送 syncookies。需要设置为1，目的是为了防止 syn flood 攻击|An operating system parameter. It controls whether to send syncookies to the peer end when the SYN queue overflows. To prevent an SYN flood, you must set this parameter to `1`., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_syncookies","paramValue":"1"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_syncookies","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6640cce3 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=221, repairable=true, repairHint=null, description=vm.swappiness 换出运行时内存的相对权重。需要设置为 0，保障系统稳定|An operating system parameter. It controls the relative weight given to swapping out of runtime memory. You must set this parameter to `0` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.swappiness","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.swappiness","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@1d40e6b0 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=222, repairable=true, repairHint=null, description=net.ipv4.tcp_tw_reuse 检查是否允许将处于 TIME-WAIT 状态的 socket（TIME-WAIT 的端口）用于新的 TCP 连接。需要设置为 1，保障系统性能|An operating system parameter. It checks whether sockets in the TIME_WAIT state can be reused for new TCP connections. You must set this parameter to `1` to ensure system performance., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_tw_reuse","paramValue":"1"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_tw_reuse","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@3898b530 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=223, repairable=true, repairHint=null, description=net.ipv4.tcp_slow_start_after_idle 检查 TCP 空闲后是否慢启动。需要设置为 0，保障系统性能|An operating system parameter. It checks whether a TCP connection starts slowly after an idle period. You must set this parameter to `0` to ensure system performance., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_slow_start_after_idle","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_slow_start_after_idle","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@205117e3 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=224, repairable=true, repairHint=null, description=vm.overcommit_memory 检查是否接受大型内存请求或拒绝的条件。需要设置为 0，保障系统稳定性|An operating system parameter. It checks the conditions for receiving large memory requests are met. You must set this parameter to `0` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.overcommit_memory","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.overcommit_memory","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4016f0de 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=225, repairable=true, repairHint=null, description=vm.nr_hugepages 检查是否开启 HugePages。需要设置为 0，保障系统稳定性|An operating system parameter. It checks whether the HugePages feature is enabled. You must set this parameter to `0` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.nr_hugepages","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.nr_hugepages","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5af9fb0a 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=238, repairable=true, repairHint=null, description=net.core.somaxconn 检查系统中每一个端口最大的监听队列的长度。建议设置在范围 [2048,16384]|check net.core.somaxconn value.Recommended value: [2048,16384], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.somaxconn","paramValue":"2048"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.somaxconn","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2735f450 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=239, repairable=true, repairHint=null, description=net.core.netdev_max_backlog 检查在每个网络接口接收数据包的速率比内核处理这些包的速率快时，允许送到队列的数据包的最大数目。建议设置在范围 [500,10000]|check net.core.netdev_max_backlog value.Recommended value: [500,10000], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.netdev_max_backlog","paramValue":"10000"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.netdev_max_backlog","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@ff3015a 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=240, repairable=true, repairHint=null, description=net.core.rmem_default 检查主机默认的 TCP 数据接收窗口大小（字节）。建议设置在范围 [65536,16777216]|check net.core.rmem_default value.Recommended value: [65536,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.rmem_default","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.rmem_default","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7814807a 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=241, repairable=true, repairHint=null, description=net.core.wmem_default 检查主机默认的 TCP 数据发送窗口大小（字节）。建议设置在范围 [65536,16777216]|check net.core.wmem_default value.Recommended value: [65536,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.wmem_default","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.wmem_default","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6da015d9 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=242, repairable=true, repairHint=null, description=net.core.rmem_max 检查主机最大的 TCP 数据接收窗口（字节）。建议设置在范围 [8388608,16777216]|check net.core.rmem_max value.Recommended value: [8388608,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.rmem_max","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.rmem_max","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@653ffb49 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=243, repairable=true, repairHint=null, description=net.core.wmem_max 检查主机最大的 TCP 数据发送窗口（字节）。建议设置在范围 [8388608,16777216]|check net.core.wmem_max value.Recommended value: [8388608,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.wmem_max","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.wmem_max","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@72d1df65 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=245, repairable=true, repairHint=null, description=net.ipv4.tcp_rmem 检查 socket 接收缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_rmem value.Recommended value: 4096 87380 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"4096 87380 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2e9a9615 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=245, repairable=true, repairHint=null, description=net.ipv4.tcp_rmem 检查 socket 接收缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_rmem value.Recommended value: 4096 87380 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"4096 87380 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@43f940c2 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=245, repairable=true, repairHint=null, description=net.ipv4.tcp_rmem 检查 socket 接收缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_rmem value.Recommended value: 4096 87380 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"4096 87380 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@59861e47 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=246, repairable=true, repairHint=null, description=net.ipv4.tcp_wmem 检查 socket 发送缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_wmem value.Recommended value: 4096 65536 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"4096 65536 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2550d8e5 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=246, repairable=true, repairHint=null, description=net.ipv4.tcp_wmem 检查 socket 发送缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_wmem value.Recommended value: 4096 65536 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"4096 65536 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5f80c864 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=246, repairable=true, repairHint=null, description=net.ipv4.tcp_wmem 检查 socket 发送缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_wmem value.Recommended value: 4096 65536 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"4096 65536 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@56f67c06 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=256, repairable=true, repairHint=null, description=net.ipv4.tcp_max_syn_backlog 检查可保存在队列中未获得对方确认的最大连接请求数目。建议设置在范围：[1024,16384]|check net.ipv4.tcp_max_syn_backlog value.Recommended value: [1024,16384], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_max_syn_backlog","paramValue":"1024"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_max_syn_backlog","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@3f6b24b9 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=257, repairable=true, repairHint=null, description=net.ipv4.tcp_fin_timeout 检查 TCP 保持在 FIN-WAIT-2 状态的时间（秒）。建议设置在范围：[15,60]|check net.ipv4.tcp_fin_timeout value.Recommended value: [15,60], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_fin_timeout","paramValue":"60"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_fin_timeout","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@1c5f5441 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=258, repairable=true, repairHint=null, description=vm.min_free_kbytes 检查保留的空闲页面池的大小。建议设置在范围：[32768,2097152]|check vm.min_free_kbytes value.Recommended value: [32768,2097152], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.min_free_kbytes","paramValue":"2097152"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.min_free_kbytes","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@45457c46 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=259, repairable=true, repairHint=null, description=vm.max_map_count 检查进程可以使用的最大内存映射区域值。建议设置为 655360|check vm.max_map_count value.Recommended value: 655360, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.max_map_count","paramValue":"655360"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.max_map_count","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@20b63ea4 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=260, repairable=true, repairHint=null, description=fs.aio-max-nr 检查同时可支持的异步 IO 数目。建议设置为 >= 1048576|check fs.aio-max-nr value.Recommended value >= 1048576, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"fs.aio-max-nr","paramValue":"1048576"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"fs.aio-max-nr","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@461e98e7 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=261, repairable=true, repairHint=null, description=kernel.numa_balancing 检查自动 numa 平衡是否禁用，为避免内存节点数据迁移可能造成的性能抖动，建议禁用|check kernel.numa_balancing value.Recommended value: 0, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"kernel.numa_balancing","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"kernel.numa_balancing","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@20052139 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=262, repairable=true, repairHint=null, description=vm.zone_reclaim_mode 检查控制内存 zone 回收模式是否禁用，为避免因回收导致的性能抖动，建议禁用|check vm.zone_reclaim_mode value.Recommended value: 0, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.zone_reclaim_mode","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.zone_reclaim_mode","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4e4cce33 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=263, repairable=true, repairHint=null, description=fs.file-max 检查整个系统能够打开的最大文件数量，建议设置为 >= 6573688|check fs.file-max value.Recommended value >= 6573688, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"fs.file-max","paramValue":"6573688"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"fs.file-max","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5f9e8a40 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=264, repairable=true, repairHint=null, description=fs.pipe-user-pages-soft 检查由单个非特权用户创建或设置的所有管道的总大小(以页为单位)的软限制，建议设置为默认值|check fs.pipe-user-pages-soft value.Recommended default value, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"fs.pipe-user-pages-soft","paramValue":"16384"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"fs.pipe-user-pages-soft","paramValue":"16384"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6d75f8f4 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=227, repairable=true, repairHint=null, description=检查主机 mtr 命令是否存在，网络测试工具，为运维或检查提供便利。建议安装|Checks whether the mtr command is available on the host. mtr is a network test tool. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"mtr"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6470108a 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=228, repairable=true, repairHint=null, description=检查主机 tar 命令是否存在，Linux 打包工具，OCP 依赖的重要的运维工具。建议安装|Checks whether the tar command is available on the host. tar is a packaging tool in Linux. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"tar"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4ce57761 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=229, repairable=true, repairHint=null, description=检查主机是否安装 curl 命令，命令行文本传输工具，为运维或检查提供便利。建议安装|Checks whether the curl command is available on the host. curl is a command-line tool for transferring data. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"curl"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@67822837 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=230, repairable=true, repairHint=null, description=检查主机是否安装 nc 命令，netcat 网络工具，OCP 依赖的重要的运维工具。建议安装|Checks whether the NetCat (nc) command is available on the host. nc is a networking tool. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"nc"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@25f2ce6b 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=231, repairable=true, repairHint=null, description=检查主机是否安装 binutils 二进制工具集，为运维或检查提供便利。建议安装|Checks whether the binutils binary toolkit is installed on the host. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"binutils"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@41e26e1f 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=232, repairable=true, repairHint=null, description=检查主机是否安装 bind-utils DNS 工具集，为运维或检查提供便利。建议安装|Checks whether the bind-utils toolkit is installed on the host. The bind-utils toolkit contains a collection of utilities for querying DNS name servers. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"bind-utils"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@265b473d 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=233, repairable=true, repairHint=null, description=检查主机是否安装 iproute 网络管理工具包，为运维或检查提供便利。建议安装|Checks whether the iproute network management toolkit is installed on the host. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"iproute"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@50ee0051 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=244, repairable=true, repairHint=null, description=libatomic 是 OceanBase 运行依赖的动态链接库，必须安装|libatomic dynamic-link library (DLL) is a dependency of OceanBase Database and must be installed., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"libatomic"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@635c0db1 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=247, repairable=false, repairHint=null, description=检查指定的用户（系统参数 ocp.operation.default.os.user）是否存在|check if user(ocp.operation.default.os.user in System Parameters) exists, repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@319c1553 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=248, repairable=true, repairHint=null, description=检查 root 用户 openFiles 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 655350| check if (user root) limit of openFiles is set. Recommend value: 655350, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"openFiles","userName":"root","expectedValue":"655350"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4d82cef2 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=249, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）openFiles 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 655350| check if (ocp.operation.default.os.user in System Parameters) limit of openFiles is set. Recommend value: 655350, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"openFiles","userName":"admin","expectedValue":"655350"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6dceb39b 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=250, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）stackSize 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 10240| check if (ocp.operation.default.os.user in System Parameters) limit of stackSize is set. Recommend value: 10240, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"stackSize","userName":"admin","expectedValue":"10240"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6efc0a16 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=251, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）maxUserProcesses 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 655360| check if (ocp.operation.default.os.user in System Parameters) limit of maxUserProcesses is set. Recommend value: 655360, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"maxUserProcesses","userName":"admin","expectedValue":"655360"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@3f9bbc17 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=252, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）coreFileSize 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 unlimited| check if (ocp.operation.default.os.user in System Parameters) limit of coreFileSize is set. Recommend value: unlimited, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"coreFileSize","userName":"admin","expectedValue":"unlimited"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@64b51048 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=253, repairable=true, repairHint=null, description=检查 SELinux 是否启用，建议用户禁用 selinux，SELinux 使用所谓的委任式访问控制 (Mandatory Access Control, MAC) ，可以针对特定的程序与特定的档案资源来进行权限的控管| check if selinux is disabled., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSelinux@1c78e5d6, parameters={"state":"disabled"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSelinux@1c78e5d6, rollbackParameters={"state":"disabled"}, rollbackReplaceField=state), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7cff352e 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=254, repairable=false, repairHint=null, description=OceanBase 数据盘文件类型，磁盘容量 > 16T 时应为 xfs 类型|File Type of OceanBase Data Disk. File system type should be xfs if disk size > 16T, repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@72dd3c1d 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=265, repairable=false, repairHint=null, description=检查操作系统版本，一般推荐使用标准化主机文档中列举的操作系统版本 |check OS version. Recommend using OS versions listed in document, repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@10444d55 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=266, repairable=true, repairHint=null, description=检查主机是否安装 fio 命令，磁盘压测工具，OBServer 依赖的重要的运维工具。建议安装|Checks whether the fio command is available on the host. fio is a disk pressure test tool. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"fio"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5c3f3845 2024-09-11 17:58:39.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=277, repairable=true, repairHint=null, description=检查主机 clock source 配置。建议设置为 tsc 类型（aarch64 推荐 arch_sys_conter）|Checks host OS clock source config. We recommend that you set it to tsc type(arch_sys_conter on aarch64)., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairClockSource@43f6ef2d, parameters={"sourceType":"tsc"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairClockSource@43f6ef2d, rollbackParameters={"isReset":"true", "sourceType":""}, rollbackReplaceField=sourceType), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@596c30a9 2024-09-11 17:58:39.747 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c7953309c70d20c6,7ecfa8a7463b2864] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts/2000010/checkSystemInfoReport client=10.31.33.41, traceId=c7953309c70d20c6, duration=183 ms] 2024-09-11 17:58:40.003 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:58:40.004 INFO 2981023 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:58:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,c09b89dc8c136054,44f79c5c9baba72a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,c09b89dc8c136054,44f79c5c9baba72a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:41.335 INFO 2981023 --- [ocp-async-4,d009e67ae717a7d4,6e632630868e19f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:41.728 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:41.728 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:42.352 INFO 2981023 --- [ocp-schedules-19,e77061f28d2e8309,1088db8b8bf4737a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:58:43.205 INFO 2981023 --- [ocp-schedules-13,f5de27600b7ec77e,334c57b7b64596c7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:58:43.224 WARN 2981023 --- [ocp-schedules-13,f5de27600b7ec77e,334c57b7b64596c7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:58:43.336 INFO 2981023 --- [ocp-async-1,a204bc6f6f296ceb,a00846790ad42151] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,9db666f0592f9236,e01efea281d8bfb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,9db666f0592f9236,e01efea281d8bfb8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:44.728 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:44.729 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:45.101 INFO 2981023 --- [prometheus-second-schedule-1,6c905b85c60ca53d,1e15e0bce75e4ba7] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:58:45.336 INFO 2981023 --- [ocp-async-9,bd1325f21f7b2ef4,01cdbb29f910c13d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:46.020 INFO 2981023 --- [ocp-schedules-21,773c8ebcb1dc46d6,546724bb95259a77] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:58:46.021 INFO 2981023 --- [host-check-clockdiff1,773c8ebcb1dc46d6,6d923a654d3b05ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:58:46.021 INFO 2981023 --- [host-check-clockdiff3,773c8ebcb1dc46d6,e10924760d16b987] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:58:46.024 INFO 2981023 --- [host-check-clockdiff2,773c8ebcb1dc46d6,e0614ab2a3435898] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:58:46.025 INFO 2981023 --- [host-check-clockdiff4,773c8ebcb1dc46d6,31097caf2b590add] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:58:46.033 INFO 2981023 --- [ocp-schedules-8,c2ae7087e58bfe86,90e23eab4115a31e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:58:46.035 INFO 2981023 --- [host-check-clockdiff3,773c8ebcb1dc46d6,e10924760d16b987] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048726 0 0 , error= 2024-09-11 17:58:46.035 INFO 2981023 --- [ocp-schedules-17,f0ee86bfa353e445,5bf6d6d543497aa9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:58:46.036 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:58:46.036 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:58:46.037 INFO 2981023 --- [host-check-agent-status1,f0ee86bfa353e445,2475ac843bfed5a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:58:46.037 INFO 2981023 --- [host-check-agent-status2,f0ee86bfa353e445,8e1fbb95470ed839] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:58:46.038 INFO 2981023 --- [host-check-agent-status3,f0ee86bfa353e445,a3d34a9bf9fd04e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:58:46.038 INFO 2981023 --- [host-check-agent-status4,f0ee86bfa353e445,c9ffcb2f4e74bc06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:58:46.040 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:58:46.040 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:58:46.060 INFO 2981023 --- [host-check-agent-status4,f0ee86bfa353e445,c9ffcb2f4e74bc06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:46.061 INFO 2981023 --- [host-check-agent-status3,f0ee86bfa353e445,a3d34a9bf9fd04e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:58:46.064 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:58:46.065 INFO 2981023 --- [host-check-agent-status4,f0ee86bfa353e445,c9ffcb2f4e74bc06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:46.065 INFO 2981023 --- [host-check-agent-status3,f0ee86bfa353e445,a3d34a9bf9fd04e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:58:46.065 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:46.066 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:58:46.066 INFO 2981023 --- [host-check-agent-status2,f0ee86bfa353e445,8e1fbb95470ed839] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:58:46.068 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:46.068 INFO 2981023 --- [host-check-agent-status1,f0ee86bfa353e445,2475ac843bfed5a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:58:46.069 INFO 2981023 --- [host-check-agent-status1,f0ee86bfa353e445,2475ac843bfed5a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:58:46.070 INFO 2981023 --- [host-check-agent-status2,f0ee86bfa353e445,8e1fbb95470ed839] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:58:46.070 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:58:46.083 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:58:46.093 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:58:46.164 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:58:46.186 INFO 2981023 --- [ocp-schedules-18,e8623241ce91c0b0,a969f61419942712] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:58:46.188 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:58:46.189 INFO 2981023 --- [ocp-schedules-27,f853260cdfcca71b,778bf9592ba92cd9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:58:46.191 INFO 2981023 --- [ocp-schedules-3,7142b81ec934f8bd,5d2109e5ad9923fd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:58:46.194 INFO 2981023 --- [ocp-schedules-2,dafa753588ab08e7,231d334f5a2f44db] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:58:46.196 INFO 2981023 --- [ocp-schedules-23,1e434d3b6e674276,1c00c2d52145ca3c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:58:46.262 INFO 2981023 --- [ocp-schedules-27,f853260cdfcca71b,778bf9592ba92cd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:58:46.272 INFO 2981023 --- [ocp-schedules-2,dafa753588ab08e7,231d334f5a2f44db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:58:46.284 INFO 2981023 --- [ocp-schedules-3,7142b81ec934f8bd,5d2109e5ad9923fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:58:46.293 INFO 2981023 --- [ocp-schedules-3,7142b81ec934f8bd,5d2109e5ad9923fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:58:46.357 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:58:46.367 INFO 2981023 --- [ocp-schedules-18,e8623241ce91c0b0,a969f61419942712] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:58:46.378 INFO 2981023 --- [ocp-schedules-12,95d2bde98ef27c35,4e501704d04d9701] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:58:46.461 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:58:46.461 INFO 2981023 --- [host-check-agent-status3,f0ee86bfa353e445,a3d34a9bf9fd04e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:58:46.463 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:58:46.463621606+08:00 2024-09-11 17:58:46.465 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:58:46.464+08:00 2024-09-11 17:58:46.465 INFO 2981023 --- [host-check-agent-status1,f0ee86bfa353e445,2475ac843bfed5a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:58:46.467 INFO 2981023 --- [host-check-agent-status3,f0ee86bfa353e445,a3d34a9bf9fd04e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:58:46.467 INFO 2981023 --- [host-check-agent-status3,f0ee86bfa353e445,a3d34a9bf9fd04e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:58:46.467 INFO 2981023 --- [host-check-agent-status3,f0ee86bfa353e445,a3d34a9bf9fd04e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:58:46.468 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:58:46.464+08:00 and status AVAILABLE 2024-09-11 17:58:46.469 INFO 2981023 --- [host-check-agent-status1,f0ee86bfa353e445,2475ac843bfed5a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:58:46.469 INFO 2981023 --- [host-check-agent-status1,f0ee86bfa353e445,2475ac843bfed5a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:58:46.469 INFO 2981023 --- [host-check-agent-status1,f0ee86bfa353e445,2475ac843bfed5a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:58:46.471 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:58:46.469496+08:00, lastProcessTime=2024-09-11T17:58:36.337611+08:00 2024-09-11 17:58:46.471 INFO 2981023 --- [host-check-agent-status4,f0ee86bfa353e445,c9ffcb2f4e74bc06] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:58:46.472 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:58:46.472 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:58:46.473 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:58:46.473 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:58:46.478640272+08:00 2024-09-11 17:58:46.473 INFO 2981023 --- [host-check-agent-status2,f0ee86bfa353e445,8e1fbb95470ed839] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:58:46.474 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T17:58:46.474061560+08:00 2024-09-11 17:58:46.474 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:58:46.479871298+08:00 2024-09-11 17:58:46.474 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:58:46.473+08:00 2024-09-11 17:58:46.476 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T17:58:46.474+08:00 2024-09-11 17:58:46.476 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:58:46.474+08:00 2024-09-11 17:58:46.476 INFO 2981023 --- [host-check-agent-status4,f0ee86bfa353e445,c9ffcb2f4e74bc06] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:58:46.476 INFO 2981023 --- [host-check-agent-status4,f0ee86bfa353e445,c9ffcb2f4e74bc06] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:58:46.476 INFO 2981023 --- [host-check-agent-status4,f0ee86bfa353e445,c9ffcb2f4e74bc06] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:58:46.476 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:58:46.478 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:58:46.474+08:00 and status AVAILABLE 2024-09-11 17:58:46.480 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:58:46.473+08:00 and status AVAILABLE 2024-09-11 17:58:46.481 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T17:58:46.474+08:00 and status AVAILABLE 2024-09-11 17:58:46.482 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.483 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:46.483 INFO 2981023 --- [host-check-agent-status2,f0ee86bfa353e445,8e1fbb95470ed839] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:58:46.483 INFO 2981023 --- [host-check-agent-status2,f0ee86bfa353e445,8e1fbb95470ed839] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:58:46.483 INFO 2981023 --- [host-check-agent-status2,f0ee86bfa353e445,8e1fbb95470ed839] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:58:46.483 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:58:46.486 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.487 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:46.487 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:58:46.488 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 17:58:46.491 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:46.494 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.494 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:58:46.501 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:46.501 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:58:46.502 INFO 2981023 --- [ocp-schedules-27,f853260cdfcca71b,778bf9592ba92cd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:58:46.559 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:58:46.560 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:58:46.561 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.562 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:58:46.562 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:46.562 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:58:46.563 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.563 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:58:46.563 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:46.565 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.565 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:46.572 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:58:46.573 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:58:46.575 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:58:46.576 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:58:46.578 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:58:46.583 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:58:46.584 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.584 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:46.610 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:58:46.617 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:58:46.629 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:58:46.634 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 17:58:46.653 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:58:46.658 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:58:46.659 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:58:46.663 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.665 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.666 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:58:46.666 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:58:46.667 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.667 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 17:58:46.670 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.671 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.671 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:58:46.677 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:58:46.683 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.684 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:58:46.685 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.685 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:58:46.691 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:58:46.701 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:58:46.702 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:58:46.709 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:58:46.718 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.719 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.719 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:46.720 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:58:46.720 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.721 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.721 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.722 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:46.723 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:58:46.723 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:58:46.723 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.726 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:58:46.727 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:58:46.728 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.736 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.737 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.737 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:58:46.738 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:58:46.738 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.738 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:58:46.739 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:58:46.739 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:58:46.739 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.746 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:58:46.753 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:58:46.758 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:58:46.758 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:58:46.761 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:58:46.762 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:46.762 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:58:46.764 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:46.764 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:58:46.769 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:58:46.775 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 17:58:46.780 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:46.781 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:46.783 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:58:46.788 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 17:58:46.788 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:58:46.790 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.790 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.790 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.791 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.791 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:46.791 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:58:46.792 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:58:46.792 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:58:46.792 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.793 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:58:46.795 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.798 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:58:46.799 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.800 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:46.802 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:58:46.804 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:58:46.806 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:58:46.808 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:58:46.808 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:58:46.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.809 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.811 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:58:46.814 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:58:46.814 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:58:46.815 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:58:46.816 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:58:46.825 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:58:46.831 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:58:46.838 INFO 2981023 --- [host-check-host-status3,c2ae7087e58bfe86,0f68cd115da341b3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:58:46.842 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:58:46.846 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:58:46.846 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:58:46.857 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:58:46.861 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:58:46.862 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.863 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.864 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:58:46.864 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:58:46.865 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.866 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.867 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 17:58:46.867 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.867 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:58:46.867 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:58:46.868 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:58:46.868 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:58:46.868 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.876 INFO 2981023 --- [host-check-host-status4,c2ae7087e58bfe86,c6c447a62cd25c43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 17:58:46.878 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:46.878 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:58:46.892 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:58:46.900 INFO 2981023 --- [host-check-host-status1,c2ae7087e58bfe86,16e7fa3a4c2764fe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:58:46.907 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:58:46.907 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:58:46.913 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:58:46.920 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.921 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.922 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.922 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:58:46.923 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:58:46.923 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:58:46.924 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.924 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:58:46.926 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:58:46.929 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:58:46.931 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:58:46.932 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:58:46.932 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:58:46.936 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:58:46.938 INFO 2981023 --- [host-check-host-status2,c2ae7087e58bfe86,ca17853cd64265cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:58:46.949 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.950 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.950 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:58:46.952 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:58:46.955 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:58:46.959 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:58:46.959 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:58:46.960 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:58:46.963 INFO 2981023 --- [ocp-schedules-4,655bb9d8136bc692,aef98dee4ffd8b33] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:58:46.970 INFO 2981023 --- [host-check-clockdiff2,773c8ebcb1dc46d6,e0614ab2a3435898] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048726 5 5 , error= 2024-09-11 17:58:46.974 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:58:46.469496+08:00 2024-09-11 17:58:46.985 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:58:46.990 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:46.991 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:46.991 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:58:46.996 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:58:46.996 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:58:46.996 INFO 2981023 --- [ocp-schedules-2,dafa753588ab08e7,231d334f5a2f44db] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:58:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,769003b9cbf0f795,0ddbd33ed3ef9ae8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,769003b9cbf0f795,0ddbd33ed3ef9ae8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:47.340 INFO 2981023 --- [ocp-async-8,ca796429d8893351,29e9bc2cdaa72b5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:47.729 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:47.730 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:47.769 INFO 2981023 --- [ocp-async-6,72a10a598d998c3e,5ff129d920ca9abf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:58:47.951 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hosts/2000010/repairSystemInfo, client 10.31.33.41, traceId 0e817ce8ba29f7e3] 2024-09-11 17:58:47.952 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hosts/2000010/repairSystemInfo?id=2000010, client=10.31.33.41, traceId=0e817ce8ba29f7e3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.repairSystemInfo(Long), args=2000010,] 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=210, repairable=false, repairHint=null, description=检查主机时钟偏移，单位(ms)。OBServer 和 OCP 的时钟偏移应保持在 50ms 内，来满足 OceanBase 集群的时钟偏移要求|Checks the clock offset of the host, in milliseconds. In an OceanBase cluster, the clock offset between OBServers and OCP must be within 50 ms., repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6eca2be2 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=211, repairable=true, repairHint=null, description=libaio 是 OceanBase 运行依赖的动态链接库，必须安装|libaio dynamic-link library (DLL) is a dependency of OceanBase Database and must be installed., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"libaio"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@23c8d6ad 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=212, repairable=true, repairHint=null, description=crond 用于设置定时任务，OceanBase 依赖的基础能力|The crond service is a basic dependency of OceanBase Database used for setting up scheduled tasks., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, parameters={"name":"crond","opType":"start"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, rollbackParameters={"name":"crond","opType":"stop"}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@32fe1ff3 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=213, repairable=true, repairHint=null, description=sshd 用于加密登录或者数据传输，OceanBase 依赖的基础能力|The sshd service is a basic dependency of OceanBase Database for encrypted logon or data transmission., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, parameters={"name":"sshd","opType":"start"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, rollbackParameters={"name":"sshd","opType":"stop"}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@15f565d1 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=214, repairable=true, repairHint=null, description=建议用户禁用 firewalld 或为 OceanBase 集群各服务添加允许规则。避免防火墙拦截|We recommend that you disable the firewalld service or add whitelist rules for services of the OceanBase cluster as needed. Otherwise, services will be blocked by the firewall., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, parameters={"name":"firewalld","opType":"stop"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairService@1e7fc0a3, rollbackParameters={"name":"sshd","opType":"start"}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2adfc9b6 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=215, repairable=true, repairHint=null, description=检查操作系统 /sys/kernel/mm/transparent_hugepage/enabled 配置。需要设置为 never，保障系统稳定性|Checks the settings of the enabled file in the /sys/kernel/mm/transparent_hugepage/ directory of the operating system. You must set this parameter to `never` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairTHP@21dccea5, parameters={"newOption":"never"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairTHP@21dccea5, rollbackParameters={"newOption":"never","isReset":true}, rollbackReplaceField=newOption), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4f471ffe 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=216, repairable=false, repairHint=null, description=检查主机安装的 Python 版本是否 2.7.x，确保相关 OceanBase 脚本能够正常运行。建议用户自行安装|Checks whether the version of Python installed on the host is 2.7.x to ensure the normal execution of relevant OceanBase scripts. We recommend that you install it by yourself., repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@d849062 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=217, repairable=true, repairHint=null, description=检查操作系统 swap 信息。要求移除 swap 分区|Checks the swap information of the operating system. Swap partitions must be removed., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSwap@44955c9b, parameters={"newOption":"off"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSwap@44955c9b, rollbackParameters={"newOption":"off"}, rollbackReplaceField=newOption), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@33e1b7ee 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=218, repairable=true, repairHint=null, description=net.ipv4.conf.default.rp_filter 检查网卡对接收到的数据包进行反向路由验证的规则。需要设置为 1，开启严格的反向路由校验，保障系统安全性|An operating system parameter. It checks the NIC rules for reverse routing verification on received packets. To ensure system security, you must set this parameter to `1` to enable strict reverse routing verification., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.conf.default.rp_filter","paramValue":"1"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.conf.default.rp_filter","paramValue":"1"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@67275a5d 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=219, repairable=true, repairHint=null, description=net.ipv4.conf.default.accept_source_route 允许数据包的发送者指定数据包的发送路径，以及返回给发送者的数据包所走的路径。需设置为 0，禁用该特性，保障系统安全性|An operating system parameter. It controls whether to allow the sender to specify the path for sending packets and the path by which packets are returned to the sender. To ensure system security, you must set this parameter to `0` to disable this feature., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.conf.default.accept_source_route","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.conf.default.accept_source_route","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@10e801a8 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=220, repairable=true, repairHint=null, description=net.ipv4.tcp_syncookies 当出现 syn 等候队列出现溢出时向对方发送 syncookies。需要设置为1，目的是为了防止 syn flood 攻击|An operating system parameter. It controls whether to send syncookies to the peer end when the SYN queue overflows. To prevent an SYN flood, you must set this parameter to `1`., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_syncookies","paramValue":"1"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_syncookies","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@39468457 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=221, repairable=true, repairHint=null, description=vm.swappiness 换出运行时内存的相对权重。需要设置为 0，保障系统稳定|An operating system parameter. It controls the relative weight given to swapping out of runtime memory. You must set this parameter to `0` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.swappiness","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.swappiness","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7260a56 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=222, repairable=true, repairHint=null, description=net.ipv4.tcp_tw_reuse 检查是否允许将处于 TIME-WAIT 状态的 socket（TIME-WAIT 的端口）用于新的 TCP 连接。需要设置为 1，保障系统性能|An operating system parameter. It checks whether sockets in the TIME_WAIT state can be reused for new TCP connections. You must set this parameter to `1` to ensure system performance., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_tw_reuse","paramValue":"1"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_tw_reuse","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5a8e5d5d 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=223, repairable=true, repairHint=null, description=net.ipv4.tcp_slow_start_after_idle 检查 TCP 空闲后是否慢启动。需要设置为 0，保障系统性能|An operating system parameter. It checks whether a TCP connection starts slowly after an idle period. You must set this parameter to `0` to ensure system performance., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_slow_start_after_idle","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_slow_start_after_idle","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@a3be2e5 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=224, repairable=true, repairHint=null, description=vm.overcommit_memory 检查是否接受大型内存请求或拒绝的条件。需要设置为 0，保障系统稳定性|An operating system parameter. It checks the conditions for receiving large memory requests are met. You must set this parameter to `0` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.overcommit_memory","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.overcommit_memory","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7d2abc9 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=225, repairable=true, repairHint=null, description=vm.nr_hugepages 检查是否开启 HugePages。需要设置为 0，保障系统稳定性|An operating system parameter. It checks whether the HugePages feature is enabled. You must set this parameter to `0` to ensure system stability., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.nr_hugepages","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.nr_hugepages","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@301165df 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=238, repairable=true, repairHint=null, description=net.core.somaxconn 检查系统中每一个端口最大的监听队列的长度。建议设置在范围 [2048,16384]|check net.core.somaxconn value.Recommended value: [2048,16384], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.somaxconn","paramValue":"2048"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.somaxconn","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@18dda235 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=239, repairable=true, repairHint=null, description=net.core.netdev_max_backlog 检查在每个网络接口接收数据包的速率比内核处理这些包的速率快时，允许送到队列的数据包的最大数目。建议设置在范围 [500,10000]|check net.core.netdev_max_backlog value.Recommended value: [500,10000], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.netdev_max_backlog","paramValue":"10000"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.netdev_max_backlog","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4da44701 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=240, repairable=true, repairHint=null, description=net.core.rmem_default 检查主机默认的 TCP 数据接收窗口大小（字节）。建议设置在范围 [65536,16777216]|check net.core.rmem_default value.Recommended value: [65536,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.rmem_default","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.rmem_default","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5fba219f 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=241, repairable=true, repairHint=null, description=net.core.wmem_default 检查主机默认的 TCP 数据发送窗口大小（字节）。建议设置在范围 [65536,16777216]|check net.core.wmem_default value.Recommended value: [65536,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.wmem_default","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.wmem_default","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@64be982a 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=242, repairable=true, repairHint=null, description=net.core.rmem_max 检查主机最大的 TCP 数据接收窗口（字节）。建议设置在范围 [8388608,16777216]|check net.core.rmem_max value.Recommended value: [8388608,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.rmem_max","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.rmem_max","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@11874282 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=243, repairable=true, repairHint=null, description=net.core.wmem_max 检查主机最大的 TCP 数据发送窗口（字节）。建议设置在范围 [8388608,16777216]|check net.core.wmem_max value.Recommended value: [8388608,16777216], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.core.wmem_max","paramValue":"16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.core.wmem_max","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7e7493fc 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=245, repairable=true, repairHint=null, description=net.ipv4.tcp_rmem 检查 socket 接收缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_rmem value.Recommended value: 4096 87380 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"4096 87380 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@70ccf374 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=245, repairable=true, repairHint=null, description=net.ipv4.tcp_rmem 检查 socket 接收缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_rmem value.Recommended value: 4096 87380 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"4096 87380 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@60a34a8f 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=245, repairable=true, repairHint=null, description=net.ipv4.tcp_rmem 检查 socket 接收缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_rmem value.Recommended value: 4096 87380 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"4096 87380 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_rmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4b1b874b 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=246, repairable=true, repairHint=null, description=net.ipv4.tcp_wmem 检查 socket 发送缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_wmem value.Recommended value: 4096 65536 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"4096 65536 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@112a16a7 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=246, repairable=true, repairHint=null, description=net.ipv4.tcp_wmem 检查 socket 发送缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_wmem value.Recommended value: 4096 65536 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"4096 65536 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@66a00a5 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=246, repairable=true, repairHint=null, description=net.ipv4.tcp_wmem 检查 socket 发送缓冲区分配的字节数。建议设置在范围 min:[4096,8192], default:[65536,174760], max:[8388608,16777216]|check net.ipv4.tcp_wmem value.Recommended value: 4096 65536 16777216, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"4096 65536 16777216"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_wmem","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7ce7d5d5 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=256, repairable=true, repairHint=null, description=net.ipv4.tcp_max_syn_backlog 检查可保存在队列中未获得对方确认的最大连接请求数目。建议设置在范围：[1024,16384]|check net.ipv4.tcp_max_syn_backlog value.Recommended value: [1024,16384], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_max_syn_backlog","paramValue":"1024"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_max_syn_backlog","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6b5134dd 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=257, repairable=true, repairHint=null, description=net.ipv4.tcp_fin_timeout 检查 TCP 保持在 FIN-WAIT-2 状态的时间（秒）。建议设置在范围：[15,60]|check net.ipv4.tcp_fin_timeout value.Recommended value: [15,60], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"net.ipv4.tcp_fin_timeout","paramValue":"60"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"net.ipv4.tcp_fin_timeout","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@2a0f883e 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=258, repairable=true, repairHint=null, description=vm.min_free_kbytes 检查保留的空闲页面池的大小。建议设置在范围：[32768,2097152]|check vm.min_free_kbytes value.Recommended value: [32768,2097152], repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.min_free_kbytes","paramValue":"2097152"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.min_free_kbytes","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@7cb6daf2 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=259, repairable=true, repairHint=null, description=vm.max_map_count 检查进程可以使用的最大内存映射区域值。建议设置为 655360|check vm.max_map_count value.Recommended value: 655360, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.max_map_count","paramValue":"655360"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.max_map_count","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@3e3de1cc 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=260, repairable=true, repairHint=null, description=fs.aio-max-nr 检查同时可支持的异步 IO 数目。建议设置为 >= 1048576|check fs.aio-max-nr value.Recommended value >= 1048576, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"fs.aio-max-nr","paramValue":"1048576"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"fs.aio-max-nr","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5c28dd0a 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=261, repairable=true, repairHint=null, description=kernel.numa_balancing 检查自动 numa 平衡是否禁用，为避免内存节点数据迁移可能造成的性能抖动，建议禁用|check kernel.numa_balancing value.Recommended value: 0, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"kernel.numa_balancing","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"kernel.numa_balancing","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@ab0e0c9 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=262, repairable=true, repairHint=null, description=vm.zone_reclaim_mode 检查控制内存 zone 回收模式是否禁用，为避免因回收导致的性能抖动，建议禁用|check vm.zone_reclaim_mode value.Recommended value: 0, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"vm.zone_reclaim_mode","paramValue":"0"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"vm.zone_reclaim_mode","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@401bda27 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=263, repairable=true, repairHint=null, description=fs.file-max 检查整个系统能够打开的最大文件数量，建议设置为 >= 6573688|check fs.file-max value.Recommended value >= 6573688, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"fs.file-max","paramValue":"6573688"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"fs.file-max","paramValue":"0"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@11796069 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=264, repairable=true, repairHint=null, description=fs.pipe-user-pages-soft 检查由单个非特权用户创建或设置的所有管道的总大小(以页为单位)的软限制，建议设置为默认值|check fs.pipe-user-pages-soft value.Recommended default value, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, parameters={"paramKey":"fs.pipe-user-pages-soft","paramValue":"16384"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairKernelParams@492c2ea9, rollbackParameters={"paramKey":"fs.pipe-user-pages-soft","paramValue":"16384"}, rollbackReplaceField=paramValue), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@ba32fc1 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=227, repairable=true, repairHint=null, description=检查主机 mtr 命令是否存在，网络测试工具，为运维或检查提供便利。建议安装|Checks whether the mtr command is available on the host. mtr is a network test tool. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"mtr"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@61830d4 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=228, repairable=true, repairHint=null, description=检查主机 tar 命令是否存在，Linux 打包工具，OCP 依赖的重要的运维工具。建议安装|Checks whether the tar command is available on the host. tar is a packaging tool in Linux. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"tar"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@184c18d2 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=229, repairable=true, repairHint=null, description=检查主机是否安装 curl 命令，命令行文本传输工具，为运维或检查提供便利。建议安装|Checks whether the curl command is available on the host. curl is a command-line tool for transferring data. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"curl"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@42ec599a 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=230, repairable=true, repairHint=null, description=检查主机是否安装 nc 命令，netcat 网络工具，OCP 依赖的重要的运维工具。建议安装|Checks whether the NetCat (nc) command is available on the host. nc is a networking tool. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"nc"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@5023ab21 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=231, repairable=true, repairHint=null, description=检查主机是否安装 binutils 二进制工具集，为运维或检查提供便利。建议安装|Checks whether the binutils binary toolkit is installed on the host. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"binutils"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@177aaf61 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=232, repairable=true, repairHint=null, description=检查主机是否安装 bind-utils DNS 工具集，为运维或检查提供便利。建议安装|Checks whether the bind-utils toolkit is installed on the host. The bind-utils toolkit contains a collection of utilities for querying DNS name servers. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"bind-utils"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6f08bfd4 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=233, repairable=true, repairHint=null, description=检查主机是否安装 iproute 网络管理工具包，为运维或检查提供便利。建议安装|Checks whether the iproute network management toolkit is installed on the host. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"iproute"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@4f081b26 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=244, repairable=true, repairHint=null, description=libatomic 是 OceanBase 运行依赖的动态链接库，必须安装|libatomic dynamic-link library (DLL) is a dependency of OceanBase Database and must be installed., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"libatomic"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@23fcb45 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=247, repairable=false, repairHint=null, description=检查指定的用户（系统参数 ocp.operation.default.os.user）是否存在|check if user(ocp.operation.default.os.user in System Parameters) exists, repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@407c6dc4 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=248, repairable=true, repairHint=null, description=检查 root 用户 openFiles 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 655350| check if (user root) limit of openFiles is set. Recommend value: 655350, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"openFiles","userName":"root","expectedValue":"655350"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6aa08c00 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=249, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）openFiles 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 655350| check if (ocp.operation.default.os.user in System Parameters) limit of openFiles is set. Recommend value: 655350, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"openFiles","userName":"admin","expectedValue":"655350"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@657e7b59 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=250, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）stackSize 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 10240| check if (ocp.operation.default.os.user in System Parameters) limit of stackSize is set. Recommend value: 10240, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"stackSize","userName":"admin","expectedValue":"10240"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@19e7f150 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=251, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）maxUserProcesses 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 655360| check if (ocp.operation.default.os.user in System Parameters) limit of maxUserProcesses is set. Recommend value: 655360, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"maxUserProcesses","userName":"admin","expectedValue":"655360"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@3a56bde5 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=252, repairable=true, repairHint=null, description=检查指定用户（系统参数 ocp.operation.default.os.user）coreFileSize 系统限制。系统限制主要用于限制 shell 启动进程所占用的资源，建议设置为 unlimited| check if (ocp.operation.default.os.user in System Parameters) limit of coreFileSize is set. Recommend value: unlimited, repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, parameters={"option":"coreFileSize","userName":"admin","expectedValue":"unlimited"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairProcessLimits@44249f1a, rollbackParameters={"isReset":true}, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@54f8f623 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=253, repairable=true, repairHint=null, description=检查 SELinux 是否启用，建议用户禁用 selinux，SELinux 使用所谓的委任式访问控制 (Mandatory Access Control, MAC) ，可以针对特定的程序与特定的档案资源来进行权限的控管| check if selinux is disabled., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSelinux@1c78e5d6, parameters={"state":"disabled"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairSelinux@1c78e5d6, rollbackParameters={"state":"disabled"}, rollbackReplaceField=state), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@42caeb56 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=254, repairable=false, repairHint=null, description=OceanBase 数据盘文件类型，磁盘容量 > 16T 时应为 xfs 类型|File Type of OceanBase Data Disk. File system type should be xfs if disk size > 16T, repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6401ef2e 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=265, repairable=false, repairHint=null, description=检查操作系统版本，一般推荐使用标准化主机文档中列举的操作系统版本 |check OS version. Recommend using OS versions listed in document, repairMethod=null, parameters=, rollbackRepairMethod=null, rollbackParameters=, rollbackReplaceField=), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6b37cfde 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=266, repairable=true, repairHint=null, description=检查主机是否安装 fio 命令，磁盘压测工具，OBServer 依赖的重要的运维工具。建议安装|Checks whether the fio command is available on the host. fio is a disk pressure test tool. We recommend that you install it., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairPackageNetInstall@2914f4a, parameters={"name":"fio"}, rollbackRepairMethod=null, rollbackParameters=null, rollbackReplaceField=name), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@78ed7575 2024-09-11 17:58:48.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.c.i.h.SystemCheckerHelperImpl : repairPlan:ItemRepairPlan(checkItemId=277, repairable=true, repairHint=null, description=检查主机 clock source 配置。建议设置为 tsc 类型（aarch64 推荐 arch_sys_conter）|Checks host OS clock source config. We recommend that you set it to tsc type(arch_sys_conter on aarch64)., repairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairClockSource@43f6ef2d, parameters={"sourceType":"tsc"}, rollbackRepairMethod=com.oceanbase.ocp.compute.checker.internal.repair.methods.RepairClockSource@43f6ef2d, rollbackParameters={"isReset":"true", "sourceType":""}, rollbackReplaceField=sourceType), itemOverview:com.oceanbase.ocp.inspection.model.InspectionItem@6de8b165 2024-09-11 17:58:48.179 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=repair system, nodeCount=4, taskCount=4 2024-09-11 17:58:48.179 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:58:48.179 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='repair system', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=null, tenantId=null, hostId=2000010, creator='admin', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false, system_checker_operation=REPAIR, system_checker_record_id=48, host_id=2000010}, listMap={}}, createTime=null, startTime=2024-09-11T17:58:48.179+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=SELinux config, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.RepairCheckItemTask, seriesId=1, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Before repair system, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.BeforeRepairSystemInfoTask, seriesId=0, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name= clock source checks, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.RepairCheckItemTask, seriesId=2, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=After repair system, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.compute.checker.internal.task.AfterRepairSystemInfoTask, seriesId=3, startTime=null, endTime=null}], template=null, traceId='0e817ce8ba29f7e3', operation=EXECUTE} 2024-09-11 17:58:48.225 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=4048202, taskName=repair system, argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false, system_checker_operation=REPAIR, system_checker_record_id=48, host_id=2000010}, listMap={}} 2024-09-11 17:58:48.267 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0e817ce8ba29f7e3,86d055e5ea79c636] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hosts/2000010/repairSystemInfo client=10.31.33.41, traceId=0e817ce8ba29f7e3, duration=319 ms] 2024-09-11 17:58:48.388 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,4f14eff63f36d2e3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048197, operation:EXECUTE, state: RUNNING 2024-09-11 17:58:48.427 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,e70fbf18c4f005bb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048197, context=Context{parallelIdx=-1, stringMap={system_checker_operation=REPAIR, task_instance_id=4048202, task_operation=execute, latest_execution_start_time=2024-09-11T17:58:48.384+08:00, sub_task_instance_name=Before repair system, system_checker_record_id=48, sub_task_instance_id=4048197, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:58:48.436 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,e70fbf18c4f005bb] c.o.o.c.c.i.t.BeforeRepairSystemInfoTask : update system checker status successfully, hostId: 2000010, recordId: 48 2024-09-11 17:58:48.447 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,e70fbf18c4f005bb] c.o.o.c.c.i.t.BeforeRepairSystemInfoTask : update system checker record successfully, hostId: 2000010, recordId: 48 2024-09-11 17:58:48.459 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,e70fbf18c4f005bb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048197, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:58:48.548 INFO 2981023 --- [ocp-schedules-5,2d963c7be1e80beb,07e3c7a50735ddb1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:58:48.619 WARN 2981023 --- [scanner-13582543-1,2d963c7be1e80beb,9453406fae3165f7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:58:48.637 INFO 2981023 --- [alarm-detect-17,2d963c7be1e80beb,3e5930bdc2cb8527] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645094, data is empty. 2024-09-11 17:58:48.643 INFO 2981023 --- [alarm-detect-17,2d963c7be1e80beb,3e5930bdc2cb8527] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645094, data is empty. 2024-09-11 17:58:48.648 INFO 2981023 --- [alarm-detect-17,2d963c7be1e80beb,3e5930bdc2cb8527] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645094, data is empty. 2024-09-11 17:58:48.666 INFO 2981023 --- [alarm-detect-1,2d963c7be1e80beb,8e2bdc67e5250250] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645094, data is empty. 2024-09-11 17:58:48.666 INFO 2981023 --- [alarm-detect-15,2d963c7be1e80beb,e4e40f5425d73a96] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645094, data is empty. 2024-09-11 17:58:48.670 INFO 2981023 --- [alarm-detect-15,2d963c7be1e80beb,e4e40f5425d73a96] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645094, data is empty. 2024-09-11 17:58:48.672 INFO 2981023 --- [alarm-detect-1,2d963c7be1e80beb,8e2bdc67e5250250] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645094, data is empty. 2024-09-11 17:58:48.678 INFO 2981023 --- [alarm-detect-15,2d963c7be1e80beb,e4e40f5425d73a96] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645094, data is empty. 2024-09-11 17:58:48.679 INFO 2981023 --- [alarm-detect-1,2d963c7be1e80beb,8e2bdc67e5250250] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645094, data is empty. 2024-09-11 17:58:48.755 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3e8fe6e34443bc07,0159504d2f98c420] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=3e8fe6e34443bc07, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:58:48.760 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3e8fe6e34443bc07,0159504d2f98c420] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:58:48.777 INFO 2981023 --- [alarm-detect-7,2d963c7be1e80beb,94b8ef8b51924e92] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645094, data is empty. 2024-09-11 17:58:48.781 INFO 2981023 --- [alarm-detect-7,2d963c7be1e80beb,94b8ef8b51924e92] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645094, data is empty. 2024-09-11 17:58:48.783 INFO 2981023 --- [alarm-detect-7,2d963c7be1e80beb,94b8ef8b51924e92] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645094, data is empty. 2024-09-11 17:58:48.820 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3e8fe6e34443bc07,0159504d2f98c420] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=3e8fe6e34443bc07, duration=74 ms] 2024-09-11 17:58:49.324 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048196] 2024-09-11 17:58:49.336 INFO 2981023 --- [ocp-async-3,364f64fa2556f84b,5382c978247b044b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:49.416 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,7293a53cc3bc86ea] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048196, operation:EXECUTE, state: RUNNING 2024-09-11 17:58:49.441 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,9922877efb096da8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048196, context=Context{parallelIdx=-1, stringMap={system_checker_operation=REPAIR, task_instance_id=4048202, task_operation=execute, latest_execution_start_time=2024-09-11T17:58:49.411+08:00, sub_task_instance_name=SELinux config, system_checker_record_id=48, sub_task_instance_id=4048196, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:58:49.463 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,9922877efb096da8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:49.471 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,9922877efb096da8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:49.530 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,9922877efb096da8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/setSelinux, request body:SetSelinuxRequest(state=disabled), params:null 2024-09-11 17:58:49.556 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,9922877efb096da8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048196, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:58:49.737 INFO 2981023 --- [host-check-clockdiff1,773c8ebcb1dc46d6,6d923a654d3b05ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048729 6 6 , error= 2024-09-11 17:58:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:58:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:58:50.286 INFO 2981023 --- [ocp-schedules-20,2ea43ce6e81cf882,7a15f86be2706534] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:58:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,ec4ba07b7b711092,992083ec76fda805] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,ec4ba07b7b711092,992083ec76fda805] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:50.337 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048198] 2024-09-11 17:58:50.432 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,9468d54947ed106d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048198, operation:EXECUTE, state: RUNNING 2024-09-11 17:58:50.453 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,d68e0ac34264f11a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048198, context=Context{parallelIdx=-1, stringMap={system_checker_operation=REPAIR, task_instance_id=4048202, task_operation=execute, latest_execution_start_time=2024-09-11T17:58:50.431+08:00, sub_task_instance_name= clock source checks, system_checker_record_id=48, sub_task_instance_id=4048198, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:58:50.465 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,d68e0ac34264f11a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:58:50.470 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,d68e0ac34264f11a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:58:50.484 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,d68e0ac34264f11a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/setClockSource, request body:SetClockSourceRequest(sourceType=tsc), params:null 2024-09-11 17:58:50.506 ERROR 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,d68e0ac34264f11a] c.o.o.c.c.i.r.methods.RepairClockSource : set clock source to tsc failed: [AgentClient]:http request is failed, response:Unexpected error: symlink /usr/lib/systemd/system/set_clocksource.service /etc/systemd/system/multi-user.target.wants/set_clocksource.service: file exists 2024-09-11 17:58:50.515 ERROR 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,d68e0ac34264f11a] c.o.o.c.c.i.h.SystemCheckerHelperImpl : Failed to repair 277. Please see the log for details 2024-09-11 17:58:50.520 ERROR 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,d68e0ac34264f11a] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.common.unexpected, and args [2000010] 2024-09-11 17:58:50.524 ERROR 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,d68e0ac34264f11a] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : An unknown error has occurred. Cause: 2,000,010. Error message: {1}. Contact the administrator. com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=COMMON_UNEXPECTED, args=2000010 at sun.reflect.GeneratedConstructorAccessor732.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:71) at com.oceanbase.ocp.compute.checker.internal.task.RepairCheckItemTask.run(RepairCheckItemTask.java:59) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 17:58:50.533 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,d68e0ac34264f11a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048198, operation:EXECUTE, state: DISREGARDED 2024-09-11 17:58:50.730 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:50.732 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:51.336 INFO 2981023 --- [ocp-async-7,f1ae479e80be24cb,e0a5bda684aa592b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:51.349 INFO 2981023 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048199] 2024-09-11 17:58:51.451 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,d4d83dff6a9a985f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048199, operation:EXECUTE, state: RUNNING 2024-09-11 17:58:51.470 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,137248ab08fd6512] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048199, context=Context{parallelIdx=-1, stringMap={system_checker_operation=REPAIR, task_instance_id=4048202, task_operation=execute, latest_execution_start_time=2024-09-11T17:58:51.449+08:00, sub_task_instance_name=After repair system, system_checker_record_id=48, sub_task_instance_id=4048199, host_id=2000010}, listMap={}}, executor=10.31.35.10 2024-09-11 17:58:51.475 WARN 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,137248ab08fd6512] c.o.o.c.c.i.t.AfterRepairSystemInfoTask : Some check items failed to be repaired on host(hostId: 2000010). Please see the log for details 2024-09-11 17:58:51.489 INFO 2981023 --- [manual-subtask-executor16,0e817ce8ba29f7e3,137248ab08fd6512] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048199, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:58:51.585 INFO 2981023 --- [ocp-schedules-18,03e70379c6d14c81,0b0e39d558a538ff] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:58:51.586 INFO 2981023 --- [ocp-schedules-18,03e70379c6d14c81,0b0e39d558a538ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:58:51.586 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:58:51.586 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:58:51.586 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:58:51.586 INFO 2981023 --- [ocp-schedules-18,03e70379c6d14c81,0b0e39d558a538ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:58:51.996 INFO 2981023 --- [host-check-clockdiff4,773c8ebcb1dc46d6,31097caf2b590add] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048731 0 0 , error= 2024-09-11 17:58:52.355 INFO 2981023 --- [ocp-schedules-12,e52a0f1415c23006,ddddbf5e70ae7230] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:58:52.406 INFO 2981023 --- [task-coordinator3,,] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"activeAt":"2024-09-11T09:57:49Z","alarmTemplateId":0,"alarmType":"host_operation","appType":"OCP","containSelfcure":false,"description":"Host: 10.31.35.7, alert: repair system SUCCESS, operator was admin, result was success.","gmtCreate":"2024-09-11T09:57:49Z","gmtModified":"2024-09-11T09:57:49.001476Z","id":6007106,"labels":{"app_type":"OCP","svr_ip":"10.31.35.7","alarm_status_native":"Active","alarm_level":"5","operator":"admin","alarm_summary_zh_tw":"alarm_template_id=0:host=10.31.35.7:operation_name=repair system SUCCESS","alarm_summary_en_us":"alarm_template_id=0:host=10.31.35.7:operation_name=repair system SUCCESS","alarm_type":"host_operation","alarm_status":"Active","alarm_group_zh_tw":"提示","host":"10.31.35.7","alarm_scope":"Host","task_result":"success","alarm_scope_native":"Host","alarm_description_zh_tw":"伺服器：10.31.35.7，警示：repair system SUCCESS, 操作者 admin，運行結果：success。","app":"OCP","alarm_duration":"63","alarm_name_en_us":"Host operation info","operation_result":"SUCCESS","message_type":"Alarm","alarm_status_zh_cn":"告警中","alarm_scope_zh_tw":"主機","alarm_group_en_us":"info","alarm_description_en_us":"Host: 10.31.35.7, alert: repair system SUCCESS, operator was admin, result was success.","__host.id":"2000010","alarm_scope_zh_cn":"主机","alarm_level_zh_cn":"提醒","alarm_summary_zh_cn":"alarm_template_id=0:host=10.31.35.7:operation_name=repair system SUCCESS","alarm_description":"Host: 10.31.35.7, alert: repair system SUCCESS, operator was admin, result was success.","ocp_ip":"10.31.35.10","alarm_name":"Host operation info","alarm_status_zh_tw":"警示中","task_id":"4048202","alarm_level_zh_tw":"提醒","alarm_level_native":"5","alarm_scope_en_us":"Host","alarm_level_en_us":"Info","auto_resolved":"1","alarm_summary":"alarm_template_id=0:host=10.31.35.7:operation_name=repair system SUCCESS","alarm_active_at":"2024-09-11T09:57:49Z","alarm_target":"alarm_template_id=0:host=10.31.35.7:operation_name=repair system","alarm_name_zh_tw":"伺服器運維操作提醒","alarm_is_build_in_rule":"true","host_ip":"10.3 2024-09-11 17:58:52.406 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048202, status:SUCCESS 2024-09-11 17:58:52.409 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048202, status:SUCCESS 2024-09-11 17:58:53.208 INFO 2981023 --- [ocp-schedules-22,8306d1bd9ecfe121,20cc5f9e040d8593] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:58:53.208 INFO 2981023 --- [ocp-schedules-22,8306d1bd9ecfe121,20cc5f9e040d8593] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:58:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,64a76dc638909494,5cc700797068ba7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,64a76dc638909494,5cc700797068ba7c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:53.336 INFO 2981023 --- [ocp-async-4,78398361507c4744,5f8997fa1f107025] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:53.732 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:53.734 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:53.925 INFO 2981023 --- [ocp-schedules-12,16e739cf46bd0118,1bc16b0637b3a470] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:58:53.944 WARN 2981023 --- [ocp-schedules-12,16e739cf46bd0118,1bc16b0637b3a470] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:58:54.276 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,485a813b7bd08fb7,45f7a751ea96cca3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=485a813b7bd08fb7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:58:54.277 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-8,485a813b7bd08fb7,45f7a751ea96cca3] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:58:54.307 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,485a813b7bd08fb7,45f7a751ea96cca3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=485a813b7bd08fb7, duration=35 ms] 2024-09-11 17:58:54.346 INFO 2981023 --- [ocp-schedules-28,3f80b5a3c053bc7e,48182217f11d7e4d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:58:54.759 INFO 2981023 --- [ocp-schedules-24,31bd29d012115a6e,dd1582eb029ffa86] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T17:58:54.759+08:00, tenantSize=6 2024-09-11 17:58:54.761 INFO 2981023 --- [ocp-schedules-24,31bd29d012115a6e,dd1582eb029ffa86] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726048734759000 2024-09-11 17:58:54.761 INFO 2981023 --- [ocp-schedules-24,31bd29d012115a6e,dd1582eb029ffa86] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726048434736000 2024-09-11 17:58:54.781 INFO 2981023 --- [ocp-schedules-29,fb39867dab85b27e,13ef13c728ad3c53] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:58:54.792 INFO 2981023 --- [ocp-schedules-24,31bd29d012115a6e,dd1582eb029ffa86] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 17:58:54.797 INFO 2981023 --- [ocp-schedules-29,fb39867dab85b27e,13ef13c728ad3c53] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:58:54.797 INFO 2981023 --- [ocp-schedules-29,fb39867dab85b27e,13ef13c728ad3c53] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:58:55.336 INFO 2981023 --- [ocp-async-5,c5fd019b60f8088e,3fb5e30e87aa686f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:55.816 INFO 2981023 --- [ocp-schedules-9,2dbc65da3a8cd07b,450e6e54ecacdbc1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:58:55.816 INFO 2981023 --- [ocp-schedules-9,2dbc65da3a8cd07b,450e6e54ecacdbc1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690420679617835008 2024-09-11 17:58:55.817 INFO 2981023 --- [ocp-schedules-9,2dbc65da3a8cd07b,450e6e54ecacdbc1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:58:55.817 INFO 2981023 --- [ocp-schedules-9,2dbc65da3a8cd07b,450e6e54ecacdbc1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:58:55.817 INFO 2981023 --- [ocp-schedules-9,2dbc65da3a8cd07b,450e6e54ecacdbc1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:58:55.817 INFO 2981023 --- [ocp-schedules-9,2dbc65da3a8cd07b,450e6e54ecacdbc1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:58:55.943 INFO 2981023 --- [ocp-schedules-3,7f681395ac9c75d0,7a27eb47ea95814c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:58:55.950 INFO 2981023 --- [ocp-schedules-3,7f681395ac9c75d0,7a27eb47ea95814c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:58:55.950 INFO 2981023 --- [ocp-schedules-3,7f681395ac9c75d0,7a27eb47ea95814c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:58:56.218 INFO 2981023 --- [ocp-schedules-20,8c81917b0211a828,58a9e1c6b0f651fe] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:58:56.218+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:58:56.221 INFO 2981023 --- [ocp-schedules-20,8c81917b0211a828,58a9e1c6b0f651fe] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:58:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,40a8a93303b8ecbe,4a4cf22d0df1c9df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:56.320 INFO 2981023 --- [prometheus-minute-schedule-1,40a8a93303b8ecbe,4a4cf22d0df1c9df] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:56.357 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:58:56.356115+08:00, lastProcessTime=2024-09-11T17:58:46.469496+08:00 2024-09-11 17:58:56.381 INFO 2981023 --- [ocp-schedules-20,8c81917b0211a828,58a9e1c6b0f651fe] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 54 to SqlAggregateData 2024-09-11 17:58:56.383 INFO 2981023 --- [ocp-schedules-20,8c81917b0211a828,58a9e1c6b0f651fe] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:58:56.488 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:58:56.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:58:56.356115+08:00 2024-09-11 17:58:56.734 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:56.735 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:58:57.028 INFO 2981023 --- [ocp-schedules-12,6f78b52d38d404ff,58ed6eda716966d9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:58:57.029 INFO 2981023 --- [ocp-schedules-12,6f78b52d38d404ff,58ed6eda716966d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:58:57.068 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:58:57.069 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.069 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:58:57.069 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:58:57.069 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.069 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.072 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:58:57.073 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:58:57.073 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:58:57.076 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.076 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.076 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.077 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.077 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.077 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.084 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.084 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.088 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.088 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.089 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.089 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.090 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.091 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:58:57.097 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.097 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:58:57.100 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.100 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.100 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:58:57.101 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4171D723-0-0] 2024-09-11 17:58:57.101 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:58:57.101 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.101 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:58:57.103 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.103 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.103 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:58:57.103 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.110 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.110 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.110 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:58:57.110 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:58:57.111 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.111 INFO 2981023 --- [ocp-schedules-27,91485a1a78f2ba91,e6be1cb3b3d458a9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:58:57.111 INFO 2981023 --- [ocp-schedules-27,91485a1a78f2ba91,e6be1cb3b3d458a9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:58:57.112 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:58:57.112 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.113 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:58:57.113 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:58:57.114 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.114 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:58:57.117 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:58:57.124 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.125 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:58:57.125 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.126 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:58:57.126 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.127 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:58:57.127 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.128 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:58:57.128 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.128 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:58:57.129 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:58:57.129 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.129 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4171D72D-0-0] 2024-09-11 17:58:57.129 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:58:57.129 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:58:57.130 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.131 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:58:57.131 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.132 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:58:57.132 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.133 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:58:57.133 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.134 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:58:57.134 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:58:57.137 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:57.137 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.137 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:58:57.138 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:58:57.139 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.140 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.140 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:58:57.140 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.140 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.141 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:58:57.142 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.142 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:58:57.143 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.144 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:58:57.145 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.146 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:58:57.146 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:58:57.149 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:58:57.156 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:58:57.157 INFO 2981023 --- [Collect all tenants major compactions1,91485a1a78f2ba91,1ecb962abffdcf9b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:58:57.159 INFO 2981023 --- [Collect all tenants major compactions2,91485a1a78f2ba91,ce22e72e78962dcc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:58:57.160 INFO 2981023 --- [Collect all tenants major compactions3,91485a1a78f2ba91,d2226d5e19b418dd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:58:57.160 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:57.160 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:58:57.160 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:58:57.160 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:58:57.162 INFO 2981023 --- [Collect all tenants major compactions5,91485a1a78f2ba91,78046d55284e2e7b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:58:57.163 INFO 2981023 --- [Collect all tenants major compactions4,91485a1a78f2ba91,9eeb468469613e76] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:58:57.164 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.164 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:58:57.165 INFO 2981023 --- [Collect all tenants major compactions6,91485a1a78f2ba91,6697a858d93f1743] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:58:57.172 INFO 2981023 --- [Collect all tenants major compactions3,91485a1a78f2ba91,d2226d5e19b418dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:58:57.173 INFO 2981023 --- [Collect all tenants major compactions4,91485a1a78f2ba91,9eeb468469613e76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.173 INFO 2981023 --- [Collect all tenants major compactions3,91485a1a78f2ba91,d2226d5e19b418dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:58:57.174 INFO 2981023 --- [Collect all tenants major compactions5,91485a1a78f2ba91,78046d55284e2e7b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.174 INFO 2981023 --- [Collect all tenants major compactions4,91485a1a78f2ba91,9eeb468469613e76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:58:57.175 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:58:57.175 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.177 INFO 2981023 --- [Collect all tenants major compactions4,91485a1a78f2ba91,9eeb468469613e76] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:58:57.177 INFO 2981023 --- [Collect all tenants major compactions4,91485a1a78f2ba91,9eeb468469613e76] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:58:57.177 INFO 2981023 --- [Collect all tenants major compactions3,91485a1a78f2ba91,d2226d5e19b418dd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:58:57.177 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:58:57.177 INFO 2981023 --- [Collect all tenants major compactions3,91485a1a78f2ba91,d2226d5e19b418dd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:58:57.179 INFO 2981023 --- [Collect all tenants major compactions1,91485a1a78f2ba91,1ecb962abffdcf9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.180 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:58:57.180 INFO 2981023 --- [Collect all tenants major compactions1,91485a1a78f2ba91,1ecb962abffdcf9b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:58:57.182 INFO 2981023 --- [Collect all tenants major compactions1,91485a1a78f2ba91,1ecb962abffdcf9b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:58:57.182 INFO 2981023 --- [Collect all tenants major compactions1,91485a1a78f2ba91,1ecb962abffdcf9b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:58:57.182 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.183 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:58:57.183 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:58:57.184 INFO 2981023 --- [Collect all tenants major compactions5,91485a1a78f2ba91,78046d55284e2e7b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:58:57.185 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:58:57.186 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.187 INFO 2981023 --- [Collect all tenants major compactions5,91485a1a78f2ba91,78046d55284e2e7b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:58:57.187 INFO 2981023 --- [Collect all tenants major compactions5,91485a1a78f2ba91,78046d55284e2e7b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:58:57.192 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.194 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:58:57.196 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:58:57.196 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.197 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:58:57.199 INFO 2981023 --- [Collect all tenants major compactions6,91485a1a78f2ba91,6697a858d93f1743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.200 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:58:57.200 INFO 2981023 --- [Collect all tenants major compactions6,91485a1a78f2ba91,6697a858d93f1743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:58:57.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:58:57.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:58:57.202 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:58:57.202 INFO 2981023 --- [Collect all tenants major compactions2,91485a1a78f2ba91,ce22e72e78962dcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:58:57.204 INFO 2981023 --- [Collect all tenants major compactions2,91485a1a78f2ba91,ce22e72e78962dcc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:58:57.206 INFO 2981023 --- [Collect all tenants major compactions2,91485a1a78f2ba91,ce22e72e78962dcc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:58:57.206 INFO 2981023 --- [Collect all tenants major compactions2,91485a1a78f2ba91,ce22e72e78962dcc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:58:57.208 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:58:57.210 INFO 2981023 --- [Collect all tenants major compactions6,91485a1a78f2ba91,6697a858d93f1743] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:58:57.210 INFO 2981023 --- [Collect all tenants major compactions6,91485a1a78f2ba91,6697a858d93f1743] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:58:57.211 INFO 2981023 --- [ocp-schedules-27,91485a1a78f2ba91,e6be1cb3b3d458a9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:58:57.212 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:58:57.212 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:58:57.212 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:58:57.214 INFO 2981023 --- [ocp-schedules-12,6f78b52d38d404ff,58ed6eda716966d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:58:57.336 INFO 2981023 --- [ocp-async-6,8431a6b33a88e9be,ffe291f054166cf1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:57.769 INFO 2981023 --- [ocp-async-5,33e64262acd33fd2,50ac27429f217b5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:58:57.864 INFO 2981023 --- [ocp-schedules-25,680886f8b4d65a70,3d5dd0386d6e2966] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:58:58.841 INFO 2981023 --- [ocp-schedules-16,cdd9fb97e09c29cc,6916afa00d4965a4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:58:58.908 WARN 2981023 --- [scanner-13582908-1,cdd9fb97e09c29cc,574e96fe6e858d6d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:58:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,01b8db4bbb420c89,2d3a82255526ba00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048740, interval=60 2024-09-11 17:58:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,01b8db4bbb420c89,2d3a82255526ba00] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:58:59.336 INFO 2981023 --- [ocp-async-4,f19984d8800db196,98fc431a1fe192a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:58:59.735 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:58:59.737 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:01.338 INFO 2981023 --- [ocp-async-1,eecbc115728a7e45,d30695bbb9b10db9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,6900a8c7103f6463,6b7bf3bea7c722d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,6900a8c7103f6463,6b7bf3bea7c722d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:02.359 INFO 2981023 --- [ocp-schedules-30,8bb12834a531a1dc,2503fc8152ab380e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:59:02.737 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:02.737 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:03.337 INFO 2981023 --- [ocp-async-10,5b675f7ea0028614,3f37bc5ce3a04201] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,1b8f10641c4ea23f,006d859ca797f94e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,1b8f10641c4ea23f,006d859ca797f94e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:05.337 INFO 2981023 --- [ocp-async-9,e05522cb660fbf6c,5efee5c93c70805f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:05.738 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:05.739 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:59:06.352152+08:00, lastProcessTime=2024-09-11T17:58:56.356115+08:00 2024-09-11 17:59:06.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:59:06.352152+08:00 2024-09-11 17:59:07.117 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:55:06.757, end=2024-09-11T17:57:07.112} 2024-09-11 17:59:07.123 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdoc 2024-09-11 17:59:07.124 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:59:07.125 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:55:06.757, end=2024-09-11T17:57:07.112},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:59:07.125 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:59:07.125 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:59:07.126 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:59:07.128 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:55:06.757, end=2024-09-11T17:57:07.112},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:59:07.128 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:59:07.128 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:59:07.189 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347189000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.189Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.193 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347193000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.193Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.193 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347193000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.193Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.206 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347206000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.206Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.206 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347206000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.206Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.212 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:59:07.212 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.215 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:59:07.215 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.215 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347215000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.215Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.220 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347220000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.220Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.220 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347220000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.220Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.221 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:59:07.221 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.228 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:59:07.228 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.228 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:59:07.228 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.229 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.229 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.232 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.232 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.236 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.236 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.239 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:59:07.239 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.240 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.240 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.243 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 64 2024-09-11 17:59:07.243 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.248 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:59:07.248 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.249 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.249 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.250 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.250 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.250 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.250 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.250 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.250 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.250 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.251 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.254 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.254 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.254 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.255 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.257 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.257 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.262 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.262 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.262 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.262 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.265 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.265 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.266 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.266 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.266 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.267 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.267 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.268 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.268 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.268 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.268 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.268 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.268 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.270 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.270 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.270 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.271 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.271 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.271 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.272 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.272 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.273 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.273 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.206, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.276 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.189, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.277 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.193, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.279 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.206, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.279 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.193, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.281 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.215, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.286 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.286 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.291 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.291 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.291 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.304 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347304000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.304Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.305 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347305000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.305Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.307 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.314 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:59:07.314 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.316 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347316000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.316Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.319 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347319000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.319Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.320 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.22, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.326 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.326 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.328 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.22, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.330 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347330000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.330Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.331 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.331 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.331 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:59:07.331 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.332 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347332000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.332Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.334 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:59:07.334 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.337 INFO 2981023 --- [ocp-async-4,3180678ea7f5f015,760c1764fc073261] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:07.338 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:59:07.338 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.344 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.344 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.345 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.345 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.348 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.348 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.350 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.355 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 17:59:07.355 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.357 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:59:07.357 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.369 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.369 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.370 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.370 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.370 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347370000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.370Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.370 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.370 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.374 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.374 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.375 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.375 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.376 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.377 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.304, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.379 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.379 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.379 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.379 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.382 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.382 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.383 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.384 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 17:59:07.384 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.384 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.384 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.384 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.384 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.387 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.305, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.388 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.316, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.392 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.392 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.393 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.393 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.394 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.394 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.394 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.395 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.395 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.396 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.396 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.398 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.398 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.399 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.332, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.400 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.33, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.401 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.319, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.404 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.404 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.405 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 17:59:07.408 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.37, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.409 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:59:07.411 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:55:06.757, end=2024-09-11T17:57:07.112},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:59:07.411 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:59:07.411 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:59:07.421 INFO 2981023 --- [plan_change_detector18,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:59:07.421 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:59:07.421 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:55:06.757, end=2024-09-11T17:57:07.112},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:59:07.421 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:59:07.421 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:59:07.422 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:59:07.424 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:55:06.757, end=2024-09-11T17:57:07.112},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:59:07.424 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:59:07.424 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:59:07.435 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 17:59:07.450 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962347450000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T09:59:07.450Z, endTime=2024-09-11T09:57:07.112Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 17:59:07.452 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 17:59:07.452 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 17:59:07.458 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.458 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.459 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 17:59:07.459 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 17:59:07.465 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 17:59:07.465 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048627112000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 17:59:07.466 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 17:59:07.466 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T17:59:07.45, end=2024-09-11T17:57:07.112}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 17:59:07.466 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 17:59:07.467 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:55:06.757, end=2024-09-11T17:57:07.112},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 17:59:07.467 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 17:59:07.467 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 17:59:07.468 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 17:59:07.671 INFO 2981023 --- [ocp-schedules-24,5eb435e78565b8a5,0613c43cb59eff1a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 17:59:07.745 INFO 2981023 --- [ocp-schedules-11,9571f82fed18939a,97e76a6ef71cced5] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T17:59:07.745+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:59:07.745 INFO 2981023 --- [ocp-schedules-11,9571f82fed18939a,97e76a6ef71cced5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 17:59:07.747 INFO 2981023 --- [ocp-schedules-11,9571f82fed18939a,97e76a6ef71cced5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 17:59:07.747 INFO 2981023 --- [ocp-schedules-11,9571f82fed18939a,97e76a6ef71cced5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 17:59:07.770 INFO 2981023 --- [ocp-async-9,d8ff30a173f514b7,cec6f69eece27417] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:59:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,edee5dedba21eb1e,88bd1ed761c45865] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,edee5dedba21eb1e,88bd1ed761c45865] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:08.739 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:08.741 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:09.045 INFO 2981023 --- [ocp-schedules-28,eb61b52617f9acf3,c149fb707877f2b7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:59:09.101 WARN 2981023 --- [scanner-13583126-1,eb61b52617f9acf3,608678f4cc056bb4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:59:09.116 INFO 2981023 --- [alarm-detect-6,eb61b52617f9acf3,faae797e20f83273] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645094, data is empty. 2024-09-11 17:59:09.120 INFO 2981023 --- [alarm-detect-6,eb61b52617f9acf3,faae797e20f83273] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645148, data is empty. 2024-09-11 17:59:09.125 INFO 2981023 --- [alarm-detect-6,eb61b52617f9acf3,faae797e20f83273] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645148, data is empty. 2024-09-11 17:59:09.143 INFO 2981023 --- [alarm-detect-12,eb61b52617f9acf3,336f120ceb1450ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645094, data is empty. 2024-09-11 17:59:09.145 INFO 2981023 --- [alarm-detect-12,eb61b52617f9acf3,336f120ceb1450ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645148, data is empty. 2024-09-11 17:59:09.185 INFO 2981023 --- [alarm-detect-12,eb61b52617f9acf3,336f120ceb1450ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645148, data is empty. 2024-09-11 17:59:09.192 INFO 2981023 --- [alarm-detect-20,eb61b52617f9acf3,1c48f98151f36c65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645094, data is empty. 2024-09-11 17:59:09.198 INFO 2981023 --- [alarm-detect-20,eb61b52617f9acf3,1c48f98151f36c65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645148, data is empty. 2024-09-11 17:59:09.206 INFO 2981023 --- [alarm-detect-20,eb61b52617f9acf3,1c48f98151f36c65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645148, data is empty. 2024-09-11 17:59:09.284 INFO 2981023 --- [alarm-detect-1,eb61b52617f9acf3,2b1800ca7ec3ea5a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645094, data is empty. 2024-09-11 17:59:09.288 INFO 2981023 --- [alarm-detect-1,eb61b52617f9acf3,2b1800ca7ec3ea5a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645148, data is empty. 2024-09-11 17:59:09.292 INFO 2981023 --- [alarm-detect-1,eb61b52617f9acf3,2b1800ca7ec3ea5a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645148, data is empty. 2024-09-11 17:59:09.337 INFO 2981023 --- [ocp-async-10,35e0fc82a5ba76c1,5af402d26f2ac8a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:59:10.003 INFO 2981023 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:59:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 17:59:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 17:59:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,1f40dcbe325f6db8,f338593dfa4f8c1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,1f40dcbe325f6db8,f338593dfa4f8c1d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:11.337 INFO 2981023 --- [ocp-async-9,a29fd10a13e6fb69,306173545e7dd239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:11.359 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048680, nodeCount=285, affectRows=285 2024-09-11 17:59:11.655 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048680, nodeCount=195, affectRows=195 2024-09-11 17:59:11.741 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:11.742 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:12.362 INFO 2981023 --- [ocp-schedules-15,849953b6304abfe6,9599eb5b0aeec606] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:59:12.909 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048680, nodeCount=195, validCount=195 2024-09-11 17:59:13.018 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f9e94694f453e8e4,04b31245311fb182] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts/2000010/repairSystemInfoReport?id=2000010, client=10.31.33.41, traceId=f9e94694f453e8e4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.getRepairSystemInfoReport(Long), args=2000010,] 2024-09-11 17:59:13.023 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f9e94694f453e8e4,04b31245311fb182] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts/2000010/repairSystemInfoReport client=10.31.33.41, traceId=f9e94694f453e8e4, duration=10 ms] 2024-09-11 17:59:13.169 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048680, nodeCount=285, validCount=285 2024-09-11 17:59:13.255 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048680, nodeCount=1000, affectRows=1000 2024-09-11 17:59:13.257 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048680, nodeCount=999, validCount=999 2024-09-11 17:59:13.258 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048680, nodeCount=1000, affectRows=1000 2024-09-11 17:59:13.261 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048680, nodeCount=1000, validCount=1000 2024-09-11 17:59:13.267 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048680, nodeCount=48, affectRows=48 2024-09-11 17:59:13.267 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048680, nodeCount=48, validCount=48 2024-09-11 17:59:13.337 INFO 2981023 --- [ocp-async-3,bb42739c8617378e,a338ef532e6499f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,023259376802b16c,e970338648b1b191] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,97ea3bff55af7e97,a162982133d02808] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=3600 2024-09-11 17:59:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,023259376802b16c,e970338648b1b191] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:14.725 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,96dc8fb2f869fd70,0f5157473a84e999] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hosts/2000010/ignoreSystemInfoCheck, client 10.31.33.41, traceId 96dc8fb2f869fd70] 2024-09-11 17:59:14.727 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,96dc8fb2f869fd70,0f5157473a84e999] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hosts/2000010/ignoreSystemInfoCheck?id=2000010, client=10.31.33.41, traceId=96dc8fb2f869fd70, method=NoDataResponse com.oceanbase.ocp.server.common.controller.ComputeController.ignoreSystemInfoCheck(Long), args=2000010,] 2024-09-11 17:59:14.739 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,96dc8fb2f869fd70,0f5157473a84e999] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hosts/2000010/ignoreSystemInfoCheck client=10.31.33.41, traceId=96dc8fb2f869fd70, duration=16 ms] 2024-09-11 17:59:14.742 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:14.743 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:14.799 INFO 2981023 --- [ocp-schedules-21,9a9b437dd854c2c2,fbb53be45e1d8983] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:59:14.807 INFO 2981023 --- [ocp-schedules-21,9a9b437dd854c2c2,fbb53be45e1d8983] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:59:14.807 INFO 2981023 --- [ocp-schedules-21,9a9b437dd854c2c2,fbb53be45e1d8983] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:59:15.252 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8ab133e9ea382e4d,81b56f61bb0c7d95] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.31.33.41, traceId=8ab133e9ea382e4d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:59:15.253 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8ab133e9ea382e4d,81b56f61bb0c7d95] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:59:15.282 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8ab133e9ea382e4d,81b56f61bb0c7d95] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=8ab133e9ea382e4d, duration=32 ms] 2024-09-11 17:59:15.338 INFO 2981023 --- [ocp-async-1,7ecd9d990287364e,53086da87663ace5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:16.017 INFO 2981023 --- [ocp-schedules-7,d9b9f15b878bb344,b60920639b9bf3bf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:59:16.017 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:59:16.018 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:59:16.019 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:59:16.021 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:59:16.056 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:59:16.056 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:59:16.060 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:59:16.060 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:59:16.060 INFO 2981023 --- [ocp-schedules-15,a8b2827f243b0c6a,b1fc03e0c0766de4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:59:16.061 INFO 2981023 --- [ocp-schedules-16,a36823d10c32a687,80f1213e66c536d5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:59:16.063 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:16.063 INFO 2981023 --- [host-check-clockdiff1,a8b2827f243b0c6a,4fadc3697923a59f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:59:16.063 INFO 2981023 --- [host-check-agent-status1,a36823d10c32a687,dcb56a9632200e21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:59:16.064 INFO 2981023 --- [host-check-agent-status2,a36823d10c32a687,7671ddfa34c3e16e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:59:16.064 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:16.066 INFO 2981023 --- [host-check-agent-status3,a36823d10c32a687,c9b810ae272dd0ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:59:16.069 INFO 2981023 --- [host-check-agent-status4,a36823d10c32a687,af522bfefd2b21cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:59:16.069 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:16.069 INFO 2981023 --- [host-check-clockdiff2,a8b2827f243b0c6a,30a7a7f6721e820f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:59:16.072 INFO 2981023 --- [host-check-agent-status2,a36823d10c32a687,7671ddfa34c3e16e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:16.074 INFO 2981023 --- [host-check-agent-status2,a36823d10c32a687,7671ddfa34c3e16e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:16.086 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:16.095 INFO 2981023 --- [host-check-clockdiff3,a8b2827f243b0c6a,13d8cb0db7eeb220] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:59:16.100 INFO 2981023 --- [host-check-clockdiff4,a8b2827f243b0c6a,d2ef2e394d4530f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:59:16.100 INFO 2981023 --- [host-check-agent-status4,a36823d10c32a687,af522bfefd2b21cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:16.187 INFO 2981023 --- [host-check-agent-status4,a36823d10c32a687,af522bfefd2b21cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:16.195 INFO 2981023 --- [ocp-schedules-17,2ed138d230284684,d7b484d96e8fcab6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:59:16.195 INFO 2981023 --- [ocp-schedules-6,69413465ab2a6a50,d044b3ad3efc2933] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:59:16.195 INFO 2981023 --- [ocp-schedules-29,a606384f6bff2054,d2cb20f7bc07f2f2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:59:16.195 INFO 2981023 --- [ocp-schedules-22,5e0ac11879013c14,4e02f24e4bdcbcd8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:59:16.233 INFO 2981023 --- [host-check-clockdiff2,a8b2827f243b0c6a,30a7a7f6721e820f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048756 5 5 , error= 2024-09-11 17:59:16.233 INFO 2981023 --- [host-check-clockdiff3,a8b2827f243b0c6a,13d8cb0db7eeb220] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048756 0 0 , error= 2024-09-11 17:59:16.260 INFO 2981023 --- [ocp-schedules-29,a606384f6bff2054,d2cb20f7bc07f2f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:59:16.260 INFO 2981023 --- [ocp-schedules-6,69413465ab2a6a50,d044b3ad3efc2933] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:59:16.262 INFO 2981023 --- [host-check-agent-status1,a36823d10c32a687,dcb56a9632200e21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:59:16.265 INFO 2981023 --- [host-check-agent-status1,a36823d10c32a687,dcb56a9632200e21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:59:16.267 INFO 2981023 --- [host-check-agent-status3,a36823d10c32a687,c9b810ae272dd0ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:59:16.270 INFO 2981023 --- [host-check-agent-status3,a36823d10c32a687,c9b810ae272dd0ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:59:16.273 INFO 2981023 --- [ocp-schedules-17,2ed138d230284684,d7b484d96e8fcab6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:59:16.278 INFO 2981023 --- [ocp-schedules-21,a495a463bd3de9f3,aebcfe448fbfd8d7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:59:16.298 INFO 2981023 --- [host-check-agent-status4,a36823d10c32a687,af522bfefd2b21cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:59:16.303 INFO 2981023 --- [host-check-agent-status4,a36823d10c32a687,af522bfefd2b21cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:59:16.303 INFO 2981023 --- [host-check-agent-status4,a36823d10c32a687,af522bfefd2b21cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:59:16.303 INFO 2981023 --- [host-check-agent-status4,a36823d10c32a687,af522bfefd2b21cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:59:16.305 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:59:16.307 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:59:16.307188629+08:00 2024-09-11 17:59:16.308 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:59:16.307+08:00 2024-09-11 17:59:16.311 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:59:16.311 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:59:16.313 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:59:16.318591344+08:00 2024-09-11 17:59:16.316 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T17:59:16.311728920+08:00 2024-09-11 17:59:16.316 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:59:16.313+08:00 2024-09-11 17:59:16.317 INFO 2981023 --- [host-check-agent-status1,a36823d10c32a687,dcb56a9632200e21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:59:16.318 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:59:16.307+08:00 and status AVAILABLE 2024-09-11 17:59:16.319 INFO 2981023 --- [host-check-agent-status3,a36823d10c32a687,c9b810ae272dd0ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:59:16.321 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:59:16.313+08:00 and status AVAILABLE 2024-09-11 17:59:16.321 INFO 2981023 --- [host-check-agent-status1,a36823d10c32a687,dcb56a9632200e21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:59:16.321 INFO 2981023 --- [host-check-agent-status1,a36823d10c32a687,dcb56a9632200e21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:59:16.321 INFO 2981023 --- [host-check-agent-status1,a36823d10c32a687,dcb56a9632200e21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:59:16.324 INFO 2981023 --- [host-check-agent-status3,a36823d10c32a687,c9b810ae272dd0ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:59:16.325 INFO 2981023 --- [host-check-agent-status3,a36823d10c32a687,c9b810ae272dd0ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:59:16.325 INFO 2981023 --- [host-check-agent-status3,a36823d10c32a687,c9b810ae272dd0ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:59:16.326 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T17:59:16.316+08:00 2024-09-11 17:59:16.327 INFO 2981023 --- [ocp-schedules-6,69413465ab2a6a50,d044b3ad3efc2933] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:59:16.354 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:59:16.355 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T17:59:16.316+08:00 and status AVAILABLE 2024-09-11 17:59:16.356 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:59:16.359 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:59:16.361 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:59:16.366717974+08:00 2024-09-11 17:59:16.363 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:59:16.361+08:00 2024-09-11 17:59:16.365 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:59:16.361+08:00 and status AVAILABLE 2024-09-11 17:59:16.375 INFO 2981023 --- [host-check-agent-status2,a36823d10c32a687,7671ddfa34c3e16e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:59:16.377 INFO 2981023 --- [host-check-agent-status2,a36823d10c32a687,7671ddfa34c3e16e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:59:16.377 INFO 2981023 --- [host-check-agent-status2,a36823d10c32a687,7671ddfa34c3e16e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:59:16.377 INFO 2981023 --- [host-check-agent-status2,a36823d10c32a687,7671ddfa34c3e16e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:59:16.386 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.387 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:16.388 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:59:16.395 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 17:59:16.396 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:59:16.400 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:59:16.401 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:59:16.400515+08:00, lastProcessTime=2024-09-11T17:59:06.352152+08:00 2024-09-11 17:59:16.402 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:59:16.405 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.406 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:16.406 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:59:16.407 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:59:16.407 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:16.407 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:59:16.408 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:16.409 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.410 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:16.410 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:16.410 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:16.410 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:59:16.411 INFO 2981023 --- [ocp-schedules-17,2ed138d230284684,d7b484d96e8fcab6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:59:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:16.457 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.457 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:16.479 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:59:16.480 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:59:16.480 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:59:16.481 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:59:16.493 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.493 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:16.501 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 17:59:16.505 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:59:16.509 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:59:16.516 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:59:16.517 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:59:16.518 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:59:16.520 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 17:59:16.526 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:59:16.554 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:59:16.561 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.561 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.562 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:59:16.566 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:16.568 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:59:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.571 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:59:16.572 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:59:16.574 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:59:16.583 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:16.583 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:59:16.583 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:59:16.584 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.585 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.586 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:16.590 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:59:16.591 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.594 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:59:16.595 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:59:16.605 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 17:59:16.606 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:16.606 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:59:16.608 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:59:16.613 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:16.614 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 17:59:16.619 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:59:16.621 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:16.624 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.624 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.625 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:16.626 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:59:16.633 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:59:16.634 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:59:16.634 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:59:16.637 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:59:16.638 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:59:16.640 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.640 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:59:16.642 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.642 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:16.645 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:59:16.650 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:59:16.654 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:59:16.655 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:59:16.655 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:59:16.660 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:59:16.662 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:16.669 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.669 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.670 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:59:16.671 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:59:16.679 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 17:59:16.694 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:59:16.694 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:59:16.695 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:59:16.704 INFO 2981023 --- [host-check-host-status4,d9b9f15b878bb344,f34f4cdbf10abda8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 17:59:16.711 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.711 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.713 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:16.714 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:59:16.714 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.716 INFO 2981023 --- [host-check-host-status1,d9b9f15b878bb344,1647b17be9aed47d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:59:16.716 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:59:16.717 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:59:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:16.727 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:59:16.727 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:59:16.733 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:59:16.741 INFO 2981023 --- [host-check-host-status3,d9b9f15b878bb344,2fe1d2ff45a0ac9c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:59:16.744 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:16.752 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:16.754 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:59:16.757 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:59:16.758 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:59:16.760 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:59:16.761 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:59:16.766 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:59:16.768 INFO 2981023 --- [host-check-host-status2,d9b9f15b878bb344,04c396ff6992d9da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:59:16.768 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:59:16.400515+08:00 2024-09-11 17:59:16.772 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:59:16.776 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.777 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.777 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:59:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:59:16.779 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.780 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.780 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:59:16.784 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:59:16.784 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:59:16.786 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:59:16.786 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:59:16.808 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:59:16.813 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:16.813 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:16.814 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:59:16.819 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:59:16.819 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:59:16.819 INFO 2981023 --- [ocp-schedules-29,a606384f6bff2054,d2cb20f7bc07f2f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:59:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,86df97f201f78e8c,1d0dce7069c8d170] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,86df97f201f78e8c,1d0dce7069c8d170] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:17.338 INFO 2981023 --- [ocp-async-6,672f25e2295955fd,c6a7f1dcd53a298d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:17.743 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:17.743 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:17.770 INFO 2981023 --- [ocp-async-7,199a5e82a2abe1c9,3463cc47a3add54d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:59:17.862 INFO 2981023 --- [host-check-clockdiff1,a8b2827f243b0c6a,4fadc3697923a59f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048757 5 5 , error= 2024-09-11 17:59:19.337 INFO 2981023 --- [ocp-schedules-16,b595f3cff23e6d66,68d9516dd4987bc8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:59:19.338 INFO 2981023 --- [ocp-async-5,2d44abe28c08daba,f06329f29bb97e10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:19.397 WARN 2981023 --- [scanner-13584946-1,b595f3cff23e6d66,7da24b73d4cde9c3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:59:19.466 INFO 2981023 --- [alarm-detect-1,b595f3cff23e6d66,3d35a0950df0f9cf] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726048725 ~ 1726048749 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726048740, value=1.663)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.663, values={1726048740=1.663}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 17:59:19.467 INFO 2981023 --- [alarm-detect-1,b595f3cff23e6d66,3d35a0950df0f9cf] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Firing 2024-09-11 17:59:19.467 INFO 2981023 --- [alarm-detect-1,b595f3cff23e6d66,3d35a0950df0f9cf] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-3] 2024-09-11 17:59:19.505 INFO 2981023 --- [alarm-detect-1,b595f3cff23e6d66,3d35a0950df0f9cf] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007108 2024-09-11 17:59:19.669 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,cd84e6f846069c3e,dfe8ef6e53f90701] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.31.33.41, traceId=cd84e6f846069c3e, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2024-09-11 17:59:19.675 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,cd84e6f846069c3e,dfe8ef6e53f90701] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.31.33.41, traceId=cd84e6f846069c3e, duration=10 ms] 2024-09-11 17:59:19.684 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,a6cd78a3018e9d77,8846e8fdd12d881f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP&maxPoints=360, client=10.31.33.41, traceId=a6cd78a3018e9d77, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,] 2024-09-11 17:59:19.697 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,a6cd78a3018e9d77,8846e8fdd12d881f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=a6cd78a3018e9d77, duration=16 ms] 2024-09-11 17:59:19.764 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,49ace290ed4cdc92,6ab133e31e876294] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2024-09-11T16%3A59%3A19%2B08%3A00&endTime=2024-09-11T17%3A59%3A19%2B08%3A00&maxPoints=360, client=10.31.33.41, traceId=49ace290ed4cdc92, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T16:59:19+08:00,2024-09-11T17:59:19+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2024-09-11 17:59:19.781 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,51d1a2bccc8b63a9,7c4a9c6be8939e67] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.31.33.41, traceId=51d1a2bccc8b63a9, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2024-09-11 17:59:19.785 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,49ace290ed4cdc92,6ab133e31e876294] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=49ace290ed4cdc92, duration=24 ms] 2024-09-11 17:59:19.793 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,51d1a2bccc8b63a9,7c4a9c6be8939e67] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=51d1a2bccc8b63a9, duration=14 ms] 2024-09-11 17:59:19.814 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9cf1b6a9ef07f279,e4fa5dce78151725] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=9cf1b6a9ef07f279, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 17:59:19.824 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,0612b0b8f97a5227,f64a1659655827db] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.31.33.41, traceId=0612b0b8f97a5227, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2024-09-11 17:59:19.839 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,0612b0b8f97a5227,f64a1659655827db] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.31.33.41, traceId=0612b0b8f97a5227, duration=17 ms] 2024-09-11 17:59:19.859 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9cf1b6a9ef07f279,e4fa5dce78151725] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=9cf1b6a9ef07f279, duration=46 ms] 2024-09-11 17:59:19.868 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:19.869 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:19.871 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:59:19.871 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:19.871 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:59:19.873 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:59:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,24910095c11fb6ae,b2c82012c96a2182] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,24910095c11fb6ae,b2c82012c96a2182] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:20.743 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:20.745 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:21.338 INFO 2981023 --- [ocp-async-1,a08cca24c7e53db9,4a769a3c6042f1ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:22.090 INFO 2981023 --- [host-check-clockdiff4,a8b2827f243b0c6a,d2ef2e394d4530f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048762 -1 -1 , error= 2024-09-11 17:59:22.364 INFO 2981023 --- [ocp-schedules-18,d665f4bc37df67f0,bad79fe4af0dcebb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:59:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,0672b2e218981958,9ca9f41199464035] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:23.315 INFO 2981023 --- [prometheus-minute-schedule-1,0672b2e218981958,9ca9f41199464035] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:23.338 INFO 2981023 --- [ocp-async-8,c87a85f696efcd9d,c4d0c42046cc11b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:23.746 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:23.746 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:24.064 INFO 2981023 --- [ocp-schedules-14,b455c349fb4aa7f7,4796d3af474d4e27] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:59:24.064 INFO 2981023 --- [ocp-schedules-14,b455c349fb4aa7f7,4796d3af474d4e27] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 17:59:24.064 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 17:59:24.064 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 17:59:24.064 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 17:59:24.068 INFO 2981023 --- [ocp-schedules-14,b455c349fb4aa7f7,4796d3af474d4e27] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:59:24.068 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:59:24.069 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:24.072 INFO 2981023 --- [ocp-schedules-14,b455c349fb4aa7f7,4796d3af474d4e27] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:59:24.072 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:24.073 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:59:24.073 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:24.074 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:24.096 INFO 2981023 --- [ocp-schedules-14,b455c349fb4aa7f7,4796d3af474d4e27] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 17:59:24.096 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 17:59:24.098 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 17:59:24.098 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 17:59:24.101 INFO 2981023 --- [ocp-schedules-14,b455c349fb4aa7f7,4796d3af474d4e27] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 17:59:24.281 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,581bdbdc366daa16,08122007c5158e7d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.31.33.41, traceId=581bdbdc366daa16, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-09-11 17:59:24.284 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,581bdbdc366daa16,08122007c5158e7d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.31.33.41, traceId=581bdbdc366daa16, duration=5 ms] 2024-09-11 17:59:24.290 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,174b4e8d545d6a6d,aa977f16b8531448] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.31.33.41, traceId=174b4e8d545d6a6d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 17:59:24.293 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,adbfde43e3fd2702,c09d751e9ee82708] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=adbfde43e3fd2702, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:59:24.293 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,7b2b2238d59b93d3,47c630b7d4124a1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.31.33.41, traceId=7b2b2238d59b93d3, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-09-11 17:59:24.293 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,7b2b2238d59b93d3,47c630b7d4124a1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.31.33.41, traceId=7b2b2238d59b93d3, duration=3 ms] 2024-09-11 17:59:24.296 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,260dc1bc747056cd,880a2d792abff0c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.31.33.41, traceId=260dc1bc747056cd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-09-11 17:59:24.296 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,260dc1bc747056cd,880a2d792abff0c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.31.33.41, traceId=260dc1bc747056cd, duration=4 ms] 2024-09-11 17:59:24.301 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,174b4e8d545d6a6d,aa977f16b8531448] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=174b4e8d545d6a6d, duration=14 ms] 2024-09-11 17:59:24.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,adbfde43e3fd2702,c09d751e9ee82708] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=adbfde43e3fd2702, duration=13 ms] 2024-09-11 17:59:24.349 INFO 2981023 --- [ocp-schedules-26,819484b77a019db5,21f1a91c47c7f7f3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:59:24.400 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,8d5c91d85c2b95e8,d7d984665d93f971] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.31.33.41, traceId=8d5c91d85c2b95e8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-09-11 17:59:24.400 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,8d5c91d85c2b95e8,d7d984665d93f971] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.31.33.41, traceId=8d5c91d85c2b95e8, duration=2 ms] 2024-09-11 17:59:24.436 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7bc0232f44581ba3,8980816ce27ff2ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.31.33.41, traceId=7bc0232f44581ba3, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-09-11 17:59:24.436 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,7bc0232f44581ba3,8980816ce27ff2ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.31.33.41, traceId=7bc0232f44581ba3, duration=1 ms] 2024-09-11 17:59:24.438 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,2ba318c0602046df,66f14d25196f5f48] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.31.33.41, traceId=2ba318c0602046df, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-09-11 17:59:24.441 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,2ba318c0602046df,66f14d25196f5f48] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.31.33.41, traceId=2ba318c0602046df, duration=5 ms] 2024-09-11 17:59:24.517 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,0e256ef1c5e8517b,8e080ea1b40b1226] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.31.33.41, traceId=0e256ef1c5e8517b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 17:59:24.530 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 17:59:24.530 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, client=10.31.33.41, traceId=] 2024-09-11 17:59:24.530 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 17:59:24.533 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 17:59:24.533 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, client=10.31.33.41, traceId=] 2024-09-11 17:59:24.533 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 17:59:24.568 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0295267412cbdc0b,5ca98f123e29435a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/summary?id=1000006, client=10.31.33.41, traceId=0295267412cbdc0b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1000006,] 2024-09-11 17:59:24.573 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e46fc75470b22b76,64cefef423c969de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/topSql/switch?id=1000006, client=10.31.33.41, traceId=e46fc75470b22b76, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1000006,] 2024-09-11 17:59:24.596 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e46fc75470b22b76,64cefef423c969de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/topSql/switch client=10.31.33.41, traceId=e46fc75470b22b76, duration=26 ms] 2024-09-11 17:59:24.608 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:24.609 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 17:59:24.614 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:24.614 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0295267412cbdc0b,5ca98f123e29435a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:24.615 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT ZONE, SVR_IP, SVR_PORT, NAME, VALUE, DATA_TYPE, SECTION, EDIT_LEVEL, INFO FROM GV$OB_PARAMETERS WHERE SCOPE = 'CLUSTER' AND name = ?, args: [_lcl_op_interval] 2024-09-11 17:59:24.616 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0295267412cbdc0b,5ca98f123e29435a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:59:24.625 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0295267412cbdc0b,5ca98f123e29435a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/summary client=10.31.33.41, traceId=0295267412cbdc0b, duration=60 ms] 2024-09-11 17:59:24.635 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:24.635 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:24.635 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [resource_hard_limit] 2024-09-11 17:59:24.635 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2024-09-11 17:59:24.643 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:24.643 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e9e8ad58d22b5749,1346d0e0cfee918b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/stats?id=1000006, client=10.31.33.41, traceId=e9e8ad58d22b5749, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1000006,] 2024-09-11 17:59:24.647 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1] 2024-09-11 17:59:24.658 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e9e8ad58d22b5749,1346d0e0cfee918b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:24.660 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e9e8ad58d22b5749,1346d0e0cfee918b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:59:24.661 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e9e8ad58d22b5749,1346d0e0cfee918b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/stats client=10.31.33.41, traceId=e9e8ad58d22b5749, duration=21 ms] 2024-09-11 17:59:24.669 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:24.673 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:24.677 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1] 2024-09-11 17:59:24.732 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,0e256ef1c5e8517b,8e080ea1b40b1226] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.31.33.41, traceId=0e256ef1c5e8517b, duration=218 ms] 2024-09-11 17:59:25.193 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,22e19c9537795651,b977f819baf9e74d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.31.33.41, traceId=22e19c9537795651, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-09-11 17:59:25.199 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,22e19c9537795651,b977f819baf9e74d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.31.33.41, traceId=22e19c9537795651, duration=8 ms] 2024-09-11 17:59:25.338 INFO 2981023 --- [ocp-async-2,e15fb41e915bc337,6c25ce4689dba776] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:26.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a687c7cfb192581b,76ce03a1ae35e1fb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.31.33.41, traceId=a687c7cfb192581b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 17:59:26.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a687c7cfb192581b,76ce03a1ae35e1fb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=a687c7cfb192581b, duration=9 ms] 2024-09-11 17:59:26.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,22030caec11de2ab,f6afcdbf77d850f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/alarmSummaries?id=1000006, client=10.31.33.41, traceId=22030caec11de2ab, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=1000006,] 2024-09-11 17:59:26.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c8ac23b6ae8ea3ad,5c5a7b32304612da] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/eventMeta?id=1000006, client=10.31.33.41, traceId=c8ac23b6ae8ea3ad, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1000006,] 2024-09-11 17:59:26.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,68119cbfbcc58845,a458527b073cda93] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?status=AVAILABLE&status=ONLINE&excludedServiceType=OB_CLUSTER&excludedServiceType=BACKUP_AGENT&excludedServiceType=RESTORE_AGENT&excludedServiceType=BACKUP_RESTORE_AGENT&excludedServiceType=ARBITRATION&excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=68119cbfbcc58845, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[AVAILABLE, ONLINE],[OB_CLUSTER, BACKUP_AGENT, RESTORE_AGENT, BACKUP_RESTORE_AGENT, ARBITRATION, BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:26.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c8ac23b6ae8ea3ad,5c5a7b32304612da] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/eventMeta client=10.31.33.41, traceId=c8ac23b6ae8ea3ad, duration=3 ms] 2024-09-11 17:59:26.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,1b9dc2810c07a18e,d22207dae21a6bef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/serverStats?id=1000006, client=10.31.33.41, traceId=1b9dc2810c07a18e, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=1000006,] 2024-09-11 17:59:26.114 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,68119cbfbcc58845,a458527b073cda93] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:59:26.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,22030caec11de2ab,f6afcdbf77d850f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/alarmSummaries client=10.31.33.41, traceId=22030caec11de2ab, duration=15 ms] 2024-09-11 17:59:26.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,1b9dc2810c07a18e,d22207dae21a6bef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:26.135 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,1b9dc2810c07a18e,d22207dae21a6bef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:59:26.136 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,1b9dc2810c07a18e,d22207dae21a6bef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/serverStats client=10.31.33.41, traceId=1b9dc2810c07a18e, duration=25 ms] 2024-09-11 17:59:26.141 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,68119cbfbcc58845,a458527b073cda93] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=68119cbfbcc58845, duration=35 ms] 2024-09-11 17:59:26.221 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ca05fb741f859a7a,0b7c55489b10fc16] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.31.33.41, traceId=ca05fb741f859a7a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:26.223 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ca05fb741f859a7a,0b7c55489b10fc16] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:59:26.231 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bf114902c7b3dd14,baabe0c908cc2ff9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=bf114902c7b3dd14, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:59:26.236 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bf114902c7b3dd14,baabe0c908cc2ff9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=bf114902c7b3dd14, duration=7 ms] 2024-09-11 17:59:26.242 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e75be489adb2ac7c,265e7c888dd51474] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top/realtime?metrics=active_session%2Cqps%2Ctps&labels=app%3AOB%2Cob_cluster_name%3Alinkrtd%2Cob_cluster_id%3A1719994443&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&queryAsync=true&endTime=2024-09-11T17%3A59%3A25%2B08%3A00&limit=1&maxPoints=360, client=10.31.33.41, traceId=e75be489adb2ac7c, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryRealtimeTopMetrics(OffsetDateTime,Long,String,String,Long,Long,Integer,Integer,String,Boolean,QueryRouteType), args=2024-09-11T17:59:25+08:00,60,active_session,qps,tps,app,ob_cluster_name,ob_cluster_id,0,0,360,1,app:OB,ob_cluster_name:linkrtd,ob_cluster_id:1719994443,true,NOT_ROUTED,] 2024-09-11 17:59:26.247 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e75be489adb2ac7c,265e7c888dd51474] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = qps 2024-09-11 17:59:26.247 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e75be489adb2ac7c,265e7c888dd51474] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = qps, minTimestamp = 0, minTimeBuffer = 0, startTime = 1726048710 2024-09-11 17:59:26.253 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2348fcdc05063fd1,db1d40f2c4a5d69f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/topSql/status/events?null, client=10.31.33.41, traceId=2348fcdc05063fd1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitchEvents(Long,OffsetDateTime,OffsetDateTime,Pageable), args=1000006,Page request [number: 0, size 5000, sort: timestamp: ASC],] 2024-09-11 17:59:26.255 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e75be489adb2ac7c,265e7c888dd51474] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top/realtime client=10.31.33.41, traceId=e75be489adb2ac7c, duration=16 ms] 2024-09-11 17:59:26.257 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ca05fb741f859a7a,0b7c55489b10fc16] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=ca05fb741f859a7a, duration=39 ms] 2024-09-11 17:59:26.262 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2348fcdc05063fd1,db1d40f2c4a5d69f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/topSql/status/events client=10.31.33.41, traceId=2348fcdc05063fd1, duration=13 ms] 2024-09-11 17:59:26.265 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7138ca410ec05300,8347393c39ae6012] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=7138ca410ec05300, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:26.269 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7138ca410ec05300,8347393c39ae6012] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:59:26.277 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7138ca410ec05300,8347393c39ae6012] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=7138ca410ec05300, duration=14 ms] 2024-09-11 17:59:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,473072ac431adf3f,1b3d484e7c46166f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:26.315 INFO 2981023 --- [prometheus-minute-schedule-1,473072ac431adf3f,1b3d484e7c46166f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:26.350 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8631a4e88f29e432,b1a83f1622efe642] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top/realtime?metrics=active_session%2Cqps%2Ctps&labels=app%3AOB%2Cob_cluster_name%3Alinkrtd%2Cob_cluster_id%3A1719994443&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&queryAsync=true&minStep=1&endTime=2024-09-11T17%3A59%3A25%2B08%3A00&limit=1&maxPoints=360, client=10.31.33.41, traceId=8631a4e88f29e432, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryRealtimeTopMetrics(OffsetDateTime,Long,String,String,Long,Long,Integer,Integer,String,Boolean,QueryRouteType), args=2024-09-11T17:59:25+08:00,60,active_session,qps,tps,app,ob_cluster_name,ob_cluster_id,0,1,360,1,app:OB,ob_cluster_name:linkrtd,ob_cluster_id:1719994443,true,NOT_ROUTED,] 2024-09-11 17:59:26.352 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8631a4e88f29e432,b1a83f1622efe642] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = qps 2024-09-11 17:59:26.352 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8631a4e88f29e432,b1a83f1622efe642] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = qps, minTimestamp = 0, minTimeBuffer = 0, startTime = 1726048710 2024-09-11 17:59:26.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:59:26.353515+08:00, lastProcessTime=2024-09-11T17:59:16.400515+08:00 2024-09-11 17:59:26.355 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8631a4e88f29e432,b1a83f1622efe642] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top/realtime client=10.31.33.41, traceId=8631a4e88f29e432, duration=8 ms] 2024-09-11 17:59:26.379 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,4c6f57b2ed020256,ee4e13361f1139d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.31.33.41, traceId=4c6f57b2ed020256, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:26.383 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,4c6f57b2ed020256,ee4e13361f1139d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.31.33.41, traceId=4c6f57b2ed020256, duration=5 ms] 2024-09-11 17:59:26.394 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,282a6a900f74af28,efa8c4fc94e81c66] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=282a6a900f74af28, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:59:26.397 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,282a6a900f74af28,efa8c4fc94e81c66] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=282a6a900f74af28, duration=6 ms] 2024-09-11 17:59:26.445 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,590ba598523a7799,711b95152e4999fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/events?startTime=2024-09-04T17%3A59%3A25%2B08%3A00&endTime=2024-09-11T17%3A59%3A25%2B08%3A00&page=1&size=7, client=10.31.33.41, traceId=590ba598523a7799, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1000006,2024-09-04T17:59:25+08:00,2024-09-11T17:59:25+08:00,Page request [number: 0, size 7, sort: timestamp: DESC],] 2024-09-11 17:59:26.463 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,590ba598523a7799,711b95152e4999fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/events client=10.31.33.41, traceId=590ba598523a7799, duration=22 ms] 2024-09-11 17:59:26.501 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 17:59:26.507 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:59:26.353515+08:00 2024-09-11 17:59:26.524 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,79254a699b67fa49,430c779a12852dec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=79254a699b67fa49, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:59:26.526 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,79254a699b67fa49,430c779a12852dec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=79254a699b67fa49, duration=4 ms] 2024-09-11 17:59:26.746 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:26.747 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:27.338 INFO 2981023 --- [ocp-async-7,f771f779494e8abd,42b13f45653f9e89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:27.770 INFO 2981023 --- [ocp-async-3,06795609ac8f0146,aa6d35a1892ebd8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:59:28.204 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,c2cc804641fa8353,7ea08203fba0a59f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?status=AVAILABLE&status=ONLINE&excludedServiceType=OB_CLUSTER&excludedServiceType=BACKUP_AGENT&excludedServiceType=RESTORE_AGENT&excludedServiceType=BACKUP_RESTORE_AGENT&excludedServiceType=ARBITRATION&excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=c2cc804641fa8353, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[AVAILABLE, ONLINE],[OB_CLUSTER, BACKUP_AGENT, RESTORE_AGENT, BACKUP_RESTORE_AGENT, ARBITRATION, BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:28.204 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4beee864f19affb1,3c3e2f03cb94f3ad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/software-packages?null, client=10.31.33.41, traceId=4beee864f19affb1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.SoftwarePackageController.listPackages(String,List,List,List,List,boolean,Pageable), args=true,Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:28.206 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-3,c2cc804641fa8353,7ea08203fba0a59f] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:59:28.215 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,c2cc804641fa8353,7ea08203fba0a59f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=c2cc804641fa8353, duration=14 ms] 2024-09-11 17:59:28.249 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4beee864f19affb1,3c3e2f03cb94f3ad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/software-packages client=10.31.33.41, traceId=4beee864f19affb1, duration=48 ms] 2024-09-11 17:59:28.364 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,48a775845625fece,b83b7f691ca42f8f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/software-packages?null, client=10.31.33.41, traceId=48a775845625fece, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.SoftwarePackageController.listPackages(String,List,List,List,List,boolean,Pageable), args=true,Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:28.396 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,48a775845625fece,b83b7f691ca42f8f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/software-packages client=10.31.33.41, traceId=48a775845625fece, duration=34 ms] 2024-09-11 17:59:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,d540c33d6592cf74,a2a6ec5cd737d332] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,d540c33d6592cf74,a2a6ec5cd737d332] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:29.339 INFO 2981023 --- [ocp-async-6,0c8f1ddc709b4ca7,17ab030b668f1428] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:29.555 INFO 2981023 --- [ocp-schedules-9,95cd9ce43a6d30ad,d4c5e8c45c633a65] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:59:29.570 WARN 2981023 --- [scanner-13585111-1,95cd9ce43a6d30ad,daa7b5304498a61d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:59:29.624 INFO 2981023 --- [alarm-detect-5,95cd9ce43a6d30ad,4d832083718390d5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645148, data is empty. 2024-09-11 17:59:29.628 INFO 2981023 --- [alarm-detect-5,95cd9ce43a6d30ad,4d832083718390d5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645148, data is empty. 2024-09-11 17:59:29.634 INFO 2981023 --- [alarm-detect-5,95cd9ce43a6d30ad,4d832083718390d5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645148, data is empty. 2024-09-11 17:59:29.663 INFO 2981023 --- [alarm-detect-9,95cd9ce43a6d30ad,91d6fa175b1f205d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645148, data is empty. 2024-09-11 17:59:29.665 INFO 2981023 --- [alarm-detect-4,95cd9ce43a6d30ad,de2cf020b01dab05] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645148, data is empty. 2024-09-11 17:59:29.668 INFO 2981023 --- [alarm-detect-9,95cd9ce43a6d30ad,91d6fa175b1f205d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645148, data is empty. 2024-09-11 17:59:29.670 INFO 2981023 --- [alarm-detect-4,95cd9ce43a6d30ad,de2cf020b01dab05] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645148, data is empty. 2024-09-11 17:59:29.674 INFO 2981023 --- [alarm-detect-9,95cd9ce43a6d30ad,91d6fa175b1f205d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645148, data is empty. 2024-09-11 17:59:29.676 INFO 2981023 --- [alarm-detect-4,95cd9ce43a6d30ad,de2cf020b01dab05] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645148, data is empty. 2024-09-11 17:59:29.747 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:29.752 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:29.770 INFO 2981023 --- [alarm-detect-11,95cd9ce43a6d30ad,89d078d1a9abd7e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645148, data is empty. 2024-09-11 17:59:29.773 INFO 2981023 --- [alarm-detect-11,95cd9ce43a6d30ad,89d078d1a9abd7e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645148, data is empty. 2024-09-11 17:59:29.777 INFO 2981023 --- [alarm-detect-11,95cd9ce43a6d30ad,89d078d1a9abd7e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645148, data is empty. 2024-09-11 17:59:31.339 INFO 2981023 --- [ocp-async-2,f2d984543f5ec7d8,54f5fb8984223019] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,c0b3ef9f812fffdc,5fcb429682e4fcab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,c0b3ef9f812fffdc,5fcb429682e4fcab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:32.367 INFO 2981023 --- [ocp-schedules-2,9f081a3297a93664,ef3785d7bc63b2b6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:59:32.752 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:32.752 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:33.339 INFO 2981023 --- [ocp-async-4,303fc81a66e85762,e16a5d0017f52a75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:33.719 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,04bbf92e79961fcb,fadbfa220baa10aa] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/1000006/checkServerResource, client 10.31.33.41, traceId 04bbf92e79961fcb] 2024-09-11 17:59:33.721 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,04bbf92e79961fcb,fadbfa220baa10aa] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/1000006/checkServerResource?id=1000006, client=10.31.33.41, traceId=04bbf92e79961fcb, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.checkServerResourceForCluster(Long,CheckServerResourceParam), args=1000006,CheckServerResourceParam(hostIdList=[2000010], logDir=/oceanbase/redo, dataDir=/data/oceanbase),] 2024-09-11 17:59:33.767 INFO 2981023 --- [pool-556-thread-1,04bbf92e79961fcb,3bb4d2978eca0db5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:33.771 INFO 2981023 --- [pool-556-thread-1,04bbf92e79961fcb,3bb4d2978eca0db5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:33.832 INFO 2981023 --- [pool-556-thread-1,04bbf92e79961fcb,3bb4d2978eca0db5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/hostInfo, request body:{}, params:null 2024-09-11 17:59:34.810 INFO 2981023 --- [ocp-schedules-8,444603f3466a9ec3,0a4b82d4e7f977bc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:59:34.818 INFO 2981023 --- [ocp-schedules-8,444603f3466a9ec3,0a4b82d4e7f977bc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:59:34.818 INFO 2981023 --- [ocp-schedules-8,444603f3466a9ec3,0a4b82d4e7f977bc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:59:34.837 INFO 2981023 --- [pool-556-thread-1,04bbf92e79961fcb,3bb4d2978eca0db5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/disk/usage, request body:GetDiskUsageRequest(path=/oceanbase/redo), params:null 2024-09-11 17:59:34.842 INFO 2981023 --- [pool-556-thread-1,04bbf92e79961fcb,3bb4d2978eca0db5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/disk/usage, request body:GetDiskUsageRequest(path=/data/oceanbase), params:null 2024-09-11 17:59:34.846 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,04bbf92e79961fcb,fadbfa220baa10aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/1000006/checkServerResource client=10.31.33.41, traceId=04bbf92e79961fcb, duration=1131 ms] 2024-09-11 17:59:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,9d3e58c4eefccffb,5f2e08ea52eee223] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,9d3e58c4eefccffb,5f2e08ea52eee223] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:35.339 INFO 2981023 --- [ocp-async-10,56e6b7d528137650,59b3208a2a758409] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:35.753 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:35.753 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:59:36.352648+08:00, lastProcessTime=2024-09-11T17:59:26.353515+08:00 2024-09-11 17:59:36.483 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:59:36.352648+08:00 2024-09-11 17:59:37.314 INFO 2981023 --- [prometheus-cache-monitor1,246b061467b204d9,3541c31e3e169ea2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7719,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":5002} 2024-09-11 17:59:37.315 INFO 2981023 --- [prometheus-cache-monitor1,246b061467b204d9,3541c31e3e169ea2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:59:37.339 INFO 2981023 --- [ocp-async-9,b3787f511792f379,97524b9087e85740] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:37.770 INFO 2981023 --- [ocp-async-10,fb83983f938f0195,1c9cc363129ac779] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:59:38.256 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,472a66512457eb76,58311668b9e7ad37] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hosts/batchCheckHostUserExists, client 10.31.33.41, traceId 472a66512457eb76] 2024-09-11 17:59:38.257 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,472a66512457eb76,58311668b9e7ad37] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hosts/batchCheckHostUserExists?null, client=10.31.33.41, traceId=472a66512457eb76, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.batchCheckHostUserExists(BatchCheckHostUserExistsRequest), args=BatchCheckHostUserExistsRequest(username=admin, hostIdList=[2000010]),] 2024-09-11 17:59:38.260 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,472a66512457eb76,58311668b9e7ad37] c.o.o.c.host.service.HostServiceImpl : validHostIdList=[2000010] 2024-09-11 17:59:38.265 INFO 2981023 --- [pool-557-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:38.266 INFO 2981023 --- [pool-557-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:38.278 INFO 2981023 --- [pool-557-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/userExists, request body:CheckUserExistsRequest(name=admin), params:null 2024-09-11 17:59:38.281 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,472a66512457eb76,58311668b9e7ad37] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hosts/batchCheckHostUserExists client=10.31.33.41, traceId=472a66512457eb76, duration=27 ms] 2024-09-11 17:59:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,85ac2af6abf7fc63,a04a59f0937f7d13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,85ac2af6abf7fc63,a04a59f0937f7d13] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:38.753 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:38.753 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:39.339 INFO 2981023 --- [ocp-async-7,c03ba038971019ec,fa500530231114af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:39.821 INFO 2981023 --- [ocp-schedules-17,fe2e3573b08ae7f1,38af20f4f2637003] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:59:39.867 WARN 2981023 --- [scanner-13585439-1,fe2e3573b08ae7f1,bd4e730bab880c02] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:59:40.006 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 17:59:40.007 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 17:59:40.066 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a4c2e659c1f2a8fd,b6d6fd1244bc2ebf] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hosts/batchCheckHostDirectoryPermissions, client 10.31.33.41, traceId a4c2e659c1f2a8fd] 2024-09-11 17:59:40.067 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a4c2e659c1f2a8fd,b6d6fd1244bc2ebf] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hosts/batchCheckHostDirectoryPermissions?null, client=10.31.33.41, traceId=a4c2e659c1f2a8fd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.batchCheckHostDirectoryPermissions(BatchCheckHostDirectoryRequest), args=BatchCheckHostDirectoryRequest(username=admin, hostIdList=[2000010], testDirectoryList=[HostDirectory(key=dataDiskPath, directory=/data/oceanbase), HostDirectory(key=logDiskPath, directory=/oceanbase/redo)], permissionList=[ACCESS_READ, ACCESS_EXECUTE], onlyExistedAncestor=false),] 2024-09-11 17:59:40.069 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a4c2e659c1f2a8fd,b6d6fd1244bc2ebf] c.o.o.c.host.service.HostServiceImpl : validHostIdList=[2000010] 2024-09-11 17:59:40.075 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:40.075 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:40.086 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/file/checkDirectoryPermission, request body:CheckDirectoryPermissionRequest(directory=/data/oceanbase, user=admin, permission=ACCESS_READ, onlyExistingAncestor=false), params:null 2024-09-11 17:59:40.108 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:40.109 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:40.120 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/file/checkDirectoryPermission, request body:CheckDirectoryPermissionRequest(directory=/data/oceanbase, user=admin, permission=ACCESS_EXECUTE, onlyExistingAncestor=false), params:null 2024-09-11 17:59:40.139 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:40.140 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:40.151 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/file/checkDirectoryPermission, request body:CheckDirectoryPermissionRequest(directory=/oceanbase/redo, user=admin, permission=ACCESS_READ, onlyExistingAncestor=false), params:null 2024-09-11 17:59:40.169 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:40.170 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:40.181 INFO 2981023 --- [pool-558-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/file/checkDirectoryPermission, request body:CheckDirectoryPermissionRequest(directory=/oceanbase/redo, user=admin, permission=ACCESS_EXECUTE, onlyExistingAncestor=false), params:null 2024-09-11 17:59:40.197 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,a4c2e659c1f2a8fd,b6d6fd1244bc2ebf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hosts/batchCheckHostDirectoryPermissions client=10.31.33.41, traceId=a4c2e659c1f2a8fd, duration=133 ms] 2024-09-11 17:59:40.573 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4cebe74acacc9b96,e299facadf7b5e13] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hosts/batchCheckHostDirectoryPermissions, client 10.31.33.41, traceId 4cebe74acacc9b96] 2024-09-11 17:59:40.574 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4cebe74acacc9b96,e299facadf7b5e13] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hosts/batchCheckHostDirectoryPermissions?null, client=10.31.33.41, traceId=4cebe74acacc9b96, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.batchCheckHostDirectoryPermissions(BatchCheckHostDirectoryRequest), args=BatchCheckHostDirectoryRequest(username=admin, hostIdList=[2000010], testDirectoryList=[HostDirectory(key=installPath, directory=/data/rtdob/oceanbase)], permissionList=[ACCESS_READ, ACCESS_EXECUTE], onlyExistedAncestor=true),] 2024-09-11 17:59:40.577 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4cebe74acacc9b96,e299facadf7b5e13] c.o.o.c.host.service.HostServiceImpl : validHostIdList=[2000010] 2024-09-11 17:59:40.581 INFO 2981023 --- [pool-559-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:40.582 INFO 2981023 --- [pool-559-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:40.593 INFO 2981023 --- [pool-559-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/file/checkDirectoryPermission, request body:CheckDirectoryPermissionRequest(directory=/data/rtdob/oceanbase, user=admin, permission=ACCESS_READ, onlyExistingAncestor=true), params:null 2024-09-11 17:59:40.612 INFO 2981023 --- [pool-559-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:40.613 INFO 2981023 --- [pool-559-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:40.624 INFO 2981023 --- [pool-559-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/file/checkDirectoryPermission, request body:CheckDirectoryPermissionRequest(directory=/data/rtdob/oceanbase, user=admin, permission=ACCESS_EXECUTE, onlyExistingAncestor=true), params:null 2024-09-11 17:59:40.639 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4cebe74acacc9b96,e299facadf7b5e13] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hosts/batchCheckHostDirectoryPermissions client=10.31.33.41, traceId=4cebe74acacc9b96, duration=67 ms] 2024-09-11 17:59:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,03af98d7e16c028d,77d9128dbf74a964] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,03af98d7e16c028d,77d9128dbf74a964] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:41.339 INFO 2981023 --- [ocp-async-1,e17298df6e464b65,ff937520a5e33902] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:41.673 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,57c94318fdf726f8,6d9d2cdeab007f06] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hosts/batchCheckPortOccupied, client 10.31.33.41, traceId 57c94318fdf726f8] 2024-09-11 17:59:41.673 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,57c94318fdf726f8,6d9d2cdeab007f06] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hosts/batchCheckPortOccupied?null, client=10.31.33.41, traceId=57c94318fdf726f8, method=IterableResponse com.oceanbase.ocp.server.common.controller.ComputeController.batchCheckPortOccupied(BatchCheckPortOccupiedRequest), args=BatchCheckPortOccupiedRequest(hostIdList=[2000010], portList=[2881, 2882]),] 2024-09-11 17:59:41.673 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,57c94318fdf726f8,6d9d2cdeab007f06] c.o.o.c.host.service.HostServiceImpl : Batch check host ports occupied, request:BatchCheckPortOccupiedRequest(hostIdList=[2000010], portList=[2881, 2882]) 2024-09-11 17:59:41.679 INFO 2981023 --- [pool-560-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:41.679 INFO 2981023 --- [pool-560-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:41.690 INFO 2981023 --- [pool-560-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/checkPortOccupied, request body:CheckPortOccupiedRequest(portList=[2881, 2882]), params:null 2024-09-11 17:59:41.707 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,57c94318fdf726f8,6d9d2cdeab007f06] c.o.o.c.host.service.HostServiceImpl : Success check host ports occupied, result:[CheckHostPortOccupiedResult(hostId=null, hostIpAddress=10.31.35.7, resultList=[CheckPortOccupiedResult.OccupiedCheckResult(port=2881, occupied=false), CheckPortOccupiedResult.OccupiedCheckResult(port=2882, occupied=false)])] 2024-09-11 17:59:41.707 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,57c94318fdf726f8,6d9d2cdeab007f06] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hosts/batchCheckPortOccupied client=10.31.33.41, traceId=57c94318fdf726f8, duration=36 ms] 2024-09-11 17:59:41.754 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:41.754 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:42.370 INFO 2981023 --- [ocp-schedules-16,341fd2bc5042f876,f6b93775e255ecd9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:59:43.339 INFO 2981023 --- [ocp-async-6,181adebaa99b9347,728db25ee0ddd073] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:43.358 INFO 2981023 --- [ocp-schedules-3,02525c9f2ef01ac3,e7e9f44a82b4a86d] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:59:43.377 WARN 2981023 --- [ocp-schedules-3,02525c9f2ef01ac3,e7e9f44a82b4a86d] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:59:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,44a064816ad97322,53d0a93d42702dbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,44a064816ad97322,53d0a93d42702dbf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:44.754 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:44.754 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:45.101 INFO 2981023 --- [prometheus-second-schedule-1,45de01769a016dc9,7891133a2bc9ce90] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 17:59:45.339 INFO 2981023 --- [ocp-async-5,03d78fc88af10a0d,ac979c2444f17066] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:46.020 INFO 2981023 --- [ocp-schedules-15,7b353cc9d98b2a18,eb999ce479536a07] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:59:46.020 INFO 2981023 --- [ocp-schedules-26,15b7f68ab17812e4,923afbda738a56e1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:59:46.021 INFO 2981023 --- [host-check-agent-status2,7b353cc9d98b2a18,e9556b1e18f7cffb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:59:46.021 INFO 2981023 --- [host-check-agent-status1,7b353cc9d98b2a18,5432d89401881053] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:59:46.021 INFO 2981023 --- [host-check-clockdiff1,15b7f68ab17812e4,6c527b8d6dffadd1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 17:59:46.021 INFO 2981023 --- [host-check-clockdiff2,15b7f68ab17812e4,2ec9bc0527547efe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 17:59:46.021 INFO 2981023 --- [host-check-clockdiff3,15b7f68ab17812e4,5fca1855cf3dfcfb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 17:59:46.021 INFO 2981023 --- [host-check-clockdiff4,15b7f68ab17812e4,cef7a834fb880687] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 17:59:46.025 INFO 2981023 --- [host-check-agent-status3,7b353cc9d98b2a18,86d4a8af3db2c800] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:59:46.025 INFO 2981023 --- [host-check-agent-status4,7b353cc9d98b2a18,8906af83827a00cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:59:46.025 INFO 2981023 --- [ocp-schedules-4,764cace3cb85894a,c805c26683ab80b2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 17:59:46.028 INFO 2981023 --- [host-check-agent-status2,7b353cc9d98b2a18,e9556b1e18f7cffb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:46.029 INFO 2981023 --- [host-check-agent-status2,7b353cc9d98b2a18,e9556b1e18f7cffb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:46.030 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 17:59:46.031 INFO 2981023 --- [host-check-agent-status3,7b353cc9d98b2a18,86d4a8af3db2c800] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:59:46.032 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 17:59:46.032 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 17:59:46.032 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=AVAILABLE 2024-09-11 17:59:46.033 INFO 2981023 --- [host-check-agent-status3,7b353cc9d98b2a18,86d4a8af3db2c800] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:59:46.035 INFO 2981023 --- [host-check-agent-status4,7b353cc9d98b2a18,8906af83827a00cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:46.036 INFO 2981023 --- [host-check-agent-status4,7b353cc9d98b2a18,8906af83827a00cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:46.037 INFO 2981023 --- [host-check-clockdiff3,15b7f68ab17812e4,5fca1855cf3dfcfb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048786 0 0 , error= 2024-09-11 17:59:46.038 INFO 2981023 --- [host-check-agent-status1,7b353cc9d98b2a18,5432d89401881053] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:59:46.039 INFO 2981023 --- [host-check-agent-status1,7b353cc9d98b2a18,5432d89401881053] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:59:46.041 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:46.041 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:46.042 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:46.043 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:46.048 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:59:46.049 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:59:46.050 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:59:46.051 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:59:46.067 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 17:59:46.070 INFO 2981023 --- [host-check-agent-status2,7b353cc9d98b2a18,e9556b1e18f7cffb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 17:59:46.070 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 17:59:46.071 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T17:59:46.069660413+08:00 2024-09-11 17:59:46.071 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T17:59:46.071+08:00 2024-09-11 17:59:46.072 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T17:59:46.077611146+08:00 2024-09-11 17:59:46.073 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T17:59:46.071+08:00 and status AVAILABLE 2024-09-11 17:59:46.073 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T17:59:46.072+08:00 2024-09-11 17:59:46.074 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T17:59:46.072+08:00 and status AVAILABLE 2024-09-11 17:59:46.077 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 17:59:46.078 INFO 2981023 --- [host-check-agent-status3,7b353cc9d98b2a18,86d4a8af3db2c800] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 17:59:46.079 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T17:59:46.079119888+08:00 2024-09-11 17:59:46.079 INFO 2981023 --- [host-check-agent-status4,7b353cc9d98b2a18,8906af83827a00cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 17:59:46.080 INFO 2981023 --- [host-check-agent-status2,7b353cc9d98b2a18,e9556b1e18f7cffb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 17:59:46.080 INFO 2981023 --- [host-check-agent-status2,7b353cc9d98b2a18,e9556b1e18f7cffb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 17:59:46.080 INFO 2981023 --- [host-check-agent-status2,7b353cc9d98b2a18,e9556b1e18f7cffb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 17:59:46.081 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 17:59:46.083 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T17:59:46.088332595+08:00 2024-09-11 17:59:46.083 INFO 2981023 --- [host-check-agent-status4,7b353cc9d98b2a18,8906af83827a00cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 17:59:46.083 INFO 2981023 --- [host-check-agent-status4,7b353cc9d98b2a18,8906af83827a00cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 17:59:46.083 INFO 2981023 --- [host-check-agent-status4,7b353cc9d98b2a18,8906af83827a00cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 17:59:46.083 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T17:59:46.083+08:00 2024-09-11 17:59:46.083 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T17:59:46.079+08:00 2024-09-11 17:59:46.084 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 17:59:46.091 INFO 2981023 --- [host-check-agent-status3,7b353cc9d98b2a18,86d4a8af3db2c800] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 17:59:46.091 INFO 2981023 --- [host-check-agent-status3,7b353cc9d98b2a18,86d4a8af3db2c800] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 17:59:46.091 INFO 2981023 --- [host-check-agent-status3,7b353cc9d98b2a18,86d4a8af3db2c800] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 17:59:46.091 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T17:59:46.083+08:00 and status AVAILABLE 2024-09-11 17:59:46.101 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T17:59:46.079+08:00 and status AVAILABLE 2024-09-11 17:59:46.103 INFO 2981023 --- [host-check-agent-status1,7b353cc9d98b2a18,5432d89401881053] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 17:59:46.105 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:46.106 INFO 2981023 --- [host-check-agent-status1,7b353cc9d98b2a18,5432d89401881053] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 17:59:46.106 INFO 2981023 --- [host-check-agent-status1,7b353cc9d98b2a18,5432d89401881053] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 17:59:46.106 INFO 2981023 --- [host-check-agent-status1,7b353cc9d98b2a18,5432d89401881053] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 17:59:46.107 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:46.116 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 17:59:46.120 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 17:59:46.124 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:46.125 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:46.128 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:59:46.129 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 17:59:46.129 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:59:46.133 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:59:46.140 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:59:46.144 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:59:46.145 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:59:46.150 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 17:59:46.167 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 17:59:46.169 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 17:59:46.173 INFO 2981023 --- [ocp-schedules-20,26fac48b8fb2772c,94b423488130d8f3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 17:59:46.178 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:59:46.183 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 17:59:46.184 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 17:59:46.187 INFO 2981023 --- [ocp-schedules-25,3a3fd03f4d668d8d,b30fe19f98287642] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 17:59:46.187 INFO 2981023 --- [ocp-schedules-9,65a64bd2c37fbcae,c5b760df532e780a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 17:59:46.188 INFO 2981023 --- [ocp-schedules-1,d9faabfeedb4c32a,8618dfa091007476] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 17:59:46.188 INFO 2981023 --- [ocp-schedules-11,5cbc489ba4713236,10a07fd3e05eb778] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 17:59:46.194 INFO 2981023 --- [ocp-schedules-9,65a64bd2c37fbcae,c5b760df532e780a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:59:46.198 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 17:59:46.198 INFO 2981023 --- [ocp-schedules-1,d9faabfeedb4c32a,8618dfa091007476] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:59:46.204 INFO 2981023 --- [ocp-schedules-25,3a3fd03f4d668d8d,b30fe19f98287642] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:59:46.204 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 17:59:46.204 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 17:59:46.214 INFO 2981023 --- [ocp-schedules-20,26fac48b8fb2772c,94b423488130d8f3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 17:59:46.219 INFO 2981023 --- [ocp-schedules-7,587ae3d75604d12c,5abc22ab3e28739d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 17:59:46.224 INFO 2981023 --- [ocp-schedules-9,65a64bd2c37fbcae,c5b760df532e780a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 17:59:46.228 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 17:59:46.237 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:59:46.241 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:59:46.262 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 17:59:46.263 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:59:46.265 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.265 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:59:46.266 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:59:46.266 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.277 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 17:59:46.282 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 17:59:46.290 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.290 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:46.291 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 17:59:46.291 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 17:59:46.292 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 17:59:46.292 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 17:59:46.295 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.296 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:46.304 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 17:59:46.328 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.329 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:46.329 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 17:59:46.338 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 17:59:46.342 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/1000006/transferToDistributed, client 10.31.33.41, traceId 81ea8781302714e9] 2024-09-11 17:59:46.344 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/1000006/transferToDistributed?id=1000006, client=10.31.33.41, traceId=81ea8781302714e9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.transferToDistributed(Long,TransferToDistributedParam), args=1000006,TransferToDistributedParam(expandAllTenantReplica=true, addZoneParamList=[AddZoneParam(name=zone2, idcName=default_idc, servers=[2000010], architecture=x86_64, startupParameters=[])]),] 2024-09-11 17:59:46.347 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:59:46.350 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:59:46.360 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:59:46.354365+08:00, lastProcessTime=2024-09-11T17:59:36.352648+08:00 2024-09-11 17:59:46.362 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:59:46.365 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 17:59:46.368 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.369 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.369 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 17:59:46.370 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.370 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:59:46.371 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:59:46.371 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.374 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:46.374 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.376 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 17:59:46.380 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.381 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:59:46.382 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=ob-cluster-1000006, method=addZone 2024-09-11 17:59:46.386 INFO 2981023 --- [host-check-host-status4,764cace3cb85894a,8e4d6dadda11f287] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 17:59:46.388 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.388 INFO 2981023 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:46.389 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000006, obVersion=4.2.4.0 2024-09-11 17:59:46.390 INFO 2981023 --- [ocp-schedules-25,3a3fd03f4d668d8d,b30fe19f98287642] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:59:46.390 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.390 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 17:59:46.391 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:46.400 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 17:59:46.401 INFO 2981023 --- [host-check-host-status1,764cace3cb85894a,e2649a7176a4bedb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 17:59:46.411 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 17:59:46.411 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 17:59:46.417 INFO 2981023 --- [host-check-host-status2,764cace3cb85894a,1d1ae7ac44e2f7d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 17:59:46.426 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.426 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.427 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:46.428 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:59:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.429 INFO 2981023 --- [pool-561-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:59:46.431 INFO 2981023 --- [pool-561-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:59:46.438 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 17:59:46.445 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.446 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:46.446 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:59:46.451 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 17:59:46.459 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.462 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.462 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 17:59:46.463 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 17:59:46.463 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.463 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:59:46.464 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:59:46.464 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.466 INFO 2981023 --- [pool-561-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/userExists, request body:CheckUserExistsRequest(name=admin), params:null 2024-09-11 17:59:46.470 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:46.470 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 17:59:46.472 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.473 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.473 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:59:46.474 INFO 2981023 --- [pool-562-thread-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:46.475 INFO 2981023 --- [pool-562-thread-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:46.478 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.479 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.479 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:46.480 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 17:59:46.486 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 17:59:46.502 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 17:59:46.504 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.505 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.505 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:46.505 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 17:59:46.508 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.511 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:59:46.512 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:59:46.514 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 17:59:46.515 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 17:59:46.516 INFO 2981023 --- [pool-562-thread-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/system/commandExists, request body:CheckCommandExistsRequest(name=python2.7), params:null 2024-09-11 17:59:46.518 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 17:59:46.524 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.525 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select version() 2024-09-11 17:59:46.525 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000024, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:46.525 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:59:46.529 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.533 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.533 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:59:46.534 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:59:46.534 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.538 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:59:46.549 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 17:59:46.560 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:25.803588+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:46.561 INFO 2981023 --- [host-check-host-status3,764cace3cb85894a,d9d4665d0c70119f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 17:59:46.572 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.572 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.573 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:46.573 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.573 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.573 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 17:59:46.574 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:46.575 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkrtd 2024-09-11 17:59:46.578 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 17:59:46.580 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 2024-09-11 17:59:46.585 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:46.587 INFO 2981023 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:46.591 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.591 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.592 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select version() 2024-09-11 17:59:46.603 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 17:59:46.605 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 17:59:46.609 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.614 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.614 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 17:59:46.614 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.615 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.615 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 17:59:46.615 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 17:59:46.615 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.623 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 17:59:46.623 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.627 INFO 2981023 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:59:46.632 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 17:59:46.633 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 17:59:46.642 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:46.642 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 17:59:46.642 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 17:59:46.643 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/package/info, request body:GetPackageInfoRequest(name=oceanbase-huaweicloud-obs), params:null 2024-09-11 17:59:46.650 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.650 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.651 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:59:46.657 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 17:59:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:59:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 17:59:46.666 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 17:59:46.668 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 17:59:46.671 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 17:59:46.672 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 17:59:46.675 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 17:59:46.676 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 17:59:46.689 INFO 2981023 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000007, sqlPort: 2881 got ObProcessInfoResult(pid=80926, runPath=/data/rtdob/oceanbase, runUser=admin, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:59:46.690 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 17:59:46.692 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 17:59:46.696 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.696 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.697 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:59:46.698 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 17:59:46.699 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:59:46.354365+08:00 2024-09-11 17:59:46.701 INFO 2981023 --- [ocp-schedules-19,5180b7a15460c361,7c3ad1e0ea420bbc] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 17:59:46.703 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:59:46.703 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048284472000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 17:59:46.721 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 17:59:46.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:46.727 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:46.727 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 17:59:46.733 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:59:46.733 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048212661000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 17:59:46.733 INFO 2981023 --- [ocp-schedules-1,d9faabfeedb4c32a,8618dfa091007476] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:59:46.843 INFO 2981023 --- [ocp-schedules-24,a62dd9f2d78101e1,eb226146c6e1873f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7719,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":5012} 2024-09-11 17:59:46.843 INFO 2981023 --- [ocp-schedules-24,a62dd9f2d78101e1,eb226146c6e1873f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 17:59:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,2f49aca7ca1ec0d6,a3375a8e522fee52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,2f49aca7ca1ec0d6,a3375a8e522fee52] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:47.340 INFO 2981023 --- [ocp-async-1,090fa799aeec0693,61f45c53433bc0cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:47.664 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Add OB zone, nodeCount=35, taskCount=35 2024-09-11 17:59:47.664 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 17:59:47.664 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Add OB zone', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1000006, tenantId=null, hostId=null, creator='admin', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={cluster_name=linkrtd, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, prohibit_rollback=false, service_name=linkrtd:1719994443, ob_log_disk_path=/oceanbase/redo, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_cluster_id=1719994443, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, service_type=OB_CLUSTER, ob_disk_path_style=OBD, ob_svr_port=2882, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, ob_data_disk_path=/data/oceanbase, zone2.idc=default_idc, zone2.region=sys_region, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, createTime=null, startTime=2024-09-11T17:59:47.664+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Wait observer active, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.WaitObServerActiveTask, seriesId=11, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update zone status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.zone.UpdateObZoneStatusTask, seriesId=23, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=15, startTime=null, endT 2024-09-11 17:59:47.755 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:47.755 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:47.770 INFO 2981023 --- [ocp-async-7,b3d7fcd521a37623,df30dc2443e7667f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:59:47.779 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048223} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.790 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048220} BaseTagEntity{id=null, clusterName='linkrtd', clusterId=1000006, obClusterId=1719994443, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1000029, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.792 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048229} BaseTagEntity{id=null, clusterName='linkrtd', clusterId=1000006, obClusterId=1719994443, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1000029, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.795 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048209} BaseTagEntity{id=null, clusterName='linkrtd', clusterId=1000006, obClusterId=1719994443, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1000029, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.797 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048221} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.799 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048234} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.801 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048232} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.803 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048204} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.804 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048215} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.806 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048224} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.809 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048203} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.811 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048210} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2000010, svrIp='10.31.35.7', deleted=false, createTime=null} 2024-09-11 17:59:47.817 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=4048201} BaseTagEntity{id=null, clusterName='linkrtd', clusterId=1000006, obClusterId=1719994443, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null} 2024-09-11 17:59:47.824 INFO 2981023 --- [host-check-clockdiff1,15b7f68ab17812e4,6c527b8d6dffadd1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048787 5 5 , error= 2024-09-11 17:59:47.825 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=4048203, taskName=Add OB zone, argument=Argument{parallelIdx=-1, stringMap={cluster_name=linkrtd, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, prohibit_rollback=false, service_name=linkrtd:1719994443, ob_log_disk_path=/oceanbase/redo, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_cluster_id=1719994443, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, service_type=OB_CLUSTER, ob_disk_path_style=OBD, ob_svr_port=2882, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, ob_data_disk_path=/data/oceanbase, zone2.idc=default_idc, zone2.region=sys_region, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}} 2024-09-11 17:59:47.830 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.task.TaskTagService : Task tag already exist. taskId=4048203 2024-09-11 17:59:47.840 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TRANSFER_STANDALONE_TO_DISTRIBUTED, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.transferToDistributed, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=1000006, resourceName=linkrtd:1719994443, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=4048203, eventInfo={"param":{"addZoneParamList":[{"architecture":"x86_64","idcName":"default_idc","name":"zone2","servers":[2000010],"startupParameters":[]}],"expandAllTenantReplica":true},"id":1000006}, executionType=NEW, requestUrl=http://pms-cas.linkreit.com:15002/api/v2/ob/clusters/1000006/transferToDistributed, clientIp=10.31.33.41, serverIp=10.31.35.10, startTime=2024-09-11T17:59:46.375+08:00, endTime=null, role=null, aliUid=null) 2024-09-11 17:59:47.851 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,81ea8781302714e9,10a55fa637061328] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/1000006/transferToDistributed client=10.31.33.41, traceId=81ea8781302714e9, duration=1515 ms] 2024-09-11 17:59:48.548 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,81a71f244413a74f,3cc5762be179b3b4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.31.33.41, traceId=81a71f244413a74f, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-09-11 17:59:48.549 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,9beea8c782c203c1,7daa952509f34c0a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9beea8c782c203c1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:59:48.551 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,81a71f244413a74f,3cc5762be179b3b4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.31.33.41, traceId=81a71f244413a74f, duration=7 ms] 2024-09-11 17:59:48.564 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,09b15908d0577ea7,69355b0e2ac1f0f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/summary?id=1000006, client=10.31.33.41, traceId=09b15908d0577ea7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1000006,] 2024-09-11 17:59:48.598 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,9beea8c782c203c1,7daa952509f34c0a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9beea8c782c203c1, duration=54 ms] 2024-09-11 17:59:48.645 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,09b15908d0577ea7,69355b0e2ac1f0f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:48.647 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,09b15908d0577ea7,69355b0e2ac1f0f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:59:48.682 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,685ad48471fa8327] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048205, operation:EXECUTE, state: RUNNING 2024-09-11 17:59:48.685 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,09b15908d0577ea7,69355b0e2ac1f0f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/summary client=10.31.33.41, traceId=09b15908d0577ea7, duration=141 ms] 2024-09-11 17:59:48.782 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,975760f249da6280] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048205, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Prepare add zone, zone2.idc=default_idc, sub_task_instance_id=4048205, zone2.region=sys_region, cluster_name=linkrtd, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, service_name=linkrtd:1719994443, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, latest_execution_start_time=2024-09-11T17:59:48.677+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 17:59:48.793 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,975760f249da6280] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048205, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:59:49.295 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,746feda449276035,14da46d375ab9718] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&version=2&ObClusterId=1719994437, client=10.31.35.10, traceId=746feda449276035, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkprdocp,1719994437,2,{"ObRegion":"linkprdocp","ObCluster":"linkprdocp","ObRegionId":1719994437,"ObClusterId":1719994437,"Type":"PRIMARY","timestamp":1726048789294192,"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 17:59:49.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,746feda449276035,14da46d375ab9718] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkprdocp, obClusterId=1719994437, rootServerJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, groupName=linkprdocp, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 17:59:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,746feda449276035,14da46d375ab9718] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 17:59:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,746feda449276035,14da46d375ab9718] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkprdocp, clusterId=1719994437, affectRows=1 2024-09-11 17:59:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,746feda449276035,14da46d375ab9718] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.10, traceId=746feda449276035, duration=9 ms] 2024-09-11 17:59:49.340 INFO 2981023 --- [ocp-async-3,1823077a16799983,67a45ad27bd1ad22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:49.712 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048224] 2024-09-11 17:59:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:59:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 17:59:50.015 INFO 2981023 --- [ocp-schedules-5,8e43f79c10dd9e1f,95dc12c2a471b623] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 17:59:50.065 WARN 2981023 --- [scanner-13585695-1,8e43f79c10dd9e1f,2b1e87dba7b8daa9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 17:59:50.092 INFO 2981023 --- [alarm-detect-8,8e43f79c10dd9e1f,8c3c9cebf93ef208] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645148, data is empty. 2024-09-11 17:59:50.096 INFO 2981023 --- [alarm-detect-8,8e43f79c10dd9e1f,8c3c9cebf93ef208] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645148, data is empty. 2024-09-11 17:59:50.102 INFO 2981023 --- [alarm-detect-8,8e43f79c10dd9e1f,8c3c9cebf93ef208] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645148, data is empty. 2024-09-11 17:59:50.112 INFO 2981023 --- [alarm-detect-9,8e43f79c10dd9e1f,5a786b36a98ccab5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645148, data is empty. 2024-09-11 17:59:50.115 INFO 2981023 --- [alarm-detect-19,8e43f79c10dd9e1f,14448b10175b5b21] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645148, data is empty. 2024-09-11 17:59:50.118 INFO 2981023 --- [alarm-detect-19,8e43f79c10dd9e1f,14448b10175b5b21] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645148, data is empty. 2024-09-11 17:59:50.120 INFO 2981023 --- [alarm-detect-9,8e43f79c10dd9e1f,5a786b36a98ccab5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645148, data is empty. 2024-09-11 17:59:50.127 INFO 2981023 --- [alarm-detect-9,8e43f79c10dd9e1f,5a786b36a98ccab5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645148, data is empty. 2024-09-11 17:59:50.128 INFO 2981023 --- [alarm-detect-19,8e43f79c10dd9e1f,14448b10175b5b21] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645148, data is empty. 2024-09-11 17:59:50.231 INFO 2981023 --- [alarm-detect-9,8e43f79c10dd9e1f,4a45d4f885f2e5f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645148, data is empty. 2024-09-11 17:59:50.235 INFO 2981023 --- [alarm-detect-9,8e43f79c10dd9e1f,4a45d4f885f2e5f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645148, data is empty. 2024-09-11 17:59:50.238 INFO 2981023 --- [alarm-detect-9,8e43f79c10dd9e1f,4a45d4f885f2e5f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645148, data is empty. 2024-09-11 17:59:50.284 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 17:59:50.288 INFO 2981023 --- [ocp-schedules-24,ea7d54efd694506c,548e6fd91b046379] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.289 INFO 2981023 --- [ocp-schedules-1,94aac6628d54d18b,55e969571ced0092] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 17:59:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,ed5f241c2a40d872,614df6d9e7478b85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,ed5f241c2a40d872,614df6d9e7478b85] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:50.527 INFO 2981023 --- [host-check-clockdiff2,15b7f68ab17812e4,2ec9bc0527547efe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048790 5 5 , error= 2024-09-11 17:59:50.738 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,8c9a4e949c098e0d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048224, operation:EXECUTE, state: RUNNING 2024-09-11 17:59:50.755 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:50.757 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:50.822 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,3a2479221fac5ed2,b3e06d765c4926c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.31.33.41, traceId=3a2479221fac5ed2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:50.823 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-1,3a2479221fac5ed2,b3e06d765c4926c6] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:59:50.832 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6422663534f6177f,62f54ac92a1a015d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/alarmSummaries?id=1000006, client=10.31.33.41, traceId=6422663534f6177f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=1000006,] 2024-09-11 17:59:50.833 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,3dbb0687cc99556a,8d4da8e4a95de2dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?status=AVAILABLE&status=ONLINE&excludedServiceType=OB_CLUSTER&excludedServiceType=BACKUP_AGENT&excludedServiceType=RESTORE_AGENT&excludedServiceType=BACKUP_RESTORE_AGENT&excludedServiceType=ARBITRATION&excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=3dbb0687cc99556a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[AVAILABLE, ONLINE],[OB_CLUSTER, BACKUP_AGENT, RESTORE_AGENT, BACKUP_RESTORE_AGENT, ARBITRATION, BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:50.840 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,53b4d0f4b1000db9,1eadc865b5dc6703] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/serverStats?id=1000006, client=10.31.33.41, traceId=53b4d0f4b1000db9, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=1000006,] 2024-09-11 17:59:50.841 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-7,3dbb0687cc99556a,8d4da8e4a95de2dc] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:59:50.830 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,b8f4d415ebde07cd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048224, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Check ob server network, zone2.idc=default_idc, sub_task_instance_id=4048224, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:59:50.736+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 17:59:50.842 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,43b71c05dbec6a9b,83fc92a77fb2cae0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/eventMeta?id=1000006, client=10.31.33.41, traceId=43b71c05dbec6a9b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1000006,] 2024-09-11 17:59:50.842 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,43b71c05dbec6a9b,83fc92a77fb2cae0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/eventMeta client=10.31.33.41, traceId=43b71c05dbec6a9b, duration=5 ms] 2024-09-11 17:59:50.854 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6422663534f6177f,62f54ac92a1a015d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/alarmSummaries client=10.31.33.41, traceId=6422663534f6177f, duration=25 ms] 2024-09-11 17:59:50.864 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,53b4d0f4b1000db9,1eadc865b5dc6703] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:50.866 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,53b4d0f4b1000db9,1eadc865b5dc6703] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 17:59:50.867 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,53b4d0f4b1000db9,1eadc865b5dc6703] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/serverStats client=10.31.33.41, traceId=53b4d0f4b1000db9, duration=30 ms] 2024-09-11 17:59:50.841 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,b8f4d415ebde07cd] c.o.o.s.t.b.c.CheckServerNetworkTask : check server network begin, serverAddrList=[10.31.35.6:2882] 2024-09-11 17:59:50.867 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,3a2479221fac5ed2,b3e06d765c4926c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=3a2479221fac5ed2, duration=48 ms] 2024-09-11 17:59:50.871 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,3dbb0687cc99556a,8d4da8e4a95de2dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=3dbb0687cc99556a, duration=42 ms] 2024-09-11 17:59:50.871 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,b8f4d415ebde07cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:50.879 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,b8f4d415ebde07cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:50.899 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,b8f4d415ebde07cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/ob/checkConnect, request body:CheckPortConnectRequest(checkItemList=[CheckPortConnectItem(ip=10.31.35.6, portList=[2882])]), params:null 2024-09-11 17:59:50.908 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,b8f4d415ebde07cd] c.o.o.s.t.b.c.CheckServerNetworkTask : check server network end 2024-09-11 17:59:50.917 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,b8f4d415ebde07cd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048224, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:59:50.931 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b3dcc2c75b85d769,f077bf9a2af4e28c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=b3dcc2c75b85d769, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:59:50.934 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b3dcc2c75b85d769,f077bf9a2af4e28c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=b3dcc2c75b85d769, duration=5 ms] 2024-09-11 17:59:50.970 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,29b527305967980d,be30a368cdf89f6c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top/realtime?metrics=active_session%2Cqps%2Ctps&labels=app%3AOB%2Cob_cluster_name%3Alinkrtd%2Cob_cluster_id%3A1719994443&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&queryAsync=true&endTime=2024-09-11T17%3A59%3A50%2B08%3A00&limit=1&maxPoints=360, client=10.31.33.41, traceId=29b527305967980d, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryRealtimeTopMetrics(OffsetDateTime,Long,String,String,Long,Long,Integer,Integer,String,Boolean,QueryRouteType), args=2024-09-11T17:59:50+08:00,60,active_session,qps,tps,app,ob_cluster_name,ob_cluster_id,0,0,360,1,app:OB,ob_cluster_name:linkrtd,ob_cluster_id:1719994443,true,NOT_ROUTED,] 2024-09-11 17:59:50.971 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,29b527305967980d,be30a368cdf89f6c] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = active_session 2024-09-11 17:59:50.971 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,29b527305967980d,be30a368cdf89f6c] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = active_session, minTimestamp = 0, minTimeBuffer = 0, startTime = 1726048740 2024-09-11 17:59:50.973 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,29b527305967980d,be30a368cdf89f6c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top/realtime client=10.31.33.41, traceId=29b527305967980d, duration=6 ms] 2024-09-11 17:59:50.983 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a50f24b880c4ee8e,b2c12749f1a9ee0b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/topSql/status/events?null, client=10.31.33.41, traceId=a50f24b880c4ee8e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitchEvents(Long,OffsetDateTime,OffsetDateTime,Pageable), args=1000006,Page request [number: 0, size 5000, sort: timestamp: ASC],] 2024-09-11 17:59:50.992 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,83a6c1fe620b2cee,2922527dae1eed74] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top/realtime?metrics=active_session%2Cqps%2Ctps&labels=app%3AOB%2Cob_cluster_name%3Alinkrtd%2Cob_cluster_id%3A1719994443&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&queryAsync=true&minStep=1&endTime=2024-09-11T17%3A59%3A50%2B08%3A00&limit=1&maxPoints=360, client=10.31.33.41, traceId=83a6c1fe620b2cee, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryRealtimeTopMetrics(OffsetDateTime,Long,String,String,Long,Long,Integer,Integer,String,Boolean,QueryRouteType), args=2024-09-11T17:59:50+08:00,60,active_session,qps,tps,app,ob_cluster_name,ob_cluster_id,0,1,360,1,app:OB,ob_cluster_name:linkrtd,ob_cluster_id:1719994443,true,NOT_ROUTED,] 2024-09-11 17:59:50.992 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2b5a538761ff4f39,abab5fa5788e4416] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=2b5a538761ff4f39, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:50.993 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a50f24b880c4ee8e,b2c12749f1a9ee0b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/topSql/status/events client=10.31.33.41, traceId=a50f24b880c4ee8e, duration=12 ms] 2024-09-11 17:59:50.993 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-4,83a6c1fe620b2cee,2922527dae1eed74] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = active_session 2024-09-11 17:59:50.993 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-4,83a6c1fe620b2cee,2922527dae1eed74] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = active_session, minTimestamp = 0, minTimeBuffer = 0, startTime = 1726048740 2024-09-11 17:59:50.997 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2b5a538761ff4f39,abab5fa5788e4416] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 17:59:50.997 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,83a6c1fe620b2cee,2922527dae1eed74] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top/realtime client=10.31.33.41, traceId=83a6c1fe620b2cee, duration=7 ms] 2024-09-11 17:59:51.004 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,787b3ed2e8cec2c5,6f9f60de135c3573] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.31.33.41, traceId=787b3ed2e8cec2c5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 17:59:51.007 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,787b3ed2e8cec2c5,6f9f60de135c3573] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.31.33.41, traceId=787b3ed2e8cec2c5, duration=6 ms] 2024-09-11 17:59:51.008 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2b5a538761ff4f39,abab5fa5788e4416] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=2b5a538761ff4f39, duration=18 ms] 2024-09-11 17:59:51.054 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,5df1c74dfcbc7928,78ae0063f648b055] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/events?startTime=2024-09-04T17%3A59%3A49%2B08%3A00&endTime=2024-09-11T17%3A59%3A49%2B08%3A00&page=1&size=7, client=10.31.33.41, traceId=5df1c74dfcbc7928, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1000006,2024-09-04T17:59:49+08:00,2024-09-11T17:59:49+08:00,Page request [number: 0, size 7, sort: timestamp: DESC],] 2024-09-11 17:59:51.061 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,5df1c74dfcbc7928,78ae0063f648b055] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/events client=10.31.33.41, traceId=5df1c74dfcbc7928, duration=9 ms] 2024-09-11 17:59:51.090 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,fd03d255fc861c1a,1b11b44941a3e41f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fd03d255fc861c1a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:59:51.095 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,fd03d255fc861c1a,1b11b44941a3e41f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fd03d255fc861c1a, duration=8 ms] 2024-09-11 17:59:51.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,69db8a59029061c8,b0632ef2da683270] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=69db8a59029061c8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:59:51.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,69db8a59029061c8,b0632ef2da683270] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=69db8a59029061c8, duration=4 ms] 2024-09-11 17:59:51.229 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ae6cf36b61968f73,b5a3fcdff6e3ce57] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=ae6cf36b61968f73, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 17:59:51.231 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ae6cf36b61968f73,b5a3fcdff6e3ce57] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=ae6cf36b61968f73, duration=5 ms] 2024-09-11 17:59:51.340 INFO 2981023 --- [ocp-async-7,873b4479ab6de385,62050e4888b740bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:51.587 INFO 2981023 --- [ocp-schedules-5,68ace0ed625d0d82,5e634eb75032b9ed] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 17:59:51.588 INFO 2981023 --- [ocp-schedules-5,68ace0ed625d0d82,5e634eb75032b9ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:59:51.588 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:59:51.588 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:59:51.588 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 17:59:51.589 INFO 2981023 --- [ocp-schedules-5,68ace0ed625d0d82,5e634eb75032b9ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 17:59:51.803 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048216] 2024-09-11 17:59:52.007 INFO 2981023 --- [host-check-clockdiff4,15b7f68ab17812e4,cef7a834fb880687] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048792 -1 -1 , error= 2024-09-11 17:59:52.372 INFO 2981023 --- [ocp-schedules-6,e4021c56db88a0bb,fbe6413d485ca7de] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 17:59:52.605 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2573d528017d435d,0580af9a2a99db3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2573d528017d435d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:59:52.612 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2573d528017d435d,0580af9a2a99db3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2573d528017d435d, duration=10 ms] 2024-09-11 17:59:52.789 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,5036f3f9ac42d751] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048216, operation:EXECUTE, state: RUNNING 2024-09-11 17:59:52.875 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,e3bd84dcc462e059] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048216, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048216, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:59:52.787+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 17:59:52.884 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,e3bd84dcc462e059] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048216, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:59:53.215 INFO 2981023 --- [ocp-schedules-29,053612ca756475f1,99496ab3afa1c942] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 17:59:53.215 INFO 2981023 --- [ocp-schedules-29,053612ca756475f1,99496ab3afa1c942] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 17:59:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,8fb3b748ae8d8484,e0365d2cc8efc882] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,8fb3b748ae8d8484,e0365d2cc8efc882] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:53.340 INFO 2981023 --- [ocp-async-8,2e10859672a0b1c2,b42ca7c7949595d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:53.757 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:53.759 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:53.886 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,92a2ff76ad3bfb99,26160b64d6b930c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=92a2ff76ad3bfb99, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:59:53.894 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048212] 2024-09-11 17:59:53.899 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,92a2ff76ad3bfb99,26160b64d6b930c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=92a2ff76ad3bfb99, duration=17 ms] 2024-09-11 17:59:53.973 INFO 2981023 --- [ocp-schedules-3,8476bdf92eb8332a,263347e3ee5daef6] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 17:59:53.992 WARN 2981023 --- [ocp-schedules-3,8476bdf92eb8332a,263347e3ee5daef6] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 17:59:54.352 INFO 2981023 --- [ocp-schedules-23,71ad9f55e5e7aa85,7e656d451d677981] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 17:59:54.821 INFO 2981023 --- [ocp-schedules-2,f9bdfdb3f232e6f2,894202f5c80f35b1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 17:59:54.831 INFO 2981023 --- [ocp-schedules-2,f9bdfdb3f232e6f2,894202f5c80f35b1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:59:54.831 INFO 2981023 --- [ocp-schedules-2,f9bdfdb3f232e6f2,894202f5c80f35b1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 17:59:54.837 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,4922b025f73fe1bb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048212, operation:EXECUTE, state: RUNNING 2024-09-11 17:59:54.922 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,4572d6c4a2a79557] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048212, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Reserve host, zone2.idc=default_idc, sub_task_instance_id=4048212, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:59:54.835+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 17:59:54.927 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,4572d6c4a2a79557] c.o.o.s.t.business.host.ReserveHostTask : reserve host 2000010 for service OB_CLUSTER linkrtd:1719994443 2024-09-11 17:59:54.956 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,4572d6c4a2a79557] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key row_lock_compute_host_2000010 2024-09-11 17:59:54.973 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,4572d6c4a2a79557] c.o.o.c.h.m.HostServiceManagerImpl : Reserved host: service [OB_CLUSTER:linkrtd:1719994443:4.2.4.0], host [2000010:10.31.35.7], status changed from [AVAILABLE] to [ONLINE] 2024-09-11 17:59:54.988 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,4572d6c4a2a79557] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048212, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:59:55.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a3d370eba14f7c0b,4eff8eee931c8be3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a3d370eba14f7c0b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:59:55.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a3d370eba14f7c0b,4eff8eee931c8be3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a3d370eba14f7c0b, duration=9 ms] 2024-09-11 17:59:55.340 INFO 2981023 --- [ocp-async-2,e12f40daa7e5f675,c84d7e265c02e38b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:55.817 INFO 2981023 --- [ocp-schedules-16,b9f152c07e558b2c,ade3f490bec5b216] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 17:59:55.817 INFO 2981023 --- [ocp-schedules-16,b9f152c07e558b2c,ade3f490bec5b216] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690420679617835008 2024-09-11 17:59:55.818 INFO 2981023 --- [ocp-schedules-16,b9f152c07e558b2c,ade3f490bec5b216] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 17:59:55.818 INFO 2981023 --- [ocp-schedules-16,b9f152c07e558b2c,ade3f490bec5b216] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 17:59:55.818 INFO 2981023 --- [ocp-schedules-16,b9f152c07e558b2c,ade3f490bec5b216] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 17:59:55.818 INFO 2981023 --- [ocp-schedules-16,b9f152c07e558b2c,ade3f490bec5b216] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 17:59:55.950 INFO 2981023 --- [ocp-schedules-23,be24cdb413b4d01a,f930e41d2920a5f9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 17:59:55.956 INFO 2981023 --- [ocp-schedules-23,be24cdb413b4d01a,f930e41d2920a5f9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 17:59:55.956 INFO 2981023 --- [ocp-schedules-23,be24cdb413b4d01a,f930e41d2920a5f9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 17:59:55.981 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048210] 2024-09-11 17:59:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,6ab3e684d7764286,3eea22637a9e96c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,6ab3e684d7764286,3eea22637a9e96c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:56.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T17:59:56.353389+08:00, lastProcessTime=2024-09-11T17:59:46.354365+08:00 2024-09-11 17:59:56.405 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,5a0acce9ca8a2c13,bb770300699b1612] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5a0acce9ca8a2c13, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:59:56.413 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,5a0acce9ca8a2c13,bb770300699b1612] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5a0acce9ca8a2c13, duration=11 ms] 2024-09-11 17:59:56.421 INFO 2981023 --- [ocp-schedules-13,6d768dc3491004de,5576bfd95144e6d5] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T17:59:56.421+08:00, clusters=[1, 2, 1000006] 2024-09-11 17:59:56.425 INFO 2981023 --- [ocp-schedules-13,6d768dc3491004de,5576bfd95144e6d5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 17:59:56.538 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T17:59:56.353389+08:00 2024-09-11 17:59:56.542 INFO 2981023 --- [ocp-schedules-13,6d768dc3491004de,5576bfd95144e6d5] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 17 to SqlAggregateData 2024-09-11 17:59:56.543 INFO 2981023 --- [ocp-schedules-13,6d768dc3491004de,5576bfd95144e6d5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 17:59:56.759 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:56.759 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:56.885 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,7ae84813d3611859] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048210, operation:EXECUTE, state: RUNNING 2024-09-11 17:59:56.966 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,bc520f4ecd3cec72] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048210, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Pre check for install ob, zone2.idc=default_idc, sub_task_instance_id=4048210, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:59:56.884+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 17:59:56.972 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,bc520f4ecd3cec72] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 17:59:56.979 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,bc520f4ecd3cec72] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 17:59:57.050 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,bc520f4ecd3cec72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/ob/install/preCheck, request body:ObClusterInstallRequest(obClusterName=linkrtd, obVersion=4.2.4.0, obPath=ObPath(installPath=/data/rtdob/oceanbase, dataPath=/data/oceanbase, logPath=/oceanbase/redo, diskPathStyle=OBD, runPath=/data/rtdob/oceanbase), runUser=null), params:null 2024-09-11 17:59:57.063 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,bc520f4ecd3cec72] .o.o.i.c.ClusterHostOperationServiceImpl : ob install pre check for observer, expect process should not exist, passed=true 2024-09-11 17:59:57.067 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,bc520f4ecd3cec72] .o.o.i.c.ClusterHostOperationServiceImpl : ob install pre check for /data/oceanbase/linkrtd/sstable/block_file, expect file should not exist, passed=true 2024-09-11 17:59:57.071 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,bc520f4ecd3cec72] .o.o.i.c.ClusterHostOperationServiceImpl : ob install pre check for /oceanbase/redo/linkrtd/clog, expect file should not exist, passed=true 2024-09-11 17:59:57.075 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,bc520f4ecd3cec72] .o.o.i.c.ClusterHostOperationServiceImpl : all ob install pre check passed 2024-09-11 17:59:57.088 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,bc520f4ecd3cec72] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048210, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:59:57.211 INFO 2981023 --- [ocp-schedules-12,7c030c0ed65d9f36,2dfbac02cd846d63] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 17:59:57.211 INFO 2981023 --- [ocp-schedules-12,7c030c0ed65d9f36,2dfbac02cd846d63] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 17:59:57.214 INFO 2981023 --- [ocp-schedules-7,dcc597b8befa47ed,547fb263bf04de0a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 17:59:57.215 INFO 2981023 --- [ocp-schedules-7,dcc597b8befa47ed,547fb263bf04de0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 17:59:57.238 INFO 2981023 --- [Collect all tenants major compactions3,7c030c0ed65d9f36,51e8d795cd530a7e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:59:57.239 INFO 2981023 --- [Collect all tenants major compactions1,7c030c0ed65d9f36,71372b97caf49fcf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:59:57.239 INFO 2981023 --- [Collect all tenants major compactions2,7c030c0ed65d9f36,0e64ed018f445565] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:59:57.241 INFO 2981023 --- [Collect all tenants major compactions4,7c030c0ed65d9f36,5e8b16aa4890fd3e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:59:57.244 INFO 2981023 --- [Collect all tenants major compactions6,7c030c0ed65d9f36,49c2aeed41ced6a7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:59:57.245 INFO 2981023 --- [Collect all tenants major compactions5,7c030c0ed65d9f36,886944ed7b49cc5b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:59:57.256 INFO 2981023 --- [Collect all tenants major compactions4,7c030c0ed65d9f36,5e8b16aa4890fd3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.257 INFO 2981023 --- [Collect all tenants major compactions4,7c030c0ed65d9f36,5e8b16aa4890fd3e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:59:57.257 INFO 2981023 --- [Collect all tenants major compactions3,7c030c0ed65d9f36,51e8d795cd530a7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.257 INFO 2981023 --- [Collect all tenants major compactions2,7c030c0ed65d9f36,0e64ed018f445565] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.257 INFO 2981023 --- [Collect all tenants major compactions1,7c030c0ed65d9f36,71372b97caf49fcf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.258 INFO 2981023 --- [Collect all tenants major compactions3,7c030c0ed65d9f36,51e8d795cd530a7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:59:57.258 INFO 2981023 --- [Collect all tenants major compactions2,7c030c0ed65d9f36,0e64ed018f445565] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 17:59:57.258 INFO 2981023 --- [Collect all tenants major compactions1,7c030c0ed65d9f36,71372b97caf49fcf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 17:59:57.261 INFO 2981023 --- [Collect all tenants major compactions4,7c030c0ed65d9f36,5e8b16aa4890fd3e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 17:59:57.261 INFO 2981023 --- [Collect all tenants major compactions3,7c030c0ed65d9f36,51e8d795cd530a7e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 17:59:57.261 INFO 2981023 --- [Collect all tenants major compactions4,7c030c0ed65d9f36,5e8b16aa4890fd3e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 17:59:57.262 INFO 2981023 --- [Collect all tenants major compactions1,7c030c0ed65d9f36,71372b97caf49fcf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 17:59:57.262 INFO 2981023 --- [Collect all tenants major compactions1,7c030c0ed65d9f36,71372b97caf49fcf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 17:59:57.261 INFO 2981023 --- [Collect all tenants major compactions3,7c030c0ed65d9f36,51e8d795cd530a7e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 17:59:57.262 INFO 2981023 --- [Collect all tenants major compactions2,7c030c0ed65d9f36,0e64ed018f445565] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 17:59:57.262 INFO 2981023 --- [Collect all tenants major compactions2,7c030c0ed65d9f36,0e64ed018f445565] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 17:59:57.266 INFO 2981023 --- [Collect all tenants major compactions5,7c030c0ed65d9f36,886944ed7b49cc5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.267 INFO 2981023 --- [Collect all tenants major compactions5,7c030c0ed65d9f36,886944ed7b49cc5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:59:57.269 INFO 2981023 --- [Collect all tenants major compactions6,7c030c0ed65d9f36,49c2aeed41ced6a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.269 INFO 2981023 --- [Collect all tenants major compactions5,7c030c0ed65d9f36,886944ed7b49cc5b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 17:59:57.269 INFO 2981023 --- [Collect all tenants major compactions5,7c030c0ed65d9f36,886944ed7b49cc5b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 17:59:57.270 INFO 2981023 --- [Collect all tenants major compactions6,7c030c0ed65d9f36,49c2aeed41ced6a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 17:59:57.272 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:59:57.272 INFO 2981023 --- [Collect all tenants major compactions6,7c030c0ed65d9f36,49c2aeed41ced6a7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 17:59:57.272 INFO 2981023 --- [Collect all tenants major compactions6,7c030c0ed65d9f36,49c2aeed41ced6a7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 17:59:57.272 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:59:57.272 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.272 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.273 INFO 2981023 --- [ocp-schedules-12,7c030c0ed65d9f36,2dfbac02cd846d63] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 17:59:57.275 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:59:57.278 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:59:57.279 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:59:57.279 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.279 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.280 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.282 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.283 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.283 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 17:59:57.286 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.286 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.291 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.291 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.291 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:59:57.292 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.293 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.294 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.301 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.301 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:59:57.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.304 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.305 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:59:57.305 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.305 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4261D6E2-0-0] 2024-09-11 17:59:57.305 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:59:57.305 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.306 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:59:57.307 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.307 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:59:57.315 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.316 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.316 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:59:57.316 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:59:57.319 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.320 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.329 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.329 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.329 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:59:57.329 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 17:59:57.329 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4261D6EC-0-0] 2024-09-11 17:59:57.330 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:59:57.330 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 17:59:57.330 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.331 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:59:57.331 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.332 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:59:57.332 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 17:59:57.337 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:59:57.337 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.338 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:59:57.339 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.339 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.339 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:59:57.340 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.340 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 17:59:57.340 INFO 2981023 --- [ocp-async-7,eabb518279bc0f00,7f20fc7dd0ad0cfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:57.341 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:59:57.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.342 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:59:57.343 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:59:57.343 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.343 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.344 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:59:57.344 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:59:57.345 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.345 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:59:57.346 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 17:59:57.347 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.348 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 17:59:57.348 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.349 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:59:57.349 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:59:57.350 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.350 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:59:57.351 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.352 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 17:59:57.353 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.354 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 17:59:57.354 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.355 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 17:59:57.356 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 17:59:57.357 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:57.358 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 17:59:57.359 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.360 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.366 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:57.368 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.368 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:59:57.373 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:59:57.375 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 17:59:57.376 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.377 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 17:59:57.377 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 17:59:57.377 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 17:59:57.377 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 17:59:57.381 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:59:57.383 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 17:59:57.383 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.383 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:59:57.384 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 17:59:57.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 17:59:57.394 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:59:57.395 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:59:57.396 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 17:59:57.396 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 17:59:57.397 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 17:59:57.397 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 17:59:57.397 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 17:59:57.397 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 17:59:57.402 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 17:59:57.403 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 17:59:57.403 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 17:59:57.403 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 17:59:57.403 INFO 2981023 --- [ocp-schedules-7,dcc597b8befa47ed,547fb263bf04de0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 17:59:57.673 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7097126ea24783e3,638284a9fc22516c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7097126ea24783e3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:59:57.678 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7097126ea24783e3,638284a9fc22516c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7097126ea24783e3, duration=8 ms] 2024-09-11 17:59:57.770 INFO 2981023 --- [ocp-async-2,3d97a631df9dc9b1,6a0dcd0a20d62eea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 17:59:58.064 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048202] 2024-09-11 17:59:58.316 INFO 2981023 --- [ocp-schedules-30,852e105a1ea99f54,040187c932f9a02d] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 17:59:58.938 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,68afcdf62128037c,32a439de37b0a3aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=68afcdf62128037c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 17:59:58.947 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,c85ae1b07c6a4eb4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048202, operation:EXECUTE, state: RUNNING 2024-09-11 17:59:58.952 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,68afcdf62128037c,32a439de37b0a3aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=68afcdf62128037c, duration=20 ms] 2024-09-11 17:59:59.040 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,196cba8f0435151a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048202, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048202, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:59:58.943+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 17:59:59.050 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,196cba8f0435151a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048202, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 17:59:59.108 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048233] 2024-09-11 17:59:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,61fc0488ceb3b96d,ce59b61a0eb3db45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048800, interval=60 2024-09-11 17:59:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,61fc0488ceb3b96d,ce59b61a0eb3db45] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 17:59:59.340 INFO 2981023 --- [ocp-async-9,b0a69eeabac6d078,4633903365dec059] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 17:59:59.759 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 17:59:59.761 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 17:59:59.994 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,2bd768ef97b42f15] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048233, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:00.076 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,44747a5cb82a6ceb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048233, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Set zone context, zone2.idc=default_idc, sub_task_instance_id=4048233, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T17:59:59.993+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:00:00.118 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,44747a5cb82a6ceb] c.o.o.s.t.b.zone.SetZoneContextTask : set operating zone to zone2 2024-09-11 18:00:00.130 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,44747a5cb82a6ceb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048233, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:00:00.203 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7839016ec19254c9,a3853d668ba1a3d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7839016ec19254c9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:00.209 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7839016ec19254c9,a3853d668ba1a3d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7839016ec19254c9, duration=9 ms] 2024-09-11 18:00:00.282 INFO 2981023 --- [ocp-schedules-18,1a486b96c47035d2,063bb138f18f7a8f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:00:00.320 WARN 2981023 --- [scanner-13585995-1,1a486b96c47035d2,011fe3fe7beb29de] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:00:00.586 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 16:00:01.0, nextRun=2024-09-11 18:00:00.0, currentDate=Wed Sep 11 18:00:00 HKT 2024 2024-09-11 18:00:00.586 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999974, name=Detect log piece, templateName=BackupDetectLogPieceSchedule, scheduleRule=0 0 */2 * * ?, lastRunTime=2024-09-11 16:00:01.0, nextRunTime=2024-09-11 18:00:00.0} 2024-09-11 18:00:00.597 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Detect log piece, nodeCount=1, taskCount=1 2024-09-11 18:00:00.597 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:00:00.597 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Detect log piece', taskDefinitionId=999974, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:00:00.597+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Detect log piece, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.schedule.DetectLogPieceTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:00:00.608 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:55:01.0, nextRun=2024-09-11 18:00:00.0, currentDate=Wed Sep 11 18:00:00 HKT 2024 2024-09-11 18:00:00.609 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:55:01.0, nextRunTime=2024-09-11 18:00:00.0} 2024-09-11 18:00:00.610 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 18:00:00.610 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:00:00.610 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:00:00.610+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:00:00.617 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:55:01.0, nextRun=2024-09-11 18:00:00.0, currentDate=Wed Sep 11 18:00:00 HKT 2024 2024-09-11 18:00:00.617 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 17:55:01.0, nextRunTime=2024-09-11 18:00:00.0} 2024-09-11 18:00:00.639 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 18:00:00.645 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 18:00:00.646 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 18:00:00.646 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:00:00.646 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T18:00:00.646+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:00:00.656 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 15:00:01.0, nextRun=2024-09-11 18:00:00.0, currentDate=Wed Sep 11 18:00:00 HKT 2024 2024-09-11 18:00:00.656 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000001, name=Collect backup files size on schedule, templateName=BackupScheduleCollectTemplateFactory, scheduleRule=0 0 */3 * * *, lastRunTime=2024-09-11 15:00:01.0, nextRunTime=2024-09-11 18:00:00.0} 2024-09-11 18:00:00.658 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Collect backup files size, nodeCount=1, taskCount=1 2024-09-11 18:00:00.658 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:00:00.658 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,0a003509e4a15384] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Collect backup files size', taskDefinitionId=1000001, state=RUNNING, type=SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:00:00.658+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Collect backup files size, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.BackupCollectSizeTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:00:01.012 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,8e3733a960f39609] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048235, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:01.022 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,8e3733a960f39609] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048236, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:01.029 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048235, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048204, task_operation=execute, latest_execution_start_time=2024-09-11T18:00:01.011+08:00, sub_task_instance_name=Detect log piece, sub_task_instance_id=4048235}, listMap={}}, executor=10.31.35.10 2024-09-11 18:00:01.031 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.o.b.i.t.schedule.DetectLogPieceTask : start detect log piece task. 2024-09-11 18:00:01.046 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,8e3733a960f39609] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048237, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:01.053 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048236, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048205, task_operation=execute, latest_execution_start_time=2024-09-11T18:00:01.021+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048236}, listMap={}}, executor=10.31.35.10 2024-09-11 18:00:01.059 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,8e3733a960f39609] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048238, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:01.061 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:01.061 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:00:01.062 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:00:01.062 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:01.064 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, round_id, piece_id AS backup_piece_id, dest_id, 0 as copy_id, start_scn_display AS start_ts, checkpoint_scn_display AS checkpoint_ts, SCN_TO_TIMESTAMP(max_scn) as max_ts, 1 as start_piece_id, status, file_status, path, output_bytes, compatible FROM CDB_OB_ARCHIVELOG_PIECE_FILES WHERE tenant_id != 1 AND status = 'FROZEN' AND file_status = 'AVAILABLE' 2024-09-11 18:00:01.073 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:00:01.078 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048237, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048206, task_operation=execute, latest_execution_start_time=2024-09-11T18:00:01.039+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048237}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 18:00:01.078 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 18:00:01.078 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.o.b.i.t.schedule.DetectLogPieceTask : skip update piece file state, filePath: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p67 2024-09-11 18:00:01.080 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.o.b.i.t.schedule.DetectLogPieceTask : skip update piece file state, filePath: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p68 2024-09-11 18:00:01.111 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 18:00:01.113 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048238, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048207, task_operation=execute, latest_execution_start_time=2024-09-11T18:00:01.057+08:00, sub_task_instance_name=Collect backup files size, sub_task_instance_id=4048238}, listMap={}}, executor=10.31.35.10 2024-09-11 18:00:01.115 INFO 2981023 --- [subtask-executor28,ddd72be6e49ae268,3f7e382c33fdeeaf] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048235, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:00:01.128 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 18:00:01.140 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.ProxyBackupStorageMonitor : collect backup storage info for clusters: rtdob 2024-09-11 18:00:01.176 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 18:00:01.222 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 18:00:01.223 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 18:00:01.252 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048207] 2024-09-11 18:00:01.257 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048204, status:SUCCESS 2024-09-11 18:00:01.254 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.BackupStrategyManagerFactory : isPhysical=true, isLogical=false, isPhysicalAndWasLogical=false 2024-09-11 18:00:01.302 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048204, status:SUCCESS 2024-09-11 18:00:01.341 INFO 2981023 --- [ocp-async-3,69ef174199ccdc17,da334026efbf37e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:01.384 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 18:00:01.460 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.PhysicalBackupStorageMonitor : tenantBackupUri=file:///obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/ 2024-09-11 18:00:01.486 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.perf.sql.internal.SqlParseService : parse 2 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 18:00:01.489 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:01.493 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:00:01.503 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:01.590 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/backup/file/exists, request body:CheckBackupFileExistsRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/), params:null 2024-09-11 18:00:01.592 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:00:01.595 INFO 2981023 --- [subtask-executor32,ddd72be6e49ae268,ea0fc6a52bbff545] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048237, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:00:01.600 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:01.605 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:01.612 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:00:01.623 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/backup/file/exists, request body:CheckBackupFileExistsRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/), params:null 2024-09-11 18:00:01.642 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:01.647 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:01.649 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:00:01.667 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,da666a783d34bb01,aeafc84692356b73] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=da666a783d34bb01, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:01.674 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,da666a783d34bb01,aeafc84692356b73] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=da666a783d34bb01, duration=14 ms] 2024-09-11 18:00:01.678 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/backup/file/find, request body:FindBackupFileRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/, recursive=false), params:null 2024-09-11 18:00:01.679 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 18:00:01.685 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 18:00:01.691 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:00:01.775 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.PhysicalBackupStorageMonitor : tenantBackupUri=file:///obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/ 2024-09-11 18:00:01.781 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:01.790 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:01.814 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:00:01.828 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/backup/file/exists, request body:CheckBackupFileExistsRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/), params:null 2024-09-11 18:00:01.845 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:01.860 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:01.867 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:00:01.878 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/backup/file/exists, request body:CheckBackupFileExistsRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/), params:null 2024-09-11 18:00:01.887 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:00:01.888 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:01.894 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:01.903 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:00:01.918 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/backup/file/find, request body:FindBackupFileRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/, recursive=false), params:null 2024-09-11 18:00:01.945 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:00:01.961 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:00:01.989 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:00:02.002 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 18:00:02.002 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3393.268094739184], b:[-5.803527553549782E12] 2024-09-11 18:00:02.006 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.BackupStrategyManagerFactory : backupAbility=PHYSICAL_BACKUP 2024-09-11 18:00:02.013 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:00:02.051 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.ProxyBackupStorageMonitor : collectInfoList=[BackupStorageCollectInfo(ocpBackupObject=OcpBackupObject(obClusterId=1718689631, clusterId=2, clusterName=rtdob, obTenantId=null, tenantId=null, tenantName=null, clusterType=PRIMARY, obVersion=4.3.1.0, backupDimension=CLUSTER), backupStorageType=BACKUP_STORAGE_FILE, collectItemList=[BackupCollectItem(clusterId=2, obClusterId=1718689631, obTenantId=1004, tenantName=linkreit, type=DATA_BACKUP, directory=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_35_full, collected=true, fileSize=31200886784), BackupCollectItem(clusterId=2, obClusterId=1718689631, obTenantId=1004, tenantName=linkreit, type=DATA_BACKUP, directory=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_11_full, collected=true, fileSize=24943947776), BackupCollectItem(clusterId=2, obClusterId=1718689631, obTenantId=1004, tenantName=linkreit, type=DATA_BACKUP, directory=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_24_full, collected=true, fileSize=40960), BackupCollectItem(clusterId=2, obClusterId=1718689631, obTenantId=1004, tenantName=linkreit, type=DATA_BACKUP, directory=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_26_full, collected=true, fileSize=29096972288), BackupCollectItem(clusterId=2, obClusterId=1718689631, obTenantId=1004, tenantName=linkreit, type=DATA_BACKUP, directory=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_31_full, collected=true, fileSize=30901018624), BackupCollectItem(clusterId=2, obClusterId=1718689631, obTenantId=1004, tenantName=linkreit, type=DATA_BACKUP, directory=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_23_full, collected=true, fileSize=40960), BackupCollectItem(clusterId=2, obClusterId=1718689631, obTenantId=1004, tenantName=linkreit, type=DATA_BACKUP, directory=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_7_full, collected=true, fileSize=23735545856), BackupCollectItem(clusterId=2, obClusterId=1718689631, obTenantId=1004, tenantName=linkreit, t 2024-09-11 18:00:02.087 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:00:02.116 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.s.BackupCollectInfoProvider : storageCollectInfoList size:[1] 2024-09-11 18:00:02.124 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.s.BackupCollectInfoProvider : online hosts: [2], and arrange hosts: [[2]] 2024-09-11 18:00:02.133 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.s.StorageCapacityExporter : Start pre hook computers 2024-09-11 18:00:02.139 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_35_full 2024-09-11 18:00:02.150 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_35_full}}, value: 31200886784, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.155 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,bdf3246fc9690331] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048207, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:02.159 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:00:02.161 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_11_full 2024-09-11 18:00:02.165 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_11_full}}, value: 24943947776, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.177 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_24_full 2024-09-11 18:00:02.182 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_24_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.178 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:00:02.191 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_26_full 2024-09-11 18:00:02.196 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_26_full}}, value: 29096972288, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.198 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:00:02.206 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_31_full 2024-09-11 18:00:02.210 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_31_full}}, value: 30901018624, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.214 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:00:02.221 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_23_full 2024-09-11 18:00:02.224 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:00:02.226 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_23_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.235 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:00:02.237 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_7_full 2024-09-11 18:00:02.238 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 18:00:02.241 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_7_full}}, value: 23735545856, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.242 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:00:02.253 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_27_full 2024-09-11 18:00:02.258 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_27_full}}, value: 29199110144, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.262 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048207, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Add zone, zone2.idc=default_idc, sub_task_instance_id=4048207, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:00:02.153+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:00:02.269 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_45_full 2024-09-11 18:00:02.272 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:00:02.272 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.o.s.t.business.zone.AddObZoneTask : begin to add zone 2024-09-11 18:00:02.277 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_45_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.281 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.o.s.t.b.zone.ObZoneTaskHandler : begin to add zone, clusterId=1000006, zone=zone2, region=sys_region 2024-09-11 18:00:02.293 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:00:02.298 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_36_full 2024-09-11 18:00:02.305 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_36_full}}, value: 31431000064, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,a3e6f372a78cd3df,c213bbee7095d47e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,a3e6f372a78cd3df,c213bbee7095d47e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:02.312 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:02.314 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,570f5353c1e0b9e4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048236, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:00:02.316 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_55_full 2024-09-11 18:00:02.339 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048205, status:SUCCESS 2024-09-11 18:00:02.340 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048205, status:SUCCESS 2024-09-11 18:00:02.321 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:02.352 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048206, status:SUCCESS 2024-09-11 18:00:02.352 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048206, status:SUCCESS 2024-09-11 18:00:02.337 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_55_full}}, value: 35799838720, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.348 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:02.375 INFO 2981023 --- [ocp-schedules-17,ddb247558b244d1f,9dce1dcf3546147a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:00:02.372 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_38_full 2024-09-11 18:00:02.373 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter system add zone ? region ?, args: [zone2, sys_region] 2024-09-11 18:00:02.376 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_38_full}}, value: 31484047360, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.388 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.o.s.t.b.zone.ObZoneTaskHandler : add zone success 2024-09-11 18:00:02.389 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_29_full 2024-09-11 18:00:02.392 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.o.o.i.helper.OcpCacheHelperImpl : Ob cluster resource has been changed, cluster 1000006, tenant null, notify ocp cache 2024-09-11 18:00:02.397 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_29_full}}, value: 29722750976, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.401 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.o.s.t.business.zone.AddObZoneTask : add zone finished 2024-09-11 18:00:02.409 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_57_full 2024-09-11 18:00:02.416 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_57_full}}, value: 36243329024, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.418 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2dd009afa310aed7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048207, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:00:02.428 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_25_full 2024-09-11 18:00:02.432 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_25_full}}, value: 28667936768, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.440 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_61_full 2024-09-11 18:00:02.445 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_61_full}}, value: 37330051072, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.452 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_18_full 2024-09-11 18:00:02.456 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_18_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.463 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_32_full 2024-09-11 18:00:02.466 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_32_full}}, value: 30166515712, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.473 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_42_full 2024-09-11 18:00:02.476 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_42_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.483 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_58_full 2024-09-11 18:00:02.486 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_58_full}}, value: 36892938240, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.495 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_12_full 2024-09-11 18:00:02.498 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_12_full}}, value: 25354481664, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.505 INFO 2981023 --- [storageCapacity3,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : ready to collect nfs capacity from hosts: [2] 2024-09-11 18:00:02.505 INFO 2981023 --- [storageCapacity3,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : idle hosts: [2] 2024-09-11 18:00:02.509 INFO 2981023 --- [storageCapacity3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:02.510 INFO 2981023 --- [storageCapacity3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:02.505 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_5_full 2024-09-11 18:00:02.512 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_5_full}}, value: 23378472960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.524 INFO 2981023 --- [storageCapacity3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/file/getDirectoryUsedBytes, request body:GetDirectoryUsedBytesRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/format.obbak, timeoutMillis=3600000), params:null 2024-09-11 18:00:02.532 INFO 2981023 --- [storageCapacity3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=7a1b1190-0db1-4411-bc67-bbc399315913), params:null 2024-09-11 18:00:02.532 INFO 2981023 --- [storageCapacity3,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:02.533 INFO 2981023 --- [storageCapacity3,,] c.o.ocp.common.lang.pattern.Retry : wait for 3 seconds 2024-09-11 18:00:02.533 INFO 2981023 --- [storageCapacity6,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : ready to collect nfs capacity from hosts: [2] 2024-09-11 18:00:02.533 INFO 2981023 --- [storageCapacity6,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : idle hosts: [2] 2024-09-11 18:00:02.536 INFO 2981023 --- [storageCapacity6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:02.532 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_63_full 2024-09-11 18:00:02.537 INFO 2981023 --- [storageCapacity6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:02.537 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_63_full}}, value: 37805322240, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.549 INFO 2981023 --- [storageCapacity6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/file/getDirectoryUsedBytes, request body:GetDirectoryUsedBytesRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_sets, timeoutMillis=3600000), params:null 2024-09-11 18:00:02.546 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_59_full 2024-09-11 18:00:02.557 INFO 2981023 --- [storageCapacity6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=3eae7caf-fa14-4695-bba9-fa91d779e101), params:null 2024-09-11 18:00:02.553 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_59_full}}, value: 36688637952, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.557 INFO 2981023 --- [storageCapacity6,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:02.557 INFO 2981023 --- [storageCapacity6,,] c.o.ocp.common.lang.pattern.Retry : wait for 3 seconds 2024-09-11 18:00:02.560 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_6_full 2024-09-11 18:00:02.564 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_6_full}}, value: 23412011008, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.571 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_22_full 2024-09-11 18:00:02.574 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_22_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.581 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_19_full 2024-09-11 18:00:02.585 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_19_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.594 INFO 2981023 --- [ocp-schedules-20,5c95a0c0a750ec4c,02a2832d2e37e1b9] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:00:02.596 INFO 2981023 --- [ocp-schedules-20,5c95a0c0a750ec4c,02a2832d2e37e1b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:00:02.595 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_47_full 2024-09-11 18:00:02.605 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_47_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.617 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:02.617 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:00:02.616 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_2_full 2024-09-11 18:00:02.623 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:02.620 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_2_full}}, value: 21938651136, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.625 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:00:02.626 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048802626000} 2024-09-11 18:00:02.627 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_4_full 2024-09-11 18:00:02.638 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:00:02.636 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_4_full}}, value: 22772150272, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.643 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000224, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:00:02.644 INFO 2981023 --- [ocp-schedules-20,5c95a0c0a750ec4c,02a2832d2e37e1b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:00:02.645 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_13_full 2024-09-11 18:00:02.649 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_13_full}}, value: 25554059264, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.656 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_64_full 2024-09-11 18:00:02.660 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_64_full}}, value: 38040580096, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.673 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_62_full 2024-09-11 18:00:02.679 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_62_full}}, value: 37494857728, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.698 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_14_full 2024-09-11 18:00:02.705 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_14_full}}, value: 25844359168, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.722 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_21_full 2024-09-11 18:00:02.728 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_21_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.735 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_16_full 2024-09-11 18:00:02.742 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_16_full}}, value: 26752696320, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.761 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:02.761 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:02.758 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_8_full 2024-09-11 18:00:02.765 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_8_full}}, value: 24011943936, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.776 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_50_full 2024-09-11 18:00:02.780 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_50_full}}, value: 34231373824, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.787 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_56_full 2024-09-11 18:00:02.793 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_56_full}}, value: 35972063232, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.803 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_1_full 2024-09-11 18:00:02.808 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_1_full}}, value: 19695378432, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.822 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_40_full 2024-09-11 18:00:02.826 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_40_full}}, value: 31889608704, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.833 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_53_full 2024-09-11 18:00:02.837 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_53_full}}, value: 35275300864, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.845 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_54_full 2024-09-11 18:00:02.849 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_54_full}}, value: 35685986304, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.855 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_28_full 2024-09-11 18:00:02.863 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:00:02.864 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:00:02.864 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:00:02.860 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_28_full}}, value: 29434007552, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.871 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:00:02.874 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000224, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:00:02.871 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_15_full 2024-09-11 18:00:02.877 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_15_full}}, value: 26138959872, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.883 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_39_full 2024-09-11 18:00:02.887 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_39_full}}, value: 31736152064, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.893 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_10_full 2024-09-11 18:00:02.897 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_10_full}}, value: 24638050304, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.904 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_3_full 2024-09-11 18:00:02.907 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_3_full}}, value: 22494461952, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.915 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_60_full 2024-09-11 18:00:02.919 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_60_full}}, value: 36941070336, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.926 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_44_full 2024-09-11 18:00:02.929 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_44_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.936 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_49_full 2024-09-11 18:00:02.940 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_49_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.947 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_34_full 2024-09-11 18:00:02.951 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_34_full}}, value: 30547877888, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.959 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4b4d774289171baf,a518b6c7a3705ba1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4b4d774289171baf, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:02.957 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_30_full 2024-09-11 18:00:02.961 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_30_full}}, value: 29941104640, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.966 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4b4d774289171baf,a518b6c7a3705ba1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4b4d774289171baf, duration=10 ms] 2024-09-11 18:00:02.969 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000224 2024-09-11 18:00:02.970 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_41_full 2024-09-11 18:00:02.976 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_41_full}}, value: 32164925440, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.984 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_52_full 2024-09-11 18:00:02.988 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_52_full}}, value: 34896809984, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:02.996 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_33_full 2024-09-11 18:00:03.000 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_33_full}}, value: 30303637504, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.007 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_46_full 2024-09-11 18:00:03.010 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_46_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.017 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_37_full 2024-09-11 18:00:03.021 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_37_full}}, value: 31407239168, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.028 INFO 2981023 --- [storageCapacity7,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : ready to collect nfs capacity from hosts: [2] 2024-09-11 18:00:03.028 INFO 2981023 --- [storageCapacity7,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : idle hosts: [2] 2024-09-11 18:00:03.031 INFO 2981023 --- [storageCapacity7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:03.028 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_17_full 2024-09-11 18:00:03.032 INFO 2981023 --- [storageCapacity7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:03.031 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_17_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.039 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_48_full 2024-09-11 18:00:03.044 INFO 2981023 --- [storageCapacity7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/file/getDirectoryUsedBytes, request body:GetDirectoryUsedBytesRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/check_file, timeoutMillis=3600000), params:null 2024-09-11 18:00:03.043 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_48_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.050 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_51_full 2024-09-11 18:00:03.063 INFO 2981023 --- [storageCapacity7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=a477d06e-2a61-4037-8000-ce71de2e6392), params:null 2024-09-11 18:00:03.064 INFO 2981023 --- [storageCapacity7,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:03.064 INFO 2981023 --- [storageCapacity7,,] c.o.ocp.common.lang.pattern.Retry : wait for 3 seconds 2024-09-11 18:00:03.056 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_51_full}}, value: 34473377792, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.070 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_9_full 2024-09-11 18:00:03.076 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_9_full}}, value: 24374763520, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.090 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_20_full 2024-09-11 18:00:03.093 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_20_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.102 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_43_full 2024-09-11 18:00:03.106 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_set_43_full}}, value: 40960, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.113 INFO 2981023 --- [storageCapacity1,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : ready to collect nfs capacity from hosts: [2] 2024-09-11 18:00:03.113 INFO 2981023 --- [storageCapacity8,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : ready to collect nfs capacity from hosts: [2] 2024-09-11 18:00:03.113 INFO 2981023 --- [storageCapacity8,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : idle hosts: [2] 2024-09-11 18:00:03.113 INFO 2981023 --- [storageCapacity1,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : idle hosts: [2] 2024-09-11 18:00:03.116 INFO 2981023 --- [storageCapacity8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:03.116 INFO 2981023 --- [storageCapacity1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:03.117 INFO 2981023 --- [storageCapacity1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:03.117 INFO 2981023 --- [storageCapacity8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:03.113 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p67 2024-09-11 18:00:03.120 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=LOG_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p67}}, value: 16599212032, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.131 INFO 2981023 --- [storageCapacity5,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : ready to collect nfs capacity from hosts: [2] 2024-09-11 18:00:03.131 INFO 2981023 --- [storageCapacity5,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : idle hosts: [2] 2024-09-11 18:00:03.133 INFO 2981023 --- [storageCapacity1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/file/getDirectoryUsedBytes, request body:GetDirectoryUsedBytesRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, timeoutMillis=3600000), params:null 2024-09-11 18:00:03.133 INFO 2981023 --- [storageCapacity8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/file/getDirectoryUsedBytes, request body:GetDirectoryUsedBytesRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/rounds, timeoutMillis=3600000), params:null 2024-09-11 18:00:03.135 INFO 2981023 --- [storageCapacity5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:03.135 INFO 2981023 --- [storageCapacity5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:03.131 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p68 2024-09-11 18:00:03.138 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=LOG_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p68}}, value: 16301498368, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.142 INFO 2981023 --- [storageCapacity8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=52dd77f0-1237-4949-8a10-b96150d7e02a), params:null 2024-09-11 18:00:03.143 INFO 2981023 --- [storageCapacity8,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"directoryUsedBytes":8192},, 2024-09-11 18:00:03.143 INFO 2981023 --- [storageCapacity8,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : nfs storage dir: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/rounds, used: '8192 bytes', capacity compute successfully in 30 ms 2024-09-11 18:00:03.143 INFO 2981023 --- [storageCapacity8,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=LOG_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/rounds}}, value: 8192, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.144 INFO 2981023 --- [storageCapacity1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=f78bedde-a1af-4300-98d9-8c0404eee733), params:null 2024-09-11 18:00:03.145 INFO 2981023 --- [storageCapacity1,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"directoryUsedBytes":4096},, 2024-09-11 18:00:03.145 INFO 2981023 --- [storageCapacity1,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : nfs storage dir: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, used: '4096 bytes', capacity compute successfully in 32 ms 2024-09-11 18:00:03.145 INFO 2981023 --- [storageCapacity1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=LOG_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak}}, value: 4096, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.147 INFO 2981023 --- [storageCapacity5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/file/getDirectoryUsedBytes, request body:GetDirectoryUsedBytesRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p69, timeoutMillis=3600000), params:null 2024-09-11 18:00:03.148 INFO 2981023 --- [storageCapacity4,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : ready to collect nfs capacity from hosts: [2] 2024-09-11 18:00:03.148 INFO 2981023 --- [storageCapacity4,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : idle hosts: [2] 2024-09-11 18:00:03.148 INFO 2981023 --- [storageCapacity2,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : ready to collect nfs capacity from hosts: [2] 2024-09-11 18:00:03.148 INFO 2981023 --- [storageCapacity2,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : idle hosts: [2] 2024-09-11 18:00:03.152 INFO 2981023 --- [storageCapacity2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:03.152 INFO 2981023 --- [storageCapacity4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:03.153 INFO 2981023 --- [storageCapacity2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:03.154 INFO 2981023 --- [storageCapacity4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:03.154 INFO 2981023 --- [storageCapacity5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=bf2aff6a-42f3-432d-8aa2-17ead2cf2d3c), params:null 2024-09-11 18:00:03.155 INFO 2981023 --- [storageCapacity5,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:03.155 INFO 2981023 --- [storageCapacity5,,] c.o.ocp.common.lang.pattern.Retry : wait for 3 seconds 2024-09-11 18:00:03.152 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:03.158 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:03.168 INFO 2981023 --- [storageCapacity4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/file/getDirectoryUsedBytes, request body:GetDirectoryUsedBytesRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, timeoutMillis=3600000), params:null 2024-09-11 18:00:03.169 INFO 2981023 --- [storageCapacity2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/file/getDirectoryUsedBytes, request body:GetDirectoryUsedBytesRequest(path=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, timeoutMillis=3600000), params:null 2024-09-11 18:00:03.177 INFO 2981023 --- [storageCapacity4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=832c0722-f99d-4c72-8e8a-f13acf7f8e0b), params:null 2024-09-11 18:00:03.178 INFO 2981023 --- [storageCapacity4,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"directoryUsedBytes":24576},, 2024-09-11 18:00:03.178 INFO 2981023 --- [storageCapacity4,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : nfs storage dir: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, used: '24576 bytes', capacity compute successfully in 30 ms 2024-09-11 18:00:03.178 INFO 2981023 --- [storageCapacity4,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=LOG_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces}}, value: 24576, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.179 INFO 2981023 --- [storageCapacity2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=aa1c5e92-bf50-4b56-b6aa-e88c1940b4eb), params:null 2024-09-11 18:00:03.180 INFO 2981023 --- [storageCapacity2,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"directoryUsedBytes":8192},, 2024-09-11 18:00:03.180 INFO 2981023 --- [storageCapacity2,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : nfs storage dir: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, used: '8192 bytes', capacity compute successfully in 32 ms 2024-09-11 18:00:03.180 INFO 2981023 --- [storageCapacity2,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=LOG_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file}}, value: 8192, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:03.176 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/disk/usage, request body:GetDiskUsageRequest(path=/obbackup/), params:null 2024-09-11 18:00:03.184 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.s.NfsStorageCapacityComputer : success to get disk usage, diskUsage:DiskUsage(mountPoint=/obbackup, totalSizeBytes=263085035520, usedSizeBytes=115507949568, availableSizeBytes=134138589184, usedPercent=46.26859645057852) 2024-09-11 18:00:03.190 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.s.NfsStorageCapacityComputer : beforeStoreMetric: collect root dir storage, totalCap:263085035520, usedBytes:115507949568 2024-09-11 18:00:03.193 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.s.NfsStorageCapacityComputer : store root dir used percent metric, root dir:[file:///obbackup/], used percent:[46.26859645057852%] 2024-09-11 18:00:03.205 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.CapacityComputerExecutor : skip compute for root or collected directory = file:///obbackup/ 2024-09-11 18:00:03.210 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.s.StorageCapacityExporter : Cluster rtdob submit 75 tasks finished, and finished size: 1 2024-09-11 18:00:03.217 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.b.i.m.s.StorageCapacityExporter : Start post hook computers 2024-09-11 18:00:03.226 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2ae5f21878511a83] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048238, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:00:03.341 INFO 2981023 --- [ocp-async-3,90ebef1d5d80dd73,7565eaa2f12fb062] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:03.361 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048207, status:SUCCESS 2024-09-11 18:00:03.362 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048207, status:SUCCESS 2024-09-11 18:00:03.400 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048217] 2024-09-11 18:00:04.213 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,e21f65a77b662b47] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048217, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:04.255 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d42a3ca8cce8a965,b04838c5f4f5aa4e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d42a3ca8cce8a965, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:04.269 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d42a3ca8cce8a965,b04838c5f4f5aa4e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d42a3ca8cce8a965, duration=19 ms] 2024-09-11 18:00:04.320 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048217, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Start zone, zone2.idc=default_idc, sub_task_instance_id=4048217, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:00:04.211+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:00:04.338 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.o.s.t.business.zone.StartObZoneTask : try to start zone, clusterName=linkrtd, obClusterId=1719994443, clusterType=PRIMARY, zoneName=zone2 2024-09-11 18:00:04.342 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.o.s.t.b.zone.ObZoneTaskHandler : begin to start zone, clusterId=1000006, zone=zone2 2024-09-11 18:00:04.360 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:04.368 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:04.376 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:04.380 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter system start zone ?, args: [zone2] 2024-09-11 18:00:04.391 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.o.s.t.b.zone.ObZoneTaskHandler : start zone success 2024-09-11 18:00:04.396 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.o.s.t.business.zone.StartObZoneTask : start zone finished 2024-09-11 18:00:04.408 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,405d3e7736c19bcb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048217, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:00:04.796 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,8a2dc029e43e1e3b] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:50:05.0, nextRun=2024-09-11 18:00:04.0, currentDate=Wed Sep 11 18:00:04 HKT 2024 2024-09-11 18:00:04.796 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,8a2dc029e43e1e3b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-09-11 17:50:05.0, nextRunTime=2024-09-11 18:00:04.0} 2024-09-11 18:00:04.800 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,8a2dc029e43e1e3b] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-09-11 18:00:04.800 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,8a2dc029e43e1e3b] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:00:04.800 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,8a2dc029e43e1e3b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:00:04.800+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:00:05.255 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,6a886204ca762bef] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048239, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:05.279 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048239, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048208, task_operation=execute, latest_execution_start_time=2024-09-11T18:00:05.253+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=4048239}, listMap={}}, executor=10.31.35.10 2024-09-11 18:00:05.287 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 18:00:05.287 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 18:00:05.288 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.288 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:00:05.289 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.290 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.290 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.291 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 18:00:05.303 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.303 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.304 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.304 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 18:00:05.313 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,e537b0e1c9337be8,c9a63ae99cd56ef4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:05.313 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.314 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.314 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 18:00:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,e537b0e1c9337be8,c9a63ae99cd56ef4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:05.332 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.332 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.335 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.336 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 18:00:05.344 INFO 2981023 --- [ocp-async-1,33f22f8a6d1483cb,e21eec84423ad377] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:05.348 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.349 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.349 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.350 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 18:00:05.355 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.355 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.356 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.356 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 18:00:05.360 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.360 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.361 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.361 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 18:00:05.364 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.364 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.365 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.365 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 18:00:05.370 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.370 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.371 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.371 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 18:00:05.374 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.375 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.375 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.376 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 18:00:05.379 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.379 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.380 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.380 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 18:00:05.385 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.386 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.386 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.387 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 18:00:05.395 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.396 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.396 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.397 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 18:00:05.400 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.401 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.401 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.401 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 18:00:05.406 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.406 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.407 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.407 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 18:00:05.410 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.411 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.411 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.411 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 18:00:05.414 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.415 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.415 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.416 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 18:00:05.418 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.419 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.419 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.419 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 18:00:05.422 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.422 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.423 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.423 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 18:00:05.426 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.426 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.427 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.427 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 18:00:05.430 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.430 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.430 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.431 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 18:00:05.434 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.434 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.434 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.435 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 18:00:05.438 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.438 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.438 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.439 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 18:00:05.441 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.442 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.442 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.442 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 18:00:05.445 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.446 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.446 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.447 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 18:00:05.450 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.450 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.451 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.451 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 18:00:05.454 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.455 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.455 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.456 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 18:00:05.460 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.460 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.461 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.461 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 18:00:05.464 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.465 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.465 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.466 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 18:00:05.469 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.469 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.470 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.470 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 18:00:05.474 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.474 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.475 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.475 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 18:00:05.479 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.479 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.480 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.480 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 18:00:05.483 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048211] 2024-09-11 18:00:05.483 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.483 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.484 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.484 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 18:00:05.487 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.487 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.488 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.488 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 18:00:05.493 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.493 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.494 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.494 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 18:00:05.498 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.499 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.499 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.499 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 18:00:05.502 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.503 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.503 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.503 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 18:00:05.507 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.507 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.508 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.508 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 18:00:05.510 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.510 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.511 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.511 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 18:00:05.514 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.514 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.514 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.515 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 18:00:05.517 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.517 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.517 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.518 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 18:00:05.520 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.520 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.521 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.521 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 18:00:05.527 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.528 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.528 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.529 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 18:00:05.533 INFO 2981023 --- [storageCapacity3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=7a1b1190-0db1-4411-bc67-bbc399315913), params:null 2024-09-11 18:00:05.535 INFO 2981023 --- [storageCapacity3,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"directoryUsedBytes":4096},, 2024-09-11 18:00:05.535 INFO 2981023 --- [storageCapacity3,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : nfs storage dir: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/format.obbak, used: '4096 bytes', capacity compute successfully in 3030 ms 2024-09-11 18:00:05.535 INFO 2981023 --- [storageCapacity3,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/format.obbak}}, value: 4096, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:05.546 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.546 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.547 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.547 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 18:00:05.552 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.552 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.552 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.553 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 18:00:05.556 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.557 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.557 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.557 INFO 2981023 --- [storageCapacity6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=3eae7caf-fa14-4695-bba9-fa91d779e101), params:null 2024-09-11 18:00:05.558 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 18:00:05.558 INFO 2981023 --- [storageCapacity6,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"directoryUsedBytes":20480},, 2024-09-11 18:00:05.559 INFO 2981023 --- [storageCapacity6,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : nfs storage dir: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_sets, used: '20480 bytes', capacity compute successfully in 3026 ms 2024-09-11 18:00:05.559 INFO 2981023 --- [storageCapacity6,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/backup_sets}}, value: 20480, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:05.562 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.562 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.563 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.563 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 18:00:05.565 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.565 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.566 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.566 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 18:00:05.569 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.569 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.569 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.570 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 18:00:05.573 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 18:00:05.573 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 18:00:05.573 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c9ac4f25893ea752,48285ac611cac301] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c9ac4f25893ea752, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:05.581 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c9ac4f25893ea752,48285ac611cac301] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c9ac4f25893ea752, duration=11 ms] 2024-09-11 18:00:05.583 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.583 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:00:05.585 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.586 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.586 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.587 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 18:00:05.591 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.592 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:05.592 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.593 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 18:00:05.595 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 18:00:05.604 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6a83ee4588b87375] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048239, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:00:05.762 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:05.763 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:00:05.985 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:00:05.989 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:00:05.994 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:00:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:00:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:00:05.997 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.997 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:00:05.998 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:05.998 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:00:06.064 INFO 2981023 --- [storageCapacity7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=a477d06e-2a61-4037-8000-ce71de2e6392), params:null 2024-09-11 18:00:06.067 INFO 2981023 --- [storageCapacity7,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"directoryUsedBytes":8192},, 2024-09-11 18:00:06.067 INFO 2981023 --- [storageCapacity7,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : nfs storage dir: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/check_file, used: '8192 bytes', capacity compute successfully in 3039 ms 2024-09-11 18:00:06.067 INFO 2981023 --- [storageCapacity7,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=DATA_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/data/check_file}}, value: 8192, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:06.128 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T17:57:41, start=2024-09-11T17:55:11, end=2024-09-11T17:57:41, updates=312, granule=120000ms, delay=150000ms 2024-09-11 18:00:06.133 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T17:57:41, target=2024-09-11T17:57:41, granule=120000ms, delay=0ms) 2024-09-11 18:00:06.133 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:57:41, target=2024-09-11T17:50:51, granule=600000ms, delay=410000ms) 2024-09-11 18:00:06.133 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=166ms, period=120000ms 2024-09-11 18:00:06.155 INFO 2981023 --- [storageCapacity5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=bf2aff6a-42f3-432d-8aa2-17ead2cf2d3c), params:null 2024-09-11 18:00:06.160 INFO 2981023 --- [storageCapacity5,,] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"directoryUsedBytes":6752346112},, 2024-09-11 18:00:06.160 INFO 2981023 --- [storageCapacity5,,] c.o.o.b.i.m.s.NfsStorageCapacityComputer : nfs storage dir: /obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p69, used: '6752346112 bytes', capacity compute successfully in 3029 ms 2024-09-11 18:00:06.161 INFO 2981023 --- [storageCapacity5,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_backup_storage_capacity_bytes', labels={app=BACKUP, backup_type=LOG_BACKUP, cluster_id=2, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p69}}, value: 6752346112, timestamp: 1726048802 : 2024-09-11 10:00:02 2024-09-11 18:00:06.307 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,af44fa582841ba25] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048211, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:06.358 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:00:06.358135+08:00, lastProcessTime=2024-09-11T17:59:56.353389+08:00 2024-09-11 18:00:06.380 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048208, status:SUCCESS 2024-09-11 18:00:06.380 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048208, status:SUCCESS 2024-09-11 18:00:06.417 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,f7c2891656e3aaf7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048211, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Set zone idc info, zone2.idc=default_idc, sub_task_instance_id=4048211, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:00:06.305+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:00:06.441 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,f7c2891656e3aaf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:06.446 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,f7c2891656e3aaf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter system alter zone ? set, region = ?, idc = ?, args: [zone2, sys_region, default_idc] 2024-09-11 18:00:06.456 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,f7c2891656e3aaf7] c.o.o.s.t.b.zone.SetZoneIdcInfoTask : alter zone, zoneName=zone2, regionName=sys_region, idcName=default_idc 2024-09-11 18:00:06.467 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,f7c2891656e3aaf7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048211, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:00:06.508 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:00:06.358135+08:00 2024-09-11 18:00:06.530 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048223] 2024-09-11 18:00:06.844 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,642c59e2d6df443e,f476321368c78d8f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=642c59e2d6df443e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:06.851 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,642c59e2d6df443e,f476321368c78d8f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=642c59e2d6df443e, duration=10 ms] 2024-09-11 18:00:07.344 INFO 2981023 --- [ocp-async-6,887f887afd34d2fc,531ae2a1773c91b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:07.360 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,6552a16710482642] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048223, operation:EXECUTE, state: RUNNING 2024-09-11 18:00:07.442 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048223, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Install dependencies, zone2.idc=default_idc, sub_task_instance_id=4048223, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:00:07.358+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:00:07.462 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.h.s.HostOperationServiceImpl : [downloadRpm]: generate agent async task token:955b53ff-8ed8-41b4-a8c9-e7f4afb3fcf2 2024-09-11 18:00:07.470 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:07.478 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:07.496 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/file/download, request body:DownloadFileRequest(source=RemoteFileAccessInfo(type=URL, url=http://10.31.35.10:8080/api/v2/object-storage/sys-package/oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm, path=null), target=rpms/oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm, validateChecksum=true, checksum=ee79e726bd34c87073c002ab297724a36d932e77), params:null 2024-09-11 18:00:07.505 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=955b53ff-8ed8-41b4-a8c9-e7f4afb3fcf2), params:null 2024-09-11 18:00:07.513 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"path":"rpms/oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm"},, 2024-09-11 18:00:07.517 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.h.s.HostOperationServiceImpl : rpm install, hostId=2000010, softwarePackageMeta=SoftwarePackageMeta(id=1000019, objectId=4000030, name=oceanbase-ce-libs, fileName=oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm, extension=rpm, size=163148, sha1=ee79e726bd34c87073c002ab297724a36d932e77, downloadUrl=http://10.31.35.10:8080/api/v2/object-storage/sys-package/oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm, type=OB_CE_LIBS_PACKAGE, version=4.2.4.0, buildNumber=100000082024070810, operatingSystem=el7, architecture=x86_64, createTime=2024-09-11T15:32:43+08:00, deletable=null, installationDataList=null, hasSignature=null, signatureValid=null), rpmFilePath=rpms/oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm, installPath=/data/rtdob/oceanbase 2024-09-11 18:00:07.521 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.h.s.HostOperationServiceImpl : [installRpm]: generate agent async task token:012ad9b0-0883-4dc6-ac53-ef32739a9c69 2024-09-11 18:00:07.527 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:07.532 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:07.549 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/package/install, request body:InstallPackageRequest(name=oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64, file=rpms/oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm, installPath=/data/rtdob/oceanbase, force=false), params:null 2024-09-11 18:00:07.556 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=012ad9b0-0883-4dc6-ac53-ef32739a9c69), params:null 2024-09-11 18:00:07.561 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:07.565 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2024-09-11 18:00:07.686 INFO 2981023 --- [ocp-schedules-12,4f7906140d3baba3,3288646ee26d8517] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:00:07.762 INFO 2981023 --- [ocp-schedules-29,2f4565f18a53ffa4,830ed4486e164fca] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:00:07.762+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:00:07.762 INFO 2981023 --- [ocp-schedules-29,2f4565f18a53ffa4,830ed4486e164fca] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:00:07.764 INFO 2981023 --- [ocp-schedules-29,2f4565f18a53ffa4,830ed4486e164fca] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:00:07.765 INFO 2981023 --- [ocp-schedules-29,2f4565f18a53ffa4,830ed4486e164fca] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:00:07.770 INFO 2981023 --- [ocp-async-10,881fe0d0a39f882d,6fe36de8b09a8b43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:00:08.091 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,da54de47ac835790,eca95074f19b94c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=da54de47ac835790, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:08.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,da54de47ac835790,eca95074f19b94c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=da54de47ac835790, duration=9 ms] 2024-09-11 18:00:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,52cd1b824a10fc04,40431bdf5079f6c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,52cd1b824a10fc04,40431bdf5079f6c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:08.763 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:08.765 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:09.344 INFO 2981023 --- [ocp-async-5,a6e24522c0d63de2,9476518cb59b4a9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:09.359 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8085d4959b4d6d9b,d182c8ef9bc73467] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8085d4959b4d6d9b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:09.372 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8085d4959b4d6d9b,d182c8ef9bc73467] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8085d4959b4d6d9b, duration=18 ms] 2024-09-11 18:00:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:00:10.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:00:10.479 INFO 2981023 --- [ocp-schedules-7,d1475ad8d210f06f,c88de885856a1fcf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:00:10.527 WARN 2981023 --- [scanner-13586403-1,d1475ad8d210f06f,424b474568cbb36a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:00:10.550 INFO 2981023 --- [alarm-detect-13,d1475ad8d210f06f,07ee62784ffc20bf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645148, data is empty. 2024-09-11 18:00:10.554 INFO 2981023 --- [alarm-detect-13,d1475ad8d210f06f,07ee62784ffc20bf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645165, data is empty. 2024-09-11 18:00:10.557 INFO 2981023 --- [alarm-detect-13,d1475ad8d210f06f,07ee62784ffc20bf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645165, data is empty. 2024-09-11 18:00:10.575 INFO 2981023 --- [alarm-detect-4,d1475ad8d210f06f,3d4ffd3ca4151be0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645148, data is empty. 2024-09-11 18:00:10.577 INFO 2981023 --- [alarm-detect-9,d1475ad8d210f06f,a5da721bbe53013a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645148, data is empty. 2024-09-11 18:00:10.578 INFO 2981023 --- [alarm-detect-4,d1475ad8d210f06f,3d4ffd3ca4151be0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645165, data is empty. 2024-09-11 18:00:10.581 INFO 2981023 --- [alarm-detect-9,d1475ad8d210f06f,a5da721bbe53013a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645165, data is empty. 2024-09-11 18:00:10.582 INFO 2981023 --- [alarm-detect-4,d1475ad8d210f06f,3d4ffd3ca4151be0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645165, data is empty. 2024-09-11 18:00:10.584 INFO 2981023 --- [alarm-detect-9,d1475ad8d210f06f,a5da721bbe53013a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645165, data is empty. 2024-09-11 18:00:10.637 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4552bb73733a73d1,3cf0d21046d3b287] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4552bb73733a73d1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:10.651 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4552bb73733a73d1,3cf0d21046d3b287] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4552bb73733a73d1, duration=19 ms] 2024-09-11 18:00:10.729 INFO 2981023 --- [alarm-detect-3,d1475ad8d210f06f,b3f3c26122988084] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645148, data is empty. 2024-09-11 18:00:10.733 INFO 2981023 --- [alarm-detect-3,d1475ad8d210f06f,b3f3c26122988084] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645165, data is empty. 2024-09-11 18:00:10.740 INFO 2981023 --- [alarm-detect-3,d1475ad8d210f06f,b3f3c26122988084] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645165, data is empty. 2024-09-11 18:00:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:00:10.922 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:00:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,4e2466a9fc42cade,bfbce5550034bccc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,4e2466a9fc42cade,bfbce5550034bccc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:11.359 INFO 2981023 --- [ocp-async-10,42bdf62c8e30daef,9df678cb0d9fc899] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:11.765 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:11.766 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:11.946 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,dfdb5042dc2a5383,e9694f08480227fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=dfdb5042dc2a5383, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:11.953 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,dfdb5042dc2a5383,e9694f08480227fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=dfdb5042dc2a5383, duration=10 ms] 2024-09-11 18:00:12.377 INFO 2981023 --- [ocp-schedules-18,2bca50736161431f,7782efe4cc10ebef] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:00:12.658 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,06033eb02aca6fe6,18461c4476c8ad82] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&version=2&ObClusterId=1718689631, client=10.31.35.5, traceId=06033eb02aca6fe6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=rtdob,1718689631,2,{"ObRegion":"rtdob","ObCluster":"rtdob","ObRegionId":1718689631,"ObClusterId":1718689631,"Type":"PRIMARY","timestamp":1726048812658727,"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:00:12.666 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,06033eb02aca6fe6,18461c4476c8ad82] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=rtdob, obClusterId=1718689631, rootServerJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, groupName=rtdob, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:00:12.666 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,06033eb02aca6fe6,18461c4476c8ad82] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:00:12.666 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,06033eb02aca6fe6,18461c4476c8ad82] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=rtdob, clusterId=1718689631, affectRows=1 2024-09-11 18:00:12.667 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,06033eb02aca6fe6,18461c4476c8ad82] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.5, traceId=06033eb02aca6fe6, duration=9 ms] 2024-09-11 18:00:13.000 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048740, nodeCount=480, affectRows=480 2024-09-11 18:00:13.224 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e94ace405916c316,343d617338bdbdac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e94ace405916c316, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:13.230 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e94ace405916c316,343d617338bdbdac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e94ace405916c316, duration=10 ms] 2024-09-11 18:00:13.358 INFO 2981023 --- [ocp-async-2,7216e004a410444a,cc40da7f3001af21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:13.393 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048740, nodeCount=1000, affectRows=1000 2024-09-11 18:00:13.393 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048740, nodeCount=1000, affectRows=1000 2024-09-11 18:00:13.396 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048740, nodeCount=1000, validCount=1000 2024-09-11 18:00:13.396 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048740, nodeCount=999, validCount=999 2024-09-11 18:00:13.403 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048740, nodeCount=9, affectRows=9 2024-09-11 18:00:13.403 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048740, nodeCount=40, affectRows=40 2024-09-11 18:00:13.403 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048740, nodeCount=9, validCount=9 2024-09-11 18:00:13.403 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048740, nodeCount=39, validCount=39 2024-09-11 18:00:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,f077f6080a8db7eb,2ef141c6dc379140] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,5cb207b406e830eb,c19eddc5c96cd096] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:00:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,f077f6080a8db7eb,2ef141c6dc379140] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:14.491 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a418ab0f02a8b024,b916d46d4bc8c34b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a418ab0f02a8b024, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:14.497 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a418ab0f02a8b024,b916d46d4bc8c34b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a418ab0f02a8b024, duration=9 ms] 2024-09-11 18:00:14.766 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:14.766 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:14.834 INFO 2981023 --- [ocp-schedules-19,7abe1e751d809697,521645e5e3ce1fe0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:00:14.843 INFO 2981023 --- [ocp-schedules-19,7abe1e751d809697,521645e5e3ce1fe0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:00:14.843 INFO 2981023 --- [ocp-schedules-19,7abe1e751d809697,521645e5e3ce1fe0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:00:15.358 INFO 2981023 --- [ocp-async-4,8380bcc0e79c602d,b9759976925c1684] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:15.741 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,c434c65f6f5f18cd,a018e8eaa2145240] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c434c65f6f5f18cd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:15.749 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,c434c65f6f5f18cd,a018e8eaa2145240] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c434c65f6f5f18cd, duration=11 ms] 2024-09-11 18:00:16.016 INFO 2981023 --- [ocp-schedules-24,903f81c8d1e5b5a7,0da4e12e6b542d2d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:00:16.016 INFO 2981023 --- [host-check-clockdiff1,903f81c8d1e5b5a7,c8bd35ac5132577b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:00:16.016 INFO 2981023 --- [host-check-clockdiff2,903f81c8d1e5b5a7,0a2be9d44c7a59a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:00:16.017 INFO 2981023 --- [host-check-clockdiff3,903f81c8d1e5b5a7,e042ecb13b1507b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:00:16.017 INFO 2981023 --- [host-check-clockdiff4,903f81c8d1e5b5a7,326b28dbe9b2dec1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:00:16.029 INFO 2981023 --- [host-check-clockdiff3,903f81c8d1e5b5a7,e042ecb13b1507b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048816 0 0 , error= 2024-09-11 18:00:16.030 INFO 2981023 --- [ocp-schedules-14,7e7e605612b82cda,0a1190013af89eaf] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:00:16.030 INFO 2981023 --- [host-check-agent-status1,7e7e605612b82cda,37e37af148a8825a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:00:16.032 INFO 2981023 --- [host-check-agent-status2,7e7e605612b82cda,b0bd4a27f50aa4e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:00:16.034 INFO 2981023 --- [host-check-agent-status4,7e7e605612b82cda,732516eff82d2d37] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:00:16.034 INFO 2981023 --- [ocp-schedules-18,22d9736032c9b2a7,0cd7620514036b32] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:00:16.035 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:00:16.035 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:00:16.036 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:00:16.037 INFO 2981023 --- [host-check-agent-status3,7e7e605612b82cda,bbf243e634640333] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:00:16.040 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:00:16.042 INFO 2981023 --- [host-check-agent-status2,7e7e605612b82cda,b0bd4a27f50aa4e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:00:16.045 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:16.046 INFO 2981023 --- [host-check-agent-status2,7e7e605612b82cda,b0bd4a27f50aa4e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:00:16.046 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:16.047 INFO 2981023 --- [host-check-agent-status3,7e7e605612b82cda,bbf243e634640333] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:00:16.049 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:00:16.049 INFO 2981023 --- [host-check-agent-status1,7e7e605612b82cda,37e37af148a8825a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:16.049 INFO 2981023 --- [host-check-agent-status4,7e7e605612b82cda,732516eff82d2d37] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:16.050 INFO 2981023 --- [host-check-agent-status3,7e7e605612b82cda,bbf243e634640333] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:00:16.050 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:00:16.051 INFO 2981023 --- [host-check-agent-status4,7e7e605612b82cda,732516eff82d2d37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:16.051 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:00:16.052 INFO 2981023 --- [host-check-agent-status1,7e7e605612b82cda,37e37af148a8825a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:16.053 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:16.054 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:00:16.056 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:16.094 INFO 2981023 --- [host-check-agent-status2,7e7e605612b82cda,b0bd4a27f50aa4e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:00:16.095 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:00:16.097 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:00:16.102545172+08:00 2024-09-11 18:00:16.097 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:00:16.098 INFO 2981023 --- [host-check-agent-status3,7e7e605612b82cda,bbf243e634640333] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:00:16.098 INFO 2981023 --- [host-check-agent-status2,7e7e605612b82cda,b0bd4a27f50aa4e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:00:16.098 INFO 2981023 --- [host-check-agent-status2,7e7e605612b82cda,b0bd4a27f50aa4e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:00:16.098 INFO 2981023 --- [host-check-agent-status2,7e7e605612b82cda,b0bd4a27f50aa4e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:00:16.098 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:00:16.097+08:00 2024-09-11 18:00:16.099 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:00:16.100 INFO 2981023 --- [host-check-agent-status4,7e7e605612b82cda,732516eff82d2d37] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:00:16.101 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:00:16.099703502+08:00 2024-09-11 18:00:16.101 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:00:16.097+08:00 and status AVAILABLE 2024-09-11 18:00:16.102 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:00:16.102 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:00:16.101+08:00 2024-09-11 18:00:16.103 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:00:16.107904916+08:00 2024-09-11 18:00:16.105 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:00:16.101117385+08:00 2024-09-11 18:00:16.105 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:00:16.105+08:00 2024-09-11 18:00:16.106 INFO 2981023 --- [host-check-agent-status1,7e7e605612b82cda,37e37af148a8825a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:00:16.106 INFO 2981023 --- [host-check-agent-status3,7e7e605612b82cda,bbf243e634640333] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:00:16.106 INFO 2981023 --- [host-check-agent-status3,7e7e605612b82cda,bbf243e634640333] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:00:16.106 INFO 2981023 --- [host-check-agent-status3,7e7e605612b82cda,bbf243e634640333] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:00:16.106 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:00:16.103+08:00 2024-09-11 18:00:16.107 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:00:16.105+08:00 and status AVAILABLE 2024-09-11 18:00:16.107 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:00:16.101+08:00 and status AVAILABLE 2024-09-11 18:00:16.108 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:00:16.103+08:00 and status AVAILABLE 2024-09-11 18:00:16.108 INFO 2981023 --- [host-check-agent-status1,7e7e605612b82cda,37e37af148a8825a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:00:16.108 INFO 2981023 --- [host-check-agent-status1,7e7e605612b82cda,37e37af148a8825a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:00:16.108 INFO 2981023 --- [host-check-agent-status1,7e7e605612b82cda,37e37af148a8825a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:00:16.112 INFO 2981023 --- [host-check-agent-status4,7e7e605612b82cda,732516eff82d2d37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:00:16.112 INFO 2981023 --- [host-check-agent-status4,7e7e605612b82cda,732516eff82d2d37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:00:16.112 INFO 2981023 --- [host-check-agent-status4,7e7e605612b82cda,732516eff82d2d37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:00:16.120 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:00:16.121 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:00:16.121 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:00:16.124 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:00:16.168 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:16.170 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:16.175 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:00:16.177 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:00:16.187 INFO 2981023 --- [ocp-schedules-25,05385a7a18d67001,782533065882eb6f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:00:16.187 INFO 2981023 --- [ocp-schedules-19,bfd5c96eb514b3b5,16827f5a365d8601] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:00:16.188 INFO 2981023 --- [ocp-schedules-8,31cf36248675dd92,201e8208b9b55ee0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:00:16.188 INFO 2981023 --- [ocp-schedules-5,7ee351a5c6bb6987,184170e26bcea01c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:00:16.188 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:00:16.189 INFO 2981023 --- [ocp-schedules-8,31cf36248675dd92,201e8208b9b55ee0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:00:16.193 INFO 2981023 --- [ocp-schedules-19,bfd5c96eb514b3b5,16827f5a365d8601] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:00:16.193 INFO 2981023 --- [ocp-schedules-25,05385a7a18d67001,782533065882eb6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:00:16.196 INFO 2981023 --- [ocp-schedules-19,bfd5c96eb514b3b5,16827f5a365d8601] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:00:16.196 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048802626000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:00:16.202 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:00:16.211 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:00:16.230 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:00:16.230 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:16.240 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:16.240 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:00:16.247 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:00:16.264 INFO 2981023 --- [ocp-schedules-30,1454a26d0670203d,3bd64990aeffaf54] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:00:16.270 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.270 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:00:16.272 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:00:16.277 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.278 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:00:16.278 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.278 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:00:16.279 INFO 2981023 --- [ocp-schedules-25,05385a7a18d67001,782533065882eb6f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:00:16.279 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.279 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:16.279 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:16.280 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.280 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:16.289 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:00:16.299 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:00:16.302 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:00:16.305 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:00:16.311 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:00:16.313 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:16.314 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.314 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:00:16.321 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:00:16.329 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:00:16.333 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:00:16.336 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:00:16.344 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:00:16.356 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:00:16.358 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:00:16.362 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:16.365 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.367 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:16.368 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:00:16.369 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:00:16.368744+08:00, lastProcessTime=2024-09-11T18:00:06.358135+08:00 2024-09-11 18:00:16.369 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:00:16.370 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.373 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:00:16.374 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:00:16.374 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048802626000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:00:16.375 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:00:16.380 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:00:16.392 INFO 2981023 --- [host-check-host-status4,22d9736032c9b2a7,48c520e51c45d5d0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:00:16.395 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:00:16.395 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:00:16.396 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:00:16.401 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:00:16.405 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:00:16.406 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:00:16.413 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:00:16.422 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:16.423 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.423 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:00:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:00:16.427 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:00:16.428 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:00:16.432 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:00:16.433 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:00:16.433 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:00:16.436 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048740, nodeCount=480, validCount=480 2024-09-11 18:00:16.436 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:00:16.447 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:00:16.450 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:00:16.453 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:00:16.485 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:00:16.492 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:16.492 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.493 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:00:16.499 INFO 2981023 --- [host-check-host-status3,22d9736032c9b2a7,083d43fdad178126] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:00:16.500 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:00:16.510 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:00:16.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:16.531 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.531 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:16.532 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:00:16.532 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.533 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:00:16.535 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:00:16.535 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:00:16.535 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:00:16.542 INFO 2981023 --- [host-check-host-status2,22d9736032c9b2a7,3ef19d48bf5a89ea] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:00:16.547 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:00:16.549 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:00:16.549 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:00:16.550 INFO 2981023 --- [host-check-host-status1,22d9736032c9b2a7,8b843021bbb0ac8f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:00:16.558 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:16.558 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.559 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:00:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:00:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:00:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:00:16.569 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:16.570 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:00:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:00:16.575 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:00:16.578 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:16.579 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:16.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:00:16.619 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:00:16.368744+08:00 2024-09-11 18:00:16.621 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:00:16.626 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:16.627 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:16.627 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:00:16.634 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:00:16.634 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:00:16.634 INFO 2981023 --- [ocp-schedules-8,31cf36248675dd92,201e8208b9b55ee0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:00:16.965 INFO 2981023 --- [host-check-clockdiff2,903f81c8d1e5b5a7,0a2be9d44c7a59a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048816 5 5 , error= 2024-09-11 18:00:16.975 INFO 2981023 --- [host-check-clockdiff1,903f81c8d1e5b5a7,c8bd35ac5132577b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048816 5 5 , error= 2024-09-11 18:00:17.028 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,1e329ff70c67fca2,0c82e530944d8cf6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1e329ff70c67fca2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:17.035 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,1e329ff70c67fca2,0c82e530944d8cf6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1e329ff70c67fca2, duration=10 ms] 2024-09-11 18:00:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,58dd7d435c50d546,0d85bb54636b2244] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,58dd7d435c50d546,0d85bb54636b2244] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:17.358 INFO 2981023 --- [ocp-async-5,a4a1995c438096d4,41b95112ed394e17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:17.569 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=012ad9b0-0883-4dc6-ac53-ef32739a9c69), params:null 2024-09-11 18:00:17.575 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"changed":true,"installPath":"/data/rtdob/oceanbase","packageInfo":{"architecture":"x86_64","buildNumber":"100000082024070810","fullPackageName":"oceanbase-ce-libs-4.2.4.0-100000082024070810.el8.x86_64","name":"oceanbase-ce-libs","os":"el8","version":"4.2.4.0"}},, 2024-09-11 18:00:17.579 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.h.s.HostOperationServiceImpl : rpm install done, package info=PackageInfo(fullPackageName=oceanbase-ce-libs-4.2.4.0-100000082024070810.el8.x86_64, name=oceanbase-ce-libs, version=4.2.4.0, buildNumber=100000082024070810, os=el8, architecture=x86_64) 2024-09-11 18:00:17.590 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.h.s.HostOperationServiceImpl : [downloadRpm]: generate agent async task token:e56ec251-a2b1-4e51-afaa-b2088b0abf8f 2024-09-11 18:00:17.615 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:17.622 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:17.638 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/file/download, request body:DownloadFileRequest(source=RemoteFileAccessInfo(type=URL, url=http://10.31.35.10:8080/api/v2/object-storage/sys-package/oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm, path=null), target=rpms/oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm, validateChecksum=true, checksum=69a7db94693b7d9d993c5244e5fa70930c61d419), params:null 2024-09-11 18:00:17.662 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=e56ec251-a2b1-4e51-afaa-b2088b0abf8f), params:null 2024-09-11 18:00:17.682 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:17.687 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.ocp.common.lang.pattern.Retry : wait for 3 seconds 2024-09-11 18:00:17.766 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:17.767 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:17.770 INFO 2981023 --- [ocp-async-7,1b56e132f2da8446,2f80c74e94ed1c45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:00:18.296 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,57e76790c8f361bf,1e3e05a98a0837ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=57e76790c8f361bf, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:18.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,57e76790c8f361bf,1e3e05a98a0837ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=57e76790c8f361bf, duration=10 ms] 2024-09-11 18:00:19.358 INFO 2981023 --- [ocp-async-9,f0e6197fc0e5feb5,326193268b7aaf7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:19.568 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,36d5bff9276a1ef7,db2174c2de7f9c18] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=36d5bff9276a1ef7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:19.574 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,36d5bff9276a1ef7,db2174c2de7f9c18] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=36d5bff9276a1ef7, duration=9 ms] 2024-09-11 18:00:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,af8e8b344df45729,3ed096bdf849e747] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,af8e8b344df45729,3ed096bdf849e747] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:20.691 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=e56ec251-a2b1-4e51-afaa-b2088b0abf8f), params:null 2024-09-11 18:00:20.699 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"path":"rpms/oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"},, 2024-09-11 18:00:20.703 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.h.s.HostOperationServiceImpl : rpm install, hostId=2000010, softwarePackageMeta=SoftwarePackageMeta(id=1000020, objectId=4000031, name=oceanbase-ce-utils, fileName=oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm, extension=rpm, size=277641484, sha1=69a7db94693b7d9d993c5244e5fa70930c61d419, downloadUrl=http://10.31.35.10:8080/api/v2/object-storage/sys-package/oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm, type=OB_CE_UTILS_PACKAGE, version=4.2.4.0, buildNumber=100000082024070810, operatingSystem=el7, architecture=x86_64, createTime=2024-09-11T15:33:20+08:00, deletable=null, installationDataList=null, hasSignature=null, signatureValid=null), rpmFilePath=rpms/oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm, installPath=/data/rtdob/oceanbase 2024-09-11 18:00:20.707 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.h.s.HostOperationServiceImpl : [installRpm]: generate agent async task token:1386a1df-7888-4ac9-9eba-58db1971c45c 2024-09-11 18:00:20.718 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:20.724 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:20.740 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/package/install, request body:InstallPackageRequest(name=oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64, file=rpms/oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm, installPath=/data/rtdob/oceanbase, force=false), params:null 2024-09-11 18:00:20.748 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=1386a1df-7888-4ac9-9eba-58db1971c45c), params:null 2024-09-11 18:00:20.767 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:20.767 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:20.765 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:20.769 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2024-09-11 18:00:20.778 INFO 2981023 --- [ocp-schedules-1,8f320231ba415f22,4650f2e32cb41c76] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:00:20.809 WARN 2981023 --- [scanner-13588215-1,8f320231ba415f22,26fe9224f336721f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:00:20.839 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e8a973c3d41372e2,7c97252aa407400f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e8a973c3d41372e2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:20.867 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e8a973c3d41372e2,7c97252aa407400f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e8a973c3d41372e2, duration=33 ms] 2024-09-11 18:00:20.913 INFO 2981023 --- [alarm-detect-9,8f320231ba415f22,0337efd364f748a7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726048785 ~ 1726048810 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726048800, value=1.676)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.676, values={1726048800=1.676}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:00:21.358 INFO 2981023 --- [ocp-async-3,a60b78e0df761491,c318628cc6316045] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:22.001 INFO 2981023 --- [host-check-clockdiff4,903f81c8d1e5b5a7,326b28dbe9b2dec1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048821 -1 -1 , error= 2024-09-11 18:00:22.151 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,46ea11cb32fdca23,a5046921a92d04b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=46ea11cb32fdca23, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:22.157 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,46ea11cb32fdca23,a5046921a92d04b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=46ea11cb32fdca23, duration=9 ms] 2024-09-11 18:00:22.380 INFO 2981023 --- [ocp-schedules-20,8fd613d65df87156,897a91b531aa96cc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:00:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,7b44e4f022aacabc,2316268866cbfec8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,7b44e4f022aacabc,2316268866cbfec8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:23.359 INFO 2981023 --- [ocp-async-8,da6c59201a550cd7,66fc79136b631780] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:23.414 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,d26d5e23f52439fc,84aa364a6ebf1c5a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d26d5e23f52439fc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:23.419 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,d26d5e23f52439fc,84aa364a6ebf1c5a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d26d5e23f52439fc, duration=8 ms] 2024-09-11 18:00:23.767 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:23.768 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:24.104 INFO 2981023 --- [ocp-schedules-20,03261cd78f7072ac,034f298a0e9bcf7d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:00:24.104 INFO 2981023 --- [ocp-schedules-20,03261cd78f7072ac,034f298a0e9bcf7d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:00:24.104 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:00:24.105 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:00:24.105 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:00:24.109 INFO 2981023 --- [ocp-schedules-20,03261cd78f7072ac,034f298a0e9bcf7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:00:24.109 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:24.110 INFO 2981023 --- [ocp-schedules-20,03261cd78f7072ac,034f298a0e9bcf7d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:00:24.110 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:24.110 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:24.111 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:24.112 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:00:24.113 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:00:24.127 INFO 2981023 --- [ocp-schedules-20,03261cd78f7072ac,034f298a0e9bcf7d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:00:24.127 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:00:24.128 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:00:24.130 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:00:24.132 INFO 2981023 --- [ocp-schedules-20,03261cd78f7072ac,034f298a0e9bcf7d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:00:24.355 INFO 2981023 --- [ocp-schedules-16,e4010205b1d612b3,5e5722b3eab1a630] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:00:24.741 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,5cbe362bb2216de5,9388d88bb8eba75b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5cbe362bb2216de5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:24.749 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,5cbe362bb2216de5,9388d88bb8eba75b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5cbe362bb2216de5, duration=12 ms] 2024-09-11 18:00:25.359 INFO 2981023 --- [ocp-async-10,590f1320754f25cb,779663ec432c22b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:26.023 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6dc6b8f4e7825a82,2ab228294f577e06] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6dc6b8f4e7825a82, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:26.029 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6dc6b8f4e7825a82,2ab228294f577e06] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6dc6b8f4e7825a82, duration=9 ms] 2024-09-11 18:00:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,43140e434c6d8a6a,b0e8b4aa3344f060] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,43140e434c6d8a6a,b0e8b4aa3344f060] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:26.339 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:00:26.338869+08:00, lastProcessTime=2024-09-11T18:00:16.368744+08:00 2024-09-11 18:00:26.471 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:00:26.338869+08:00 2024-09-11 18:00:26.768 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:26.768 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:27.284 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,fc65aa82a50c06dd,eff517bdd03b84a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fc65aa82a50c06dd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:27.289 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,fc65aa82a50c06dd,eff517bdd03b84a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fc65aa82a50c06dd, duration=8 ms] 2024-09-11 18:00:27.359 INFO 2981023 --- [ocp-async-2,d568d3d38c8ecbb9,20c11df6850f9a30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:27.771 INFO 2981023 --- [ocp-async-6,ecb70fd84c66bcf8,baf8651b709d5f0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:00:28.555 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,15ed4bd7bc55272a,8eb1fc4c6e982c22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=15ed4bd7bc55272a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:28.561 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,15ed4bd7bc55272a,8eb1fc4c6e982c22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=15ed4bd7bc55272a, duration=9 ms] 2024-09-11 18:00:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,e31cf20f0f847461,4dcb502526a0ff2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,e31cf20f0f847461,4dcb502526a0ff2d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:29.359 INFO 2981023 --- [ocp-async-1,43af74a8bd26b35d,dc9f555563e88a5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:29.768 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:29.769 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:29.827 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,70b4744bd7aa2870,6a19ea32c5b76d91] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=70b4744bd7aa2870, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:29.833 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,70b4744bd7aa2870,6a19ea32c5b76d91] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=70b4744bd7aa2870, duration=8 ms] 2024-09-11 18:00:30.773 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=1386a1df-7888-4ac9-9eba-58db1971c45c), params:null 2024-09-11 18:00:30.788 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:30.793 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2024-09-11 18:00:30.982 INFO 2981023 --- [ocp-schedules-10,f080131fbfd1a662,02b3e512758b3a79] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:00:31.010 WARN 2981023 --- [scanner-13588413-1,f080131fbfd1a662,3947436058a60a9e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:00:31.052 INFO 2981023 --- [alarm-detect-12,f080131fbfd1a662,ed08b535c1203d2f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645165, data is empty. 2024-09-11 18:00:31.056 INFO 2981023 --- [alarm-detect-12,f080131fbfd1a662,ed08b535c1203d2f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645165, data is empty. 2024-09-11 18:00:31.061 INFO 2981023 --- [alarm-detect-12,f080131fbfd1a662,ed08b535c1203d2f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645165, data is empty. 2024-09-11 18:00:31.079 INFO 2981023 --- [alarm-detect-11,f080131fbfd1a662,d0108336242afe58] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645165, data is empty. 2024-09-11 18:00:31.087 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f01483c43c6f8e5f,8bb74a08595e6f4b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f01483c43c6f8e5f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:31.088 INFO 2981023 --- [alarm-detect-11,f080131fbfd1a662,d0108336242afe58] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645165, data is empty. 2024-09-11 18:00:31.100 INFO 2981023 --- [alarm-detect-6,f080131fbfd1a662,b2f38c838725cc80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645165, data is empty. 2024-09-11 18:00:31.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f01483c43c6f8e5f,8bb74a08595e6f4b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f01483c43c6f8e5f, duration=26 ms] 2024-09-11 18:00:31.108 INFO 2981023 --- [alarm-detect-11,f080131fbfd1a662,d0108336242afe58] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645165, data is empty. 2024-09-11 18:00:31.113 INFO 2981023 --- [alarm-detect-6,f080131fbfd1a662,b2f38c838725cc80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645165, data is empty. 2024-09-11 18:00:31.121 INFO 2981023 --- [alarm-detect-6,f080131fbfd1a662,b2f38c838725cc80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645165, data is empty. 2024-09-11 18:00:31.359 INFO 2981023 --- [alarm-detect-9,f080131fbfd1a662,4b011756a509032c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645165, data is empty. 2024-09-11 18:00:31.359 INFO 2981023 --- [ocp-async-9,043e6e39f19a43f6,abc61da0c832b242] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:31.363 INFO 2981023 --- [alarm-detect-9,f080131fbfd1a662,4b011756a509032c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645165, data is empty. 2024-09-11 18:00:31.365 INFO 2981023 --- [alarm-detect-9,f080131fbfd1a662,4b011756a509032c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645165, data is empty. 2024-09-11 18:00:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,832703417c1c1205,9fc91e8828fa508b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,832703417c1c1205,9fc91e8828fa508b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:32.369 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,27780ba6c6bb62ff,8559a340d8199ce5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=27780ba6c6bb62ff, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:32.374 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,27780ba6c6bb62ff,8559a340d8199ce5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=27780ba6c6bb62ff, duration=8 ms] 2024-09-11 18:00:32.383 INFO 2981023 --- [ocp-schedules-29,4c91eee711d0f202,606262004366be85] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:00:32.769 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:32.769 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:33.359 INFO 2981023 --- [ocp-async-2,41dd2e3f29205bd9,d2f3a62fbb27c28a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:33.652 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f9672b2a277f6a7d,4e81077752aeab4a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f9672b2a277f6a7d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:33.657 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f9672b2a277f6a7d,4e81077752aeab4a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f9672b2a277f6a7d, duration=9 ms] 2024-09-11 18:00:34.848 INFO 2981023 --- [ocp-schedules-30,b1d1397eac72320f,8a534861ee404149] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:00:34.853 INFO 2981023 --- [ocp-schedules-30,b1d1397eac72320f,8a534861ee404149] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:00:34.853 INFO 2981023 --- [ocp-schedules-30,b1d1397eac72320f,8a534861ee404149] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:00:34.876 INFO 2981023 --- [ocp-schedules-24,b957e319d20c9df6,d6c056bfffd53567] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY,C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,C_rtdob-1718689631-PRIMARY/H_10.31.35.5,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkprdocp-1719994437/H_10.31.35.10,C_linkrtd-1719994443/H_10.31.35.6,C_rtdob-1718689631/H_10.31.35.5,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7, current targets ,,,,,C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY,C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkprdocp-1719994437/H_10.31.35.10,C_linkrtd-1719994443/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631/H_10.31.35.5,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7,H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:34.932 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,0ae1e1621ad7f8a8,13284631e3f57fb0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0ae1e1621ad7f8a8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:34.938 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,0ae1e1621ad7f8a8,13284631e3f57fb0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0ae1e1621ad7f8a8, duration=9 ms] 2024-09-11 18:00:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,00d9256604f2e0ed,82e382964415d56f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,00d9256604f2e0ed,82e382964415d56f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:35.359 INFO 2981023 --- [ocp-async-4,75587666d01b4fd9,58ea4761e8d286d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:35.770 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:35.770 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:36.204 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9d6d90acb2d87822,74ad4ac8283fe04d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9d6d90acb2d87822, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:36.210 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9d6d90acb2d87822,74ad4ac8283fe04d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9d6d90acb2d87822, duration=9 ms] 2024-09-11 18:00:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:00:36.337980+08:00, lastProcessTime=2024-09-11T18:00:26.338869+08:00 2024-09-11 18:00:36.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:00:36.337980+08:00 2024-09-11 18:00:36.970 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:00:36.970 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:00:36.973 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:00:36.973 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:00:36.973 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:00:36.973 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:00:36.975 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:00:36.975 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:00:36.975 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:00:36.975 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:00:37.005 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:00:37.011 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:00:37.011 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:00:37.011 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:00:37.011 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:00:37.011 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:00:37.016 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:00:37.305 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:00:37.314 INFO 2981023 --- [prometheus-cache-monitor1,d3510ff10c0fba5d,5e7e4fa052de5ffd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":7960,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":7555} 2024-09-11 18:00:37.315 INFO 2981023 --- [prometheus-cache-monitor1,d3510ff10c0fba5d,5e7e4fa052de5ffd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:00:37.360 INFO 2981023 --- [ocp-async-5,d32d891a64d2ac15,eb82d9e6f0ed7d0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:37.472 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,2de85e9c3ca67142,f7f3ce9b25d9bc00] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2de85e9c3ca67142, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:37.478 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,2de85e9c3ca67142,f7f3ce9b25d9bc00] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2de85e9c3ca67142, duration=9 ms] 2024-09-11 18:00:37.584 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.608351455S 2024-09-11 18:00:37.584 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:00:37.636 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:00:37.636 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:00:37.654 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:00:37.697 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:00:37.697 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:00:37.697 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:00:37.697 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:00:37.697 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:00:37.697 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:00:37.703 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:00:37.771 INFO 2981023 --- [ocp-async-8,aad22ee025f1f9d5,8b505992977b00d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:00:37.807 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] o.s.boot.SpringApplication : Started application in 0.833 seconds (JVM running for 84637.679) 2024-09-11 18:00:37.807 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.833819861S 2024-09-11 18:00:38.043 INFO 2981023 --- [ocp-schedules-25,33ee7a613129402b,1d735478e9461e2a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:00:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,c8658e15d8550897,3b47221af2c95398] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,c8658e15d8550897,3b47221af2c95398] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:38.754 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a4c9bff26a6ed62d,8c9178ecb995c5df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a4c9bff26a6ed62d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:38.765 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,a4c9bff26a6ed62d,8c9178ecb995c5df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a4c9bff26a6ed62d, duration=19 ms] 2024-09-11 18:00:38.770 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:38.772 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:39.360 INFO 2981023 --- [ocp-async-7,3c5c803365735c35,2faca7d96e3882f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:40.003 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:00:40.003 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:00:40.019 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,2660558971057793,ae8129e4920ac062] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2660558971057793, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:40.025 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,2660558971057793,ae8129e4920ac062] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2660558971057793, duration=9 ms] 2024-09-11 18:00:40.797 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=1386a1df-7888-4ac9-9eba-58db1971c45c), params:null 2024-09-11 18:00:40.811 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:40.816 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2024-09-11 18:00:41.277 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,6c3f0209f182f887,0263504e138ed9f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6c3f0209f182f887, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:41.282 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,6c3f0209f182f887,0263504e138ed9f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6c3f0209f182f887, duration=8 ms] 2024-09-11 18:00:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,5da17608148722fa,775c279f12b1d2d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,5da17608148722fa,775c279f12b1d2d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:41.360 INFO 2981023 --- [ocp-async-8,b30e046f593ecef3,129bca132236f0c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:41.421 INFO 2981023 --- [ocp-schedules-26,78bbf6e88d85dccf,e053f843d2b3132b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:00:41.468 WARN 2981023 --- [scanner-13588774-1,78bbf6e88d85dccf,b10fe16f0755e79b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:00:41.772 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:41.772 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:42.385 INFO 2981023 --- [ocp-schedules-8,9a65d69f715d9fad,f8c0750b02534e33] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:00:42.535 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c63d50db983a4e7f,0c75ca63ff753c73] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c63d50db983a4e7f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:42.541 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c63d50db983a4e7f,0c75ca63ff753c73] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c63d50db983a4e7f, duration=9 ms] 2024-09-11 18:00:43.360 INFO 2981023 --- [ocp-async-6,d82243975341d14a,f516aa4fd9d2ffc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:43.512 INFO 2981023 --- [ocp-schedules-3,3dba0f403b98f52d,a5f6397713462b21] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:00:43.530 WARN 2981023 --- [ocp-schedules-3,3dba0f403b98f52d,a5f6397713462b21] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:00:43.754 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,c9f1627fe074df6c,b3d649f5d3211c6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c9f1627fe074df6c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:43.768 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,c9f1627fe074df6c,b3d649f5d3211c6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c9f1627fe074df6c, duration=17 ms] 2024-09-11 18:00:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,8d2f63f808d3e432,ac45dc67e77e0dd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:44.318 INFO 2981023 --- [prometheus-minute-schedule-1,8d2f63f808d3e432,ac45dc67e77e0dd8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:44.772 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:44.774 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:45.040 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9403336145097fbd,0e1ae8e7c29d4293] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9403336145097fbd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:45.047 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9403336145097fbd,0e1ae8e7c29d4293] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9403336145097fbd, duration=10 ms] 2024-09-11 18:00:45.101 INFO 2981023 --- [prometheus-second-schedule-1,4638d6d6c146e113,ffbc6ac03bc9a3be] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:00:45.360 INFO 2981023 --- [ocp-async-10,c6483beef692eb4e,ccecb8163ebae5cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:46.023 INFO 2981023 --- [ocp-schedules-7,b5fb305d346f63a7,c6bdfb9f5c703a51] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:00:46.023 INFO 2981023 --- [host-check-agent-status1,b5fb305d346f63a7,3eb0bf7da67f0d40] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:00:46.024 INFO 2981023 --- [host-check-agent-status2,b5fb305d346f63a7,0912cc16f5c9d93e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:00:46.024 INFO 2981023 --- [host-check-agent-status3,b5fb305d346f63a7,1d683e537226ddde] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:00:46.024 INFO 2981023 --- [ocp-schedules-5,2294edae71e7c7fe,d643caa89ef259d4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:00:46.024 INFO 2981023 --- [host-check-agent-status4,b5fb305d346f63a7,c61e35ac0db1a8ec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:00:46.025 INFO 2981023 --- [host-check-clockdiff1,2294edae71e7c7fe,a35a4b78c1b52810] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:00:46.025 INFO 2981023 --- [host-check-clockdiff2,2294edae71e7c7fe,e3c0beb379843da8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:00:46.025 INFO 2981023 --- [host-check-clockdiff3,2294edae71e7c7fe,518015b7d574f607] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:00:46.025 INFO 2981023 --- [host-check-clockdiff4,2294edae71e7c7fe,51d97db6f10318e7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:00:46.054 INFO 2981023 --- [host-check-clockdiff2,2294edae71e7c7fe,e3c0beb379843da8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048846 5 5 , error= 2024-09-11 18:00:46.055 INFO 2981023 --- [host-check-agent-status1,b5fb305d346f63a7,3eb0bf7da67f0d40] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:46.056 INFO 2981023 --- [ocp-schedules-14,a8ba12e292f4f819,e76172819af7e16e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:00:46.057 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:00:46.057 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:00:46.057 INFO 2981023 --- [host-check-clockdiff3,2294edae71e7c7fe,518015b7d574f607] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048846 0 0 , error= 2024-09-11 18:00:46.059 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:00:46.059 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:00:46.060 INFO 2981023 --- [host-check-agent-status1,b5fb305d346f63a7,3eb0bf7da67f0d40] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:46.061 INFO 2981023 --- [host-check-agent-status3,b5fb305d346f63a7,1d683e537226ddde] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:00:46.063 INFO 2981023 --- [host-check-agent-status4,b5fb305d346f63a7,c61e35ac0db1a8ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:46.065 INFO 2981023 --- [host-check-agent-status2,b5fb305d346f63a7,0912cc16f5c9d93e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:00:46.065 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:00:46.065 INFO 2981023 --- [host-check-agent-status4,b5fb305d346f63a7,c61e35ac0db1a8ec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:46.066 INFO 2981023 --- [host-check-agent-status3,b5fb305d346f63a7,1d683e537226ddde] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:00:46.066 INFO 2981023 --- [host-check-agent-status2,b5fb305d346f63a7,0912cc16f5c9d93e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:00:46.067 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:00:46.067 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:00:46.068 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:00:46.099 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:46.175 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:00:46.184 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:46.185 INFO 2981023 --- [ocp-schedules-8,b5118c14b89e6d8e,81d4c96ebce8bcbc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:00:46.193 INFO 2981023 --- [ocp-schedules-16,11a938ebc7cea975,4e3b7bcf9c5f21b7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:00:46.193 INFO 2981023 --- [ocp-schedules-25,7d48c2e3060320f5,35c2f48a32be6766] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:00:46.194 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:46.194 INFO 2981023 --- [ocp-schedules-19,f4e0104978833e95,65ede2b38b2cd288] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:00:46.194 INFO 2981023 --- [ocp-schedules-27,9bba111902ee5847,8d45bcf940918437] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:00:46.199 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:46.261 INFO 2981023 --- [ocp-schedules-25,7d48c2e3060320f5,35c2f48a32be6766] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:00:46.265 INFO 2981023 --- [ocp-schedules-25,7d48c2e3060320f5,35c2f48a32be6766] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:00:46.270 INFO 2981023 --- [ocp-schedules-19,f4e0104978833e95,65ede2b38b2cd288] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:00:46.272 INFO 2981023 --- [ocp-schedules-16,11a938ebc7cea975,4e3b7bcf9c5f21b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:00:46.274 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048802626000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:00:46.288 INFO 2981023 --- [ocp-schedules-8,b5118c14b89e6d8e,81d4c96ebce8bcbc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:00:46.311 INFO 2981023 --- [ocp-schedules-21,9a8c38e7740432e9,efd1d7582077bd9b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:00:46.358 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:00:46.372 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4cee6d9669b0d5f0,9afbe1c039abe3e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4cee6d9669b0d5f0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:46.372 INFO 2981023 --- [host-check-agent-status1,b5fb305d346f63a7,3eb0bf7da67f0d40] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:00:46.375 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:00:46.376 INFO 2981023 --- [host-check-agent-status1,b5fb305d346f63a7,3eb0bf7da67f0d40] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:00:46.376 INFO 2981023 --- [host-check-agent-status1,b5fb305d346f63a7,3eb0bf7da67f0d40] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:00:46.376 INFO 2981023 --- [host-check-agent-status1,b5fb305d346f63a7,3eb0bf7da67f0d40] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:00:46.377 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:00:46.377492292+08:00 2024-09-11 18:00:46.380 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:00:46.378+08:00 2024-09-11 18:00:46.392 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:00:46.378+08:00 and status AVAILABLE 2024-09-11 18:00:46.395 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4cee6d9669b0d5f0,9afbe1c039abe3e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4cee6d9669b0d5f0, duration=63 ms] 2024-09-11 18:00:46.396 INFO 2981023 --- [host-check-agent-status2,b5fb305d346f63a7,0912cc16f5c9d93e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:00:46.403 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:00:46.404 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:00:46.405 INFO 2981023 --- [host-check-agent-status2,b5fb305d346f63a7,0912cc16f5c9d93e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:00:46.405 INFO 2981023 --- [host-check-agent-status2,b5fb305d346f63a7,0912cc16f5c9d93e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:00:46.405 INFO 2981023 --- [host-check-agent-status2,b5fb305d346f63a7,0912cc16f5c9d93e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:00:46.405 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:00:46.410716585+08:00 2024-09-11 18:00:46.409 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:00:46.405+08:00 2024-09-11 18:00:46.411 INFO 2981023 --- [host-check-agent-status4,b5fb305d346f63a7,c61e35ac0db1a8ec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:00:46.411 INFO 2981023 --- [host-check-agent-status3,b5fb305d346f63a7,1d683e537226ddde] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:00:46.413 INFO 2981023 --- [host-check-agent-status3,b5fb305d346f63a7,1d683e537226ddde] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:00:46.413 INFO 2981023 --- [host-check-agent-status3,b5fb305d346f63a7,1d683e537226ddde] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:00:46.413 INFO 2981023 --- [host-check-agent-status3,b5fb305d346f63a7,1d683e537226ddde] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:00:46.454 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:00:46.453359+08:00, lastProcessTime=2024-09-11T18:00:36.337980+08:00 2024-09-11 18:00:46.461 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:00:46.472 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:00:46.405+08:00 and status AVAILABLE 2024-09-11 18:00:46.474 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:00:46.473451051+08:00 2024-09-11 18:00:46.477 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:00:46.474+08:00 2024-09-11 18:00:46.480 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:00:46.474+08:00 and status AVAILABLE 2024-09-11 18:00:46.481 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:00:46.481 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:00:46.486980172+08:00 2024-09-11 18:00:46.482 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:00:46.483 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:00:46.481+08:00 2024-09-11 18:00:46.484 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:00:46.488 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:00:46.481+08:00 and status AVAILABLE 2024-09-11 18:00:46.489 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:00:46.491 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.492 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:00:46.493 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:00:46.494 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:00:46.498 INFO 2981023 --- [host-check-agent-status4,b5fb305d346f63a7,c61e35ac0db1a8ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:00:46.498 INFO 2981023 --- [host-check-agent-status4,b5fb305d346f63a7,c61e35ac0db1a8ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:00:46.498 INFO 2981023 --- [host-check-agent-status4,b5fb305d346f63a7,c61e35ac0db1a8ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:00:46.501 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.501 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:00:46.503 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:00:46.503 INFO 2981023 --- [ocp-schedules-16,11a938ebc7cea975,4e3b7bcf9c5f21b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:00:46.505 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:00:46.505 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:00:46.507 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.507 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:46.508 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:00:46.509 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:00:46.511 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:00:46.512 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:00:46.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:46.518 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:46.519 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.521 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:00:46.521 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:00:46.521 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.536 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:00:46.539 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:46.541 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:46.547 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:00:46.564 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:00:46.566 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:00:46.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.572 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.572 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:00:46.573 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:00:46.580 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:00:46.588 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:00:46.588 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048802626000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:00:46.594 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.595 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.596 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:00:46.596 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:00:46.596 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.605 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:00:46.605 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:00:46.606 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:00:46.617 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:00:46.618 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:00:46.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.619 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.619 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:46.620 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:00:46.621 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:00:46.623 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:00:46.623 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.625 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:00:46.626 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:00:46.626 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:00:46.634 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:00:46.644 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:00:46.662 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:00:46.662 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:00:46.669 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:00:46.670 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:00:46.681 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:00:46.686 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:00:46.690 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.690 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.691 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:00:46.692 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:00:46.693 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:00:46.696 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:00:46.699 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:00:46.699 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.699 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:00:46.700 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:00:46.702 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:00:46.715 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.715 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.716 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:00:46.716 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:00:46.720 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.721 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:00:46.722 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:00:46.722 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.724 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:00:46.724 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:00:46.726 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:00:46.727 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:00:46.728 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:00:46.733 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:00:46.735 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.735 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.736 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:00:46.740 INFO 2981023 --- [host-check-host-status3,a8ba12e292f4f819,7ec6b93503dd0a04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:00:46.747 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:00:46.750 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:00:46.756 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:00:46.764 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:00:46.777 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.778 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.778 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:00:46.778 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:00:46.779 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:00:46.779 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.781 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:00:46.781 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:00:46.783 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:00:46.792 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:00:46.794 INFO 2981023 --- [host-check-host-status2,a8ba12e292f4f819,497d4d88f6c78cdb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:00:46.795 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.796 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.796 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:00:46.797 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:00:46.797 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.802 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:00:46.802 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:00:46.806 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:00:46.813 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:00:46.818 INFO 2981023 --- [host-check-host-status1,a8ba12e292f4f819,d7329cbdfdc30146] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:00:46.819 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:00:46.819 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:00:46.827 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.830 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.830 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.831 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.831 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:00:46.832 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:00:46.832 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:00:46.837 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:00:46.839 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:00:46.841 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:00:46.841 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:00:46.841 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:00:46.842 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:00:46.453359+08:00 2024-09-11 18:00:46.842 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:00:46.844 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:00:46.852 INFO 2981023 --- [host-check-host-status4,a8ba12e292f4f819,81104c88e06e5842] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:00:46.856 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.856 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.857 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:00:46.857 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:00:46.858 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:00:46.887 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:00:46.893 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:00:46.899 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:00:46.900 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:00:46.900 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:00:46.900 INFO 2981023 --- [ocp-schedules-19,f4e0104978833e95,65ede2b38b2cd288] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:00:46.914 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:00:46.920 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:00:46.923 INFO 2981023 --- [ocp-schedules-2,51e289aa6b434b88,6563330a14793d1b] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[1]. 2024-09-11 18:00:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,2aefc572f23df9fe,25d943d1eda87e32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,2aefc572f23df9fe,25d943d1eda87e32] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:47.360 INFO 2981023 --- [ocp-async-8,dd530331e6a78b21,6640137515743772] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:47.722 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d94353a8cb3b65a5,df21b1c9a91ccf23] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d94353a8cb3b65a5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:47.727 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d94353a8cb3b65a5,df21b1c9a91ccf23] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d94353a8cb3b65a5, duration=8 ms] 2024-09-11 18:00:47.771 INFO 2981023 --- [ocp-async-4,0f951ef3a60d9c7e,8d3d6d4b239dfd63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:00:47.774 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:47.774 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:47.817 INFO 2981023 --- [host-check-clockdiff1,2294edae71e7c7fe,a35a4b78c1b52810] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048847 6 6 , error= 2024-09-11 18:00:48.985 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2bb11a0aac847c46,a06f72cba2a5b002] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2bb11a0aac847c46, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:48.996 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2bb11a0aac847c46,a06f72cba2a5b002] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2bb11a0aac847c46, duration=15 ms] 2024-09-11 18:00:49.360 INFO 2981023 --- [ocp-async-2,4549b303b4f75815,13632ba442987259] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:00:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:00:50.189 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,d547a622cf1c5f84,acf44d084587d378] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d547a622cf1c5f84, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:50.196 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,d547a622cf1c5f84,acf44d084587d378] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d547a622cf1c5f84, duration=11 ms] 2024-09-11 18:00:50.291 INFO 2981023 --- [ocp-schedules-16,bba6d87720c55e10,151f8e052f5bad4e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:00:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,a8a95ee213081120,b945769e80ed6cb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,a8a95ee213081120,b945769e80ed6cb5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:50.774 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:50.776 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:50.819 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=1386a1df-7888-4ac9-9eba-58db1971c45c), params:null 2024-09-11 18:00:50.838 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:00:50.843 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2024-09-11 18:00:51.360 INFO 2981023 --- [ocp-async-4,449e8b27f567d80a,79aa6674cd9c478e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:51.491 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,53e9541aada869e4,39389424872e9e59] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=53e9541aada869e4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:51.499 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,53e9541aada869e4,39389424872e9e59] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=53e9541aada869e4, duration=11 ms] 2024-09-11 18:00:51.589 INFO 2981023 --- [ocp-schedules-22,1e53de1bb051b491,bb5d50c8f01971d6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:00:51.590 INFO 2981023 --- [ocp-schedules-22,1e53de1bb051b491,bb5d50c8f01971d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:00:51.590 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:00:51.590 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:00:51.590 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:00:51.591 INFO 2981023 --- [ocp-schedules-22,1e53de1bb051b491,bb5d50c8f01971d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:00:51.623 INFO 2981023 --- [ocp-schedules-10,6983fd4cdcbf5eb0,c6061fa329e0e40b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:00:51.646 WARN 2981023 --- [scanner-13588954-1,6983fd4cdcbf5eb0,435eeb599cc2810d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:00:51.692 INFO 2981023 --- [alarm-detect-14,6983fd4cdcbf5eb0,e7aee76eddb9eb68] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645165, data is empty. 2024-09-11 18:00:51.696 INFO 2981023 --- [alarm-detect-14,6983fd4cdcbf5eb0,e7aee76eddb9eb68] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645165, data is empty. 2024-09-11 18:00:51.700 INFO 2981023 --- [alarm-detect-14,6983fd4cdcbf5eb0,e7aee76eddb9eb68] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645165, data is empty. 2024-09-11 18:00:51.716 INFO 2981023 --- [alarm-detect-18,6983fd4cdcbf5eb0,f949feb4e26f5e00] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645165, data is empty. 2024-09-11 18:00:51.722 INFO 2981023 --- [alarm-detect-18,6983fd4cdcbf5eb0,f949feb4e26f5e00] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645165, data is empty. 2024-09-11 18:00:51.723 INFO 2981023 --- [alarm-detect-5,6983fd4cdcbf5eb0,a3e05a03fbfd3ecd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645165, data is empty. 2024-09-11 18:00:51.728 INFO 2981023 --- [alarm-detect-5,6983fd4cdcbf5eb0,a3e05a03fbfd3ecd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645165, data is empty. 2024-09-11 18:00:51.729 INFO 2981023 --- [alarm-detect-18,6983fd4cdcbf5eb0,f949feb4e26f5e00] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645165, data is empty. 2024-09-11 18:00:51.733 INFO 2981023 --- [alarm-detect-5,6983fd4cdcbf5eb0,a3e05a03fbfd3ecd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645165, data is empty. 2024-09-11 18:00:51.830 INFO 2981023 --- [alarm-detect-8,6983fd4cdcbf5eb0,e9962bacbd62ae65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645165, data is empty. 2024-09-11 18:00:51.833 INFO 2981023 --- [alarm-detect-8,6983fd4cdcbf5eb0,e9962bacbd62ae65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645165, data is empty. 2024-09-11 18:00:51.837 INFO 2981023 --- [alarm-detect-8,6983fd4cdcbf5eb0,e9962bacbd62ae65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645165, data is empty. 2024-09-11 18:00:51.983 INFO 2981023 --- [host-check-clockdiff4,2294edae71e7c7fe,51d97db6f10318e7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048851 -1 -1 , error= 2024-09-11 18:00:52.388 INFO 2981023 --- [ocp-schedules-9,7ad5e9135bececc6,6a2fdf12e2c35b43] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:00:52.790 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1412c494e9a55a2a,5c2eaede0788a8de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1412c494e9a55a2a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:52.796 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1412c494e9a55a2a,5c2eaede0788a8de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1412c494e9a55a2a, duration=10 ms] 2024-09-11 18:00:53.218 INFO 2981023 --- [ocp-schedules-4,436f02f3c512b0c7,7eb79c774b13743b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:00:53.218 INFO 2981023 --- [ocp-schedules-4,436f02f3c512b0c7,7eb79c774b13743b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:00:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,87d49947dae840df,6299bf8188ba510c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,87d49947dae840df,6299bf8188ba510c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:53.360 INFO 2981023 --- [ocp-async-8,8e949d2f92a3b323,9b2df44328d4d298] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:53.776 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:53.777 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:54.021 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:00:54.040 WARN 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:00:54.052 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ff491b56c80aead8,a61ef9387dae5099] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ff491b56c80aead8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:54.059 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ff491b56c80aead8,a61ef9387dae5099] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ff491b56c80aead8, duration=11 ms] 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2024-09-11 18:00:54.061 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2024-09-11 18:00:54.062 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2024-09-11 18:00:54.063 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2024-09-11 18:00:54.064 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2024-09-11 18:00:54.065 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2024-09-11 18:00:54.066 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2024-09-11 18:00:54.067 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2024-09-11 18:00:54.068 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2024-09-11 18:00:54.069 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2024-09-11 18:00:54.070 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 3 2024-09-11 18:00:54.071 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2024-09-11 18:00:54.072 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2024-09-11 18:00:54.073 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2024-09-11 18:00:54.074 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2024-09-11 18:00:54.075 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2024-09-11 18:00:54.076 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2024-09-11 18:00:54.077 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2024-09-11 18:00:54.078 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/H_10.31.35.10,C_linkrtd-1719994443-PRIMARY/H_10.31.35.6,H_10.31.35.7,C_rtdob-1718689631-PRIMARY/H_10.31.35.5 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_10.31.35.10,H_10.31.35.5,H_10.31.35.6,H_10.31.35.7 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,C_linkrtd-1719994443-PRIMARY/T_sys_PRIMARY,C_rtdob-1718689631-PRIMARY/T_linkreit_PRIMARY,T_sys_PRIMARY 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_linkprdocp-1719994437-PRIMARY,C_linkrtd-1719994443-PRIMARY,C_rtdob-1718689631-PRIMARY 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2024-09-11 18:00:54.079 INFO 2981023 --- [ocp-schedules-15,df18302b660b97c1,fb5c44ca3fd8bd27] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=126, updatedCount=126, failedToUpdateCount=0 2024-09-11 18:00:54.357 INFO 2981023 --- [ocp-schedules-12,323a292ea0719eff,975b9e7560d38f5f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:00:54.856 INFO 2981023 --- [ocp-schedules-13,41c6d7dfffc2e6a6,b1f0489c0a329577] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:00:54.866 INFO 2981023 --- [ocp-schedules-13,41c6d7dfffc2e6a6,b1f0489c0a329577] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:00:54.866 INFO 2981023 --- [ocp-schedules-13,41c6d7dfffc2e6a6,b1f0489c0a329577] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:00:54.955 INFO 2981023 --- [ocp-schedules-7,ded848dbcd1cc107,ca5a5a755de6b66c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:00:54.955 INFO 2981023 --- [ocp-schedules-7,ded848dbcd1cc107,ca5a5a755de6b66c] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:00:54.960 INFO 2981023 --- [ocp-schedules-7,ded848dbcd1cc107,ca5a5a755de6b66c] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:00:54.960 INFO 2981023 --- [ocp-schedules-7,ded848dbcd1cc107,ca5a5a755de6b66c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:00:54.960 INFO 2981023 --- [ocp-schedules-7,ded848dbcd1cc107,ca5a5a755de6b66c] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:00:54.963 INFO 2981023 --- [ocp-schedules-7,ded848dbcd1cc107,ca5a5a755de6b66c] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:00:54.963 INFO 2981023 --- [ocp-schedules-7,ded848dbcd1cc107,ca5a5a755de6b66c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:00:54.963 INFO 2981023 --- [ocp-schedules-7,ded848dbcd1cc107,ca5a5a755de6b66c] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:00:54.974 INFO 2981023 --- [ocp-schedules-7,ded848dbcd1cc107,ca5a5a755de6b66c] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:00:55.320 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5d0792583379a9e8,d1c8658cc74d1fc8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5d0792583379a9e8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:55.326 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5d0792583379a9e8,d1c8658cc74d1fc8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5d0792583379a9e8, duration=9 ms] 2024-09-11 18:00:55.360 INFO 2981023 --- [ocp-async-6,30d757e04d2af331,0fc4f2106c00b9a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:55.819 INFO 2981023 --- [ocp-schedules-22,c4a15e5b472dd039,f3ac2a688b2c4141] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:00:55.819 INFO 2981023 --- [ocp-schedules-22,c4a15e5b472dd039,f3ac2a688b2c4141] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690420679617835008 2024-09-11 18:00:55.822 INFO 2981023 --- [ocp-schedules-22,c4a15e5b472dd039,f3ac2a688b2c4141] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:00:55.822 INFO 2981023 --- [ocp-schedules-22,c4a15e5b472dd039,f3ac2a688b2c4141] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:00:55.824 INFO 2981023 --- [ocp-schedules-22,c4a15e5b472dd039,f3ac2a688b2c4141] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:00:55.824 INFO 2981023 --- [ocp-schedules-22,c4a15e5b472dd039,f3ac2a688b2c4141] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:00:55.956 INFO 2981023 --- [ocp-schedules-12,cf3c1c5849479317,1e5fbad0c094e507] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:00:55.969 INFO 2981023 --- [ocp-schedules-12,cf3c1c5849479317,1e5fbad0c094e507] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:00:55.969 INFO 2981023 --- [ocp-schedules-12,cf3c1c5849479317,1e5fbad0c094e507] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:00:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,a81bfac67e46972d,5c472eb17c4a519b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,a81bfac67e46972d,5c472eb17c4a519b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007104] will be resolved. 2024-09-11 18:00:56.360 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:00:56.367 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : resolveByTimeout, timeoutCount=1 2024-09-11 18:00:56.367 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:00:56.366682+08:00, lastProcessTime=2024-09-11T18:00:46.453359+08:00 2024-09-11 18:00:56.495 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:00:56.500 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:00:56.366682+08:00 2024-09-11 18:00:56.566 INFO 2981023 --- [ocp-schedules-24,ef20375c3279850a,a65d66be387ff5eb] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:00:56.566+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:00:56.568 INFO 2981023 --- [ocp-schedules-24,ef20375c3279850a,a65d66be387ff5eb] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:00:56.585 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,db3d5ffeba9e4860,e7e76c6330b027ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=db3d5ffeba9e4860, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:56.598 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,db3d5ffeba9e4860,e7e76c6330b027ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=db3d5ffeba9e4860, duration=18 ms] 2024-09-11 18:00:56.637 INFO 2981023 --- [ocp-schedules-24,ef20375c3279850a,a65d66be387ff5eb] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 42 to SqlAggregateData 2024-09-11 18:00:56.638 INFO 2981023 --- [ocp-schedules-24,ef20375c3279850a,a65d66be387ff5eb] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:00:56.777 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:56.777 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:00:57.273 INFO 2981023 --- [ocp-schedules-9,5804dde1f78a786b,6ea7f227ad34980e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:00:57.273 INFO 2981023 --- [ocp-schedules-9,5804dde1f78a786b,6ea7f227ad34980e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:00:57.288 INFO 2981023 --- [Collect all tenants major compactions2,5804dde1f78a786b,82e3970d9fe96fba] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:00:57.288 INFO 2981023 --- [Collect all tenants major compactions1,5804dde1f78a786b,7e4a87b324e416d3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:00:57.290 INFO 2981023 --- [Collect all tenants major compactions3,5804dde1f78a786b,26f0a2a06a36f633] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:00:57.290 INFO 2981023 --- [Collect all tenants major compactions5,5804dde1f78a786b,db8834514c9aacc9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:00:57.290 INFO 2981023 --- [Collect all tenants major compactions6,5804dde1f78a786b,2ba74ceda80398c9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:00:57.290 INFO 2981023 --- [Collect all tenants major compactions4,5804dde1f78a786b,38889972a1db64d0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:00:57.312 INFO 2981023 --- [Collect all tenants major compactions2,5804dde1f78a786b,82e3970d9fe96fba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.313 INFO 2981023 --- [Collect all tenants major compactions2,5804dde1f78a786b,82e3970d9fe96fba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:00:57.314 INFO 2981023 --- [Collect all tenants major compactions3,5804dde1f78a786b,26f0a2a06a36f633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.315 INFO 2981023 --- [Collect all tenants major compactions3,5804dde1f78a786b,26f0a2a06a36f633] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:00:57.316 INFO 2981023 --- [Collect all tenants major compactions4,5804dde1f78a786b,38889972a1db64d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.316 INFO 2981023 --- [Collect all tenants major compactions5,5804dde1f78a786b,db8834514c9aacc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.317 INFO 2981023 --- [Collect all tenants major compactions4,5804dde1f78a786b,38889972a1db64d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:00:57.317 INFO 2981023 --- [Collect all tenants major compactions5,5804dde1f78a786b,db8834514c9aacc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:00:57.318 INFO 2981023 --- [Collect all tenants major compactions2,5804dde1f78a786b,82e3970d9fe96fba] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:00:57.318 INFO 2981023 --- [Collect all tenants major compactions6,5804dde1f78a786b,2ba74ceda80398c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.318 INFO 2981023 --- [Collect all tenants major compactions3,5804dde1f78a786b,26f0a2a06a36f633] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:00:57.319 INFO 2981023 --- [Collect all tenants major compactions1,5804dde1f78a786b,7e4a87b324e416d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.319 INFO 2981023 --- [Collect all tenants major compactions6,5804dde1f78a786b,2ba74ceda80398c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:00:57.320 INFO 2981023 --- [Collect all tenants major compactions4,5804dde1f78a786b,38889972a1db64d0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:00:57.321 INFO 2981023 --- [Collect all tenants major compactions1,5804dde1f78a786b,7e4a87b324e416d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:00:57.321 INFO 2981023 --- [Collect all tenants major compactions2,5804dde1f78a786b,82e3970d9fe96fba] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:00:57.322 INFO 2981023 --- [Collect all tenants major compactions5,5804dde1f78a786b,db8834514c9aacc9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:00:57.322 INFO 2981023 --- [Collect all tenants major compactions5,5804dde1f78a786b,db8834514c9aacc9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:00:57.322 INFO 2981023 --- [Collect all tenants major compactions6,5804dde1f78a786b,2ba74ceda80398c9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:00:57.322 INFO 2981023 --- [Collect all tenants major compactions6,5804dde1f78a786b,2ba74ceda80398c9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:00:57.323 INFO 2981023 --- [Collect all tenants major compactions1,5804dde1f78a786b,7e4a87b324e416d3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:00:57.323 INFO 2981023 --- [Collect all tenants major compactions1,5804dde1f78a786b,7e4a87b324e416d3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:00:57.323 INFO 2981023 --- [Collect all tenants major compactions3,5804dde1f78a786b,26f0a2a06a36f633] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:00:57.325 INFO 2981023 --- [Collect all tenants major compactions4,5804dde1f78a786b,38889972a1db64d0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:00:57.325 INFO 2981023 --- [ocp-schedules-9,5804dde1f78a786b,6ea7f227ad34980e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:00:57.361 INFO 2981023 --- [ocp-async-3,4b3514efc3412896,8445a90690acc35b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:57.404 INFO 2981023 --- [ocp-schedules-26,f90c390fcf8ce066,9e9f27b9e3c4d0f2] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:00:57.405 INFO 2981023 --- [ocp-schedules-26,f90c390fcf8ce066,9e9f27b9e3c4d0f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:00:57.435 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:00:57.435 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.437 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:00:57.437 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.438 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:00:57.438 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.439 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:00:57.440 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:00:57.441 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:00:57.447 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.448 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.448 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.448 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.449 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.449 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.455 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.455 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.460 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.460 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.462 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.463 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:00:57.463 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.464 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.471 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.471 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:00:57.473 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.474 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.474 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:00:57.474 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4361D6E0-0-0] 2024-09-11 18:00:57.474 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:00:57.475 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.475 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:00:57.477 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.478 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.478 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:00:57.481 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.485 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.485 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:00:57.487 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.487 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:00:57.494 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.495 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:00:57.495 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.496 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:00:57.496 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.497 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:00:57.497 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.497 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:00:57.498 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:00:57.498 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.499 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:00:57.499 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.499 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.500 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:00:57.500 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:00:57.500 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.501 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:00:57.501 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4361D6EA-0-0] 2024-09-11 18:00:57.501 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:00:57.501 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:00:57.501 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.502 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:00:57.502 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.503 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:00:57.503 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.504 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:00:57.504 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:00:57.505 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.506 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:00:57.509 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.509 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:00:57.510 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.510 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.511 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:00:57.511 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:00:57.512 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:00:57.512 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.513 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:00:57.513 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.514 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:00:57.515 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:00:57.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.519 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:00:57.520 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.520 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:00:57.521 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:00:57.523 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:00:57.525 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:57.527 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.528 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.532 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:57.533 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.534 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:00:57.538 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:00:57.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:00:57.541 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:00:57.542 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:00:57.545 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:00:57.545 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:00:57.545 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:00:57.551 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:00:57.553 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:00:57.553 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:00:57.553 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.553 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.553 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.554 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:00:57.560 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:00:57.562 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:00:57.562 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:00:57.565 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:00:57.570 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:00:57.572 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:00:57.572 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:00:57.572 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:00:57.572 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:00:57.574 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:00:57.574 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:00:57.574 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:00:57.574 INFO 2981023 --- [ocp-schedules-26,f90c390fcf8ce066,9e9f27b9e3c4d0f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:00:57.771 INFO 2981023 --- [ocp-async-6,4aba6f8e54d6a497,83eef16058994522] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:00:57.851 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,81782c5d92b60caa,312c99c55ca3cf20] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=81782c5d92b60caa, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:57.856 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,81782c5d92b60caa,312c99c55ca3cf20] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=81782c5d92b60caa, duration=8 ms] 2024-09-11 18:00:58.766 INFO 2981023 --- [ocp-schedules-8,8529647712b403d7,6859371b250265db] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:00:59.121 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8e33439e0383601d,957a3deebf6bcac6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8e33439e0383601d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:00:59.127 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8e33439e0383601d,957a3deebf6bcac6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8e33439e0383601d, duration=10 ms] 2024-09-11 18:00:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,ecc2fa8700b94907,af09e674ba9ea04b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048860, interval=60 2024-09-11 18:00:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,ecc2fa8700b94907,af09e674ba9ea04b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:00:59.361 INFO 2981023 --- [ocp-async-8,5782a95a0ff9c33c,36d733f4a15eef40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:00:59.777 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:00:59.778 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:00.040 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,75ea03701e93d939] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 17:01:01.0, nextRun=2024-09-11 18:01:00.0, currentDate=Wed Sep 11 18:01:00 HKT 2024 2024-09-11 18:01:00.040 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,75ea03701e93d939] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999995, name=Collect cluster snapshot, templateName=CollectClusterSnapshotSchedule, scheduleRule=0 1 * * * *, lastRunTime=2024-09-11 17:01:01.0, nextRunTime=2024-09-11 18:01:00.0} 2024-09-11 18:01:00.044 WARN 2981023 --- [task-coordinator1,ddd72be6e49ae268,75ea03701e93d939] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-09-11 18:01:00.044 WARN 2981023 --- [task-coordinator1,ddd72be6e49ae268,75ea03701e93d939] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-09-11 18:01:00.044 WARN 2981023 --- [task-coordinator1,ddd72be6e49ae268,75ea03701e93d939] c.o.o.c.t.e.c.worker.ReadyTaskWorker : failed to get template for class, action=com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule java.lang.NullPointerException: null at com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule.ready(CollectClusterSnapshotSchedule.java:74) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.getTaskMeta(ReadyTaskWorker.java:387) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.checkAndScheduleCronTask(ReadyTaskWorker.java:175) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.lambda$loopScheduleTask$1(ReadyTaskWorker.java:136) at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.core.task.engine.coordinator.worker.AbstractTaskWorker.transactionWithoutResult(AbstractTaskWorker.java:59) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.loopScheduleTask(ReadyTaskWorker.java:135) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.work(ReadyTaskWorker.java:104) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lockAndExecute(TaskCoordinator.java:137) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$startup$0(TaskCoordinator.java:75) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 18:01:00.045 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,75ea03701e93d939] c.o.o.c.t.e.c.worker.SubtaskGenerator : Split list is empty, name=Collect cluster hour metric, splitKey=cluster_ids, context=Argument{parallelIdx=-1, stringMap={}, listMap={}} 2024-09-11 18:01:00.049 WARN 2981023 --- [task-coordinator1,ddd72be6e49ae268,75ea03701e93d939] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Subtask is empty, mark task failed, name=Collect cluster hour metric, taskId=4048209 2024-09-11 18:01:00.400 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,413db5b3ac4abca2,83b6ef02ee83f958] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=413db5b3ac4abca2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:00.406 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,413db5b3ac4abca2,83b6ef02ee83f958] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=413db5b3ac4abca2, duration=9 ms] 2024-09-11 18:01:00.847 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=1386a1df-7888-4ac9-9eba-58db1971c45c), params:null 2024-09-11 18:01:00.855 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"changed":true,"installPath":"/data/rtdob/oceanbase","packageInfo":{"architecture":"x86_64","buildNumber":"100000082024070810","fullPackageName":"oceanbase-ce-utils-4.2.4.0-100000082024070810.el8.x86_64","name":"oceanbase-ce-utils","os":"el8","version":"4.2.4.0"}},, 2024-09-11 18:01:00.860 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.h.s.HostOperationServiceImpl : rpm install done, package info=PackageInfo(fullPackageName=oceanbase-ce-utils-4.2.4.0-100000082024070810.el8.x86_64, name=oceanbase-ce-utils, version=4.2.4.0, buildNumber=100000082024070810, os=el8, architecture=x86_64) 2024-09-11 18:01:00.875 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,5cdb1239fcbcee91] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048223, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:01:01.139 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048214] 2024-09-11 18:01:01.364 INFO 2981023 --- [ocp-async-6,6294c8383ba3db2b,a25b9ae19771ef86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:01.525 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,e2df3f4da881086c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048214, operation:EXECUTE, state: RUNNING 2024-09-11 18:01:01.609 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,fb262b2f59a2b364] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048214, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048214, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:01:01.523+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:01:01.619 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,fb262b2f59a2b364] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048214, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:01:01.740 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c416cffbbbaa046e,09273e257318bb19] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c416cffbbbaa046e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:01.745 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c416cffbbbaa046e,09273e257318bb19] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c416cffbbbaa046e, duration=8 ms] 2024-09-11 18:01:01.888 INFO 2981023 --- [ocp-schedules-4,55c19aae46c27650,110f163f8a1706e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:01:01.944 WARN 2981023 --- [scanner-13589301-1,55c19aae46c27650,e9cde5c60fc01efe] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:01:01.967 INFO 2981023 --- [alarm-detect-14,55c19aae46c27650,42a284fdfaa05c2e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645165, data is empty. 2024-09-11 18:01:01.970 INFO 2981023 --- [alarm-detect-14,55c19aae46c27650,42a284fdfaa05c2e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645207, data is empty. 2024-09-11 18:01:01.973 INFO 2981023 --- [alarm-detect-14,55c19aae46c27650,42a284fdfaa05c2e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645207, data is empty. 2024-09-11 18:01:02.000 INFO 2981023 --- [alarm-detect-4,55c19aae46c27650,420e88565794e871] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645165, data is empty. 2024-09-11 18:01:02.004 INFO 2981023 --- [alarm-detect-20,55c19aae46c27650,292ee54f46fb1295] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645165, data is empty. 2024-09-11 18:01:02.006 INFO 2981023 --- [alarm-detect-4,55c19aae46c27650,420e88565794e871] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645207, data is empty. 2024-09-11 18:01:02.008 INFO 2981023 --- [alarm-detect-20,55c19aae46c27650,292ee54f46fb1295] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645207, data is empty. 2024-09-11 18:01:02.011 INFO 2981023 --- [alarm-detect-4,55c19aae46c27650,420e88565794e871] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645207, data is empty. 2024-09-11 18:01:02.019 INFO 2981023 --- [alarm-detect-20,55c19aae46c27650,292ee54f46fb1295] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645207, data is empty. 2024-09-11 18:01:02.113 INFO 2981023 --- [alarm-detect-4,55c19aae46c27650,64626321b864dc87] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645165, data is empty. 2024-09-11 18:01:02.119 INFO 2981023 --- [alarm-detect-4,55c19aae46c27650,64626321b864dc87] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645207, data is empty. 2024-09-11 18:01:02.122 INFO 2981023 --- [alarm-detect-4,55c19aae46c27650,64626321b864dc87] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645207, data is empty. 2024-09-11 18:01:02.190 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048221] 2024-09-11 18:01:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,f929ec7d89fd4685,9afcb659425fe74e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,f929ec7d89fd4685,9afcb659425fe74e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:02.391 INFO 2981023 --- [ocp-schedules-9,f65023addbe587a0,a5df3a1eb05f3bfe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:01:02.576 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,bd9620b6d67f27f7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048221, operation:EXECUTE, state: RUNNING 2024-09-11 18:01:02.667 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048221, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Install OB rpm, zone2.idc=default_idc, sub_task_instance_id=4048221, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:01:02.573+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:01:02.706 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.h.s.HostOperationServiceImpl : [downloadRpm]: generate agent async task token:e04acce7-d344-4caf-a14d-35c596c0dca5 2024-09-11 18:01:02.715 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:02.722 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:02.778 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:02.778 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:02.783 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/file/download, request body:DownloadFileRequest(source=RemoteFileAccessInfo(type=URL, url=http://10.31.35.10:8080/api/v2/object-storage/sys-package/oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm, path=null), target=rpms/oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm, validateChecksum=true, checksum=b1c535e623b7dc1c20647c49ecc9eceac23f321c), params:null 2024-09-11 18:01:02.805 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=e04acce7-d344-4caf-a14d-35c596c0dca5), params:null 2024-09-11 18:01:02.825 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:01:02.847 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.ocp.common.lang.pattern.Retry : wait for 3 seconds 2024-09-11 18:01:02.999 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,de08a51fd1ec1509,762cde8752fe801b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=de08a51fd1ec1509, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:03.005 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,de08a51fd1ec1509,762cde8752fe801b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=de08a51fd1ec1509, duration=9 ms] 2024-09-11 18:01:03.361 INFO 2981023 --- [ocp-async-4,1fa458633b58b4e9,895647062358c64c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:04.270 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8625e7da0b6cd92e,61fe2881a96690fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8625e7da0b6cd92e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:04.284 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8625e7da0b6cd92e,61fe2881a96690fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8625e7da0b6cd92e, duration=19 ms] 2024-09-11 18:01:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,62ca769e5d224643,c30fd3c294325de0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:05.318 INFO 2981023 --- [prometheus-minute-schedule-1,62ca769e5d224643,c30fd3c294325de0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:05.361 INFO 2981023 --- [ocp-async-1,c1fa63d2cfaa94ef,f8a560d04da1076b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:05.570 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ccd903185f3c29d9,a86a5ed9b69854a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ccd903185f3c29d9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:05.576 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ccd903185f3c29d9,a86a5ed9b69854a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ccd903185f3c29d9, duration=9 ms] 2024-09-11 18:01:05.779 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:05.780 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:05.850 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=e04acce7-d344-4caf-a14d-35c596c0dca5), params:null 2024-09-11 18:01:05.858 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"path":"rpms/oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"},, 2024-09-11 18:01:05.862 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.h.s.HostOperationServiceImpl : rpm install, hostId=2000010, softwarePackageMeta=SoftwarePackageMeta(id=1000021, objectId=4000032, name=oceanbase-ce, fileName=oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm, extension=rpm, size=128665672, sha1=b1c535e623b7dc1c20647c49ecc9eceac23f321c, downloadUrl=http://10.31.35.10:8080/api/v2/object-storage/sys-package/oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm, type=OB_SERVER_INSTALL_PACKAGE, version=4.2.4.0, buildNumber=100000082024070810, operatingSystem=el7, architecture=x86_64, createTime=2024-09-11T15:33:44+08:00, deletable=null, installationDataList=null, hasSignature=null, signatureValid=null), rpmFilePath=rpms/oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm, installPath=/data/rtdob/oceanbase 2024-09-11 18:01:05.867 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.h.s.HostOperationServiceImpl : [installRpm]: generate agent async task token:f819809e-c7c7-4aa1-839a-9612571b73ce 2024-09-11 18:01:05.873 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:05.882 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:05.906 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/package/install, request body:InstallPackageRequest(name=oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64, file=rpms/oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm, installPath=/data/rtdob/oceanbase, force=false), params:null 2024-09-11 18:01:05.914 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=f819809e-c7c7-4aa1-839a-9612571b73ce), params:null 2024-09-11 18:01:05.919 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:01:05.923 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2024-09-11 18:01:06.356 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:01:06.356434+08:00, lastProcessTime=2024-09-11T18:00:56.366682+08:00 2024-09-11 18:01:06.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:01:06.356434+08:00 2024-09-11 18:01:06.922 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ea5751f436b9c496,c3961a76976f1e40] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ea5751f436b9c496, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:06.928 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ea5751f436b9c496,c3961a76976f1e40] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ea5751f436b9c496, duration=9 ms] 2024-09-11 18:01:07.361 INFO 2981023 --- [ocp-async-8,ed7f7e877a39b8cc,75180d1cfd4db924] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:07.509 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:57:07.112, end=2024-09-11T17:59:07.505} 2024-09-11 18:01:07.517 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, 2024-09-11 18:01:07.517 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:01:07.519 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:07.112, end=2024-09-11T17:59:07.505},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:01:07.519 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:01:07.519 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:01:07.520 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:01:07.520 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:07.112, end=2024-09-11T17:59:07.505},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:01:07.520 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:01:07.520 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:01:07.521 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:01:07.530 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:07.112, end=2024-09-11T17:59:07.505},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:01:07.530 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:01:07.530 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:01:07.547 INFO 2981023 --- [plan_change_detector18,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:01:07.564 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467564000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.564Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.566 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:01:07.566 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.573 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.573 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.575 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.575 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.581 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.581 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.582 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:01:07.582 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.564, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.583 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:01:07.585 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:07.112, end=2024-09-11T17:59:07.505},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:01:07.585 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:01:07.585 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:01:07.596 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:01:07.596 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:01:07.598 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:07.112, end=2024-09-11T17:59:07.505},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:01:07.598 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:01:07.598 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:01:07.644 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467643000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.643Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.645 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467645000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.645Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.646 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467646000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.646Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.646 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467646000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.646Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.663 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467663000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.663Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.664 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467664000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.664Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.665 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467665000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.665Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.674 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.674 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.675 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467675000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.675Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.675 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.675 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.679 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:01:07.679 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.683 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.683 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.683 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.683 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.685 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.685 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.705 INFO 2981023 --- [ocp-schedules-5,0a1697afb207ed49,c73f0f7dbd585d51] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:01:07.711 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.711 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.712 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:01:07.712 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.712 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 65 2024-09-11 18:01:07.712 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.718 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.718 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.718 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:01:07.718 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.720 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.720 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.721 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.721 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.721 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.721 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.725 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.725 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.727 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.727 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.730 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.730 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.730 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.730 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.732 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.732 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.732 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.732 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.733 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.733 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.733 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.733 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.734 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.734 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.735 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.736 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.736 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.737 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.737 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.737 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.737 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.737 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.737 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.742 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.742 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.743 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.645, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.744 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.643, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.744 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.744 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.745 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.646, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.748 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.748 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.748 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.750 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.750 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.751 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.753 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.753 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.753 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.757 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.646, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.757 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.758 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.664, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.762 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.675, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.775 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.663, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.776 INFO 2981023 --- [ocp-async-7,b41dc00436b9be59,30bc32f4db22c767] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:01:07.777 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.777 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.780 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467780000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.780Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.780 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467780000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.780Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.781 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.787 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467787000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.787Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.797 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467797000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.797Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.798 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.798 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.798 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.665, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.798 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.798 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.809 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467809000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.809Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.810 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.810 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.813 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.813 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.818 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467818000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.818Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.821 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.821 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.824 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.824 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.824 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.824 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.826 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.826 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.828 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962467828000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=9DC649D7324D0017757D3894A9682422, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:01:07.828Z, endTime=2024-09-11T09:59:07.505Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:01:07.830 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.830 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.830 INFO 2981023 --- [ocp-schedules-28,4f7d02fce186da16,9d2f8c00971922c8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:01:07.830+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:01:07.831 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.831 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.832 INFO 2981023 --- [ocp-schedules-28,4f7d02fce186da16,9d2f8c00971922c8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:01:07.832 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.832 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.837 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.837 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.838 INFO 2981023 --- [ocp-schedules-28,4f7d02fce186da16,9d2f8c00971922c8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:01:07.839 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.839 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.839 INFO 2981023 --- [ocp-schedules-28,4f7d02fce186da16,9d2f8c00971922c8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:01:07.840 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.840 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.841 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.841 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.842 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.842 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.842 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.842 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.844 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.844 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:01:07.844 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:01:07.845 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.845 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.846 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.847 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.847 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.848 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.848 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.848 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.78, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.851 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.78, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.853 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.853 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.854 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.854 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.855 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.856 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.856 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.856 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.858 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.859 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.797, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.861 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.787, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.861 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.861 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9DC649D7324D0017757D3894A9682422, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.862 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.809, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.863 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:01:07.863 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:01:07.863 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.863 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.864 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.868 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.818, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.869 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:01:07.869 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048747505000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9DC649D7324D0017757D3894A9682422, sqlIdList=null, dynamicSql=false) 2024-09-11 18:01:07.870 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:01:07.873 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:01:07.828, end=2024-09-11T17:59:07.505}, clusterId=2, tenantId=6, sqlId=9DC649D7324D0017757D3894A9682422, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:01:07.873 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:01:07.873 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:07.112, end=2024-09-11T17:59:07.505},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:01:07.874 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:01:07.874 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:01:07.874 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:01:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,67d7715d4ef59df9,b7928f5d42a11a97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,67d7715d4ef59df9,b7928f5d42a11a97] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:08.334 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,d2b5c1701f1fcba4,2bb7c8c9b72440df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d2b5c1701f1fcba4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:08.339 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,d2b5c1701f1fcba4,2bb7c8c9b72440df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d2b5c1701f1fcba4, duration=8 ms] 2024-09-11 18:01:08.780 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:08.782 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:09.361 INFO 2981023 --- [ocp-async-3,9c1b30a9c5fe9b9c,0f9f9e79590d76b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:09.517 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,c22f423a128b3ea8,6bc9cae564f6e944] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c22f423a128b3ea8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:09.528 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,c22f423a128b3ea8,6bc9cae564f6e944] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c22f423a128b3ea8, duration=16 ms] 2024-09-11 18:01:10.003 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:01:10.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:01:10.765 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,957a15935cb3f6d1,a226884572f0f978] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=957a15935cb3f6d1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:10.771 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,957a15935cb3f6d1,a226884572f0f978] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=957a15935cb3f6d1, duration=8 ms] 2024-09-11 18:01:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:01:10.918 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:01:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,4db5deacd6d1e0ad,efea11172f22737c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,4db5deacd6d1e0ad,efea11172f22737c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:11.361 INFO 2981023 --- [ocp-async-7,7e6c3f36a76c8b25,d0524a446994dfa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:11.782 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:11.782 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:12.036 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,02d77b75076f0775,475ad0e40bd484e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=02d77b75076f0775, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:12.047 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,02d77b75076f0775,475ad0e40bd484e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=02d77b75076f0775, duration=17 ms] 2024-09-11 18:01:12.161 INFO 2981023 --- [ocp-schedules-4,f0cd07bc812cb56f,3ff0d3f0b3fee047] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:01:12.228 WARN 2981023 --- [scanner-13590933-1,f0cd07bc812cb56f,12b564dc907a71ea] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:01:12.393 INFO 2981023 --- [ocp-schedules-23,4d75ddb1a0b7dbdd,35c57de82c3aed13] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:01:13.235 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048800, nodeCount=480, affectRows=480 2024-09-11 18:01:13.319 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4348dfcf12dc5443,278542cb455089d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4348dfcf12dc5443, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:13.325 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4348dfcf12dc5443,278542cb455089d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4348dfcf12dc5443, duration=10 ms] 2024-09-11 18:01:13.362 INFO 2981023 --- [ocp-async-9,540f3fbe7fcf3b13,7e9d89b86cf6ac7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:13.487 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048800, nodeCount=1000, affectRows=1000 2024-09-11 18:01:13.489 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048800, nodeCount=998, validCount=998 2024-09-11 18:01:13.490 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048800, nodeCount=1000, affectRows=1000 2024-09-11 18:01:13.492 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048800, nodeCount=1000, validCount=1000 2024-09-11 18:01:13.501 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048800, nodeCount=49, affectRows=49 2024-09-11 18:01:13.501 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048800, nodeCount=46, validCount=46 2024-09-11 18:01:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,6541441076f9a850,1eaf9666ff734291] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,fb1199b4fa15750c,3365cd998f3594fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:01:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,6541441076f9a850,1eaf9666ff734291] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:14.579 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,72f0096ea5fdc1d7,a9caa812b960b674] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=72f0096ea5fdc1d7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:14.585 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,72f0096ea5fdc1d7,a9caa812b960b674] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=72f0096ea5fdc1d7, duration=9 ms] 2024-09-11 18:01:14.783 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:14.783 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:14.869 INFO 2981023 --- [ocp-schedules-19,5b3e00aef0c6ad28,ad251e998ca21bb4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:01:14.879 INFO 2981023 --- [ocp-schedules-19,5b3e00aef0c6ad28,ad251e998ca21bb4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:01:14.879 INFO 2981023 --- [ocp-schedules-19,5b3e00aef0c6ad28,ad251e998ca21bb4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:01:15.059 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048800, nodeCount=480, validCount=480 2024-09-11 18:01:15.362 INFO 2981023 --- [ocp-async-6,bc245423188b46b1,37122b08432ec5a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:15.855 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,14574926bd491d31,d613152d8c0b535c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=14574926bd491d31, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:15.861 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,14574926bd491d31,d613152d8c0b535c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=14574926bd491d31, duration=9 ms] 2024-09-11 18:01:15.927 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=f819809e-c7c7-4aa1-839a-9612571b73ce), params:null 2024-09-11 18:01:15.946 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:false,null,, 2024-09-11 18:01:15.951 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.ocp.common.lang.pattern.Retry : wait for 10 seconds 2024-09-11 18:01:16.020 INFO 2981023 --- [ocp-schedules-3,ce56f841bd539c57,291b02e73fe5b068] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:01:16.021 INFO 2981023 --- [host-check-clockdiff1,ce56f841bd539c57,e588ffc48a4008cc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:01:16.022 INFO 2981023 --- [host-check-clockdiff2,ce56f841bd539c57,47474f203232113f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:01:16.022 INFO 2981023 --- [host-check-clockdiff3,ce56f841bd539c57,84ec685465bf52c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:01:16.023 INFO 2981023 --- [ocp-schedules-30,2e78936b7cdd2442,511113518fbb0938] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:01:16.025 INFO 2981023 --- [host-check-clockdiff4,ce56f841bd539c57,c1760c8091baefcb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:01:16.026 INFO 2981023 --- [ocp-schedules-15,b12e4d6779005940,dcdcd957ed44627e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:01:16.027 INFO 2981023 --- [host-check-agent-status1,2e78936b7cdd2442,0048751ae239489d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:01:16.028 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:01:16.028 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:01:16.028 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:01:16.029 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:01:16.032 INFO 2981023 --- [host-check-agent-status2,2e78936b7cdd2442,1b33da40f9e6e322] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:01:16.044 INFO 2981023 --- [host-check-clockdiff3,ce56f841bd539c57,84ec685465bf52c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048876 0 0 , error= 2024-09-11 18:01:16.044 INFO 2981023 --- [host-check-agent-status3,2e78936b7cdd2442,96a208369f92dc24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:01:16.047 INFO 2981023 --- [host-check-agent-status4,2e78936b7cdd2442,0fda2bb2f128fc66] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:01:16.068 INFO 2981023 --- [host-check-agent-status2,2e78936b7cdd2442,1b33da40f9e6e322] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:01:16.068 INFO 2981023 --- [host-check-agent-status3,2e78936b7cdd2442,96a208369f92dc24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:01:16.074 INFO 2981023 --- [host-check-agent-status3,2e78936b7cdd2442,96a208369f92dc24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:01:16.076 INFO 2981023 --- [host-check-agent-status2,2e78936b7cdd2442,1b33da40f9e6e322] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:01:16.077 INFO 2981023 --- [host-check-agent-status1,2e78936b7cdd2442,0048751ae239489d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:01:16.078 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:01:16.080 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:01:16.081 INFO 2981023 --- [host-check-agent-status1,2e78936b7cdd2442,0048751ae239489d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:01:16.082 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:01:16.085 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:01:16.110 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:01:16.189 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:01:16.190 INFO 2981023 --- [ocp-schedules-27,d66fdfa3b156dba6,4a6eb5343747c9f4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:01:16.190 INFO 2981023 --- [ocp-schedules-12,101571d721c01161,783ac5e82b8fdccc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:01:16.194 INFO 2981023 --- [host-check-agent-status4,2e78936b7cdd2442,0fda2bb2f128fc66] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:16.197 INFO 2981023 --- [host-check-agent-status4,2e78936b7cdd2442,0fda2bb2f128fc66] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:16.199 INFO 2981023 --- [ocp-schedules-17,a2efb3f8eb6bab3a,5ee1fed9a13ba6eb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:01:16.199 INFO 2981023 --- [ocp-schedules-2,d5d7e7fa26a25005,b87a7db1c44c0f6f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:01:16.276 INFO 2981023 --- [ocp-schedules-17,a2efb3f8eb6bab3a,5ee1fed9a13ba6eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:01:16.276 INFO 2981023 --- [ocp-schedules-27,d66fdfa3b156dba6,4a6eb5343747c9f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:01:16.281 INFO 2981023 --- [ocp-schedules-12,101571d721c01161,783ac5e82b8fdccc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:01:16.284 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048802626000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:01:16.287 INFO 2981023 --- [ocp-schedules-12,101571d721c01161,783ac5e82b8fdccc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:01:16.296 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:16.356 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:16.378 INFO 2981023 --- [ocp-schedules-19,07f94d6fe706ba4d,9dc3d2f54934843b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:01:16.470 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:01:16.469046+08:00, lastProcessTime=2024-09-11T18:01:06.356434+08:00 2024-09-11 18:01:16.476 INFO 2981023 --- [host-check-agent-status3,2e78936b7cdd2442,96a208369f92dc24] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:01:16.476 INFO 2981023 --- [host-check-agent-status4,2e78936b7cdd2442,0fda2bb2f128fc66] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:01:16.481 INFO 2981023 --- [host-check-agent-status2,2e78936b7cdd2442,1b33da40f9e6e322] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:01:16.486 INFO 2981023 --- [host-check-agent-status2,2e78936b7cdd2442,1b33da40f9e6e322] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:01:16.486 INFO 2981023 --- [host-check-agent-status2,2e78936b7cdd2442,1b33da40f9e6e322] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:01:16.486 INFO 2981023 --- [host-check-agent-status2,2e78936b7cdd2442,1b33da40f9e6e322] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:01:16.487 INFO 2981023 --- [host-check-agent-status3,2e78936b7cdd2442,96a208369f92dc24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:01:16.487 INFO 2981023 --- [host-check-agent-status3,2e78936b7cdd2442,96a208369f92dc24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:01:16.487 INFO 2981023 --- [host-check-agent-status3,2e78936b7cdd2442,96a208369f92dc24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:01:16.492 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.493 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:01:16.494 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:01:16.494 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:01:16.494 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.495 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:01:16.496 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:01:16.497 INFO 2981023 --- [ocp-schedules-27,d66fdfa3b156dba6,4a6eb5343747c9f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:01:16.497 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:01:16.501722446+08:00 2024-09-11 18:01:16.498 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:01:16.497+08:00 2024-09-11 18:01:16.499 INFO 2981023 --- [host-check-agent-status1,2e78936b7cdd2442,0048751ae239489d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:01:16.500 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.500 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:01:16.500 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:01:16.500 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:16.500 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:01:16.500354753+08:00 2024-09-11 18:01:16.500 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:01:16.497+08:00 and status AVAILABLE 2024-09-11 18:01:16.501 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:01:16.500+08:00 2024-09-11 18:01:16.502 INFO 2981023 --- [host-check-agent-status1,2e78936b7cdd2442,0048751ae239489d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:01:16.502 INFO 2981023 --- [host-check-agent-status1,2e78936b7cdd2442,0048751ae239489d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:01:16.502 INFO 2981023 --- [host-check-agent-status1,2e78936b7cdd2442,0048751ae239489d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:01:16.503 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:01:16.500+08:00 and status AVAILABLE 2024-09-11 18:01:16.504 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:01:16.504 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:01:16.509842926+08:00 2024-09-11 18:01:16.506 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:01:16.504+08:00 2024-09-11 18:01:16.507 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:01:16.506231935+08:00 2024-09-11 18:01:16.508 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:01:16.507+08:00 2024-09-11 18:01:16.509 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:01:16.504+08:00 and status AVAILABLE 2024-09-11 18:01:16.509 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.509 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.509 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:01:16.507+08:00 and status AVAILABLE 2024-09-11 18:01:16.510 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:16.515 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:01:16.555 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:01:16.555 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:01:16.560 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:01:16.562 INFO 2981023 --- [host-check-agent-status4,2e78936b7cdd2442,0fda2bb2f128fc66] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:01:16.562 INFO 2981023 --- [host-check-agent-status4,2e78936b7cdd2442,0fda2bb2f128fc66] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:01:16.562 INFO 2981023 --- [host-check-agent-status4,2e78936b7cdd2442,0fda2bb2f128fc66] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:01:16.581 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:01:16.582 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:01:16.587 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:01:16.588 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:01:16.591 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:16.591 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.592 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:01:16.593 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:01:16.598 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:01:16.607 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:01:16.636 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:01:16.655 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:01:16.655 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048802626000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:01:16.661 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:01:16.674 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:01:16.680 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:01:16.688 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:16.688 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:01:16.689 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.689 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:16.690 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:01:16.691 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.693 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:01:16.694 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:01:16.695 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:01:16.702 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:01:16.709 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:01:16.709 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:01:16.714 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:01:16.719 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:01:16.725 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:01:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:16.728 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.728 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:01:16.732 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:01:16.736 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:01:16.746 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:01:16.753 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:01:16.758 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:01:16.765 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:01:16.767 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:16.767 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.768 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:01:16.769 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:01:16.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:01:16.771 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:01:16.776 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:01:16.777 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:16.779 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:01:16.780 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:01:16.780 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.783 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:01:16.783 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:01:16.784 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:01:16.785 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:01:16.792 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:01:16.793 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:01:16.793 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:01:16.798 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:01:16.802 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:01:16.810 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:01:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:01:16.814 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:01:16.820 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:01:16.821 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:16.821 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.822 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:01:16.825 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:01:16.826 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:01:16.828 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:01:16.831 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:01:16.834 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:01:16.835 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:01:16.837 INFO 2981023 --- [host-check-host-status3,b12e4d6779005940,9dab97475e924fa9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:01:16.849 INFO 2981023 --- [host-check-host-status1,b12e4d6779005940,e6dda66719b38c86] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:01:16.851 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:01:16.851 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:01:16.869 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:01:16.876 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:01:16.894 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:01:16.902 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:01:16.903 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:16.906 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.907 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:01:16.913 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:16.913 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:16.914 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:01:16.918 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:01:16.922 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:01:16.922 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:01:16.923 INFO 2981023 --- [host-check-host-status2,b12e4d6779005940,fa20072d7d0e2ca4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:01:16.934 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:01:16.469046+08:00 2024-09-11 18:01:16.944 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:01:16.950 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:01:16.951 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:01:16.951 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:01:16.951 INFO 2981023 --- [ocp-schedules-17,a2efb3f8eb6bab3a,5ee1fed9a13ba6eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:01:16.955 INFO 2981023 --- [host-check-host-status4,b12e4d6779005940,4e870a9ca58fc5dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:01:16.967 INFO 2981023 --- [host-check-clockdiff2,ce56f841bd539c57,47474f203232113f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048876 5 5 , error= 2024-09-11 18:01:17.084 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e8c27a4b8b6717c2,b9da2f82667f2e3b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e8c27a4b8b6717c2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:17.090 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e8c27a4b8b6717c2,b9da2f82667f2e3b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e8c27a4b8b6717c2, duration=8 ms] 2024-09-11 18:01:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,ee4fd95389c6a5b1,a062d05410c3a39d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,ee4fd95389c6a5b1,a062d05410c3a39d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:17.362 INFO 2981023 --- [ocp-async-3,86f09b4da36145b0,52cecdf566357b21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:17.776 INFO 2981023 --- [ocp-async-10,96b1f39c0f04e16d,4024c63e66d383af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:01:17.783 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:17.784 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:17.811 INFO 2981023 --- [host-check-clockdiff1,ce56f841bd539c57,e588ffc48a4008cc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048877 6 6 , error= 2024-09-11 18:01:18.368 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,2e4d77218a2eb761,10831814cec82ff5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2e4d77218a2eb761, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:18.374 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,2e4d77218a2eb761,10831814cec82ff5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2e4d77218a2eb761, duration=9 ms] 2024-09-11 18:01:19.362 INFO 2981023 --- [ocp-async-2,e9c7c272d31af39a,568aab2dce4679b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:19.635 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e88b6f896dac4757,cadb73862dcaf7fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e88b6f896dac4757, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:19.642 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e88b6f896dac4757,cadb73862dcaf7fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e88b6f896dac4757, duration=11 ms] 2024-09-11 18:01:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,a8977a7bd712c107,89ffd6314041f3e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,a8977a7bd712c107,89ffd6314041f3e8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:20.784 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:20.784 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:20.885 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,213f48b1e487e168,4f48360128962850] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=213f48b1e487e168, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:20.890 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,213f48b1e487e168,4f48360128962850] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=213f48b1e487e168, duration=9 ms] 2024-09-11 18:01:21.362 INFO 2981023 --- [ocp-async-10,2ae2a4ee1035f61d,9271bb56cb7bb56f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:21.981 INFO 2981023 --- [host-check-clockdiff4,ce56f841bd539c57,c1760c8091baefcb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048881 -1 -1 , error= 2024-09-11 18:01:22.151 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,589b06d854526a24,dce1a3b0882ba8a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=589b06d854526a24, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:22.157 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,589b06d854526a24,dce1a3b0882ba8a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=589b06d854526a24, duration=9 ms] 2024-09-11 18:01:22.400 INFO 2981023 --- [ocp-schedules-5,57695b4faca7010d,620e748da39b5967] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:01:22.440 INFO 2981023 --- [ocp-schedules-7,808d136b65468dcf,10518e306eb041de] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:01:22.450 WARN 2981023 --- [scanner-13591304-1,808d136b65468dcf,36218e58fa3a7a14] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:01:22.499 INFO 2981023 --- [alarm-detect-7,808d136b65468dcf,4cb3c678acec96d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645207, data is empty. 2024-09-11 18:01:22.501 INFO 2981023 --- [alarm-detect-7,808d136b65468dcf,4cb3c678acec96d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645207, data is empty. 2024-09-11 18:01:22.505 INFO 2981023 --- [alarm-detect-7,808d136b65468dcf,4cb3c678acec96d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645207, data is empty. 2024-09-11 18:01:22.535 INFO 2981023 --- [alarm-detect-2,808d136b65468dcf,e8dce9e6a98ec305] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645207, data is empty. 2024-09-11 18:01:22.545 INFO 2981023 --- [alarm-detect-2,808d136b65468dcf,e8dce9e6a98ec305] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645207, data is empty. 2024-09-11 18:01:22.545 INFO 2981023 --- [alarm-detect-3,808d136b65468dcf,524513e1c2434dc9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645207, data is empty. 2024-09-11 18:01:22.554 INFO 2981023 --- [alarm-detect-3,808d136b65468dcf,524513e1c2434dc9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645207, data is empty. 2024-09-11 18:01:22.555 INFO 2981023 --- [alarm-detect-2,808d136b65468dcf,e8dce9e6a98ec305] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645207, data is empty. 2024-09-11 18:01:22.557 INFO 2981023 --- [alarm-detect-3,808d136b65468dcf,524513e1c2434dc9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645207, data is empty. 2024-09-11 18:01:22.652 INFO 2981023 --- [alarm-detect-10,808d136b65468dcf,8104dc9be528390f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645207, data is empty. 2024-09-11 18:01:22.657 INFO 2981023 --- [alarm-detect-10,808d136b65468dcf,8104dc9be528390f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645207, data is empty. 2024-09-11 18:01:22.662 INFO 2981023 --- [alarm-detect-10,808d136b65468dcf,8104dc9be528390f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645207, data is empty. 2024-09-11 18:01:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,54fdd2ed89b069b8,e69e4c8bd6e47542] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:23.317 INFO 2981023 --- [prometheus-minute-schedule-1,54fdd2ed89b069b8,e69e4c8bd6e47542] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:23.362 INFO 2981023 --- [ocp-async-3,772aa65430b084cb,ee3d17110966a2d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:23.441 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8e14fa799caa98d2,068d86ade217dd0c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8e14fa799caa98d2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:23.447 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8e14fa799caa98d2,068d86ade217dd0c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8e14fa799caa98d2, duration=9 ms] 2024-09-11 18:01:23.784 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:23.785 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:24.136 INFO 2981023 --- [ocp-schedules-9,9ca9bf71bc535104,b03f4d2c4d97f590] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:01:24.136 INFO 2981023 --- [ocp-schedules-9,9ca9bf71bc535104,b03f4d2c4d97f590] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:01:24.136 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:01:24.136 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:01:24.136 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:01:24.141 INFO 2981023 --- [ocp-schedules-9,9ca9bf71bc535104,b03f4d2c4d97f590] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:01:24.141 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:01:24.142 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:01:24.142 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:24.142 INFO 2981023 --- [ocp-schedules-9,9ca9bf71bc535104,b03f4d2c4d97f590] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:01:24.143 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:01:24.143 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:01:24.145 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:24.161 INFO 2981023 --- [ocp-schedules-9,9ca9bf71bc535104,b03f4d2c4d97f590] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:01:24.161 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:01:24.163 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:01:24.164 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:01:24.166 INFO 2981023 --- [ocp-schedules-9,9ca9bf71bc535104,b03f4d2c4d97f590] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:01:24.360 INFO 2981023 --- [ocp-schedules-29,90cc642e834274be,26d22ce12a397569] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:01:24.466 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,86578fcf6e74a5d2,c7b87b32bc30100e] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&version=2&ObClusterId=1719994443, client=10.31.35.6, traceId=86578fcf6e74a5d2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkrtd,1719994443,2,{"ObRegion":"linkrtd","ObCluster":"linkrtd","ObRegionId":1719994443,"ObClusterId":1719994443,"Type":"PRIMARY","timestamp":1726048884469714,"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:01:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,86578fcf6e74a5d2,c7b87b32bc30100e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkrtd, obClusterId=1719994443, rootServerJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048884466000}, groupName=linkrtd, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:01:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,86578fcf6e74a5d2,c7b87b32bc30100e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:01:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,86578fcf6e74a5d2,c7b87b32bc30100e] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkrtd, clusterId=1719994443, affectRows=1 2024-09-11 18:01:24.479 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,86578fcf6e74a5d2,c7b87b32bc30100e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.6, traceId=86578fcf6e74a5d2, duration=14 ms] 2024-09-11 18:01:24.803 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,26a0887feb8225de,cfde177ba425fe20] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=26a0887feb8225de, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:24.809 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,26a0887feb8225de,cfde177ba425fe20] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=26a0887feb8225de, duration=9 ms] 2024-09-11 18:01:25.362 INFO 2981023 --- [ocp-async-1,c3bba455005875fe,da41282f48fbc0d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:25.954 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=f819809e-c7c7-4aa1-839a-9612571b73ce), params:null 2024-09-11 18:01:25.962 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,{"changed":true,"installPath":"/data/rtdob/oceanbase","packageInfo":{"architecture":"x86_64","buildNumber":"100000082024070810","fullPackageName":"oceanbase-ce-4.2.4.0-100000082024070810.el8.x86_64","name":"oceanbase-ce","os":"el8","version":"4.2.4.0"}},, 2024-09-11 18:01:25.965 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.h.s.HostOperationServiceImpl : rpm install done, package info=PackageInfo(fullPackageName=oceanbase-ce-4.2.4.0-100000082024070810.el8.x86_64, name=oceanbase-ce, version=4.2.4.0, buildNumber=100000082024070810, os=el8, architecture=x86_64) 2024-09-11 18:01:25.979 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,cf4100aaf1d2da16] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048221, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:01:26.092 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,0b209a2b7b255bce,ec6d0c8e8d009c64] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0b209a2b7b255bce, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:26.099 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,0b209a2b7b255bce,ec6d0c8e8d009c64] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0b209a2b7b255bce, duration=10 ms] 2024-09-11 18:01:26.147 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048227] 2024-09-11 18:01:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,a269300bd7b4cd3a,9fc7464d49f4629a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,a269300bd7b4cd3a,9fc7464d49f4629a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:01:26.336761+08:00, lastProcessTime=2024-09-11T18:01:16.469046+08:00 2024-09-11 18:01:26.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:01:26.336761+08:00 2024-09-11 18:01:26.718 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,b8c921eb2c56227a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048227, operation:EXECUTE, state: RUNNING 2024-09-11 18:01:26.785 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:26.786 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:26.805 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,97c249f8217c00d0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048227, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048227, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:01:26.716+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:01:26.818 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,97c249f8217c00d0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048227, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:01:27.191 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048204] 2024-09-11 18:01:27.344 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,2712f433ae7588e9,6bbfbeb8d305c4b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2712f433ae7588e9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:27.350 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,2712f433ae7588e9,6bbfbeb8d305c4b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2712f433ae7588e9, duration=9 ms] 2024-09-11 18:01:27.362 INFO 2981023 --- [ocp-async-8,3a47658338fd63aa,ccc8b4151b6619d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:27.765 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,7b483b56c7b51eab] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048204, operation:EXECUTE, state: RUNNING 2024-09-11 18:01:27.776 INFO 2981023 --- [ocp-async-9,c68051e07ff149ca,687b805beda79e41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:01:27.850 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048204, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Make dir for ob, zone2.idc=default_idc, sub_task_instance_id=4048204, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:01:27.763+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:01:27.854 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] .o.o.i.c.ClusterHostOperationServiceImpl : ob path usage check for data path: /data/oceanbase, hostId: 2000010 2024-09-11 18:01:27.862 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:27.866 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:27.883 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/disk/usage, request body:GetDiskUsageRequest(path=/data/oceanbase), params:null 2024-09-11 18:01:27.891 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] .o.o.i.c.ClusterHostOperationServiceImpl : ob path usage check for install path: /data/rtdob/oceanbase, hostId: 2000010 2024-09-11 18:01:27.912 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:27.917 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:27.932 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/disk/usage, request body:GetDiskUsageRequest(path=/data/rtdob/oceanbase), params:null 2024-09-11 18:01:27.941 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] .o.o.i.c.ClusterHostOperationServiceImpl : ob path usage check done, all check passed 2024-09-11 18:01:27.947 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:27.952 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:27.966 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/ob/install/prepareDirectories, request body:ObClusterInstallRequest(obClusterName=linkrtd, obVersion=4.2.4.0, obPath=ObPath(installPath=/data/rtdob/oceanbase, dataPath=/data/oceanbase, logPath=/oceanbase/redo, diskPathStyle=OBD, runPath=/data/rtdob/oceanbase), runUser=admin), params:null 2024-09-11 18:01:27.982 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,2e9de83f8e511a3f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048204, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:01:28.233 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048219] 2024-09-11 18:01:28.665 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,696c58d71f5fed8f,5fb14769f613767e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=696c58d71f5fed8f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:28.672 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,696c58d71f5fed8f,5fb14769f613767e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=696c58d71f5fed8f, duration=10 ms] 2024-09-11 18:01:28.810 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,a6bff5d3aad74ed3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048219, operation:EXECUTE, state: RUNNING 2024-09-11 18:01:28.900 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,4851304372e2985e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048219, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048219, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:01:28.809+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:01:28.912 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,4851304372e2985e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048219, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:01:29.275 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048232] 2024-09-11 18:01:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,4672224bd3a428cd,03b1596006539976] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,4672224bd3a428cd,03b1596006539976] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:29.363 INFO 2981023 --- [ocp-async-5,a673683c33779bb9,8c0ffd05fe25635a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:29.786 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:29.786 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:29.859 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,7c8b0a3f372233fc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048232, operation:EXECUTE, state: RUNNING 2024-09-11 18:01:29.920 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4b41a66d651191c3,2def383c5f293a94] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4b41a66d651191c3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:29.927 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4b41a66d651191c3,2def383c5f293a94] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4b41a66d651191c3, duration=11 ms] 2024-09-11 18:01:29.944 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,7d798d1216494465] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048232, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Prepare cgroup environment, zone2.idc=default_idc, sub_task_instance_id=4048232, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:01:29.858+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:01:29.951 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,7d798d1216494465] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:29.955 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,7d798d1216494465] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:29.970 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,7d798d1216494465] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/ob/install/prepareCGroupEnv, request body:HandleCGroupEnvRequest(obVersion=4.2.4.0, obPath=ObPath(installPath=/data/rtdob/oceanbase, dataPath=/data/oceanbase, logPath=/oceanbase/redo, diskPathStyle=OBD, runPath=/data/rtdob/oceanbase), runUser=admin), params:null 2024-09-11 18:01:29.982 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,7d798d1216494465] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048232, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:01:30.315 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048231] 2024-09-11 18:01:30.913 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,e2c0cab071b1bb20] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048231, operation:EXECUTE, state: RUNNING 2024-09-11 18:01:31.005 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,a04e90086150376a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048231, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048231, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:01:30.911+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:01:31.015 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,a04e90086150376a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048231, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:01:31.230 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e0733c1213f3f32c,b2da69f512e028b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e0733c1213f3f32c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:31.237 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e0733c1213f3f32c,b2da69f512e028b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e0733c1213f3f32c, duration=11 ms] 2024-09-11 18:01:31.357 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048215] 2024-09-11 18:01:31.363 INFO 2981023 --- [ocp-async-1,7336cd164fa18744,f1cc1c7e808fb03f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:31.959 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,b8e2eb6d4490e0fb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048215, operation:EXECUTE, state: RUNNING 2024-09-11 18:01:32.044 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048215, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Start observer process with param, zone2.idc=default_idc, sub_task_instance_id=4048215, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:01:31.957+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:01:32.060 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] .t.b.s.StartObServerProcessWithParamTask : begin to start observer process with param 2024-09-11 18:01:32.069 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:32.084 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:32.102 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/ob/observer/start, request body:StartObServerProcessRequest(obPath=ObPath(installPath=/data/rtdob/oceanbase, dataPath=/data/oceanbase, logPath=/oceanbase/redo, diskPathStyle=OBD, runPath=/data/rtdob/oceanbase), withArguments=true, arguments=StartObServerProcessArguments(serverIp=10.31.35.7, ipAddressType=LOCAL_IP, zoneName=zone2, configUrl=null, rsList=10.31.35.6:2882:2881, clusterName=linkrtd, obClusterId=1719994443, sqlPort=2881, svrPort=2882, optionString=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6), runUser=admin, runGroup=null, readOnly=false), params:null 2024-09-11 18:01:32.186 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] com.oceanbase.ocp.core.util.TaskUtils : [TaskUtils] wait 60 seconds 2024-09-11 18:01:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,e11ed29a19f269c2,3485a78e1647fded] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,e11ed29a19f269c2,3485a78e1647fded] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:32.403 INFO 2981023 --- [ocp-schedules-21,917a4ba0bbbe66fe,808e3dbe17574f60] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:01:32.513 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c4072dc72d5774dc,3769b02e050f80c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c4072dc72d5774dc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:32.519 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c4072dc72d5774dc,3769b02e050f80c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c4072dc72d5774dc, duration=9 ms] 2024-09-11 18:01:32.697 INFO 2981023 --- [ocp-schedules-16,53a3a3103dea3b07,d8dcb39f2d996a8b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:01:32.728 WARN 2981023 --- [scanner-13591613-1,53a3a3103dea3b07,7592c059a1a818cb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:01:32.782 INFO 2981023 --- [alarm-detect-7,53a3a3103dea3b07,0903a36db97cded6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726048860 ~ 1726048882 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726048875, value=2.23)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.23, values={1726048875=2.23}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:01:32.782 INFO 2981023 --- [alarm-detect-7,53a3a3103dea3b07,0903a36db97cded6] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.10-3, currentState=Pending 2024-09-11 18:01:32.787 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:32.788 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:33.363 INFO 2981023 --- [ocp-async-8,f6a16096ef4412a8,beb44cce32e03208] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:33.804 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,38769a2ed928bfc3,9cec292f15d1ed1c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=38769a2ed928bfc3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:33.810 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,38769a2ed928bfc3,9cec292f15d1ed1c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=38769a2ed928bfc3, duration=10 ms] 2024-09-11 18:01:34.881 INFO 2981023 --- [ocp-schedules-1,d4d488fcf4719a2b,eb480698e907f577] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:01:34.887 INFO 2981023 --- [ocp-schedules-1,d4d488fcf4719a2b,eb480698e907f577] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:01:34.887 INFO 2981023 --- [ocp-schedules-1,d4d488fcf4719a2b,eb480698e907f577] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:01:35.050 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,42dc1d4d2b678a4b,6375840050b8b60c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=42dc1d4d2b678a4b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:35.057 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,42dc1d4d2b678a4b,6375840050b8b60c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=42dc1d4d2b678a4b, duration=9 ms] 2024-09-11 18:01:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,1a230a99e8c0b204,b4b1f1bfc20ec7f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,1a230a99e8c0b204,b4b1f1bfc20ec7f3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:35.363 INFO 2981023 --- [ocp-async-4,8f4ab0f785e2e3e8,d5ea6464a66a513b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:35.788 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:35.789 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:36.315 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,2840935b74874b59,f509510921557a41] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2840935b74874b59, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:36.321 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,2840935b74874b59,f509510921557a41] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2840935b74874b59, duration=9 ms] 2024-09-11 18:01:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:01:36.337543+08:00, lastProcessTime=2024-09-11T18:01:26.336761+08:00 2024-09-11 18:01:36.467 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:01:36.337543+08:00 2024-09-11 18:01:37.314 INFO 2981023 --- [prometheus-cache-monitor1,b656c3ef0c9d577e,39ece3395172643e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":9619} 2024-09-11 18:01:37.315 INFO 2981023 --- [prometheus-cache-monitor1,b656c3ef0c9d577e,39ece3395172643e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:01:37.372 INFO 2981023 --- [ocp-async-9,ac6276b896b3281f,fb5dbe0b942a7818] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:37.568 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,0aff79377a90d654,7d23851352b8484b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0aff79377a90d654, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:37.575 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,0aff79377a90d654,7d23851352b8484b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0aff79377a90d654, duration=11 ms] 2024-09-11 18:01:37.776 INFO 2981023 --- [ocp-async-3,bc03a5b29ea080ca,c09190e2618b99ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:01:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,2364fd31d0a8b563,57e790db57517325] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,2364fd31d0a8b563,57e790db57517325] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:38.789 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:38.791 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:38.851 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,515e56abd271db64,9071d7df604c460e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=515e56abd271db64, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:38.863 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,515e56abd271db64,9071d7df604c460e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=515e56abd271db64, duration=15 ms] 2024-09-11 18:01:39.370 INFO 2981023 --- [ocp-async-8,466e31e7c7b44ff1,f4aafd816d20ca3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:40.003 INFO 2981023 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:01:40.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:01:40.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,46e5ea1cabedd9c3,c879d296b70202fb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=46e5ea1cabedd9c3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:40.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,46e5ea1cabedd9c3,c879d296b70202fb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=46e5ea1cabedd9c3, duration=8 ms] 2024-09-11 18:01:40.450 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?backUrl=/task, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:01:40.450 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/task/4048203, client=10.31.33.41, traceId=] 2024-09-11 18:01:40.450 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:01:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,b886f4ea3294598b,c8cdc7b2711945d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,b886f4ea3294598b,c8cdc7b2711945d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:41.370 INFO 2981023 --- [ocp-async-3,76bc72b2a8b9525d,88c522bfdc144fcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:41.791 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:41.792 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:42.405 INFO 2981023 --- [ocp-schedules-19,b75a1fa9a88a8a84,2ced6bc5401cb2d4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:01:42.882 INFO 2981023 --- [ocp-schedules-4,74b31b1d3672216a,e324105dfb481e98] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:01:42.934 WARN 2981023 --- [scanner-13591888-1,74b31b1d3672216a,5344d1d7348b5632] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:01:42.951 INFO 2981023 --- [alarm-detect-2,74b31b1d3672216a,fd4696de3332b1db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645207, data is empty. 2024-09-11 18:01:42.962 INFO 2981023 --- [alarm-detect-2,74b31b1d3672216a,fd4696de3332b1db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645207, data is empty. 2024-09-11 18:01:42.966 INFO 2981023 --- [alarm-detect-2,74b31b1d3672216a,fd4696de3332b1db] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645207, data is empty. 2024-09-11 18:01:42.982 INFO 2981023 --- [alarm-detect-10,74b31b1d3672216a,99cd24a8fff94cc7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645207, data is empty. 2024-09-11 18:01:42.985 INFO 2981023 --- [alarm-detect-20,74b31b1d3672216a,b591c647a49f1e5d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645207, data is empty. 2024-09-11 18:01:42.986 INFO 2981023 --- [alarm-detect-10,74b31b1d3672216a,99cd24a8fff94cc7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645207, data is empty. 2024-09-11 18:01:42.990 INFO 2981023 --- [alarm-detect-10,74b31b1d3672216a,99cd24a8fff94cc7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645207, data is empty. 2024-09-11 18:01:42.991 INFO 2981023 --- [alarm-detect-20,74b31b1d3672216a,b591c647a49f1e5d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645207, data is empty. 2024-09-11 18:01:42.997 INFO 2981023 --- [alarm-detect-20,74b31b1d3672216a,b591c647a49f1e5d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645207, data is empty. 2024-09-11 18:01:43.091 INFO 2981023 --- [alarm-detect-1,74b31b1d3672216a,4e9a32619bcd9d83] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645207, data is empty. 2024-09-11 18:01:43.095 INFO 2981023 --- [alarm-detect-1,74b31b1d3672216a,4e9a32619bcd9d83] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645207, data is empty. 2024-09-11 18:01:43.099 INFO 2981023 --- [alarm-detect-1,74b31b1d3672216a,4e9a32619bcd9d83] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645207, data is empty. 2024-09-11 18:01:43.370 INFO 2981023 --- [ocp-async-7,ad5c69f973f45f80,fd2abc7a120425fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:43.665 INFO 2981023 --- [ocp-schedules-22,28ca5e22947da0bc,b90a97600bd0ea80] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:01:43.683 WARN 2981023 --- [ocp-schedules-22,28ca5e22947da0bc,b90a97600bd0ea80] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:01:44.218 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e9e71859c1a427b6,75c0db200ad851d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e9e71859c1a427b6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:44.224 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e9e71859c1a427b6,75c0db200ad851d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e9e71859c1a427b6, duration=9 ms] 2024-09-11 18:01:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,005ba8f8bdfac62c,6a597811afcd0402] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,005ba8f8bdfac62c,6a597811afcd0402] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:44.792 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:44.793 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:44.808 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:01:44.809 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, client=10.31.33.41, traceId=] 2024-09-11 18:01:44.809 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:01:44.811 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6d1047cdca96b848,a986dfa5a9143d6d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.31.33.41, traceId=6d1047cdca96b848, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-09-11 18:01:44.811 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6d1047cdca96b848,a986dfa5a9143d6d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.31.33.41, traceId=6d1047cdca96b848, duration=6 ms] 2024-09-11 18:01:44.819 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:01:44.819 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, client=10.31.33.41, traceId=] 2024-09-11 18:01:44.819 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:01:45.032 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8e8841304e738fc6,fc20285489815932] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.31.33.41, traceId=8e8841304e738fc6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-09-11 18:01:45.032 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8e8841304e738fc6,fc20285489815932] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.31.33.41, traceId=8e8841304e738fc6, duration=2 ms] 2024-09-11 18:01:45.041 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,662466dec3e424d7,ae335e63fca04764] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.31.33.41, traceId=662466dec3e424d7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-09-11 18:01:45.048 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,662466dec3e424d7,ae335e63fca04764] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.31.33.41, traceId=662466dec3e424d7, duration=8 ms] 2024-09-11 18:01:45.078 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f2e5348158effe3e,2b2df0dfb71bcbf5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.31.33.41, traceId=f2e5348158effe3e, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-09-11 18:01:45.078 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f2e5348158effe3e,2b2df0dfb71bcbf5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.31.33.41, traceId=f2e5348158effe3e, duration=3 ms] 2024-09-11 18:01:45.101 INFO 2981023 --- [prometheus-second-schedule-1,65175ef267ea5589,4a170c3c26a9f9d0] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:01:45.146 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c3a268ac63a16e8a,01156c2325660c9e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.31.33.41, traceId=c3a268ac63a16e8a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-09-11 18:01:45.146 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c3a268ac63a16e8a,01156c2325660c9e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.31.33.41, traceId=c3a268ac63a16e8a, duration=2 ms] 2024-09-11 18:01:45.149 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:01:45.149 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, client=10.31.33.41, traceId=] 2024-09-11 18:01:45.149 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:01:45.152 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,477d0a81ff970718,aee84819ab6f7bc2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.31.33.41, traceId=477d0a81ff970718, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-09-11 18:01:45.152 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,477d0a81ff970718,aee84819ab6f7bc2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.31.33.41, traceId=477d0a81ff970718, duration=2 ms] 2024-09-11 18:01:45.156 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:01:45.156 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, client=10.31.33.41, traceId=] 2024-09-11 18:01:45.156 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:01:45.370 INFO 2981023 --- [ocp-async-6,a8ed1d38effea493,24f525558677212e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:45.471 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b71e4e3401f996ff,c3dbe58e6dbb3495] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.31.33.41, traceId=b71e4e3401f996ff, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 18:01:45.620 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b71e4e3401f996ff,c3dbe58e6dbb3495] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.31.33.41, traceId=b71e4e3401f996ff, duration=153 ms] 2024-09-11 18:01:45.665 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,79cee2c67ed220a9,8204130c969e739e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.31.33.41, traceId=79cee2c67ed220a9, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-09-11 18:01:45.671 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,79cee2c67ed220a9,8204130c969e739e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.31.33.41, traceId=79cee2c67ed220a9, duration=8 ms] 2024-09-11 18:01:45.985 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,56469bf1312afcd8,c5056b7a3496fb47] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=56469bf1312afcd8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:01:46.028 INFO 2981023 --- [ocp-schedules-7,93487c23520f6ad2,cd8ebaccd67d839e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:01:46.028 INFO 2981023 --- [ocp-schedules-3,b722d58080e2b70e,da7a3e519f03fe25] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:01:46.029 INFO 2981023 --- [host-check-clockdiff1,93487c23520f6ad2,bcf622f4f93e215d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:01:46.029 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:01:46.029 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:01:46.031 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:01:46.031 INFO 2981023 --- [host-check-clockdiff2,93487c23520f6ad2,4d7694a37b188dbf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:01:46.031 INFO 2981023 --- [host-check-clockdiff4,93487c23520f6ad2,9fdc281921abc2a2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:01:46.045 INFO 2981023 --- [host-check-clockdiff3,93487c23520f6ad2,48eaf770ba5e9d04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:01:46.046 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:01:46.047 INFO 2981023 --- [ocp-schedules-24,3262b99eacff2cce,9df2da15611c05bc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:01:46.053 INFO 2981023 --- [host-check-clockdiff3,93487c23520f6ad2,48eaf770ba5e9d04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048906 0 0 , error= 2024-09-11 18:01:46.055 INFO 2981023 --- [host-check-agent-status1,3262b99eacff2cce,b627d0d739d5c6fc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:01:46.056 INFO 2981023 --- [host-check-agent-status2,3262b99eacff2cce,addd065d23367e6c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:01:46.056 INFO 2981023 --- [host-check-agent-status3,3262b99eacff2cce,44a5f1ad00a0a8ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:01:46.056 INFO 2981023 --- [host-check-agent-status4,3262b99eacff2cce,106ef0dbf2579619] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:01:46.065 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,56469bf1312afcd8,c5056b7a3496fb47] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=56469bf1312afcd8, duration=83 ms] 2024-09-11 18:01:46.072 INFO 2981023 --- [host-check-clockdiff2,93487c23520f6ad2,4d7694a37b188dbf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048906 5 5 , error= 2024-09-11 18:01:46.079 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:01:46.080 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:01:46.084 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:01:46.085 INFO 2981023 --- [host-check-agent-status2,3262b99eacff2cce,addd065d23367e6c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:01:46.086 INFO 2981023 --- [host-check-agent-status2,3262b99eacff2cce,addd065d23367e6c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:01:46.086 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:01:46.088 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:01:46.089 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:46.089 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:01:46.102 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:46.163 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:01:46.165 INFO 2981023 --- [ocp-schedules-22,8216ff67790dc2a9,b99f5fb9cc2a6edf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:01:46.178 INFO 2981023 --- [ocp-schedules-22,8216ff67790dc2a9,b99f5fb9cc2a6edf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:01:46.181 INFO 2981023 --- [host-check-agent-status1,3262b99eacff2cce,b627d0d739d5c6fc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:01:46.203 INFO 2981023 --- [ocp-schedules-12,4377e04a05e4edf6,8b4edb29c9b06b89] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:01:46.203 INFO 2981023 --- [host-check-agent-status1,3262b99eacff2cce,b627d0d739d5c6fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:01:46.203 INFO 2981023 --- [ocp-schedules-9,7b16fae19cd7fcf3,ff11cefd30764dae] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:01:46.203 INFO 2981023 --- [ocp-schedules-1,b0dc2a2393b614eb,ec1c4ef5838496a4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:01:46.203 INFO 2981023 --- [ocp-schedules-2,4b52d7c154e7628b,c0396cfccf55ac89] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:01:46.235 INFO 2981023 --- [ocp-schedules-18,0e3af3945bf3f7f0,0bdebdbfac96476a] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 18:01:46.235 INFO 2981023 --- [ocp-schedules-18,0e3af3945bf3f7f0,0bdebdbfac96476a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 18:01:46.272 INFO 2981023 --- [ocp-schedules-12,4377e04a05e4edf6,8b4edb29c9b06b89] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:01:46.279 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048884466000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:01:46.287 INFO 2981023 --- [host-check-agent-status3,3262b99eacff2cce,44a5f1ad00a0a8ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:01:46.290 INFO 2981023 --- [ocp-schedules-1,b0dc2a2393b614eb,ec1c4ef5838496a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:01:46.290 INFO 2981023 --- [ocp-schedules-9,7b16fae19cd7fcf3,ff11cefd30764dae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:01:46.292 INFO 2981023 --- [host-check-agent-status3,3262b99eacff2cce,44a5f1ad00a0a8ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:01:46.293 INFO 2981023 --- [ocp-schedules-9,7b16fae19cd7fcf3,ff11cefd30764dae] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:01:46.295 INFO 2981023 --- [host-check-agent-status4,3262b99eacff2cce,106ef0dbf2579619] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:46.366 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,c62fa7dfc4ebe27b,a5234c56b1f8192b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /favicon.ico?null, client=10.31.33.41, traceId=c62fa7dfc4ebe27b, method=void com.oceanbase.ocp.server.common.controller.web.WebController.dummyFavicon(), args=] 2024-09-11 18:01:46.366 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,c62fa7dfc4ebe27b,a5234c56b1f8192b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /favicon.ico client=10.31.33.41, traceId=c62fa7dfc4ebe27b, duration=172 ms] 2024-09-11 18:01:46.377 INFO 2981023 --- [host-check-agent-status4,3262b99eacff2cce,106ef0dbf2579619] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:46.382 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,47bd880c1172ae7d,d3e2848e984c60f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=47bd880c1172ae7d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:46.383 INFO 2981023 --- [host-check-agent-status2,3262b99eacff2cce,addd065d23367e6c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:01:46.387 INFO 2981023 --- [host-check-agent-status2,3262b99eacff2cce,addd065d23367e6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:01:46.387 INFO 2981023 --- [host-check-agent-status2,3262b99eacff2cce,addd065d23367e6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:01:46.387 INFO 2981023 --- [host-check-agent-status2,3262b99eacff2cce,addd065d23367e6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:01:46.392 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:01:46.459 INFO 2981023 --- [ocp-schedules-4,b154d6fdaabc005a,a01e5448cb91a3fb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 18:01:46.471 INFO 2981023 --- [ocp-schedules-4,b154d6fdaabc005a,a01e5448cb91a3fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:01:46.487 INFO 2981023 --- [ocp-schedules-21,6a9affffe02c9601,b1c0c9ec8c40c9e9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:01:46.501 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,47bd880c1172ae7d,d3e2848e984c60f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=47bd880c1172ae7d, duration=298 ms] 2024-09-11 18:01:46.557 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:01:46.556188+08:00, lastProcessTime=2024-09-11T18:01:36.337543+08:00 2024-09-11 18:01:46.567 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:01:46.569 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:01:46.568927924+08:00 2024-09-11 18:01:46.572 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:01:46.569+08:00 2024-09-11 18:01:46.575 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:01:46.569+08:00 and status AVAILABLE 2024-09-11 18:01:46.583 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:01:46.586 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:01:46.585834842+08:00 2024-09-11 18:01:46.587 INFO 2981023 --- [host-check-agent-status4,3262b99eacff2cce,106ef0dbf2579619] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:01:46.590 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:01:46.586+08:00 2024-09-11 18:01:46.591 INFO 2981023 --- [host-check-agent-status4,3262b99eacff2cce,106ef0dbf2579619] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:01:46.591 INFO 2981023 --- [host-check-agent-status4,3262b99eacff2cce,106ef0dbf2579619] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:01:46.591 INFO 2981023 --- [host-check-agent-status4,3262b99eacff2cce,106ef0dbf2579619] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:01:46.591 INFO 2981023 --- [host-check-agent-status1,3262b99eacff2cce,b627d0d739d5c6fc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:01:46.593 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:01:46.586+08:00 and status AVAILABLE 2024-09-11 18:01:46.594 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.595 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:01:46.595 INFO 2981023 --- [pool-10-thread-11,97bd521d6482bc1e,55b26efc77bd8a82] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 18:01:46.595 INFO 2981023 --- [pool-10-thread-11,97bd521d6482bc1e,55b26efc77bd8a82] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 18:01:46.596 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:01:46.596 INFO 2981023 --- [host-check-agent-status1,3262b99eacff2cce,b627d0d739d5c6fc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:01:46.596 INFO 2981023 --- [host-check-agent-status1,3262b99eacff2cce,b627d0d739d5c6fc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:01:46.596 INFO 2981023 --- [host-check-agent-status1,3262b99eacff2cce,b627d0d739d5c6fc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:01:46.598 INFO 2981023 --- [host-check-agent-status3,3262b99eacff2cce,44a5f1ad00a0a8ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:01:46.599 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:01:46.600 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:01:46.604991667+08:00 2024-09-11 18:01:46.600 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:01:46.602 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:01:46.602 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:01:46.602 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:01:46.600+08:00 2024-09-11 18:01:46.603 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:01:46.608562560+08:00 2024-09-11 18:01:46.604 INFO 2981023 --- [host-check-agent-status3,3262b99eacff2cce,44a5f1ad00a0a8ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:01:46.604 INFO 2981023 --- [host-check-agent-status3,3262b99eacff2cce,44a5f1ad00a0a8ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:01:46.604 INFO 2981023 --- [host-check-agent-status3,3262b99eacff2cce,44a5f1ad00a0a8ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:01:46.609 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:01:46.603+08:00 2024-09-11 18:01:46.610 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:01:46.600+08:00 and status AVAILABLE 2024-09-11 18:01:46.620 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:01:46.603+08:00 and status AVAILABLE 2024-09-11 18:01:46.660 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.660 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:01:46.661 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:01:46.661 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.661 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:46.662 INFO 2981023 --- [ocp-schedules-12,4377e04a05e4edf6,8b4edb29c9b06b89] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:01:46.670 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:01:46.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.671 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.671 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:01:46.671 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:46.673 INFO 2981023 --- [pool-10-thread-10,97bd521d6482bc1e,ce7f1b432805ced4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 18:01:46.678 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,cdb222dc29e93213,f3f7cf784a88926c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=cdb222dc29e93213, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 18:01:46.678 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,cdb222dc29e93213,f3f7cf784a88926c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=cdb222dc29e93213, duration=1 ms] 2024-09-11 18:01:46.680 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.683 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:01:46.683 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:01:46.685 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:01:46.685 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.686 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:01:46.686 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.686 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:01:46.687 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:01:46.687 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.688 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:01:46.694 INFO 2981023 --- [pool-10-thread-9,97bd521d6482bc1e,2e96c717a0f174c9] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 18:01:46.694 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.697 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 18:01:46.697 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.698 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:01:46.700 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:01:46.701 INFO 2981023 --- [pool-10-thread-10,97bd521d6482bc1e,ce7f1b432805ced4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 18:01:46.701 INFO 2981023 --- [pool-10-thread-10,97bd521d6482bc1e,ce7f1b432805ced4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 18:01:46.701 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:46.708 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:01:46.709 INFO 2981023 --- [pool-10-thread-12,97bd521d6482bc1e,2a4e78a0e2148054] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 18:01:46.725 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ef9130ba735e7825,8ed3cbcc898c6b89] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=ef9130ba735e7825, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 18:01:46.725 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ef9130ba735e7825,8ed3cbcc898c6b89] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=ef9130ba735e7825, duration=1 ms] 2024-09-11 18:01:46.726 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:01:46.730 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:01:46.736 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.737 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:01:46.741 INFO 2981023 --- [ForkJoinPool-6-worker-9,97bd521d6482bc1e,8149426f12d1e697] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 18:01:46.741 INFO 2981023 --- [ForkJoinPool-6-worker-9,97bd521d6482bc1e,8149426f12d1e697] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 18:01:46.742 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.744 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 18:01:46.746 INFO 2981023 --- [ocp-schedules-4,b154d6fdaabc005a,a01e5448cb91a3fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 18:01:46.746 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 18:01:46.746 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 18:01:46.755 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:01:46.757 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 18:01:46.757 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 18:01:46.758 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:01:46.772 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:46.772 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:01:46.777 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.777 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:01:46.785 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:01:46.785 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048884466000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:01:46.809 WARN 2981023 --- [ForkJoinPool-6-worker-9,97bd521d6482bc1e,8149426f12d1e697] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 18:01:46.817 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:01:46.822 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.824 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:01:46.825 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.826 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:01:46.826 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.828 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:01:46.829 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:01:46.835 INFO 2981023 --- [ForkJoinPool-6-worker-14,97bd521d6482bc1e,39d1ada498ce9687] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.836 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:46.837 INFO 2981023 --- [ForkJoinPool-6-worker-14,97bd521d6482bc1e,39d1ada498ce9687] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:01:46.838 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.838 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:01:46.838 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:01:46.839 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.839 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:01:46.841 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:01:46.843 INFO 2981023 --- [ocp-schedules-29,79f4a604caeb751e,ef7476c0118d88d3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2049,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":9619} 2024-09-11 18:01:46.843 INFO 2981023 --- [ocp-schedules-29,79f4a604caeb751e,ef7476c0118d88d3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:01:46.847 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:01:46.853 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:01:46.863 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:01:46.864 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.865 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:01:46.867 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:01:46.868 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.869 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.869 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:01:46.870 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:01:46.871 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.873 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:01:46.873 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:01:46.874 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 18:01:46.877 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:01:46.877 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.878 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:01:46.879 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.880 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:01:46.881 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 18:01:46.887 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:01:46.893 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:01:46.899 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.902 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:01:46, 2024-09-11 18:01:46] 2024-09-11 18:01:46.906 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:01:46.911 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.916 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 18:01:46.922 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:46.923 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:01:46.923 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:01:46.923 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:01:46.924 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.924 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.925 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:46.925 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:01:46.926 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.926 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:01:46, 2024-09-11 18:01:46] 2024-09-11 18:01:46.928 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:01:46.930 INFO 2981023 --- [ForkJoinPool-6-worker-14,97bd521d6482bc1e,39d1ada498ce9687] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:01:46.930 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:01:46.931 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 18:01:46.931 INFO 2981023 --- [ForkJoinPool-6-worker-0,97bd521d6482bc1e,d6866aea304ce81b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 18:01:46.932 INFO 2981023 --- [ForkJoinPool-6-worker-14,97bd521d6482bc1e,39d1ada498ce9687] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.933 INFO 2981023 --- [ForkJoinPool-6-worker-14,97bd521d6482bc1e,39d1ada498ce9687] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:01:46.937 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:01:46.937 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:01:46.938 INFO 2981023 --- [ForkJoinPool-6-worker-14,97bd521d6482bc1e,39d1ada498ce9687] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.938 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.939 INFO 2981023 --- [ForkJoinPool-6-worker-14,97bd521d6482bc1e,39d1ada498ce9687] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:01:46.939 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:01:46, 2024-09-11 18:01:46] 2024-09-11 18:01:46.940 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 18:01:46.940 INFO 2981023 --- [ForkJoinPool-6-worker-2,97bd521d6482bc1e,b013154f4e601136] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 18:01:46.946 INFO 2981023 --- [pool-10-thread-12,97bd521d6482bc1e,2a4e78a0e2148054] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 18:01:46.967 INFO 2981023 --- [pool-10-thread-11,97bd521d6482bc1e,55b26efc77bd8a82] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 18:01:46.968 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:46.970 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:01:46.970 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:01:46.970 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.972 INFO 2981023 --- [pool-10-thread-11,97bd521d6482bc1e,55b26efc77bd8a82] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 18:01:46.974 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:01:46.985 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:01:46.995 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:01:46.996 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:46.997 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:01:47.000 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.001 INFO 2981023 --- [ForkJoinPool-6-worker-12,97bd521d6482bc1e,b869a4b3c4e22392] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:01:47.002 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:01:47.006 INFO 2981023 --- [pool-10-thread-9,97bd521d6482bc1e,2e96c717a0f174c9] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 18:01:47.008 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.009 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.009 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:01:47.010 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:01:47.011 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:01:47.012 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.012 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:01:47.013 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:01:47.014 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.014 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:01:47.015 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:01:47.015 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.018 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.023 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 18:01:47.023 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:01:47.024 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:01:47.024 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.025 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.025 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:01:47.025 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:01:47.029 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:01:47.030 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:01:47.031 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.032 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.032 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:01:47.033 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:01:47.035 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:01:47.040 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:01:47.045 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:01:47.046 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:01:47.049 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:01:47.053 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:01:47.060 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:01:47.062 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:01:47.064 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:01:47.072 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.073 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.073 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:01:47.073 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.073 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.074 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:01:47.074 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.074 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:01:47.077 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:01:47.077 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:01:47.077 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.077 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:01:47.093 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:01:47.100 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:01:47.100 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:01:47.101 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:01:47.114 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:01:47.115 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:01:47.116 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:01:47.120 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:01:47.123 INFO 2981023 --- [host-check-host-status3,b722d58080e2b70e,584da9fb3dc1e89e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:01:47.124 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:01:47.135 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.135 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.136 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:01:47.137 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:01:47.138 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:01:47.138 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.141 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.141 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:01:47.142 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:01:47.142 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.144 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:01:47.146 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:01:47.154 INFO 2981023 --- [host-check-host-status1,b722d58080e2b70e,60f181c830e8495c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:01:47.154 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:01:47.157 INFO 2981023 --- [host-check-host-status4,b722d58080e2b70e,009376b82c8aae7d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:01:47.163 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:01:47.164 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:01:47.165 INFO 2981023 --- [host-check-host-status2,b722d58080e2b70e,f341b3b59541a7d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:01:47.183 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:01:47.189 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:01:47.193 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.194 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.194 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:01:47.200 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:01:46.556188+08:00 2024-09-11 18:01:47.202 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:01:47.202 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:01:47.206 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:01:47.218 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:01:47.223 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:01:47.225 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:47.226 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:47.226 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:01:47.231 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:01:47.233 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:01:47.233 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:01:47.234 INFO 2981023 --- [ocp-schedules-1,b0dc2a2393b614eb,ec1c4ef5838496a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:01:47.234 INFO 2981023 --- [ocp-schedules-26,7a186e2840e0d4d8,9f2a9b5c6cdd3f06] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[1]. 2024-09-11 18:01:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,0bda0f3d5a002456,e00bbca341e92bdd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,0bda0f3d5a002456,e00bbca341e92bdd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:47.370 INFO 2981023 --- [ocp-async-3,4db06dc3d12dc62e,60e380b03329cfbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:47.722 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,1908867703c338e0,0de70011bcc7ae84] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048215, client=10.31.33.41, traceId=1908867703c338e0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048215,] 2024-09-11 18:01:47.730 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,1908867703c338e0,0de70011bcc7ae84] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments client=10.31.33.41, traceId=1908867703c338e0, duration=10 ms] 2024-09-11 18:01:47.776 INFO 2981023 --- [ocp-async-10,e3edb8df00f59083,3ec12b189766236d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:01:47.793 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:47.793 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:47.822 INFO 2981023 --- [host-check-clockdiff1,93487c23520f6ad2,bcf622f4f93e215d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048907 5 5 , error= 2024-09-11 18:01:49.370 INFO 2981023 --- [ocp-async-1,260a2abf407a6e1e,fb260dd1fbd5dde5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:49.897 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,6393652ff7a17222,91097f2ed4661690] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=6393652ff7a17222, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:01:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:01:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:01:49.922 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,6393652ff7a17222,91097f2ed4661690] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=6393652ff7a17222, duration=29 ms] 2024-09-11 18:01:50.254 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ece59ea0af7517e8,0f10dbfa9dbcbacb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048215, client=10.31.33.41, traceId=ece59ea0af7517e8, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048215,] 2024-09-11 18:01:50.260 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ece59ea0af7517e8,0f10dbfa9dbcbacb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments client=10.31.33.41, traceId=ece59ea0af7517e8, duration=9 ms] 2024-09-11 18:01:50.293 INFO 2981023 --- [ocp-schedules-16,7c4b374e3b20e9de,92fa53463f92f623] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:01:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,9ea734b6016fe632,d36b61533887fc7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,9ea734b6016fe632,d36b61533887fc7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:50.793 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:50.796 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:51.371 INFO 2981023 --- [ocp-async-4,425f57489c430432,0afe07d3e5136b68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:51.454 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1c653adb0355ff0a,4312e1d06bef3d58] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1c653adb0355ff0a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:51.472 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1c653adb0355ff0a,4312e1d06bef3d58] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1c653adb0355ff0a, duration=22 ms] 2024-09-11 18:01:51.591 INFO 2981023 --- [ocp-schedules-8,3fc5d943bc33b87d,c6d87425a012c9a8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:01:51.592 INFO 2981023 --- [ocp-schedules-8,3fc5d943bc33b87d,c6d87425a012c9a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:01:51.592 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:01:51.593 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:01:51.593 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:01:51.593 INFO 2981023 --- [ocp-schedules-8,3fc5d943bc33b87d,c6d87425a012c9a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:01:52.008 INFO 2981023 --- [host-check-clockdiff4,93487c23520f6ad2,9fdc281921abc2a2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048912 0 0 , error= 2024-09-11 18:01:52.408 INFO 2981023 --- [ocp-schedules-30,446396c1dd328344,bbf03986298434c0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:01:52.447 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0f10afd895687c3f,3658eab03e54cdc5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=0f10afd895687c3f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:01:52.457 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,0f10afd895687c3f,3658eab03e54cdc5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=0f10afd895687c3f, duration=13 ms] 2024-09-11 18:01:52.782 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,c92af93acb237511,f60940a643116806] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048215, client=10.31.33.41, traceId=c92af93acb237511, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048215,] 2024-09-11 18:01:52.785 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,c92af93acb237511,f60940a643116806] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments client=10.31.33.41, traceId=c92af93acb237511, duration=6 ms] 2024-09-11 18:01:53.144 INFO 2981023 --- [ocp-schedules-14,0739944b06ed2348,4e4da0595178553c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:01:53.171 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,de5d0fb0a57715f1,252114273b489093] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=de5d0fb0a57715f1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:53.185 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,de5d0fb0a57715f1,252114273b489093] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=de5d0fb0a57715f1, duration=32 ms] 2024-09-11 18:01:53.207 WARN 2981023 --- [scanner-13592196-1,0739944b06ed2348,040b26db71262569] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:01:53.225 INFO 2981023 --- [ocp-schedules-8,e480e8bd717e9afc,7b78e5c8ccd91c3a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:01:53.225 INFO 2981023 --- [ocp-schedules-8,e480e8bd717e9afc,7b78e5c8ccd91c3a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:01:53.250 INFO 2981023 --- [alarm-detect-10,0739944b06ed2348,aefd1e76dcff00b7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726048875 ~ 1726048903 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726048890, value=1.997)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.997, values={1726048890=1.997}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:01:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,007446dd57af058a,877971620ae258b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:53.318 INFO 2981023 --- [prometheus-minute-schedule-1,007446dd57af058a,877971620ae258b4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:53.371 INFO 2981023 --- [ocp-async-5,c2e4437b6c6c0bd3,7f38752d7ca1f694] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:53.796 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:53.798 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:54.100 INFO 2981023 --- [ocp-schedules-25,fe08abec1730ba31,606acb6c59bc0da1] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:01:54.119 WARN 2981023 --- [ocp-schedules-25,fe08abec1730ba31,606acb6c59bc0da1] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:01:54.363 INFO 2981023 --- [ocp-schedules-26,986ec0d5dfd7042d,8899529a117a1f4f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:01:54.890 INFO 2981023 --- [ocp-schedules-16,0f1a030a5715521a,1e3e5ca43c4b5950] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:01:54.912 INFO 2981023 --- [ocp-schedules-16,0f1a030a5715521a,1e3e5ca43c4b5950] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:01:54.912 INFO 2981023 --- [ocp-schedules-16,0f1a030a5715521a,1e3e5ca43c4b5950] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:01:55.371 INFO 2981023 --- [ocp-async-2,c6b181676d224fbe,50531c794366a688] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:55.562 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,8a2814fd09b1c7c6,3452f778bff77ede] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=8a2814fd09b1c7c6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:01:55.587 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,8a2814fd09b1c7c6,3452f778bff77ede] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=8a2814fd09b1c7c6, duration=28 ms] 2024-09-11 18:01:55.826 INFO 2981023 --- [ocp-schedules-13,a7677b1b3cf4a549,be7dda7a0a5c36ce] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:01:55.826 INFO 2981023 --- [ocp-schedules-13,a7677b1b3cf4a549,be7dda7a0a5c36ce] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690420679617835008 2024-09-11 18:01:55.827 INFO 2981023 --- [ocp-schedules-13,a7677b1b3cf4a549,be7dda7a0a5c36ce] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:01:55.827 INFO 2981023 --- [ocp-schedules-13,a7677b1b3cf4a549,be7dda7a0a5c36ce] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:01:55.827 INFO 2981023 --- [ocp-schedules-13,a7677b1b3cf4a549,be7dda7a0a5c36ce] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:01:55.827 INFO 2981023 --- [ocp-schedules-13,a7677b1b3cf4a549,be7dda7a0a5c36ce] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:01:55.942 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f90f5f3f62d97c56,6ab26fd7be26cfe8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048215, client=10.31.33.41, traceId=f90f5f3f62d97c56, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048215,] 2024-09-11 18:01:55.949 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f90f5f3f62d97c56,6ab26fd7be26cfe8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments client=10.31.33.41, traceId=f90f5f3f62d97c56, duration=10 ms] 2024-09-11 18:01:55.969 INFO 2981023 --- [ocp-schedules-23,05f4ba5b8d8be63d,8f9adc00ad0e3f73] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:01:55.976 INFO 2981023 --- [ocp-schedules-23,05f4ba5b8d8be63d,8f9adc00ad0e3f73] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:01:55.976 INFO 2981023 --- [ocp-schedules-23,05f4ba5b8d8be63d,8f9adc00ad0e3f73] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:01:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,b05149ba4b6e1616,836db6fb2896ec1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,b05149ba4b6e1616,836db6fb2896ec1e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:01:56.353042+08:00, lastProcessTime=2024-09-11T18:01:46.556188+08:00 2024-09-11 18:01:56.501 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:01:56.353042+08:00 2024-09-11 18:01:56.523 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e26c157dcd63cd7f,1d7d94e2f7ebb96f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e26c157dcd63cd7f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:56.535 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e26c157dcd63cd7f,1d7d94e2f7ebb96f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e26c157dcd63cd7f, duration=14 ms] 2024-09-11 18:01:56.670 INFO 2981023 --- [ocp-schedules-18,e6ca426e6656c964,2da2d828f41a9ea1] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:01:56.670+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:01:56.675 INFO 2981023 --- [ocp-schedules-18,e6ca426e6656c964,2da2d828f41a9ea1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:01:56.798 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:56.798 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:01:56.849 INFO 2981023 --- [ocp-schedules-18,e6ca426e6656c964,2da2d828f41a9ea1] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 55 to SqlAggregateData 2024-09-11 18:01:56.850 INFO 2981023 --- [ocp-schedules-18,e6ca426e6656c964,2da2d828f41a9ea1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:01:57.325 INFO 2981023 --- [ocp-schedules-19,520f2aa555bfd993,b2f9a35e807977f5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:01:57.325 INFO 2981023 --- [ocp-schedules-19,520f2aa555bfd993,b2f9a35e807977f5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:01:57.344 INFO 2981023 --- [Collect all tenants major compactions3,520f2aa555bfd993,eeeddaa20f4336b3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:01:57.344 INFO 2981023 --- [Collect all tenants major compactions5,520f2aa555bfd993,85b216f095b3f537] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:01:57.350 INFO 2981023 --- [Collect all tenants major compactions4,520f2aa555bfd993,5b78ad988bbb4c13] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:01:57.350 INFO 2981023 --- [Collect all tenants major compactions1,520f2aa555bfd993,8cd8750f48e93d79] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:01:57.351 INFO 2981023 --- [Collect all tenants major compactions2,520f2aa555bfd993,014aa61ac7d877aa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:01:57.352 INFO 2981023 --- [Collect all tenants major compactions6,520f2aa555bfd993,599c3d9893d2fbca] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:01:57.366 INFO 2981023 --- [Collect all tenants major compactions3,520f2aa555bfd993,eeeddaa20f4336b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.367 INFO 2981023 --- [Collect all tenants major compactions3,520f2aa555bfd993,eeeddaa20f4336b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:01:57.368 INFO 2981023 --- [Collect all tenants major compactions2,520f2aa555bfd993,014aa61ac7d877aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.369 INFO 2981023 --- [Collect all tenants major compactions2,520f2aa555bfd993,014aa61ac7d877aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:01:57.370 INFO 2981023 --- [Collect all tenants major compactions3,520f2aa555bfd993,eeeddaa20f4336b3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:01:57.370 INFO 2981023 --- [Collect all tenants major compactions3,520f2aa555bfd993,eeeddaa20f4336b3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:01:57.370 INFO 2981023 --- [Collect all tenants major compactions5,520f2aa555bfd993,85b216f095b3f537] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.371 INFO 2981023 --- [Collect all tenants major compactions2,520f2aa555bfd993,014aa61ac7d877aa] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:01:57.371 INFO 2981023 --- [Collect all tenants major compactions2,520f2aa555bfd993,014aa61ac7d877aa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:01:57.371 INFO 2981023 --- [ocp-async-4,1a6b6dc9a8fd5d2d,040bf3c9edf92555] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:57.371 INFO 2981023 --- [Collect all tenants major compactions5,520f2aa555bfd993,85b216f095b3f537] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:01:57.373 INFO 2981023 --- [Collect all tenants major compactions6,520f2aa555bfd993,599c3d9893d2fbca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.374 INFO 2981023 --- [Collect all tenants major compactions5,520f2aa555bfd993,85b216f095b3f537] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:01:57.374 INFO 2981023 --- [Collect all tenants major compactions5,520f2aa555bfd993,85b216f095b3f537] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:01:57.374 INFO 2981023 --- [Collect all tenants major compactions6,520f2aa555bfd993,599c3d9893d2fbca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:01:57.375 INFO 2981023 --- [Collect all tenants major compactions1,520f2aa555bfd993,8cd8750f48e93d79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.375 INFO 2981023 --- [Collect all tenants major compactions4,520f2aa555bfd993,5b78ad988bbb4c13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.376 INFO 2981023 --- [Collect all tenants major compactions1,520f2aa555bfd993,8cd8750f48e93d79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:01:57.376 INFO 2981023 --- [Collect all tenants major compactions4,520f2aa555bfd993,5b78ad988bbb4c13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:01:57.376 INFO 2981023 --- [Collect all tenants major compactions6,520f2aa555bfd993,599c3d9893d2fbca] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:01:57.376 INFO 2981023 --- [Collect all tenants major compactions6,520f2aa555bfd993,599c3d9893d2fbca] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:01:57.380 INFO 2981023 --- [Collect all tenants major compactions1,520f2aa555bfd993,8cd8750f48e93d79] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:01:57.380 INFO 2981023 --- [Collect all tenants major compactions4,520f2aa555bfd993,5b78ad988bbb4c13] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:01:57.380 INFO 2981023 --- [Collect all tenants major compactions1,520f2aa555bfd993,8cd8750f48e93d79] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:01:57.380 INFO 2981023 --- [Collect all tenants major compactions4,520f2aa555bfd993,5b78ad988bbb4c13] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:01:57.381 INFO 2981023 --- [ocp-schedules-19,520f2aa555bfd993,b2f9a35e807977f5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:01:57.574 INFO 2981023 --- [ocp-schedules-23,7ef854fdf1b1a5dd,1c68a97d04e59470] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:01:57.575 INFO 2981023 --- [ocp-schedules-23,7ef854fdf1b1a5dd,1c68a97d04e59470] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:01:57.607 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:01:57.607 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:01:57.607 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.607 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.610 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:01:57.610 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:01:57.610 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.610 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:01:57.613 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:01:57.614 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.614 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.615 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.615 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.617 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.617 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.621 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.621 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.626 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.626 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:01:57.627 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.628 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.632 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.632 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.633 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.634 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:01:57.638 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.638 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.641 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.641 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:01:57.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.648 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:01:57.648 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4481D6DF-0-0] 2024-09-11 18:01:57.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:01:57.648 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:01:57.648 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.649 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:01:57.653 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.654 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.666 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.667 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.668 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:01:57.668 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:01:57.669 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.669 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:01:57.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:01:57.670 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.670 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:01:57.671 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.674 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:01:57.675 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.675 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:01:57.676 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:01:57.681 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.681 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:01:57.682 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:01:57.682 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.682 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:01:57.682 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4481D6E9-0-0] 2024-09-11 18:01:57.682 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:01:57.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:01:57.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:01:57.684 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:01:57.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:01:57.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:01:57.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.687 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:01:57.687 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:01:57.690 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:01:57.696 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.697 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:01:57.698 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.698 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:01:57.699 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.701 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:01:57.702 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.703 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:01:57.704 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.705 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:01:57.705 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.706 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:01:57.706 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:57.706 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:01:57.708 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.708 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.709 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:01:57.710 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:57.711 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.712 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:01:57.718 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:01:57.722 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:01:57.722 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:01:57.722 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:01:57.723 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:01:57.723 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:01:57.724 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:01:57.724 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:01:57.725 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.726 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:01:57.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:01:57.731 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:01:57.731 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.731 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.738 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:01:57.739 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:01:57.739 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:01:57.739 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:01:57.740 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:01:57.741 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:01:57.741 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:01:57.742 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:01:57.752 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:01:57.753 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:01:57.753 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:01:57.753 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:01:57.753 INFO 2981023 --- [ocp-schedules-23,7ef854fdf1b1a5dd,1c68a97d04e59470] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:01:57.776 INFO 2981023 --- [ocp-async-2,f4fbd6f0e2608a9d,c35a5ed6ce164648] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:01:58.307 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,872c3e1309c69989,138f8f4663360e89] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=872c3e1309c69989, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:01:58.316 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,872c3e1309c69989,138f8f4663360e89] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=872c3e1309c69989, duration=12 ms] 2024-09-11 18:01:58.544 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,01ff807f4612d709,ab15aabc33b7caca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048215, client=10.31.33.41, traceId=01ff807f4612d709, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048215,] 2024-09-11 18:01:58.547 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,01ff807f4612d709,ab15aabc33b7caca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments client=10.31.33.41, traceId=01ff807f4612d709, duration=7 ms] 2024-09-11 18:01:58.548 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e3869d1ae4ec512c,db8fa5a230744897] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e3869d1ae4ec512c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:01:58.555 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e3869d1ae4ec512c,db8fa5a230744897] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e3869d1ae4ec512c, duration=11 ms] 2024-09-11 18:01:59.236 INFO 2981023 --- [ocp-schedules-15,729eb4742b13390f,a931d59ff53c93b3] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:01:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,60cfe24410dd51e9,9f57fc6dde7eb421] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048920, interval=60 2024-09-11 18:01:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,60cfe24410dd51e9,9f57fc6dde7eb421] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:01:59.371 INFO 2981023 --- [ocp-async-3,bc3917afa60dc141,da6f19483b99ac68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:01:59.799 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:01:59.799 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:01.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,fe6a6299f2cde2c5,fd62f482fb09d6b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fe6a6299f2cde2c5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:01.126 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,fe6a6299f2cde2c5,fd62f482fb09d6b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fe6a6299f2cde2c5, duration=37 ms] 2024-09-11 18:02:01.371 INFO 2981023 --- [ocp-async-7,02fc7d2a44d8ac0a,61b3f733afb3ec17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:02.160 INFO 2981023 --- [ocp-schedules-7,50b85855c2548d87,d3ad81c23671fa7a] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:02:02.160 INFO 2981023 --- [ocp-schedules-7,50b85855c2548d87,d3ad81c23671fa7a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:02:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,fd26510888e36664,a93f73dc3d371f92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,fd26510888e36664,a93f73dc3d371f92] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:02.411 INFO 2981023 --- [ocp-schedules-17,c44303fee5e2e14e,cfeb7f8c69207225] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:02:02.644 INFO 2981023 --- [ocp-schedules-11,73a772712d519a4d,c95bc212a0fe0758] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:02:02.645 INFO 2981023 --- [ocp-schedules-11,73a772712d519a4d,c95bc212a0fe0758] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:02.666 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:02.667 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:02:02.672 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:02.674 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:02:02.674 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048922674000} 2024-09-11 18:02:02.680 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:02:02.684 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000225, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:02:02.684 INFO 2981023 --- [ocp-schedules-11,73a772712d519a4d,c95bc212a0fe0758] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:02:02.799 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:02.801 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:03.009 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:02:03.009 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:02:03.009 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:02:03.015 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000225, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:02:03.015 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:02:03.096 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6b478672a656f056,602095cd8c974d9f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6b478672a656f056, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:03.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6b478672a656f056,602095cd8c974d9f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6b478672a656f056, duration=9 ms] 2024-09-11 18:02:03.363 INFO 2981023 --- [ocp-schedules-30,8342cbaf5643db31,5f3a5ca3bc79ddfc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:02:03.371 INFO 2981023 --- [ocp-async-10,9e84388d1c76e200,8f17e268fb58d582] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:03.418 WARN 2981023 --- [scanner-13592422-1,8342cbaf5643db31,237772ee3dc12bfc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:02:03.441 INFO 2981023 --- [alarm-detect-19,8342cbaf5643db31,013c4305d713bea1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645207, data is empty. 2024-09-11 18:02:03.447 INFO 2981023 --- [alarm-detect-19,8342cbaf5643db31,013c4305d713bea1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645262, data is empty. 2024-09-11 18:02:03.452 INFO 2981023 --- [alarm-detect-19,8342cbaf5643db31,013c4305d713bea1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645262, data is empty. 2024-09-11 18:02:03.466 INFO 2981023 --- [alarm-detect-9,8342cbaf5643db31,013cd49b806aae3c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645207, data is empty. 2024-09-11 18:02:03.470 INFO 2981023 --- [alarm-detect-6,8342cbaf5643db31,59c3866c81d3be89] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645207, data is empty. 2024-09-11 18:02:03.472 INFO 2981023 --- [alarm-detect-9,8342cbaf5643db31,013cd49b806aae3c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645262, data is empty. 2024-09-11 18:02:03.475 INFO 2981023 --- [alarm-detect-6,8342cbaf5643db31,59c3866c81d3be89] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645262, data is empty. 2024-09-11 18:02:03.480 INFO 2981023 --- [alarm-detect-9,8342cbaf5643db31,013cd49b806aae3c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645262, data is empty. 2024-09-11 18:02:03.483 INFO 2981023 --- [alarm-detect-6,8342cbaf5643db31,59c3866c81d3be89] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645262, data is empty. 2024-09-11 18:02:03.508 INFO 2981023 --- [alarm-detect-2,8342cbaf5643db31,17c3658c51933238] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726048890 ~ 1726048913 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726048905, value=1.657)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.657, values={1726048905=1.657}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:02:03.631 INFO 2981023 --- [alarm-detect-4,8342cbaf5643db31,ae95bd1e7ec94d40] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645207, data is empty. 2024-09-11 18:02:03.635 INFO 2981023 --- [alarm-detect-4,8342cbaf5643db31,ae95bd1e7ec94d40] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645262, data is empty. 2024-09-11 18:02:03.642 INFO 2981023 --- [alarm-detect-4,8342cbaf5643db31,ae95bd1e7ec94d40] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645262, data is empty. 2024-09-11 18:02:03.986 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000225 2024-09-11 18:02:05.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fcd9e652e4cac438,346691695c2a5264] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fcd9e652e4cac438, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:05.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,fcd9e652e4cac438,346691695c2a5264] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fcd9e652e4cac438, duration=9 ms] 2024-09-11 18:02:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,dfe728f16f826912,fd50027a4de9ac53] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,dfe728f16f826912,fd50027a4de9ac53] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:05.372 INFO 2981023 --- [ocp-async-9,6d84739c47b01cf2,cf15da027c88d6f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:05.801 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:05.801 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:02:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:02:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:02:05.985 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:02:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:02:05.996 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:02:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:02:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:02:05.999 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T17:59:41, target=2024-09-11T17:57:41, granule=120000ms, delay=120000ms) 2024-09-11 18:02:05.999 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T17:57:41, target=2024-09-11T17:50:51, granule=600000ms, delay=410000ms) 2024-09-11 18:02:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=29ms, period=120000ms 2024-09-11 18:02:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:02:06.337479+08:00, lastProcessTime=2024-09-11T18:01:56.353042+08:00 2024-09-11 18:02:06.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:02:06.337479+08:00 2024-09-11 18:02:07.094 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,b4b6d50b19fb8290,f05779e6aa33e250] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b4b6d50b19fb8290, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:07.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,b4b6d50b19fb8290,f05779e6aa33e250] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b4b6d50b19fb8290, duration=10 ms] 2024-09-11 18:02:07.372 INFO 2981023 --- [ocp-async-1,679910d404707f92,aaeb67b09fec374a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:07.720 INFO 2981023 --- [ocp-schedules-15,2569d91f71ffd6b6,3b29d22e2704fb0b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:02:07.777 INFO 2981023 --- [ocp-async-6,2c3aac0d67b44652,b1604552fc0041fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:02:07.853 INFO 2981023 --- [ocp-schedules-17,cc0846354941d840,7cb387f0ef38b1f2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:02:07.853+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:02:07.854 INFO 2981023 --- [ocp-schedules-17,cc0846354941d840,7cb387f0ef38b1f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:02:07.855 INFO 2981023 --- [ocp-schedules-17,cc0846354941d840,7cb387f0ef38b1f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:02:07.855 INFO 2981023 --- [ocp-schedules-17,cc0846354941d840,7cb387f0ef38b1f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:02:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,7356671bf3456b00,d6339cbf6315b34d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,7356671bf3456b00,d6339cbf6315b34d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:08.801 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:08.802 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:09.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,30c38ca681664943,a70e4dac47ae7213] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=30c38ca681664943, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:09.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,30c38ca681664943,a70e4dac47ae7213] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=30c38ca681664943, duration=9 ms] 2024-09-11 18:02:09.372 INFO 2981023 --- [ocp-async-8,29908bee03410a8b,255ebf3e65835fe4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:09.917 INFO 2981023 --- [ocp-schedules-12,d3148df643933670,f12f1cc7b137e07f] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-09-11 18:02:09.917 INFO 2981023 --- [ocp-schedules-12,d3148df643933670,f12f1cc7b137e07f] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-09-11 18:02:09.917 INFO 2981023 --- [ocp-schedules-12,d3148df643933670,f12f1cc7b137e07f] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=11688 2024-09-11 18:02:10.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:02:10.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:02:10.096 INFO 2981023 --- [ocp-schedules-12,d3148df643933670,f12f1cc7b137e07f] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=11688 2024-09-11 18:02:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:02:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:02:11.094 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1e2c0e444b5f026f,bb72b411879fcf22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1e2c0e444b5f026f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:11.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1e2c0e444b5f026f,bb72b411879fcf22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1e2c0e444b5f026f, duration=11 ms] 2024-09-11 18:02:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,90a27642d9f808bf,d28b9184dd03b65d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,90a27642d9f808bf,d28b9184dd03b65d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:11.372 INFO 2981023 --- [ocp-async-9,3e8d689989e44c50,aaa52f0a63a3edee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:11.802 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:11.803 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:12.414 INFO 2981023 --- [ocp-schedules-1,f5c5efdc43c7d484,c977a00183860efb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:02:13.086 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 18:02:13.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1714dbf312af51eb,46f0b2a0349b779b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1714dbf312af51eb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:13.099 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:13.104 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:13.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1714dbf312af51eb,46f0b2a0349b779b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1714dbf312af51eb, duration=12 ms] 2024-09-11 18:02:13.164 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:02:13.165 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:02:13.175 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:13.175 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:13.186 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:02:13.187 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:02:13.197 INFO 2981023 --- [ocp-schedules-6,e53724f98673c097,ee5404b651523b60] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2024-09-11 18:02:13.372 INFO 2981023 --- [ocp-async-3,f044c7de6c4f1080,98f9607a30f4ec46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:13.682 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048860, nodeCount=1000, affectRows=1000 2024-09-11 18:02:13.682 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048860, nodeCount=1000, affectRows=1000 2024-09-11 18:02:13.684 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048860, nodeCount=999, validCount=999 2024-09-11 18:02:13.685 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048860, nodeCount=1000, validCount=1000 2024-09-11 18:02:13.687 INFO 2981023 --- [ocp-schedules-14,40ab6de03dfd27e4,56174ef5825aa563] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:02:13.737 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048860, nodeCount=49, affectRows=49 2024-09-11 18:02:13.738 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048860, nodeCount=47, validCount=47 2024-09-11 18:02:13.750 WARN 2981023 --- [scanner-13594234-1,40ab6de03dfd27e4,f73e31b029dc8a5c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:02:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,e6598ca9340efac4,d1d427e3b4b48aa6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,afe7a04417f6509f,ace1a5c5a3b185dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:02:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,e6598ca9340efac4,d1d427e3b4b48aa6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:14.803 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:14.804 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:14.915 INFO 2981023 --- [ocp-schedules-8,6de06aaa58e31746,a2a218c01a713667] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:02:14.924 INFO 2981023 --- [ocp-schedules-8,6de06aaa58e31746,a2a218c01a713667] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:02:14.924 INFO 2981023 --- [ocp-schedules-8,6de06aaa58e31746,a2a218c01a713667] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:02:15.094 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,344132a448da3359,4cb585e8c499d677] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=344132a448da3359, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:15.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,344132a448da3359,4cb585e8c499d677] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=344132a448da3359, duration=23 ms] 2024-09-11 18:02:15.181 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048860, nodeCount=480, affectRows=480 2024-09-11 18:02:15.372 INFO 2981023 --- [ocp-async-7,a6183ac1b5e48bd2,f7a8b72d83451237] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:16.022 INFO 2981023 --- [ocp-schedules-2,1df499749fc49c68,5d077830d4d62689] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:02:16.023 INFO 2981023 --- [host-check-clockdiff1,1df499749fc49c68,79a3681f4b42776f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:02:16.023 INFO 2981023 --- [host-check-clockdiff2,1df499749fc49c68,6b5a95966f68d4d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:02:16.023 INFO 2981023 --- [host-check-clockdiff3,1df499749fc49c68,1ca48ebf377641fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:02:16.023 INFO 2981023 --- [host-check-clockdiff4,1df499749fc49c68,06b2706766c819a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:02:16.028 INFO 2981023 --- [ocp-schedules-16,8e13bb85ad79d6e0,ef659d0586c8785d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:02:16.029 INFO 2981023 --- [ocp-schedules-10,8c84b062505d60b6,e36739a0d35382cf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:02:16.032 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:02:16.033 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:02:16.033 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:02:16.034 INFO 2981023 --- [host-check-agent-status1,8e13bb85ad79d6e0,cb7f7c870a337f8d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:02:16.034 INFO 2981023 --- [host-check-agent-status2,8e13bb85ad79d6e0,af505f368e71bcdd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:02:16.048 INFO 2981023 --- [host-check-agent-status3,8e13bb85ad79d6e0,37d80ca3da4aabe9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:02:16.048 INFO 2981023 --- [host-check-agent-status4,8e13bb85ad79d6e0,c4dfb3a0a93bc10f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:02:16.049 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:02:16.062 INFO 2981023 --- [host-check-clockdiff3,1df499749fc49c68,1ca48ebf377641fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048936 0 0 , error= 2024-09-11 18:02:16.063 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:02:16.063 INFO 2981023 --- [host-check-agent-status1,8e13bb85ad79d6e0,cb7f7c870a337f8d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:16.063 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:16.067 INFO 2981023 --- [host-check-agent-status1,8e13bb85ad79d6e0,cb7f7c870a337f8d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:16.067 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:02:16.068 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:16.068 INFO 2981023 --- [host-check-agent-status3,8e13bb85ad79d6e0,37d80ca3da4aabe9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:16.073 INFO 2981023 --- [host-check-agent-status3,8e13bb85ad79d6e0,37d80ca3da4aabe9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:16.096 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:02:16.098 INFO 2981023 --- [host-check-agent-status2,8e13bb85ad79d6e0,af505f368e71bcdd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:02:16.100 INFO 2981023 --- [host-check-agent-status2,8e13bb85ad79d6e0,af505f368e71bcdd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:02:16.100 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:02:16.103 INFO 2981023 --- [host-check-agent-status4,8e13bb85ad79d6e0,c4dfb3a0a93bc10f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:02:16.107 INFO 2981023 --- [host-check-agent-status4,8e13bb85ad79d6e0,c4dfb3a0a93bc10f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:02:16.109 INFO 2981023 --- [host-check-agent-status3,8e13bb85ad79d6e0,37d80ca3da4aabe9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:02:16.119 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:02:16.122 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:02:16.121856320+08:00 2024-09-11 18:02:16.124 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:16.124 INFO 2981023 --- [host-check-agent-status3,8e13bb85ad79d6e0,37d80ca3da4aabe9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:02:16.125 INFO 2981023 --- [host-check-agent-status3,8e13bb85ad79d6e0,37d80ca3da4aabe9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:02:16.125 INFO 2981023 --- [host-check-agent-status3,8e13bb85ad79d6e0,37d80ca3da4aabe9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:02:16.125 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:02:16.122+08:00 2024-09-11 18:02:16.128 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:02:16.122+08:00 and status AVAILABLE 2024-09-11 18:02:16.133 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:16.164 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:02:16.166 INFO 2981023 --- [host-check-agent-status2,8e13bb85ad79d6e0,af505f368e71bcdd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:02:16.167 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:02:16.171990346+08:00 2024-09-11 18:02:16.169 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:02:16.167+08:00 2024-09-11 18:02:16.169 INFO 2981023 --- [host-check-agent-status2,8e13bb85ad79d6e0,af505f368e71bcdd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:02:16.169 INFO 2981023 --- [host-check-agent-status2,8e13bb85ad79d6e0,af505f368e71bcdd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:02:16.169 INFO 2981023 --- [host-check-agent-status2,8e13bb85ad79d6e0,af505f368e71bcdd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:02:16.170 INFO 2981023 --- [host-check-agent-status4,8e13bb85ad79d6e0,c4dfb3a0a93bc10f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:02:16.170 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:02:16.171 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:02:16.167+08:00 and status AVAILABLE 2024-09-11 18:02:16.172 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:02:16.177514506+08:00 2024-09-11 18:02:16.172 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:02:16.172+08:00 2024-09-11 18:02:16.173 INFO 2981023 --- [host-check-agent-status4,8e13bb85ad79d6e0,c4dfb3a0a93bc10f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:02:16.173 INFO 2981023 --- [host-check-agent-status4,8e13bb85ad79d6e0,c4dfb3a0a93bc10f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:02:16.173 INFO 2981023 --- [host-check-agent-status4,8e13bb85ad79d6e0,c4dfb3a0a93bc10f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:02:16.174 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:02:16.172+08:00 and status AVAILABLE 2024-09-11 18:02:16.177 INFO 2981023 --- [host-check-agent-status1,8e13bb85ad79d6e0,cb7f7c870a337f8d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:02:16.179 INFO 2981023 --- [host-check-agent-status1,8e13bb85ad79d6e0,cb7f7c870a337f8d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:02:16.179 INFO 2981023 --- [host-check-agent-status1,8e13bb85ad79d6e0,cb7f7c870a337f8d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:02:16.179 INFO 2981023 --- [host-check-agent-status1,8e13bb85ad79d6e0,cb7f7c870a337f8d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:02:16.182 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:02:16.192 INFO 2981023 --- [ocp-schedules-15,13c8aeedfff305d8,33a430686eb65726] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:02:16.193 INFO 2981023 --- [ocp-schedules-21,c2acf8692b60530b,20388350959b6a24] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:02:16.193 INFO 2981023 --- [ocp-schedules-3,8ebf50cdf87f50a9,9ff77a1e31df4742] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:02:16.193 INFO 2981023 --- [ocp-schedules-13,d1028b0732be4139,170abf7ae513fcef] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:02:16.200 INFO 2981023 --- [ocp-schedules-21,c2acf8692b60530b,20388350959b6a24] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:16.205 INFO 2981023 --- [ocp-schedules-15,13c8aeedfff305d8,33a430686eb65726] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:16.210 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:02:16.210 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:02:16.211 INFO 2981023 --- [ocp-schedules-3,8ebf50cdf87f50a9,9ff77a1e31df4742] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:16.217 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:02:16.218 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:02:16.236 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:02:16.244 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:02:16.253 INFO 2981023 --- [ocp-schedules-28,3cf2927dc102d065,89cdd7abefbe4588] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:02:16.254 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:16.256 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:02:16.259 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:02:16.263 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:16.264 INFO 2981023 --- [ocp-schedules-21,c2acf8692b60530b,20388350959b6a24] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:02:16.264 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:02:16.265 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:02:16.265349904+08:00 2024-09-11 18:02:16.266 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048922674000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:02:16.279 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:02:16.265+08:00 2024-09-11 18:02:16.284 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:02:16.265+08:00 and status AVAILABLE 2024-09-11 18:02:16.305 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:02:16.319 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.319 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.320 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:02:16.320 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:02:16.321 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:02:16.321 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:02:16.322 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:02:16.323 INFO 2981023 --- [ocp-schedules-15,13c8aeedfff305d8,33a430686eb65726] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:02:16.326 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:02:16.333 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.333 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:16.339 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:02:16.341 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:16.342 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:16.349 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.349 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:16.352 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.354 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:16.356 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:02:16.364 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:02:16.366 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:02:16.365805+08:00, lastProcessTime=2024-09-11T18:02:06.337479+08:00 2024-09-11 18:02:16.370 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:02:16.372 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:02:16.379 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:02:16.383 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:02:16.405 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:02:16.443 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:02:16.452 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:02:16.455 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:02:16.455 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048922674000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:02:16.458 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:02:16.465 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:16.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:02:16.469 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:02:16.469 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:02:16.478 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:02:16.485 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:02:16.488 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:02:16.494 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:02:16.506 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:02:16.514 INFO 2981023 --- [host-check-host-status4,8c84b062505d60b6,f89b46830aa7fc5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:02:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:16.516 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.516 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:16.517 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:02:16.517 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:02:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.519 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:16.520 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:02:16.520 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:02:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:02:16.522 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:02:16.534 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:02:16.535 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:02:16.535 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:02:16.543 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:02:16.545 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:02:16.553 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:16.553 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:02:16.553 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.554 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:16.555 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:02:16.555 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:02:16.555 INFO 2981023 --- [host-check-host-status1,8c84b062505d60b6,a2ff7cfb78b86d76] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:02:16.555 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.556 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:02:16.558 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:02:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:02:16.563 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:02:16.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:02:16.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:02:16.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:02:16.568 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:02:16.571 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:02:16.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:16.577 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:02:16.587 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:16.587 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.588 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:02:16.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:02:16.591 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:02:16.594 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:16.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:16.598 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:02:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:02:16.600 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:02:16.609 INFO 2981023 --- [host-check-host-status3,8c84b062505d60b6,d1560ebe433d1702] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:02:16.615 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:02:16.617 INFO 2981023 --- [host-check-host-status2,8c84b062505d60b6,0df55162e471851a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:02:16.644 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:02:16.653 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:16.656 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.657 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:02:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:02:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:02:16.666 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:02:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:16.672 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:16.673 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:02:16.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:02:16.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:02:16.679 INFO 2981023 --- [ocp-schedules-3,8ebf50cdf87f50a9,9ff77a1e31df4742] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:02:16.687 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:02:16.365805+08:00 2024-09-11 18:02:17.084 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,3858f5d985466d88,95c8a5270392ea9f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=3858f5d985466d88, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:17.090 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,3858f5d985466d88,95c8a5270392ea9f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=3858f5d985466d88, duration=9 ms] 2024-09-11 18:02:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,e003d0adea82e72d,c7cb5905076e6bf8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,e003d0adea82e72d,c7cb5905076e6bf8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:17.372 INFO 2981023 --- [ocp-async-10,b97fc3669e2d356e,854aa75eee852237] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:17.389 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048860, nodeCount=480, validCount=480 2024-09-11 18:02:17.777 INFO 2981023 --- [ocp-async-4,84bcb3b456d4f264,f30efb328fba3133] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:02:17.805 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:17.805 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:17.811 INFO 2981023 --- [host-check-clockdiff2,1df499749fc49c68,6b5a95966f68d4d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048937 5 5 , error= 2024-09-11 18:02:17.814 INFO 2981023 --- [host-check-clockdiff1,1df499749fc49c68,79a3681f4b42776f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048937 6 6 , error= 2024-09-11 18:02:18.312 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,577c14308306177f,3c72f04590688741] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=577c14308306177f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:02:18.334 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,577c14308306177f,3c72f04590688741] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=577c14308306177f, duration=25 ms] 2024-09-11 18:02:19.092 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a960f04da476edd9,0cc7de9ca9065f3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a960f04da476edd9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:19.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a960f04da476edd9,0cc7de9ca9065f3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a960f04da476edd9, duration=18 ms] 2024-09-11 18:02:19.372 INFO 2981023 --- [ocp-async-1,d3a23345fbf963fc,c17f5841c725a8f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,52527a92f3398f45,d1fd0f80e32fe6c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,52527a92f3398f45,d1fd0f80e32fe6c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:20.805 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:20.807 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:21.092 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,5988f12f39a33f99,1beb85114a50da3a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5988f12f39a33f99, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:21.098 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,5988f12f39a33f99,1beb85114a50da3a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5988f12f39a33f99, duration=9 ms] 2024-09-11 18:02:21.268 INFO 2981023 --- [host-check-clockdiff4,1df499749fc49c68,06b2706766c819a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048941 0 0 , error= 2024-09-11 18:02:21.372 INFO 2981023 --- [ocp-async-4,bd3a512a6c8ebffb,bf501e439099578c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:22.416 INFO 2981023 --- [ocp-schedules-23,11a1c19b2602638b,0eddfd76561527b6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:02:23.094 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,0448081b6e070fd4,3d910c3493e2b3b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0448081b6e070fd4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:23.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,0448081b6e070fd4,3d910c3493e2b3b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0448081b6e070fd4, duration=9 ms] 2024-09-11 18:02:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,9a9056a4ea9e9b7a,1a9c31f438c1c7d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,9a9056a4ea9e9b7a,1a9c31f438c1c7d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:23.373 INFO 2981023 --- [ocp-async-10,f7b2c4456cae8036,5c10c56670e3ec7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:23.807 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:23.808 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:23.902 INFO 2981023 --- [ocp-schedules-12,2623b04735ae7366,d77944b80307394f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:02:23.946 WARN 2981023 --- [scanner-13594450-1,2623b04735ae7366,396e678f77dc0f17] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:02:23.972 INFO 2981023 --- [alarm-detect-3,2623b04735ae7366,5da1559bd26b8143] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645262, data is empty. 2024-09-11 18:02:23.976 INFO 2981023 --- [alarm-detect-3,2623b04735ae7366,5da1559bd26b8143] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645262, data is empty. 2024-09-11 18:02:23.979 INFO 2981023 --- [alarm-detect-3,2623b04735ae7366,5da1559bd26b8143] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645262, data is empty. 2024-09-11 18:02:23.997 INFO 2981023 --- [alarm-detect-16,2623b04735ae7366,e78bd9d3e23ddd30] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645262, data is empty. 2024-09-11 18:02:23.998 INFO 2981023 --- [alarm-detect-12,2623b04735ae7366,d7d8f96db6a85a9e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645262, data is empty. 2024-09-11 18:02:24.001 INFO 2981023 --- [alarm-detect-16,2623b04735ae7366,e78bd9d3e23ddd30] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645262, data is empty. 2024-09-11 18:02:24.004 INFO 2981023 --- [alarm-detect-16,2623b04735ae7366,e78bd9d3e23ddd30] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645262, data is empty. 2024-09-11 18:02:24.004 INFO 2981023 --- [alarm-detect-12,2623b04735ae7366,d7d8f96db6a85a9e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645262, data is empty. 2024-09-11 18:02:24.008 INFO 2981023 --- [alarm-detect-12,2623b04735ae7366,d7d8f96db6a85a9e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645262, data is empty. 2024-09-11 18:02:24.121 INFO 2981023 --- [alarm-detect-20,2623b04735ae7366,a5655a9f7be16206] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645262, data is empty. 2024-09-11 18:02:24.124 INFO 2981023 --- [alarm-detect-20,2623b04735ae7366,a5655a9f7be16206] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645262, data is empty. 2024-09-11 18:02:24.129 INFO 2981023 --- [alarm-detect-20,2623b04735ae7366,a5655a9f7be16206] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645262, data is empty. 2024-09-11 18:02:24.177 INFO 2981023 --- [ocp-schedules-10,d59803b0e29851d3,1e95851be28ab008] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:02:24.177 INFO 2981023 --- [ocp-schedules-10,d59803b0e29851d3,1e95851be28ab008] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:02:24.177 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:02:24.177 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:02:24.177 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:02:24.196 INFO 2981023 --- [ocp-schedules-10,d59803b0e29851d3,1e95851be28ab008] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:24.197 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:02:24.201 INFO 2981023 --- [ocp-schedules-10,d59803b0e29851d3,1e95851be28ab008] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:24.201 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:02:24.202 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:24.203 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:24.272 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:02:24.274 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:02:24.373 INFO 2981023 --- [ocp-schedules-4,6e192870aee99ee5,0bead2a0b080d7e8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:02:24.457 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:02:24.462 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:02:24.466 INFO 2981023 --- [ocp-schedules-10,d59803b0e29851d3,1e95851be28ab008] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:02:24.475 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:02:24.477 INFO 2981023 --- [ocp-schedules-10,d59803b0e29851d3,1e95851be28ab008] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:02:25.095 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,96f1445d3e2931e6,250ad0521a8ceaa8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=96f1445d3e2931e6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:25.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,96f1445d3e2931e6,250ad0521a8ceaa8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=96f1445d3e2931e6, duration=17 ms] 2024-09-11 18:02:25.373 INFO 2981023 --- [ocp-async-5,26350b446d6ceb49,d152a5c6d769d733] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,9439ff25aecf90bb,82093c6643c801c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,9439ff25aecf90bb,82093c6643c801c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:26.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:02:26.354623+08:00, lastProcessTime=2024-09-11T18:02:16.365805+08:00 2024-09-11 18:02:26.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:02:26.354623+08:00 2024-09-11 18:02:26.808 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:26.809 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:27.087 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,ca65ca8997f5bce6,ccb38fe5fe40ec34] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ca65ca8997f5bce6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:27.094 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,ca65ca8997f5bce6,ccb38fe5fe40ec34] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ca65ca8997f5bce6, duration=10 ms] 2024-09-11 18:02:27.373 INFO 2981023 --- [ocp-async-7,b3c1f0652aae9c52,9f7cb11e8b82930d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:27.777 INFO 2981023 --- [ocp-async-3,984ffe044df01f01,65f7064d265e8aa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:02:29.089 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,092f0c1dec8ed08f,c2d1ffe6852c7fb8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=092f0c1dec8ed08f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:29.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,092f0c1dec8ed08f,c2d1ffe6852c7fb8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=092f0c1dec8ed08f, duration=11 ms] 2024-09-11 18:02:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,c047cd2a1b9076e2,ff52c173d3c8e6c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,c047cd2a1b9076e2,ff52c173d3c8e6c0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:29.373 INFO 2981023 --- [ocp-async-6,b9f5f1cb65374042,44ff6e4a0ed1fb4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:29.809 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:29.809 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:31.090 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8d2cb73de9556b66,ad35726a37c6e63b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8d2cb73de9556b66, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:31.125 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8d2cb73de9556b66,ad35726a37c6e63b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8d2cb73de9556b66, duration=38 ms] 2024-09-11 18:02:31.373 INFO 2981023 --- [ocp-async-5,6e56e39b1fe6ba86,e6ea3e8d9125c2b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:32.200 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:02:32.206 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:02:32.221 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/exists, request body:CheckProcessExistsRequest(name=observer), params:null 2024-09-11 18:02:32.233 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] .t.b.s.StartObServerProcessWithParamTask : start observer process finished 2024-09-11 18:02:32.246 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,79dbcab680d83777] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048215, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:02:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,7dc44b6a29db82dc,a3ae0715b9675159] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,7dc44b6a29db82dc,a3ae0715b9675159] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:32.419 INFO 2981023 --- [ocp-schedules-3,36060cc1fc0b5a70,d30de1e9483c5af6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:02:32.809 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:32.810 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:32.991 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048213] 2024-09-11 18:02:33.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9813393972ab9031,f39717bed8d39d01] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9813393972ab9031, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:33.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9813393972ab9031,f39717bed8d39d01] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9813393972ab9031, duration=9 ms] 2024-09-11 18:02:33.171 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,2828a76445ebd6c2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048213, operation:EXECUTE, state: RUNNING 2024-09-11 18:02:33.254 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,0f623453f8c2c127] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048213, context=Context{parallelIdx=-1, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Wait node, zone2.idc=default_idc, sub_task_instance_id=4048213, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:02:33.169+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:02:33.264 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,0f623453f8c2c127] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048213, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:02:33.373 INFO 2981023 --- [ocp-async-2,23ec719cbaae6249,4f6e57d639c1bdbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:34.034 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set running subtask to ready, subtaskIds=[4048209] 2024-09-11 18:02:34.168 INFO 2981023 --- [ocp-schedules-29,b5037d2455ef7762,96815816cb979a35] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:02:34.208 WARN 2981023 --- [scanner-13594745-1,b5037d2455ef7762,1b32ee8d288ce093] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:02:34.277 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,6dc9e7372d6cf2ce] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048209, operation:EXECUTE, state: RUNNING 2024-09-11 18:02:34.423 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048209, context=Context{parallelIdx=0, stringMap={ob_log_disk_path=/oceanbase/redo, task_instance_id=4048203, task_operation=execute, ob_run_path=/data/rtdob/oceanbase, zone_name=zone2, service_version=4.2.4.0, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, ob_disk_path_style=OBD, ob_svr_port=2882, sub_task_instance_name=Add observer, zone2.idc=default_idc, sub_task_instance_id=4048209, zone2.region=sys_region, cluster_name=linkrtd, target_server_status=RUNNING, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, subtask_splitter=host_ids, service_name=linkrtd:1719994443, target_zone_status=RUNNING, ob_cluster_id=1719994443, service_type=OB_CLUSTER, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, target_cluster_status=RUNNING, latest_execution_start_time=2024-09-11T18:02:34.274+08:00, ob_data_disk_path=/data/oceanbase, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, executor=10.31.35.10 2024-09-11 18:02:34.475 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.o.s.t.b.server.ObServerTaskHandler : begin to add server, zoneName=zone2, svrIp=10.31.35.7, svrPort=2882 2024-09-11 18:02:34.480 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:34.486 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.7, 2882] 2024-09-11 18:02:34.493 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:34.497 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [600000000] 2024-09-11 18:02:34.502 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter system add server ? zone ?, args: [10.31.35.7:2882, zone2] 2024-09-11 18:02:34.927 INFO 2981023 --- [ocp-schedules-20,746be8c71fbd2314,fbc538536b5e29bd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:02:34.941 INFO 2981023 --- [ocp-schedules-20,746be8c71fbd2314,fbc538536b5e29bd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:02:34.941 INFO 2981023 --- [ocp-schedules-20,746be8c71fbd2314,fbc538536b5e29bd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:02:35.098 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,eb13eca9b9878f63,645ddde6497587ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=eb13eca9b9878f63, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:35.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,eb13eca9b9878f63,645ddde6497587ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=eb13eca9b9878f63, duration=9 ms] 2024-09-11 18:02:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,db1a91b4957d251b,881172f545c003ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,db1a91b4957d251b,881172f545c003ef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:35.373 INFO 2981023 --- [ocp-async-8,5c06a4725ab4dcd7,91b58d1488695de4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:35.810 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:35.810 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:02:36.337301+08:00, lastProcessTime=2024-09-11T18:02:26.354623+08:00 2024-09-11 18:02:36.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:02:36.337301+08:00 2024-09-11 18:02:37.094 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,986b65f55aada2c9,6488976cec435f4d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=986b65f55aada2c9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:37.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,986b65f55aada2c9,6488976cec435f4d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=986b65f55aada2c9, duration=9 ms] 2024-09-11 18:02:37.314 INFO 2981023 --- [prometheus-cache-monitor1,7104120905ce0d5d,a9b1833965e3a949] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5069,"secondWriteCacheSize":2048,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":9626} 2024-09-11 18:02:37.315 INFO 2981023 --- [prometheus-cache-monitor1,7104120905ce0d5d,a9b1833965e3a949] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:02:37.373 INFO 2981023 --- [ocp-async-3,de4c44f69e7191e2,f499b7596d2d7864] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:37.777 INFO 2981023 --- [ocp-async-10,29c8b59775579c28,9cc79f4c52fcba08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:02:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,700a659e876f35da,f50cf5d13c2fc50c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,700a659e876f35da,f50cf5d13c2fc50c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:38.810 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:38.811 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:39.086 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,22199ec9680dafd7,8b3cdc00b9a7ab3f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=22199ec9680dafd7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:39.092 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,22199ec9680dafd7,8b3cdc00b9a7ab3f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=22199ec9680dafd7, duration=9 ms] 2024-09-11 18:02:39.373 INFO 2981023 --- [ocp-async-2,1bfe9e609f3947be,a51f898cf509b21e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:40.003 INFO 2981023 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:02:40.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:02:40.321 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6f511f2e0f0d21f7,3c6e64b888935a62] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=6f511f2e0f0d21f7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:02:40.342 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6f511f2e0f0d21f7,3c6e64b888935a62] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=6f511f2e0f0d21f7, duration=24 ms] 2024-09-11 18:02:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,3eae596aae639cfb,232698c9f2d64a37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,3eae596aae639cfb,232698c9f2d64a37] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:41.374 INFO 2981023 --- [ocp-async-10,39a91bc70f7918e6,59f90b554b34bdc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:41.744 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d1024f1f4177f1f1,9f2454fd9616e03b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048215, client=10.31.33.41, traceId=d1024f1f4177f1f1, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048215,] 2024-09-11 18:02:41.751 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d1024f1f4177f1f1,9f2454fd9616e03b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048215/logSegments client=10.31.33.41, traceId=d1024f1f4177f1f1, duration=10 ms] 2024-09-11 18:02:41.781 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,8b198a1e905c3a38,05c88b1dbb4ca2bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8b198a1e905c3a38, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:41.788 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,8b198a1e905c3a38,05c88b1dbb4ca2bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8b198a1e905c3a38, duration=10 ms] 2024-09-11 18:02:41.811 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:41.812 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:42.422 INFO 2981023 --- [ocp-schedules-7,17998f98069ed3c7,8123aea76880cacc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:02:43.374 INFO 2981023 --- [ocp-async-3,e4b1058b4ecf8c48,88a4b87b91c7a055] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:43.834 INFO 2981023 --- [ocp-schedules-1,ceb84bb7fa438c68,19a0f97f8521c322] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:02:43.852 WARN 2981023 --- [ocp-schedules-1,ceb84bb7fa438c68,19a0f97f8521c322] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:02:44.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,0ed8d827b05df6c8,ed41c05b15cfd9e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0ed8d827b05df6c8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:44.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,0ed8d827b05df6c8,ed41c05b15cfd9e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0ed8d827b05df6c8, duration=9 ms] 2024-09-11 18:02:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,c7c16395982b7697,a6e80b47844341ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,c7c16395982b7697,a6e80b47844341ea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:44.380 INFO 2981023 --- [ocp-schedules-20,4a9488ba5e925982,f90744e3065632d1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:02:44.437 WARN 2981023 --- [scanner-13594943-1,4a9488ba5e925982,d34b1d5782a8ce0b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:02:44.461 INFO 2981023 --- [alarm-detect-1,4a9488ba5e925982,c9f00a7e5296d720] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645262, data is empty. 2024-09-11 18:02:44.467 INFO 2981023 --- [alarm-detect-1,4a9488ba5e925982,c9f00a7e5296d720] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645262, data is empty. 2024-09-11 18:02:44.471 INFO 2981023 --- [alarm-detect-1,4a9488ba5e925982,c9f00a7e5296d720] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645262, data is empty. 2024-09-11 18:02:44.485 INFO 2981023 --- [alarm-detect-6,4a9488ba5e925982,04d45d8af9e39063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645262, data is empty. 2024-09-11 18:02:44.485 INFO 2981023 --- [alarm-detect-20,4a9488ba5e925982,43a2fc13142f030e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645262, data is empty. 2024-09-11 18:02:44.491 INFO 2981023 --- [alarm-detect-6,4a9488ba5e925982,04d45d8af9e39063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645262, data is empty. 2024-09-11 18:02:44.492 INFO 2981023 --- [alarm-detect-20,4a9488ba5e925982,43a2fc13142f030e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645262, data is empty. 2024-09-11 18:02:44.496 INFO 2981023 --- [alarm-detect-20,4a9488ba5e925982,43a2fc13142f030e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645262, data is empty. 2024-09-11 18:02:44.496 INFO 2981023 --- [alarm-detect-6,4a9488ba5e925982,04d45d8af9e39063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645262, data is empty. 2024-09-11 18:02:44.624 INFO 2981023 --- [alarm-detect-20,4a9488ba5e925982,bd09342c3435fd6f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645262, data is empty. 2024-09-11 18:02:44.628 INFO 2981023 --- [alarm-detect-20,4a9488ba5e925982,bd09342c3435fd6f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645262, data is empty. 2024-09-11 18:02:44.635 INFO 2981023 --- [alarm-detect-20,4a9488ba5e925982,bd09342c3435fd6f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645262, data is empty. 2024-09-11 18:02:44.812 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:44.812 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:45.101 INFO 2981023 --- [prometheus-second-schedule-1,0b78b2bd6e45228c,cf1db5fa2f611b49] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:02:45.374 INFO 2981023 --- [ocp-async-7,70a2e64ac51b3e02,66f6cd229e338f47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:45.405 INFO 2981023 --- [ocp-schedules-1,d3db1d704c41678d,abc076ac3651c2fe] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-09-11 18:02:45.406 INFO 2981023 --- [ocp-schedules-1,d3db1d704c41678d,abc076ac3651c2fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:45.491 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:45.492 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:45.494 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:45.495 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 02:03:06.281634, 2024-09-11 18:02:45.439, 1000] 2024-09-11 18:02:45.495 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 10:40:18.658505, 2024-09-11 18:02:45.439, 1000] 2024-09-11 18:02:45.496 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-09-11 02:03:06.281634, endTime 2024-09-11 18:02:45.439 2024-09-11 18:02:45.498 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2024-09-11 10:40:18.658505, endTime 2024-09-11 18:02:45.439 2024-09-11 18:02:45.499 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:21:42.083265, 2024-09-11 18:02:45.44, 1000] 2024-09-11 18:02:45.535 INFO 2981023 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:45.535 INFO 2981023 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:45.538 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:45.540 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:45.540 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 6 system events to database for cluster 1000006 2024-09-11 18:02:45.540 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 6 system events for cluster 1000006, startTime 2024-09-11 17:21:42.083265, endTime 2024-09-11 18:02:45.44, limit 1000, offset 0 2024-09-11 18:02:45.547 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:45.548 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:45.551 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 17:21:42.083265, 2024-09-11 18:02:45.44, 1000, 1000] 2024-09-11 18:02:45.552 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000006, startTime 2024-09-11 17:21:42.083265, endTime 2024-09-11 18:02:45.44 2024-09-11 18:02:45.561 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:45.562 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:45.576 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 08:00:10.321716, 2024-09-11 18:02:45.526, 1000] 2024-09-11 18:02:45.577 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-09-11 08:00:10.321716, endTime 2024-09-11 18:02:45.526 2024-09-11 18:02:45.589 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 17:21:42.083371, 2024-09-11 18:02:45.554, 1000] 2024-09-11 18:02:45.589 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 16:48:30.801707, 2024-09-11 18:02:45.525, 1000] 2024-09-11 18:02:45.590 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1 server events for cluster 2, startTime 2024-09-11 16:48:30.801707, endTime 2024-09-11 18:02:45.525, limit 1000, offset 0 2024-09-11 18:02:45.600 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 114 server events for cluster 1000006, startTime 2024-09-11 17:21:42.083371, endTime 2024-09-11 18:02:45.554, limit 1000, offset 0 2024-09-11 18:02:45.601 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1 server events to database for cluster 2 2024-09-11 18:02:45.608 INFO 2981023 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:45.608 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:45.631 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 16:48:30.801707, 2024-09-11 18:02:45.525, 1000, 1000] 2024-09-11 18:02:45.631 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2024-09-11 16:48:30.801707, endTime 2024-09-11 18:02:45.525 2024-09-11 18:02:45.656 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 114 server events to database for cluster 1000006 2024-09-11 18:02:45.663 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:45.664 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:45.682 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 17:21:42.083371, 2024-09-11 18:02:45.554, 1000, 1000] 2024-09-11 18:02:45.683 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000006, startTime 2024-09-11 17:21:42.083371, endTime 2024-09-11 18:02:45.554 2024-09-11 18:02:45.683 INFO 2981023 --- [ocp-schedules-1,d3db1d704c41678d,abc076ac3651c2fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, tryCount=3, successCount=3 2024-09-11 18:02:46.017 INFO 2981023 --- [ocp-schedules-6,24d52acaeacaf7de,c6d49312cf61d879] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:02:46.021 INFO 2981023 --- [host-check-clockdiff1,24d52acaeacaf7de,66fc72eb9d112574] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:02:46.021 INFO 2981023 --- [host-check-clockdiff2,24d52acaeacaf7de,783b60448f6f5ace] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:02:46.022 INFO 2981023 --- [host-check-clockdiff3,24d52acaeacaf7de,f1f37173c4054cee] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:02:46.022 INFO 2981023 --- [host-check-clockdiff4,24d52acaeacaf7de,f81b38f62f528861] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:02:46.025 INFO 2981023 --- [ocp-schedules-18,3fc46675e15fd840,67b4635d734b5e3c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:02:46.028 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:02:46.028 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:02:46.028 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:02:46.028 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:02:46.031 INFO 2981023 --- [ocp-schedules-23,79ab9087758ba760,f89d62c8bd0e4749] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:02:46.051 INFO 2981023 --- [host-check-clockdiff2,24d52acaeacaf7de,783b60448f6f5ace] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048966 5 5 , error= 2024-09-11 18:02:46.051 INFO 2981023 --- [host-check-agent-status1,79ab9087758ba760,7a3346a994fe924b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:02:46.052 INFO 2981023 --- [host-check-agent-status2,79ab9087758ba760,0d9d51e234bb354e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:02:46.052 INFO 2981023 --- [host-check-agent-status3,79ab9087758ba760,e67ada063f820634] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:02:46.052 INFO 2981023 --- [host-check-agent-status4,79ab9087758ba760,3ac14f7bbb395267] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:02:46.056 INFO 2981023 --- [host-check-clockdiff3,24d52acaeacaf7de,f1f37173c4054cee] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048966 0 0 , error= 2024-09-11 18:02:46.058 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:02:46.058 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:02:46.058 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:46.059 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:46.063 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:02:46.064 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:02:46.064 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:46.064 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:46.065 INFO 2981023 --- [host-check-agent-status3,79ab9087758ba760,e67ada063f820634] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:46.066 INFO 2981023 --- [host-check-agent-status3,79ab9087758ba760,e67ada063f820634] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:46.069 INFO 2981023 --- [host-check-agent-status4,79ab9087758ba760,3ac14f7bbb395267] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:02:46.071 INFO 2981023 --- [host-check-agent-status2,79ab9087758ba760,0d9d51e234bb354e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:02:46.075 INFO 2981023 --- [host-check-agent-status4,79ab9087758ba760,3ac14f7bbb395267] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:02:46.098 INFO 2981023 --- [host-check-agent-status1,79ab9087758ba760,7a3346a994fe924b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:46.099 INFO 2981023 --- [host-check-agent-status2,79ab9087758ba760,0d9d51e234bb354e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:02:46.161 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:02:46.186 INFO 2981023 --- [ocp-schedules-11,fe7eee95616e480c,00f6d2a71e4756d0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:02:46.196 INFO 2981023 --- [ocp-schedules-2,098647f0727fce25,d10b58218180ba6b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:02:46.196 INFO 2981023 --- [host-check-agent-status1,79ab9087758ba760,7a3346a994fe924b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:46.196 INFO 2981023 --- [ocp-schedules-19,df607512b808e974,d156eeea11ca89e6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:02:46.196 INFO 2981023 --- [ocp-schedules-4,e7e302b31b3038b3,6b9d182a38366717] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:02:46.196 INFO 2981023 --- [ocp-schedules-15,c553ffd0ea950c47,ef8b828f4679b7c4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:02:46.205 INFO 2981023 --- [ocp-schedules-2,098647f0727fce25,d10b58218180ba6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:46.216 INFO 2981023 --- [ocp-schedules-4,e7e302b31b3038b3,6b9d182a38366717] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:46.218 INFO 2981023 --- [ocp-schedules-19,df607512b808e974,d156eeea11ca89e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:46.226 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2f87df898d4fc4a9,0dca504294a5cf4d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2f87df898d4fc4a9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:46.234 INFO 2981023 --- [ocp-schedules-19,df607512b808e974,d156eeea11ca89e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:02:46.234 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048922674000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:02:46.260 INFO 2981023 --- [ocp-schedules-11,fe7eee95616e480c,00f6d2a71e4756d0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:02:46.288 INFO 2981023 --- [host-check-agent-status3,79ab9087758ba760,e67ada063f820634] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:02:46.291 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:02:46.294 INFO 2981023 --- [host-check-agent-status3,79ab9087758ba760,e67ada063f820634] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:02:46.294 INFO 2981023 --- [host-check-agent-status3,79ab9087758ba760,e67ada063f820634] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:02:46.294 INFO 2981023 --- [host-check-agent-status3,79ab9087758ba760,e67ada063f820634] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:02:46.303 INFO 2981023 --- [ocp-schedules-24,319db092c10c00f0,81f2312c71159f80] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:02:46.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2f87df898d4fc4a9,0dca504294a5cf4d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2f87df898d4fc4a9, duration=192 ms] 2024-09-11 18:02:46.339 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.341 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.341 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:46.343 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:02:46.343 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:02:46.345 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:02:46.350887384+08:00 2024-09-11 18:02:46.346 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:02:46.346+08:00 2024-09-11 18:02:46.347 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:02:46.348 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:02:46.346+08:00 and status AVAILABLE 2024-09-11 18:02:46.349 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.350 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:46.353 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.354 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:02:46.354 INFO 2981023 --- [host-check-agent-status2,79ab9087758ba760,0d9d51e234bb354e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:02:46.355 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:02:46.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007105] will be resolved. 2024-09-11 18:02:46.356 INFO 2981023 --- [ocp-schedules-2,098647f0727fce25,d10b58218180ba6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:02:46.360 INFO 2981023 --- [host-check-agent-status4,79ab9087758ba760,3ac14f7bbb395267] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:02:46.362 INFO 2981023 --- [host-check-agent-status2,79ab9087758ba760,0d9d51e234bb354e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:02:46.362 INFO 2981023 --- [host-check-agent-status2,79ab9087758ba760,0d9d51e234bb354e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:02:46.362 INFO 2981023 --- [host-check-agent-status2,79ab9087758ba760,0d9d51e234bb354e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:02:46.362 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:02:46.363 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:02:46.362790263+08:00 2024-09-11 18:02:46.364 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:02:46.364 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.364 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:46.365 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:02:46.365 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:02:46.363+08:00 2024-09-11 18:02:46.367 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:02:46.367031224+08:00 2024-09-11 18:02:46.367 INFO 2981023 --- [host-check-agent-status4,79ab9087758ba760,3ac14f7bbb395267] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:02:46.367 INFO 2981023 --- [host-check-agent-status4,79ab9087758ba760,3ac14f7bbb395267] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:02:46.367 INFO 2981023 --- [host-check-agent-status4,79ab9087758ba760,3ac14f7bbb395267] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:02:46.369 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:02:46.363+08:00 and status AVAILABLE 2024-09-11 18:02:46.370 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:02:46.371 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:02:46.367+08:00 2024-09-11 18:02:46.372 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:02:46.372 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:02:46.377564241+08:00 2024-09-11 18:02:46.374 INFO 2981023 --- [host-check-agent-status1,79ab9087758ba760,7a3346a994fe924b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:02:46.376 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:02:46.367+08:00 and status AVAILABLE 2024-09-11 18:02:46.376 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:02:46.372+08:00 2024-09-11 18:02:46.377 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : resolveByTimeout, timeoutCount=1 2024-09-11 18:02:46.377 INFO 2981023 --- [host-check-agent-status1,79ab9087758ba760,7a3346a994fe924b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:02:46.377 INFO 2981023 --- [host-check-agent-status1,79ab9087758ba760,7a3346a994fe924b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:02:46.377 INFO 2981023 --- [host-check-agent-status1,79ab9087758ba760,7a3346a994fe924b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:02:46.378 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:02:46.377627+08:00, lastProcessTime=2024-09-11T18:02:36.337301+08:00 2024-09-11 18:02:46.379 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:02:46.372+08:00 and status AVAILABLE 2024-09-11 18:02:46.382 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.383 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:02:46.383 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:02:46.384 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:02:46.384 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.402 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:02:46.402 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:02:46.403 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:02:46.404 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:02:46.406 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:46.406 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:02:46.407 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:46.420 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:02:46.421 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.422 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:46.422 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:02:46.423 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:46.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:02:46.453 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:02:46.453 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048922674000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:02:46.460 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:02:46.461 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:02:46.464 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:02:46.477 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:02:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.484 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:46.484 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:02:46.485 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.486 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:02:46.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:02:46.488 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:02:46.489 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.489 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.490 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:02:46.490 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:02:46.491 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.491 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:02:46.506 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:02:46.509 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:02:46.517 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:02:46.519 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:02:46.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:02:46.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:02:46.528 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:02:46.529 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:02:46.530 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:02:46.563 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:02:46.572 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.573 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:02:46.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:02:46.578 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.579 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.579 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:02:46.583 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.583 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.585 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:02:46.586 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:02:46.586 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.586 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:02:46.586 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:02:46.587 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:02:46.587 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:02:46.587 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.589 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.590 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:02:46.590 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:02:46.593 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:02:46.594 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:02:46.600 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:02:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:02:46.606 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:02:46.607 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:02:46.610 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:02:46.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:02:46.614 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:02:46.614 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.614 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:02:46.616 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:02:46.617 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:02:46.617 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:02:46.624 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:02:46.627 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:02:46.628 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:02:46.630 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:02:46.631 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:02:46.636 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:02:46.656 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:02:46.663 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:02:46.668 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.669 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.669 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:02:46.670 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:02:46.670 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.677 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:02:46.683 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:02:46.686 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:02:46.691 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:02:46.692 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:02:46.692 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.692 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.693 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:02:46.693 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:02:46.697 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:02:46.701 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:02:46.702 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:02:46.703 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:02:46.704 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:02:46.707 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:02:46.712 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:02:46.716 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:02:46.723 INFO 2981023 --- [host-check-host-status2,3fc46675e15fd840,fd4478cc0b47cb70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:02:46.724 INFO 2981023 --- [host-check-host-status4,3fc46675e15fd840,fd22002ce72293d2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:02:46.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:02:46.730 INFO 2981023 --- [host-check-host-status1,3fc46675e15fd840,e8d937258e58ea71] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:02:46.736 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.738 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.739 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:02:46.739 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:02:46.740 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:02:46.740 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.753 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:02:46.754 INFO 2981023 --- [host-check-host-status3,3fc46675e15fd840,b7593d0b5d92d8d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:02:46.757 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:02:46.765 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.765 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.766 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:02:46.766 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:46.767 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:46.768 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:02:46.774 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:02:46.774 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:02:46.777 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:02:46.777 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:02:46.781 INFO 2981023 --- [ocp-schedules-4,e7e302b31b3038b3,6b9d182a38366717] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:02:46.789 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:02:46.792 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:02:46.794 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:02:46.377627+08:00 2024-09-11 18:02:46.807 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:02:46.813 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:02:46.816 INFO 2981023 --- [ocp-schedules-17,2bf554e36509e4eb,c299f369f9febd13] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:02:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,7760f1e895a87339,197afc636654bc14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,7760f1e895a87339,197afc636654bc14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:47.374 INFO 2981023 --- [ocp-async-10,1048e70d22b25ffe,c5bdce18d2c6f90c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:47.471 INFO 2981023 --- [ocp-schedules-11,31c1ffd1a044437f,6b334ec5c7f9bd65] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 2 2024-09-11 18:02:47.474 INFO 2981023 --- [ocp-schedules-11,31c1ffd1a044437f,6b334ec5c7f9bd65] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-09-11 18:02:47.474 INFO 2981023 --- [ocp-schedules-11,31c1ffd1a044437f,6b334ec5c7f9bd65] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 2 2024-09-11 18:02:47.777 INFO 2981023 --- [ocp-async-7,323e41daa7b4cc95,fc4912591e412e3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:02:47.813 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:47.813 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:48.099 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,bc5f237f1281b0e5,3171b72c5e17a6d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bc5f237f1281b0e5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:48.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,bc5f237f1281b0e5,3171b72c5e17a6d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bc5f237f1281b0e5, duration=9 ms] 2024-09-11 18:02:49.374 INFO 2981023 --- [ocp-async-5,4560fe766e590e1d,517f3a7d5c5c7005] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:02:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:02:50.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d81f54bd97bdf0f9,be9decb94f37e1ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d81f54bd97bdf0f9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:50.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d81f54bd97bdf0f9,be9decb94f37e1ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d81f54bd97bdf0f9, duration=17 ms] 2024-09-11 18:02:50.292 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:02:50.296 INFO 2981023 --- [ocp-schedules-27,e26ebf6aa59ef418,a2f65019df0a0d50] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.297 INFO 2981023 --- [ocp-schedules-15,5c2037d8de7b9798,3456b370b969106c] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:02:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,123b87326de4d316,acdc5d760828e743] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,123b87326de4d316,acdc5d760828e743] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:50.813 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:50.815 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:50.869 INFO 2981023 --- [host-check-clockdiff1,24d52acaeacaf7de,66fc72eb9d112574] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048970 5 5 , error= 2024-09-11 18:02:51.374 INFO 2981023 --- [ocp-async-1,d12b96a3b2bb48c2,34c25a4d2735e1fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:51.593 INFO 2981023 --- [ocp-schedules-30,efa2f4d0f60d5ab4,ca8fa73891844a87] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:02:51.594 INFO 2981023 --- [ocp-schedules-30,efa2f4d0f60d5ab4,ca8fa73891844a87] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:51.594 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:02:51.594 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:02:51.595 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:02:51.595 INFO 2981023 --- [ocp-schedules-30,efa2f4d0f60d5ab4,ca8fa73891844a87] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:02:52.005 INFO 2981023 --- [host-check-clockdiff4,24d52acaeacaf7de,f81b38f62f528861] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048972 0 0 , error= 2024-09-11 18:02:52.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,0af3654628dadf91,2bb7050c3bbb0dc9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0af3654628dadf91, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:52.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,0af3654628dadf91,2bb7050c3bbb0dc9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0af3654628dadf91, duration=11 ms] 2024-09-11 18:02:52.424 INFO 2981023 --- [ocp-schedules-14,01efcb7d57f3b161,f242bef64b992698] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:02:53.238 INFO 2981023 --- [ocp-schedules-2,dcfb17e510bd7dae,e3974e96a9fcd93c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:02:53.238 INFO 2981023 --- [ocp-schedules-2,dcfb17e510bd7dae,e3974e96a9fcd93c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:02:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,03ac0a9b0d531e74,453a3e8ecf3f2ffd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,03ac0a9b0d531e74,453a3e8ecf3f2ffd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:53.374 INFO 2981023 --- [ocp-async-8,f7ebd500faf252ac,e1ad0b001e05d116] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:53.815 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:53.817 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:54.098 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,7f15341999a63de9,1e76c38ad9b868b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7f15341999a63de9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:54.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,7f15341999a63de9,1e76c38ad9b868b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7f15341999a63de9, duration=21 ms] 2024-09-11 18:02:54.158 INFO 2981023 --- [ocp-schedules-18,11ba4f3b0b818b24,fd5b3abc8e2defa8] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:02:54.177 WARN 2981023 --- [ocp-schedules-18,11ba4f3b0b818b24,fd5b3abc8e2defa8] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:02:54.376 INFO 2981023 --- [ocp-schedules-12,6e4f64b4ca109136,85ffb5dacca242ab] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:02:54.666 INFO 2981023 --- [ocp-schedules-22,e46242d5d561a3f3,8de6889e07dbe1c1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:02:54.713 WARN 2981023 --- [scanner-13595272-1,e46242d5d561a3f3,8a9e786385797435] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:02:54.944 INFO 2981023 --- [ocp-schedules-26,3c24a8e6ffe12787,041dab963996a0a9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:02:54.953 INFO 2981023 --- [ocp-schedules-26,3c24a8e6ffe12787,041dab963996a0a9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:02:54.953 INFO 2981023 --- [ocp-schedules-26,3c24a8e6ffe12787,041dab963996a0a9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:02:55.374 INFO 2981023 --- [ocp-async-3,de9ca6df2f18ed0a,f309753b7b0f6134] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:55.827 INFO 2981023 --- [ocp-schedules-18,b832fc21b5f610ca,92bc5a5c74516a56] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:02:55.827 INFO 2981023 --- [ocp-schedules-18,b832fc21b5f610ca,92bc5a5c74516a56] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690420679617835008 2024-09-11 18:02:55.828 INFO 2981023 --- [ocp-schedules-18,b832fc21b5f610ca,92bc5a5c74516a56] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:02:55.828 INFO 2981023 --- [ocp-schedules-18,b832fc21b5f610ca,92bc5a5c74516a56] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:02:55.828 INFO 2981023 --- [ocp-schedules-18,b832fc21b5f610ca,92bc5a5c74516a56] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:02:55.828 INFO 2981023 --- [ocp-schedules-18,b832fc21b5f610ca,92bc5a5c74516a56] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:02:55.976 INFO 2981023 --- [ocp-schedules-16,4aa324d399c56f10,8b7c353fd66e52e4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:02:55.983 INFO 2981023 --- [ocp-schedules-16,4aa324d399c56f10,8b7c353fd66e52e4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:02:55.983 INFO 2981023 --- [ocp-schedules-16,4aa324d399c56f10,8b7c353fd66e52e4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:02:56.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71fd07e2e19c1eaf,b9efa5e4df665e48] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=71fd07e2e19c1eaf, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:56.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71fd07e2e19c1eaf,b9efa5e4df665e48] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=71fd07e2e19c1eaf, duration=10 ms] 2024-09-11 18:02:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,0d9c6b9ce8f321c9,3ed0607dfe33de01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,0d9c6b9ce8f321c9,3ed0607dfe33de01] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:56.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:02:56.354257+08:00, lastProcessTime=2024-09-11T18:02:46.377627+08:00 2024-09-11 18:02:56.496 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:02:56.354257+08:00 2024-09-11 18:02:56.817 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:56.818 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:02:56.883 INFO 2981023 --- [ocp-schedules-10,43e280244cf46f06,59a3ee2900475c09] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:02:56.883+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:02:56.886 INFO 2981023 --- [ocp-schedules-10,43e280244cf46f06,59a3ee2900475c09] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:02:56.989 INFO 2981023 --- [ocp-schedules-10,43e280244cf46f06,59a3ee2900475c09] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 21 to SqlAggregateData 2024-09-11 18:02:56.991 INFO 2981023 --- [ocp-schedules-10,43e280244cf46f06,59a3ee2900475c09] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:02:57.374 INFO 2981023 --- [ocp-async-4,be241607bc4006a1,f06130e2f52c7c6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:57.381 INFO 2981023 --- [ocp-schedules-28,1a0a18db2ff3c7d0,a5ab71f5c29920aa] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:02:57.381 INFO 2981023 --- [ocp-schedules-28,1a0a18db2ff3c7d0,a5ab71f5c29920aa] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:02:57.397 INFO 2981023 --- [Collect all tenants major compactions1,1a0a18db2ff3c7d0,0ab1dcc2912dbea1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:02:57.397 INFO 2981023 --- [Collect all tenants major compactions6,1a0a18db2ff3c7d0,76d23709151d372a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:02:57.398 INFO 2981023 --- [Collect all tenants major compactions3,1a0a18db2ff3c7d0,86cca54db51d01ea] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:02:57.399 INFO 2981023 --- [Collect all tenants major compactions2,1a0a18db2ff3c7d0,1813a772af0582d8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:02:57.400 INFO 2981023 --- [Collect all tenants major compactions4,1a0a18db2ff3c7d0,1fcefd0720d095db] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:02:57.401 INFO 2981023 --- [Collect all tenants major compactions5,1a0a18db2ff3c7d0,368b03f957fee1f0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:02:57.421 INFO 2981023 --- [Collect all tenants major compactions5,1a0a18db2ff3c7d0,368b03f957fee1f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.421 INFO 2981023 --- [Collect all tenants major compactions4,1a0a18db2ff3c7d0,1fcefd0720d095db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.421 INFO 2981023 --- [Collect all tenants major compactions6,1a0a18db2ff3c7d0,76d23709151d372a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.421 INFO 2981023 --- [Collect all tenants major compactions3,1a0a18db2ff3c7d0,86cca54db51d01ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.422 INFO 2981023 --- [Collect all tenants major compactions5,1a0a18db2ff3c7d0,368b03f957fee1f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:02:57.422 INFO 2981023 --- [Collect all tenants major compactions1,1a0a18db2ff3c7d0,0ab1dcc2912dbea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.422 INFO 2981023 --- [Collect all tenants major compactions4,1a0a18db2ff3c7d0,1fcefd0720d095db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:02:57.422 INFO 2981023 --- [Collect all tenants major compactions2,1a0a18db2ff3c7d0,1813a772af0582d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.422 INFO 2981023 --- [Collect all tenants major compactions6,1a0a18db2ff3c7d0,76d23709151d372a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:02:57.423 INFO 2981023 --- [Collect all tenants major compactions3,1a0a18db2ff3c7d0,86cca54db51d01ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:02:57.423 INFO 2981023 --- [Collect all tenants major compactions1,1a0a18db2ff3c7d0,0ab1dcc2912dbea1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:02:57.424 INFO 2981023 --- [Collect all tenants major compactions2,1a0a18db2ff3c7d0,1813a772af0582d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:02:57.425 INFO 2981023 --- [Collect all tenants major compactions5,1a0a18db2ff3c7d0,368b03f957fee1f0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:02:57.426 INFO 2981023 --- [Collect all tenants major compactions5,1a0a18db2ff3c7d0,368b03f957fee1f0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:02:57.426 INFO 2981023 --- [Collect all tenants major compactions1,1a0a18db2ff3c7d0,0ab1dcc2912dbea1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:02:57.426 INFO 2981023 --- [Collect all tenants major compactions1,1a0a18db2ff3c7d0,0ab1dcc2912dbea1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:02:57.426 INFO 2981023 --- [Collect all tenants major compactions4,1a0a18db2ff3c7d0,1fcefd0720d095db] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:02:57.426 INFO 2981023 --- [Collect all tenants major compactions4,1a0a18db2ff3c7d0,1fcefd0720d095db] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:02:57.426 INFO 2981023 --- [Collect all tenants major compactions6,1a0a18db2ff3c7d0,76d23709151d372a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:02:57.426 INFO 2981023 --- [Collect all tenants major compactions3,1a0a18db2ff3c7d0,86cca54db51d01ea] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:02:57.426 INFO 2981023 --- [Collect all tenants major compactions6,1a0a18db2ff3c7d0,76d23709151d372a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:02:57.426 INFO 2981023 --- [Collect all tenants major compactions3,1a0a18db2ff3c7d0,86cca54db51d01ea] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:02:57.427 INFO 2981023 --- [Collect all tenants major compactions2,1a0a18db2ff3c7d0,1813a772af0582d8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:02:57.427 INFO 2981023 --- [Collect all tenants major compactions2,1a0a18db2ff3c7d0,1813a772af0582d8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:02:57.427 INFO 2981023 --- [ocp-schedules-28,1a0a18db2ff3c7d0,a5ab71f5c29920aa] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:02:57.753 INFO 2981023 --- [ocp-schedules-29,d7098993480fc3c2,4af3a27711d85815] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:02:57.754 INFO 2981023 --- [ocp-schedules-29,d7098993480fc3c2,4af3a27711d85815] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:02:57.777 INFO 2981023 --- [ocp-async-9,17f76616c1e6cd06,f17e383d05be799b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:02:57.786 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:02:57.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.788 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:02:57.788 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.790 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:02:57.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:02:57.794 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:02:57.794 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.795 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.795 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.797 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:02:57.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.800 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.801 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.805 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.805 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.805 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.805 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.812 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.813 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:02:57.814 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.815 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.815 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:02:57.815 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4551D817-0-0] 2024-09-11 18:02:57.815 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:02:57.816 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.816 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:02:57.816 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.817 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.821 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.821 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:02:57.828 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.829 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:02:57.829 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.829 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:02:57.834 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.834 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.836 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.837 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:02:57.840 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.841 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:02:57.841 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:02:57.841 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4551D821-0-0] 2024-09-11 18:02:57.841 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:02:57.848 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.849 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:02:57.850 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.851 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:02:57.851 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.852 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.852 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:02:57.852 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:02:57.852 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.853 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.853 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:02:57.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:02:57.854 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:02:57.854 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.854 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.855 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:02:57.855 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:02:57.855 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.855 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:02:57.855 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.855 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:02:57.856 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:02:57.856 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:02:57.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:02:57.857 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.858 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:02:57.860 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.860 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:02:57.860 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:02:57.861 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:02:57.862 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.862 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:02:57.864 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.865 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:02:57.866 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.867 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:02:57.867 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.868 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:02:57.868 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:02:57.871 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:02:57.874 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:57.876 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.877 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:02:57.880 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:57.882 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.883 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.885 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:02:57.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:02:57.892 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:02:57.894 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.894 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:02:57.897 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:02:57.898 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.898 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.899 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:02:57.899 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:02:57.899 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:02:57.900 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:02:57.902 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:02:57.902 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.903 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:02:57.910 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:02:57.911 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:02:57.912 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:02:57.912 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:02:57.912 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:02:57.912 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:02:57.912 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:02:57.912 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:02:57.918 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:02:57.919 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:02:57.919 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:02:57.919 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:02:57.919 INFO 2981023 --- [ocp-schedules-29,d7098993480fc3c2,4af3a27711d85815] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:02:58.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,423541dc58016457,b57503038f65cccb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=423541dc58016457, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:02:58.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,423541dc58016457,b57503038f65cccb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=423541dc58016457, duration=11 ms] 2024-09-11 18:02:58.789 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.795 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obCluste 2024-09-11 18:02:58.804 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.812 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.815 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.822 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.823 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.824 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.824 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.825 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.826 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.827 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.827 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.828 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.831 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.834 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.837 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.839 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.843 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.846 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.849 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.852 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.853 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.854 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.855 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.855 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:56:58.698, end=2024-09-11T18:01:58.786} 2024-09-11 18:02:58.855 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 18:02:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,95ebc1b27f727c7f,515f9fd572ec0d34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726048980, interval=60 2024-09-11 18:02:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,95ebc1b27f727c7f,515f9fd572ec0d34] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:02:59.375 INFO 2981023 --- [ocp-async-8,03fc4e48999dba0c,3784438bd3abf60e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:02:59.707 INFO 2981023 --- [ocp-schedules-22,37f0b1aff0866d58,6b6ab0f372f7f218] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:02:59.818 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:02:59.820 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:00.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,9f5e678f68d9cb53,ad1e6961f6ef4ce8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9f5e678f68d9cb53, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:00.136 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,9f5e678f68d9cb53,ad1e6961f6ef4ce8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9f5e678f68d9cb53, duration=24 ms] 2024-09-11 18:03:01.376 INFO 2981023 --- [ocp-async-9,a133f3ed4b4fe428,da8bc996d0a05adf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:02.096 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,669ec70f0065a2e0,da22be52501caf61] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=669ec70f0065a2e0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:02.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,669ec70f0065a2e0,da22be52501caf61] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=669ec70f0065a2e0, duration=10 ms] 2024-09-11 18:03:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,2799e27f8fc90617,4be9db4eb18847d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,2799e27f8fc90617,4be9db4eb18847d4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:02.427 INFO 2981023 --- [ocp-schedules-30,93da31b6c3180041,aa205a4db4da1856] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:03:02.820 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:02.821 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:03.375 INFO 2981023 --- [ocp-async-7,5f41cf5bf6b96315,92ad0c800ce29989] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:04.351 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,110c1632f17d9ec4,53da6292153b5867] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=110c1632f17d9ec4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:03:04.359 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0ef050a14dd3d765,1030bc962085b27b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0ef050a14dd3d765, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:04.366 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0ef050a14dd3d765,1030bc962085b27b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0ef050a14dd3d765, duration=10 ms] 2024-09-11 18:03:04.376 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,110c1632f17d9ec4,53da6292153b5867] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=110c1632f17d9ec4, duration=28 ms] 2024-09-11 18:03:04.855 INFO 2981023 --- [ocp-schedules-2,3c018a312bbe2d88,1cf42d4a0b1bad82] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:03:04.908 WARN 2981023 --- [scanner-13595533-1,3c018a312bbe2d88,1845f50fb95d35af] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:03:04.934 INFO 2981023 --- [alarm-detect-2,3c018a312bbe2d88,0ba301e98bb09fe4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645262, data is empty. 2024-09-11 18:03:04.938 INFO 2981023 --- [alarm-detect-2,3c018a312bbe2d88,0ba301e98bb09fe4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645283, data is empty. 2024-09-11 18:03:04.946 INFO 2981023 --- [alarm-detect-2,3c018a312bbe2d88,0ba301e98bb09fe4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645283, data is empty. 2024-09-11 18:03:04.955 INFO 2981023 --- [alarm-detect-13,3c018a312bbe2d88,366a5274ec83951f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645262, data is empty. 2024-09-11 18:03:04.962 INFO 2981023 --- [alarm-detect-10,3c018a312bbe2d88,65b03cdf98b7f365] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645262, data is empty. 2024-09-11 18:03:04.962 INFO 2981023 --- [alarm-detect-13,3c018a312bbe2d88,366a5274ec83951f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645283, data is empty. 2024-09-11 18:03:04.966 INFO 2981023 --- [alarm-detect-10,3c018a312bbe2d88,65b03cdf98b7f365] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645283, data is empty. 2024-09-11 18:03:04.968 INFO 2981023 --- [alarm-detect-13,3c018a312bbe2d88,366a5274ec83951f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645283, data is empty. 2024-09-11 18:03:04.971 INFO 2981023 --- [alarm-detect-10,3c018a312bbe2d88,65b03cdf98b7f365] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645283, data is empty. 2024-09-11 18:03:05.068 INFO 2981023 --- [alarm-detect-17,3c018a312bbe2d88,e1c7884d9c6744ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645262, data is empty. 2024-09-11 18:03:05.073 INFO 2981023 --- [alarm-detect-17,3c018a312bbe2d88,e1c7884d9c6744ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645283, data is empty. 2024-09-11 18:03:05.080 INFO 2981023 --- [alarm-detect-17,3c018a312bbe2d88,e1c7884d9c6744ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645283, data is empty. 2024-09-11 18:03:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,e651d9c2b46a2b85,e9b6708b19462f6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,e651d9c2b46a2b85,e9b6708b19462f6b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:05.375 INFO 2981023 --- [ocp-async-6,85b37ce4bfa5e8d7,58f757bb2ea2d4fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:05.821 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:05.822 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:06.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f5053364009b6ff7,fd60ad0724c3a713] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f5053364009b6ff7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:06.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f5053364009b6ff7,fd60ad0724c3a713] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f5053364009b6ff7, duration=9 ms] 2024-09-11 18:03:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:03:06.351498+08:00, lastProcessTime=2024-09-11T18:02:56.354257+08:00 2024-09-11 18:03:06.497 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:03:06.351498+08:00 2024-09-11 18:03:07.375 INFO 2981023 --- [ocp-async-8,a1cbc03e1b76fc8d,7ff15ef2014b991a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:07.741 INFO 2981023 --- [ocp-schedules-12,c7ab018c50f42e88,350911597c4479e3] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:03:07.777 INFO 2981023 --- [ocp-async-10,6ced9b4b7e831077,9f02d426366baaad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:03:07.869 INFO 2981023 --- [ocp-schedules-23,babebd533f3c9173,f9f89660bb0bc60b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:03:07.869+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:03:07.870 INFO 2981023 --- [ocp-schedules-23,babebd533f3c9173,f9f89660bb0bc60b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:03:07.871 INFO 2981023 --- [ocp-schedules-23,babebd533f3c9173,f9f89660bb0bc60b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:03:07.872 INFO 2981023 --- [ocp-schedules-23,babebd533f3c9173,f9f89660bb0bc60b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:03:07.887 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T17:59:07.505, end=2024-09-11T18:01:07.884} 2024-09-11 18:03:07.890 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719 2024-09-11 18:03:07.890 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:03:07.892 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:59:07.505, end=2024-09-11T18:01:07.884},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:03:07.892 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:03:07.892 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:03:07.901 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:03:07.918 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962587918000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:07.918Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:07.920 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:03:07.920 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:07.925 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:07.925 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:07.927 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:07.927 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:07.932 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:07.932 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:07.933 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:03:07.933 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:07.918, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:07.933 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:03:07.935 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:59:07.505, end=2024-09-11T18:01:07.884},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:03:07.936 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:03:07.936 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:03:07.975 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962587975000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:07.975Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:07.982 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962587982000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:07.982Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:07.983 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962587983000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:07.983Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:07.984 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962587984000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:07.984Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:07.989 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962587989000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:07.989Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:07.993 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962587993000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:07.993Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:07.994 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962587994000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:07.994Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:07.994 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962587994000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:07.994Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:08.002 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:03:08.002 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.005 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:03:08.005 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.008 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:03:08.008 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.013 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.013 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.013 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:03:08.013 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.017 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:03:08.017 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.018 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:03:08.018 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.018 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 65 2024-09-11 18:03:08.018 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.020 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:03:08.020 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.022 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.022 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.022 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.022 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.026 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.026 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.028 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.028 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.028 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.028 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.030 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.030 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.031 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.031 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.035 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.035 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.036 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.036 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.036 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.036 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.038 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.038 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.038 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.038 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.039 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.039 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.039 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.040 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.040 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.040 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.040 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.040 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.041 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.041 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.041 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.041 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.044 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.044 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.045 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.045 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.045 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.049 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:07.984, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.049 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:07.975, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.050 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.050 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:07.982, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.053 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.053 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.053 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.053 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.054 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.054 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.057 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.057 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.059 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:07.983, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.059 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.060 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.060 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.064 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:07.993, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.064 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.068 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:07.994, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.069 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:07.989, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.071 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:07.994, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.084 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962588083000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:08.083Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:08.085 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962588085000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:08.085Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:08.086 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962588086000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:08.086Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:08.091 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962588091000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:08.091Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:08.092 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962588092000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:03:08.092Z, endTime=2024-09-11T10:01:07.884Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:03:08.093 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:03:08.093 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.100 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:03:08.100 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.100 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:03:08.100 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,8af2e1fa8a97e5ac,4a8d9d0291cc53be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8af2e1fa8a97e5ac, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:08.108 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.108 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.110 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.110 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.111 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:03:08.111 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.111 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.111 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.113 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.113 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.113 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.113 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.114 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:03:08.114 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:03:08.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,8af2e1fa8a97e5ac,4a8d9d0291cc53be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8af2e1fa8a97e5ac, duration=18 ms] 2024-09-11 18:03:08.119 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.119 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.122 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.122 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.122 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.123 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.123 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.124 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.124 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.124 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.125 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.125 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.125 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.125 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.127 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.127 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:03:08.127 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:03:08.128 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:08.085, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.129 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:08.086, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.131 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:08.083, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.133 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.133 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.135 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.135 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:03:08.135 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048867884000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:03:08.136 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:03:08.138 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:08.091, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.139 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:03:08.092, end=2024-09-11T18:01:07.884}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:03:08.139 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:03:08.140 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:59:07.505, end=2024-09-11T18:01:07.884},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:03:08.140 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:03:08.140 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:03:08.141 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:03:08.143 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:59:07.505, end=2024-09-11T18:01:07.884},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:03:08.143 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:03:08.143 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:03:08.156 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-17C7C851E992BE05A06EDD724C1F7787 2024-09-11 18:03:08.156 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-E54F1BEF0112539FF1A71D265ECE56D0 2024-09-11 18:03:08.156 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-75288971EEEC9A5222481B1D10AAA785 2024-09-11 18:03:08.156 INFO 2981023 --- [plan_change_detector18,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:03:08.157 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:03:08.157 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:59:07.505, end=2024-09-11T18:01:07.884},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:03:08.157 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:03:08.157 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:03:08.158 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:03:08.159 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:59:07.505, end=2024-09-11T18:01:07.884},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:03:08.159 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:03:08.159 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:03:08.160 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:03:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,cd63825bf46fb8b7,6585e5bf8345e985] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,cd63825bf46fb8b7,6585e5bf8345e985] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:08.822 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:08.824 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:09.382 INFO 2981023 --- [ocp-async-1,be9f72e8af485043,5f51d978e87eead2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:09.643 INFO 2981023 --- [metric-archive-minute-4,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=864 2024-09-11 18:03:09.643 INFO 2981023 --- [metric-archive-minute-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1726045200, nodeCount=1000, validCount=864 2024-09-11 18:03:09.683 INFO 2981023 --- [metric-archive-minute-1,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=893 2024-09-11 18:03:09.683 INFO 2981023 --- [metric-archive-minute-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1726045200, nodeCount=1000, validCount=893 2024-09-11 18:03:09.755 INFO 2981023 --- [metric-archive-minute-1,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=22, affectRows=22 2024-09-11 18:03:09.755 INFO 2981023 --- [metric-archive-minute-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1726045200, nodeCount=22, validCount=22 2024-09-11 18:03:09.798 INFO 2981023 --- [metric-archive-minute-2,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2024-09-11 18:03:09.798 INFO 2981023 --- [metric-archive-minute-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1726045200, nodeCount=1000, validCount=1000 2024-09-11 18:03:09.812 INFO 2981023 --- [metric-archive-minute-3,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=994 2024-09-11 18:03:09.812 INFO 2981023 --- [metric-archive-minute-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1726045200, nodeCount=1000, validCount=994 2024-09-11 18:03:09.885 INFO 2981023 --- [metric-archive-minute-4,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2024-09-11 18:03:09.885 INFO 2981023 --- [metric-archive-minute-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1726045200, nodeCount=1000, validCount=1000 2024-09-11 18:03:10.004 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:03:10.004 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:03:10.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,004cbebd7206c2b1,e48e68e392712df1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=004cbebd7206c2b1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:10.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,004cbebd7206c2b1,e48e68e392712df1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=004cbebd7206c2b1, duration=17 ms] 2024-09-11 18:03:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:03:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:03:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,0d42f782561d4406,cfd5968b07d06aa3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,0d42f782561d4406,cfd5968b07d06aa3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:11.376 INFO 2981023 --- [ocp-async-10,de4801196d948fe8,039ab06215ed4580] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:11.824 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:11.825 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:12.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,04c701909963d068,932fef567a0be9c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=04c701909963d068, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:12.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,04c701909963d068,932fef567a0be9c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=04c701909963d068, duration=10 ms] 2024-09-11 18:03:12.430 INFO 2981023 --- [ocp-schedules-27,e6de228471c6d0b9,571a8453cfcc0ae6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:03:12.488 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048920, nodeCount=480, affectRows=480 2024-09-11 18:03:13.376 INFO 2981023 --- [ocp-async-8,b1fa4320ef21dd39,ee24d1be1314d6bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:13.887 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048920, nodeCount=1000, affectRows=1000 2024-09-11 18:03:13.890 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048920, nodeCount=999, validCount=999 2024-09-11 18:03:13.911 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048920, nodeCount=49, affectRows=49 2024-09-11 18:03:13.912 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048920, nodeCount=47, validCount=47 2024-09-11 18:03:13.914 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048920, nodeCount=1000, affectRows=1000 2024-09-11 18:03:13.916 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048920, nodeCount=1000, validCount=1000 2024-09-11 18:03:14.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,7095a4e79d9fb27b,b4d13a79747c56b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7095a4e79d9fb27b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:14.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,7095a4e79d9fb27b,b4d13a79747c56b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7095a4e79d9fb27b, duration=10 ms] 2024-09-11 18:03:14.308 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048920, nodeCount=480, validCount=480 2024-09-11 18:03:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,af99527fe57840fb,b1f1ba57bd43a3d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,dfc9c6d34ef52764,a86845e026d242ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:03:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,af99527fe57840fb,b1f1ba57bd43a3d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:14.825 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:14.826 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:14.956 INFO 2981023 --- [ocp-schedules-18,5ca523c9c840ba59,57c8d4edfc4a011d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:03:14.964 INFO 2981023 --- [ocp-schedules-18,5ca523c9c840ba59,57c8d4edfc4a011d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:03:14.964 INFO 2981023 --- [ocp-schedules-18,5ca523c9c840ba59,57c8d4edfc4a011d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:03:15.130 INFO 2981023 --- [ocp-schedules-4,4754c5b27636f5c5,d6371ebcba4a004b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:03:15.180 WARN 2981023 --- [scanner-13597357-1,4754c5b27636f5c5,b42cbb50e8226b79] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:03:15.376 INFO 2981023 --- [ocp-async-5,4b58b6ddc784ff7b,3f275aa5657752a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:16.022 INFO 2981023 --- [ocp-schedules-21,713381fdc8eb8a49,1681ff7d4aeff55a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:03:16.022 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:03:16.022 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:03:16.024 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:03:16.036 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:03:16.038 INFO 2981023 --- [ocp-schedules-5,fd9f69d70225b687,ec5b11e2b1d4c32b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:03:16.039 INFO 2981023 --- [host-check-agent-status1,fd9f69d70225b687,a78dd644101a2f17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:03:16.039 INFO 2981023 --- [host-check-agent-status2,fd9f69d70225b687,196051df16da6edb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:03:16.039 INFO 2981023 --- [host-check-agent-status3,fd9f69d70225b687,bdd4375a2e3d13e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:03:16.039 INFO 2981023 --- [host-check-agent-status4,fd9f69d70225b687,5ba08981e15fb80d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:03:16.058 INFO 2981023 --- [ocp-schedules-12,53dc9d10c027386a,a845a2955b5dc31d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:03:16.059 INFO 2981023 --- [host-check-clockdiff1,53dc9d10c027386a,00264219b87aa1b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:03:16.063 INFO 2981023 --- [host-check-clockdiff2,53dc9d10c027386a,ed8ba2276a880309] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:03:16.064 INFO 2981023 --- [host-check-clockdiff3,53dc9d10c027386a,29ceddb104cbbfd9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:03:16.067 INFO 2981023 --- [host-check-agent-status3,fd9f69d70225b687,bdd4375a2e3d13e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:03:16.067 INFO 2981023 --- [host-check-agent-status1,fd9f69d70225b687,a78dd644101a2f17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:03:16.072 INFO 2981023 --- [host-check-agent-status3,fd9f69d70225b687,bdd4375a2e3d13e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:03:16.072 INFO 2981023 --- [host-check-agent-status1,fd9f69d70225b687,a78dd644101a2f17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:03:16.079 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:03:16.080 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:03:16.081 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:03:16.087 INFO 2981023 --- [host-check-agent-status2,fd9f69d70225b687,196051df16da6edb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:03:16.088 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:03:16.103 INFO 2981023 --- [host-check-clockdiff4,53dc9d10c027386a,1b7b7e33d617e8c1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:03:16.146 INFO 2981023 --- [host-check-clockdiff3,53dc9d10c027386a,29ceddb104cbbfd9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048996 0 0 , error= 2024-09-11 18:03:16.147 INFO 2981023 --- [host-check-clockdiff2,53dc9d10c027386a,ed8ba2276a880309] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726048996 6 6 , error= 2024-09-11 18:03:16.170 INFO 2981023 --- [host-check-agent-status2,fd9f69d70225b687,196051df16da6edb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:03:16.177 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:03:16.190 INFO 2981023 --- [ocp-schedules-30,1d5f30239da05aaf,d05f30b6c8a0dec9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:03:16.195 INFO 2981023 --- [host-check-agent-status4,fd9f69d70225b687,5ba08981e15fb80d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:03:16.195 INFO 2981023 --- [ocp-schedules-1,eb92f5755ef06905,4260d0b202e5909c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:03:16.197 INFO 2981023 --- [ocp-schedules-24,022de146c1198ffe,d352a5ecc03ce465] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:03:16.197 INFO 2981023 --- [ocp-schedules-26,5e233e03f571abf6,344dbdda49e55919] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:03:16.264 INFO 2981023 --- [ocp-schedules-30,1d5f30239da05aaf,d05f30b6c8a0dec9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:03:16.265 INFO 2981023 --- [host-check-agent-status4,fd9f69d70225b687,5ba08981e15fb80d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:03:16.268 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ddd605dd7a713860,574429a889f669fb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ddd605dd7a713860, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:16.275 INFO 2981023 --- [ocp-schedules-24,022de146c1198ffe,d352a5ecc03ce465] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:03:16.277 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:03:16.277 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:03:16.279 INFO 2981023 --- [ocp-schedules-1,eb92f5755ef06905,4260d0b202e5909c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:03:16.282 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048922674000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:03:16.286 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:03:16.296 INFO 2981023 --- [ocp-schedules-15,37351f810489c5a8,4bc687e5f9c9f3d1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:03:16.305 INFO 2981023 --- [ocp-schedules-1,eb92f5755ef06905,4260d0b202e5909c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:03:16.355 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ddd605dd7a713860,574429a889f669fb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ddd605dd7a713860, duration=247 ms] 2024-09-11 18:03:16.389 INFO 2981023 --- [host-check-agent-status3,fd9f69d70225b687,bdd4375a2e3d13e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:03:16.396 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:03:16.398 INFO 2981023 --- [host-check-agent-status3,fd9f69d70225b687,bdd4375a2e3d13e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:03:16.398 INFO 2981023 --- [host-check-agent-status3,fd9f69d70225b687,bdd4375a2e3d13e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:03:16.398 INFO 2981023 --- [host-check-agent-status3,fd9f69d70225b687,bdd4375a2e3d13e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:03:16.399 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:03:16.403872869+08:00 2024-09-11 18:03:16.401 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:03:16.399+08:00 2024-09-11 18:03:16.407 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:03:16.399+08:00 and status AVAILABLE 2024-09-11 18:03:16.408 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:03:16.408 INFO 2981023 --- [host-check-agent-status4,fd9f69d70225b687,5ba08981e15fb80d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:03:16.409 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.409 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:03:16.410 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:03:16.409038104+08:00 2024-09-11 18:03:16.410 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:03:16.412 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:03:16.410+08:00 2024-09-11 18:03:16.413 INFO 2981023 --- [host-check-agent-status4,fd9f69d70225b687,5ba08981e15fb80d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:03:16.413 INFO 2981023 --- [host-check-agent-status4,fd9f69d70225b687,5ba08981e15fb80d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:03:16.413 INFO 2981023 --- [host-check-agent-status4,fd9f69d70225b687,5ba08981e15fb80d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:03:16.454 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007107] will be resolved. 2024-09-11 18:03:16.456 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:03:16.456 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:03:16.410+08:00 and status AVAILABLE 2024-09-11 18:03:16.458 INFO 2981023 --- [host-check-agent-status1,fd9f69d70225b687,a78dd644101a2f17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:03:16.458 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:03:16.463501178+08:00 2024-09-11 18:03:16.459 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:03:16.458+08:00 2024-09-11 18:03:16.459 INFO 2981023 --- [host-check-agent-status2,fd9f69d70225b687,196051df16da6edb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:03:16.460 INFO 2981023 --- [host-check-agent-status1,fd9f69d70225b687,a78dd644101a2f17] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:03:16.460 INFO 2981023 --- [host-check-agent-status1,fd9f69d70225b687,a78dd644101a2f17] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:03:16.460 INFO 2981023 --- [host-check-agent-status1,fd9f69d70225b687,a78dd644101a2f17] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:03:16.462 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:03:16.458+08:00 and status AVAILABLE 2024-09-11 18:03:16.462 INFO 2981023 --- [host-check-agent-status2,fd9f69d70225b687,196051df16da6edb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:03:16.462 INFO 2981023 --- [host-check-agent-status2,fd9f69d70225b687,196051df16da6edb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:03:16.462 INFO 2981023 --- [host-check-agent-status2,fd9f69d70225b687,196051df16da6edb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:03:16.463 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:03:16.465 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:03:16.465252356+08:00 2024-09-11 18:03:16.465 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.465 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:03:16.466 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:03:16.465+08:00 2024-09-11 18:03:16.466 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:03:16.468 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:03:16.468 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:03:16.465+08:00 and status AVAILABLE 2024-09-11 18:03:16.468 INFO 2981023 --- [ocp-schedules-30,1d5f30239da05aaf,d05f30b6c8a0dec9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:03:16.471 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.471 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:03:16.471 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:16.473 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.473 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:16.477 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:03:16.477 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:03:16.478 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.478 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:16.484 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : resolveByTimeout, timeoutCount=1 2024-09-11 18:03:16.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:03:16.478076+08:00, lastProcessTime=2024-09-11T18:03:06.351498+08:00 2024-09-11 18:03:16.495 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:03:16.498 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:03:16.510 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:03:16.511 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:03:16.514 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:03:16.514 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:03:16.554 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:03:16.554 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:03:16.554 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:03:16.555 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:03:16.573 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:03:16.578 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:03:16.590 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:03:16.593 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:03:16.594 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:03:16.596 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:03:16.596 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048922674000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:03:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:16.600 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:03:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:03:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:03:16.610 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:03:16.618 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:03:16.623 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:03:16.630 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:03:16.630 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:03:16.632 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:03:16.638 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:03:16.644 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:03:16.657 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:03:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:03:16.665 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:03:16.668 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:03:16.670 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:03:16.672 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:03:16.676 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:03:16.676 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:03:16.677 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:16.678 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.678 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:03:16.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:03:16.680 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:03:16.681 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:03:16.689 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:03:16.691 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:03:16.702 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:03:16.702 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:03:16.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:03:16.708 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:03:16.712 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:03:16.716 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:16.717 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:16.718 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:03:16.718 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:03:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.721 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:03:16.722 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:03:16.723 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:03:16.735 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:03:16.737 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:03:16.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:03:16.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:03:16.744 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:03:16.744 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:03:16.751 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:03:16.758 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:03:16.760 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:03:16.763 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:16.763 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.764 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:03:16.765 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:03:16.768 INFO 2981023 --- [host-check-host-status4,713381fdc8eb8a49,db8b9b30f226104a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:03:16.769 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:03:16.774 INFO 2981023 --- [host-check-host-status1,713381fdc8eb8a49,39e7b9678caa881e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:03:16.774 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:03:16.777 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:03:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:03:16.780 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:03:16.785 INFO 2981023 --- [host-check-host-status2,713381fdc8eb8a49,f18e94fbe28cf392] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:03:16.790 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:03:16.793 INFO 2981023 --- [host-check-host-status3,713381fdc8eb8a49,706f0a37469f721b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:03:16.803 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:16.804 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.804 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:03:16.805 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:03:16.813 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:03:16.813 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:03:16.823 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:03:16.827 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:03:16.478076+08:00 2024-09-11 18:03:16.834 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:03:16.840 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:16.841 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:16.842 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:03:16.848 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:03:16.848 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:03:16.849 INFO 2981023 --- [ocp-schedules-24,022de146c1198ffe,d352a5ecc03ce465] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:03:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,acc314f5df4b2a6c,bc80d91e9a4ed337] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,acc314f5df4b2a6c,bc80d91e9a4ed337] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:17.376 INFO 2981023 --- [ocp-async-6,40e716a3bc570113,6a5a7934fd9d95ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:17.777 INFO 2981023 --- [ocp-async-7,c4ae728c7acefc56,2ad4db2840406d07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:03:17.826 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:17.826 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:18.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,3b89f97f8d3b5887,e0b40b18053ba828] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=3b89f97f8d3b5887, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:18.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,3b89f97f8d3b5887,e0b40b18053ba828] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=3b89f97f8d3b5887, duration=9 ms] 2024-09-11 18:03:19.376 INFO 2981023 --- [ocp-async-8,5bf119e6e0e1fb23,be009fb4d9472f0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:20.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,686aa4d05e4f61b7,08edda2123d00cd9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=686aa4d05e4f61b7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:20.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,686aa4d05e4f61b7,08edda2123d00cd9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=686aa4d05e4f61b7, duration=17 ms] 2024-09-11 18:03:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,d4b51b4a606dbdc8,50844ef69f0dca14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,d4b51b4a606dbdc8,50844ef69f0dca14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:20.579 INFO 2981023 --- [host-check-clockdiff1,53dc9d10c027386a,00264219b87aa1b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049000 5 5 , error= 2024-09-11 18:03:20.826 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:20.828 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:21.376 INFO 2981023 --- [ocp-async-7,020ed6f638a747c2,5e05ef9993b33528] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:22.082 INFO 2981023 --- [host-check-clockdiff4,53dc9d10c027386a,1b7b7e33d617e8c1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049002 0 0 , error= 2024-09-11 18:03:22.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1856f369049b7eca,71ef316f3be7e526] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1856f369049b7eca, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:22.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1856f369049b7eca,71ef316f3be7e526] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1856f369049b7eca, duration=10 ms] 2024-09-11 18:03:22.433 INFO 2981023 --- [ocp-schedules-9,5ed6c1ba0e2c644a,f9eaf34b4b2a8797] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:03:23.007 INFO 2981023 --- [ocp-schedules-27,b74da46ea187df95,0cdcd126debadd6b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 18:03:23.007 INFO 2981023 --- [ocp-schedules-27,b74da46ea187df95,0cdcd126debadd6b] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 18:03:23.030 INFO 2981023 --- [pool-563-thread-2,b74da46ea187df95,83dd117b745ff80d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:03:23.032 INFO 2981023 --- [pool-563-thread-1,b74da46ea187df95,830eae384ea6ef92] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:03:23.032 INFO 2981023 --- [pool-563-thread-3,b74da46ea187df95,b450dd9130ef9c5e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:03:23.053 INFO 2981023 --- [pool-563-thread-2,b74da46ea187df95,83dd117b745ff80d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:23.055 INFO 2981023 --- [pool-563-thread-1,b74da46ea187df95,830eae384ea6ef92] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:23.055 INFO 2981023 --- [pool-563-thread-2,b74da46ea187df95,83dd117b745ff80d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726048102000000, 1726049003000000] 2024-09-11 18:03:23.055 INFO 2981023 --- [pool-563-thread-3,b74da46ea187df95,b450dd9130ef9c5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:23.057 INFO 2981023 --- [pool-563-thread-1,b74da46ea187df95,830eae384ea6ef92] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726048102000000, 1726049003000000] 2024-09-11 18:03:23.058 INFO 2981023 --- [pool-563-thread-3,b74da46ea187df95,b450dd9130ef9c5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726048102000000, 1726049003000000] 2024-09-11 18:03:23.062 INFO 2981023 --- [pool-563-thread-3,b74da46ea187df95,b450dd9130ef9c5e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:03:23.148 INFO 2981023 --- [pool-563-thread-1,b74da46ea187df95,830eae384ea6ef92] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:03:23.200 INFO 2981023 --- [pool-563-thread-2,b74da46ea187df95,83dd117b745ff80d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:03:23.200 INFO 2981023 --- [ocp-schedules-27,b74da46ea187df95,0cdcd126debadd6b] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 18:03:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,844234a8c004ac69,31be7a768737e5bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,844234a8c004ac69,31be7a768737e5bd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:23.376 INFO 2981023 --- [ocp-async-1,285260a683a45833,8320e128fa6792a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:23.828 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:23.829 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:24.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,028023387df23dff,7fc18bed231b49ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=028023387df23dff, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:24.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,028023387df23dff,7fc18bed231b49ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=028023387df23dff, duration=10 ms] 2024-09-11 18:03:24.379 INFO 2981023 --- [ocp-schedules-16,8e68739758629959,994414dfe360adf3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:03:24.480 INFO 2981023 --- [ocp-schedules-14,e6c0a8b988a6009e,4e21bfa05a6f3118] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:03:24.480 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:03:24.480 INFO 2981023 --- [ocp-schedules-14,e6c0a8b988a6009e,4e21bfa05a6f3118] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:03:24.480 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:03:24.480 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:03:24.505 INFO 2981023 --- [ocp-schedules-14,e6c0a8b988a6009e,4e21bfa05a6f3118] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:03:24.505 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:03:24.508 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:03:24.508 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:03:24.509 INFO 2981023 --- [ocp-schedules-14,e6c0a8b988a6009e,4e21bfa05a6f3118] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:03:24.510 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:03:24.518 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:03:24.561 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:03:24.756 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:03:24.758 INFO 2981023 --- [ocp-schedules-14,e6c0a8b988a6009e,4e21bfa05a6f3118] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:03:24.761 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:03:24.773 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:03:24.776 INFO 2981023 --- [ocp-schedules-14,e6c0a8b988a6009e,4e21bfa05a6f3118] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:03:25.343 INFO 2981023 --- [ocp-schedules-15,7ce1edc83a25a391,8713c6662801222b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:03:25.377 INFO 2981023 --- [ocp-async-2,06c125d79c4b0b29,2321aa59fb78e2e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:25.385 WARN 2981023 --- [scanner-13597566-1,7ce1edc83a25a391,7e08b8b3e8caec35] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:03:25.406 INFO 2981023 --- [alarm-detect-5,7ce1edc83a25a391,c377306cde155226] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645283, data is empty. 2024-09-11 18:03:25.409 INFO 2981023 --- [alarm-detect-5,7ce1edc83a25a391,c377306cde155226] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645283, data is empty. 2024-09-11 18:03:25.414 INFO 2981023 --- [alarm-detect-5,7ce1edc83a25a391,c377306cde155226] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645283, data is empty. 2024-09-11 18:03:25.445 INFO 2981023 --- [alarm-detect-15,7ce1edc83a25a391,0ad948089470ebed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645283, data is empty. 2024-09-11 18:03:25.445 INFO 2981023 --- [alarm-detect-8,7ce1edc83a25a391,0eae49ac20745450] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645283, data is empty. 2024-09-11 18:03:25.449 INFO 2981023 --- [alarm-detect-15,7ce1edc83a25a391,0ad948089470ebed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645283, data is empty. 2024-09-11 18:03:25.451 INFO 2981023 --- [alarm-detect-8,7ce1edc83a25a391,0eae49ac20745450] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645283, data is empty. 2024-09-11 18:03:25.454 INFO 2981023 --- [alarm-detect-15,7ce1edc83a25a391,0ad948089470ebed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645283, data is empty. 2024-09-11 18:03:25.455 INFO 2981023 --- [alarm-detect-8,7ce1edc83a25a391,0eae49ac20745450] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645283, data is empty. 2024-09-11 18:03:25.590 INFO 2981023 --- [alarm-detect-20,7ce1edc83a25a391,d7758eb6b725bfd0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645283, data is empty. 2024-09-11 18:03:25.594 INFO 2981023 --- [alarm-detect-20,7ce1edc83a25a391,d7758eb6b725bfd0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645283, data is empty. 2024-09-11 18:03:25.599 INFO 2981023 --- [alarm-detect-20,7ce1edc83a25a391,d7758eb6b725bfd0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645283, data is empty. 2024-09-11 18:03:26.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b98edc5ed5694c9c,82c2705a83ddb9e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b98edc5ed5694c9c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:26.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b98edc5ed5694c9c,82c2705a83ddb9e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b98edc5ed5694c9c, duration=10 ms] 2024-09-11 18:03:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,03bc13f49ff45b89,d97f3396ec4e554e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,03bc13f49ff45b89,d97f3396ec4e554e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:26.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:03:26.351213+08:00, lastProcessTime=2024-09-11T18:03:16.478076+08:00 2024-09-11 18:03:26.484 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:03:26.351213+08:00 2024-09-11 18:03:26.829 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:26.829 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:27.377 INFO 2981023 --- [ocp-async-9,1c75d4d8067416d0,4385f5ecdce07e9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:27.778 INFO 2981023 --- [ocp-async-10,e3eebfe8e8398a8a,95400c1a078b1b26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:03:27.856 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.859 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClust 2024-09-11 18:03:27.870 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.870 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:03:27.877 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.885 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 5 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.898 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.899 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.906 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.907 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.908 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.908 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.913 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.917 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 42 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.982 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 10 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:27.996 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 167 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.027 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.032 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.033 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:03:28.035 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.038 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.045 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.046 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.047 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.048 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.048 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.049 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.050 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.050 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:03:28.051 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.051 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.052 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T17:57:27.661, end=2024-09-11T18:02:27.853} 2024-09-11 18:03:28.052 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 5 dubious sqls 2024-09-11 18:03:28.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,b816ea9d34df4161,f60c85da579fdc9d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b816ea9d34df4161, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:28.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,b816ea9d34df4161,f60c85da579fdc9d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b816ea9d34df4161, duration=9 ms] 2024-09-11 18:03:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,53d67ca89c7dc1ae,f473a65cb8c950a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,53d67ca89c7dc1ae,f473a65cb8c950a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:29.377 INFO 2981023 --- [ocp-async-6,0412dfc51bdb015e,60711451cc2733c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:29.829 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:29.830 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:30.425 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,cd97c0fe1a3d3494,61f9a9f825ef1d01] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=cd97c0fe1a3d3494, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:03:30.447 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,cd97c0fe1a3d3494,61f9a9f825ef1d01] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=cd97c0fe1a3d3494, duration=25 ms] 2024-09-11 18:03:31.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4ddbf9e05a74556e,614fe032f7c5976b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4ddbf9e05a74556e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:31.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4ddbf9e05a74556e,614fe032f7c5976b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4ddbf9e05a74556e, duration=11 ms] 2024-09-11 18:03:31.377 INFO 2981023 --- [ocp-async-10,3333f8ec4ddb6b28,6e45009fceec9977] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,a1765fca9aaf7331,38a6bed365ffa6b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,a1765fca9aaf7331,38a6bed365ffa6b4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:32.436 INFO 2981023 --- [ocp-schedules-2,749f9b33233fe106,bfc94201661c9f85] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:03:32.830 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:32.830 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:33.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f3f1ba1731d11f0d,17cd693ef411ed2f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f3f1ba1731d11f0d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:33.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f3f1ba1731d11f0d,17cd693ef411ed2f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f3f1ba1731d11f0d, duration=10 ms] 2024-09-11 18:03:33.377 INFO 2981023 --- [ocp-async-8,e855d22599b25e4e,ceb76200ce4c112d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:34.967 INFO 2981023 --- [ocp-schedules-14,97d48c247e07ef99,6eb70834b8dd6dfa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:03:34.975 INFO 2981023 --- [ocp-schedules-14,97d48c247e07ef99,6eb70834b8dd6dfa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:03:34.975 INFO 2981023 --- [ocp-schedules-14,97d48c247e07ef99,6eb70834b8dd6dfa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:03:35.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f174c2b639d3c99b,8df4b2ec561e8205] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f174c2b639d3c99b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:35.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f174c2b639d3c99b,8df4b2ec561e8205] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f174c2b639d3c99b, duration=9 ms] 2024-09-11 18:03:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,14f74b88b778c89a,57c3bf5875927d96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,14f74b88b778c89a,57c3bf5875927d96] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:35.377 INFO 2981023 --- [ocp-async-7,60c5ac29c56e84ef,6b0fc9dfbcb65b43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:35.631 INFO 2981023 --- [ocp-schedules-21,8fdc24c0b6592e0b,d5ee3bb87d8503b0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:03:35.678 WARN 2981023 --- [scanner-13597866-1,8fdc24c0b6592e0b,10ce6da497963b83] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:03:35.830 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:35.831 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:03:36.337180+08:00, lastProcessTime=2024-09-11T18:03:26.351213+08:00 2024-09-11 18:03:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:03:36.337180+08:00 2024-09-11 18:03:37.099 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,7197224381384005,520761db30aa38fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7197224381384005, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:37.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,7197224381384005,520761db30aa38fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7197224381384005, duration=8 ms] 2024-09-11 18:03:37.314 INFO 2981023 --- [prometheus-cache-monitor1,4bc84f0b56beefe0,92c78815de6652e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":4605} 2024-09-11 18:03:37.315 INFO 2981023 --- [prometheus-cache-monitor1,4bc84f0b56beefe0,92c78815de6652e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:03:37.377 INFO 2981023 --- [ocp-async-1,60ff36eef9110a9c,ed21f04a18d028e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:37.778 INFO 2981023 --- [ocp-async-4,764b115275cb876d,18c6072d68a945f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:03:38.044 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:03:38.044 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:03:38.046 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:03:38.046 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:03:38.046 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:03:38.046 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:03:38.049 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:03:38.049 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:03:38.049 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:03:38.049 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:03:38.073 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:03:38.077 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:03:38.077 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:03:38.077 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:03:38.077 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:03:38.077 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:03:38.081 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:03:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,c360349b136104fb,198473edcb2aa4fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,c360349b136104fb,198473edcb2aa4fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:38.340 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:03:38.646 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.597361578S 2024-09-11 18:03:38.646 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:03:38.700 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:03:38.700 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:03:38.722 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:03:38.759 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:03:38.759 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:03:38.759 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:03:38.759 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:03:38.759 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:03:38.760 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:03:38.764 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:03:38.831 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:38.832 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:38.851 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] o.s.boot.SpringApplication : Started application in 0.804 seconds (JVM running for 84818.723) 2024-09-11 18:03:38.851 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.804903382S 2024-09-11 18:03:39.051 INFO 2981023 --- [ocp-schedules-16,7e6cfed460ebfaf9,a9ad049c0ab3f4e1] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:03:39.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,75628e3ac3ad0c7b,6c6a91880fc87a2a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=75628e3ac3ad0c7b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:39.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,75628e3ac3ad0c7b,6c6a91880fc87a2a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=75628e3ac3ad0c7b, duration=10 ms] 2024-09-11 18:03:39.377 INFO 2981023 --- [ocp-async-9,6817d6a9f89ac0db,e1814efc03f8616c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:40.003 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:03:40.003 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:03:41.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,d54146369bbe6aa5,cd86ee27cc305c84] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d54146369bbe6aa5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:41.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,d54146369bbe6aa5,cd86ee27cc305c84] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d54146369bbe6aa5, duration=12 ms] 2024-09-11 18:03:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,0df1c053d2854add,dc9341453271bcda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,0df1c053d2854add,dc9341453271bcda] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:41.378 INFO 2981023 --- [ocp-async-4,a5a20f5c7197dd8d,52f47e8564fb0382] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:41.832 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:41.832 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:42.439 INFO 2981023 --- [ocp-schedules-6,785bfd33716ec93a,257b106b513e5078] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:03:43.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,a5fe3f0f224f76b4,ad944f02400cfda5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a5fe3f0f224f76b4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:43.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,a5fe3f0f224f76b4,ad944f02400cfda5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a5fe3f0f224f76b4, duration=11 ms] 2024-09-11 18:03:43.378 INFO 2981023 --- [ocp-async-3,05cca7d458a2c256,c88c6625c1b04500] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:44.002 INFO 2981023 --- [ocp-schedules-21,9825afb3802bd139,3aa26b2e2f5141d7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:03:44.020 WARN 2981023 --- [ocp-schedules-21,9825afb3802bd139,3aa26b2e2f5141d7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:03:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,e75938cea0c599ca,51ac7b4ee04a0462] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:44.317 INFO 2981023 --- [prometheus-minute-schedule-1,e75938cea0c599ca,51ac7b4ee04a0462] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:44.832 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:44.833 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:45.101 INFO 2981023 --- [prometheus-second-schedule-1,c30a489554581263,b323920b37c73338] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:03:45.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6df0e7fc01fbcc47,dad56474074a5f0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6df0e7fc01fbcc47, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:45.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6df0e7fc01fbcc47,dad56474074a5f0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6df0e7fc01fbcc47, duration=22 ms] 2024-09-11 18:03:45.378 INFO 2981023 --- [ocp-async-8,4ffbf15a17dcfdd2,5b0ee20cc06fbf2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:45.833 INFO 2981023 --- [ocp-schedules-9,6ad38fe4b84dea98,d2645cb4a282d282] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:03:45.908 WARN 2981023 --- [scanner-13598083-1,6ad38fe4b84dea98,c533b888ee506f9d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:03:45.934 INFO 2981023 --- [alarm-detect-17,6ad38fe4b84dea98,b4c4de2334b81f4a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645283, data is empty. 2024-09-11 18:03:45.939 INFO 2981023 --- [alarm-detect-17,6ad38fe4b84dea98,b4c4de2334b81f4a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645283, data is empty. 2024-09-11 18:03:45.945 INFO 2981023 --- [alarm-detect-17,6ad38fe4b84dea98,b4c4de2334b81f4a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645283, data is empty. 2024-09-11 18:03:45.974 INFO 2981023 --- [alarm-detect-2,6ad38fe4b84dea98,87f88cd35d3168c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645283, data is empty. 2024-09-11 18:03:45.981 INFO 2981023 --- [alarm-detect-2,6ad38fe4b84dea98,87f88cd35d3168c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645283, data is empty. 2024-09-11 18:03:45.992 INFO 2981023 --- [alarm-detect-2,6ad38fe4b84dea98,87f88cd35d3168c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645283, data is empty. 2024-09-11 18:03:45.997 INFO 2981023 --- [alarm-detect-6,6ad38fe4b84dea98,3170187265866e55] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645283, data is empty. 2024-09-11 18:03:46.007 INFO 2981023 --- [alarm-detect-6,6ad38fe4b84dea98,3170187265866e55] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645283, data is empty. 2024-09-11 18:03:46.028 INFO 2981023 --- [alarm-detect-6,6ad38fe4b84dea98,3170187265866e55] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645283, data is empty. 2024-09-11 18:03:46.032 INFO 2981023 --- [ocp-schedules-18,ffca5ccd3ab7b669,9302622e5f2b4c98] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:03:46.032 INFO 2981023 --- [ocp-schedules-10,a4e95ac1c1287aef,fe88afc5c017c6ff] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:03:46.033 INFO 2981023 --- [ocp-schedules-27,89db8b3e8bbcdaa2,89ea396b31c99689] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:03:46.047 INFO 2981023 --- [host-check-clockdiff1,89db8b3e8bbcdaa2,d8a44f7f6663158a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:03:46.050 INFO 2981023 --- [host-check-clockdiff2,89db8b3e8bbcdaa2,46991165e9b01d7d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:03:46.051 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:03:46.051 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:03:46.053 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:03:46.053 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:03:46.053 INFO 2981023 --- [host-check-agent-status1,ffca5ccd3ab7b669,40fe4422dbf13aa0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:03:46.054 INFO 2981023 --- [host-check-agent-status2,ffca5ccd3ab7b669,edde324d27c6c838] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:03:46.056 INFO 2981023 --- [host-check-agent-status3,ffca5ccd3ab7b669,2d83a3fed6a183ca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:03:46.056 INFO 2981023 --- [host-check-agent-status4,ffca5ccd3ab7b669,1e671ac557c4706d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:03:46.093 INFO 2981023 --- [host-check-agent-status3,ffca5ccd3ab7b669,2d83a3fed6a183ca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:03:46.098 INFO 2981023 --- [host-check-agent-status1,ffca5ccd3ab7b669,40fe4422dbf13aa0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:03:46.099 INFO 2981023 --- [host-check-agent-status2,ffca5ccd3ab7b669,edde324d27c6c838] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:03:46.099 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:03:46.099 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:03:46.100 INFO 2981023 --- [host-check-agent-status3,ffca5ccd3ab7b669,2d83a3fed6a183ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:03:46.100 INFO 2981023 --- [host-check-agent-status1,ffca5ccd3ab7b669,40fe4422dbf13aa0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:03:46.100 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:03:46.105 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:03:46.105 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:03:46.106 INFO 2981023 --- [host-check-agent-status2,ffca5ccd3ab7b669,edde324d27c6c838] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:03:46.107 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:03:46.108 INFO 2981023 --- [host-check-clockdiff3,89db8b3e8bbcdaa2,69e8422fb7b92cce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:03:46.116 INFO 2981023 --- [host-check-clockdiff4,89db8b3e8bbcdaa2,e5717e5311560864] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:03:46.166 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:03:46.193 INFO 2981023 --- [ocp-schedules-19,f31f0e6238bb8525,e087d990d447da0b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:03:46.205 INFO 2981023 --- [ocp-schedules-12,f272e4eaad9a764c,c7f8f505850f308f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:03:46.206 INFO 2981023 --- [ocp-schedules-11,caf2850910904fa3,287ddf6c4d052c51] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:03:46.206 INFO 2981023 --- [ocp-schedules-26,725799b46df7dc84,e3a30970f401264d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:03:46.207 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:03:46.270 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:03:46.270 INFO 2981023 --- [ocp-schedules-11,caf2850910904fa3,287ddf6c4d052c51] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:03:46.274 INFO 2981023 --- [host-check-agent-status4,ffca5ccd3ab7b669,1e671ac557c4706d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:03:46.288 INFO 2981023 --- [ocp-schedules-12,f272e4eaad9a764c,c7f8f505850f308f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:03:46.288 INFO 2981023 --- [ocp-schedules-5,ddf8f0c5b94b47e2,759da2f9b3adae56] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:03:46.288 INFO 2981023 --- [ocp-schedules-19,f31f0e6238bb8525,e087d990d447da0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:03:46.291 INFO 2981023 --- [host-check-agent-status4,ffca5ccd3ab7b669,1e671ac557c4706d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:03:46.316 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048922674000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:03:46.318 INFO 2981023 --- [ocp-schedules-12,f272e4eaad9a764c,c7f8f505850f308f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:03:46.394 INFO 2981023 --- [host-check-clockdiff2,89db8b3e8bbcdaa2,46991165e9b01d7d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049026 6 6 , error= 2024-09-11 18:03:46.421 INFO 2981023 --- [host-check-clockdiff3,89db8b3e8bbcdaa2,69e8422fb7b92cce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049026 0 0 , error= 2024-09-11 18:03:46.458 INFO 2981023 --- [ocp-schedules-5,ddf8f0c5b94b47e2,759da2f9b3adae56] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:03:46.477 INFO 2981023 --- [ocp-schedules-13,64c95e6a4df9366d,384dccdca37e4736] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:03:46.490 INFO 2981023 --- [host-check-agent-status3,ffca5ccd3ab7b669,2d83a3fed6a183ca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:03:46.539 INFO 2981023 --- [host-check-agent-status3,ffca5ccd3ab7b669,2d83a3fed6a183ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:03:46.539 INFO 2981023 --- [host-check-agent-status3,ffca5ccd3ab7b669,2d83a3fed6a183ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:03:46.539 INFO 2981023 --- [host-check-agent-status3,ffca5ccd3ab7b669,2d83a3fed6a183ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:03:46.540 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:03:46.546 INFO 2981023 --- [alarm-detect-18,6ad38fe4b84dea98,70cabbf43d6db2ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645283, data is empty. 2024-09-11 18:03:46.549 INFO 2981023 --- [alarm-detect-18,6ad38fe4b84dea98,70cabbf43d6db2ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645283, data is empty. 2024-09-11 18:03:46.556 INFO 2981023 --- [alarm-detect-18,6ad38fe4b84dea98,70cabbf43d6db2ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645283, data is empty. 2024-09-11 18:03:46.574 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:03:46.578 INFO 2981023 --- [host-check-agent-status1,ffca5ccd3ab7b669,40fe4422dbf13aa0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:03:46.578 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:03:46.583928084+08:00 2024-09-11 18:03:46.582 INFO 2981023 --- [host-check-agent-status1,ffca5ccd3ab7b669,40fe4422dbf13aa0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:03:46.582 INFO 2981023 --- [host-check-agent-status1,ffca5ccd3ab7b669,40fe4422dbf13aa0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:03:46.582 INFO 2981023 --- [host-check-agent-status1,ffca5ccd3ab7b669,40fe4422dbf13aa0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:03:46.583 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:03:46.578+08:00 2024-09-11 18:03:46.589 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:03:46.578+08:00 and status AVAILABLE 2024-09-11 18:03:46.843 INFO 2981023 --- [ocp-schedules-8,9f004445a0d18f8e,daaceb824abb10a9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2052,"seriesIdKeySize":11688,"secondWriteQueue":0,"minuteWriteCacheSize":4605} 2024-09-11 18:03:46.843 INFO 2981023 --- [ocp-schedules-8,9f004445a0d18f8e,daaceb824abb10a9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:03:46.857 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:03:46.958 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:46.959 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:03:46.962 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:03:47.042 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:03:47.060 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:03:47.115 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.116 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:03:47.117 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:03:47.145 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:03:47.146 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a2445c2660399fc1,79c3ab20b0639b3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a2445c2660399fc1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:47.149 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:03:47.147639+08:00, lastProcessTime=2024-09-11T18:03:36.337180+08:00 2024-09-11 18:03:47.150 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:03:47.147755669+08:00 2024-09-11 18:03:47.152 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:03:47.150+08:00 2024-09-11 18:03:47.155 INFO 2981023 --- [host-check-clockdiff1,89db8b3e8bbcdaa2,d8a44f7f6663158a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049026 5 5 , error= 2024-09-11 18:03:47.157 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:03:47.161 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:03:47.164723120+08:00 2024-09-11 18:03:47.162 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:03:47.161+08:00 2024-09-11 18:03:47.163 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:03:47.161+08:00 and status AVAILABLE 2024-09-11 18:03:47.164 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:03:47.167 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:03:47.167 INFO 2981023 --- [host-check-agent-status2,ffca5ccd3ab7b669,edde324d27c6c838] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:03:47.167 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:03:47.167677035+08:00 2024-09-11 18:03:47.168 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:03:47.167+08:00 2024-09-11 18:03:47.171 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:03:47.150+08:00 and status AVAILABLE 2024-09-11 18:03:47.171 INFO 2981023 --- [ocp-schedules-19,f31f0e6238bb8525,e087d990d447da0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:03:47.172 INFO 2981023 --- [host-check-agent-status2,ffca5ccd3ab7b669,edde324d27c6c838] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:03:47.172 INFO 2981023 --- [host-check-agent-status2,ffca5ccd3ab7b669,edde324d27c6c838] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:03:47.172 INFO 2981023 --- [host-check-agent-status2,ffca5ccd3ab7b669,edde324d27c6c838] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:03:47.173 INFO 2981023 --- [host-check-agent-status4,ffca5ccd3ab7b669,1e671ac557c4706d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:03:47.174 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:03:47.175 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.175 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:47.175 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.176 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:03:47.167+08:00 and status AVAILABLE 2024-09-11 18:03:47.178 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:03:47.179 INFO 2981023 --- [host-check-agent-status4,ffca5ccd3ab7b669,1e671ac557c4706d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:03:47.179 INFO 2981023 --- [host-check-agent-status4,ffca5ccd3ab7b669,1e671ac557c4706d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:03:47.179 INFO 2981023 --- [host-check-agent-status4,ffca5ccd3ab7b669,1e671ac557c4706d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:03:47.179 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.180 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:03:47.180 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:03:47.183 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.187 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:03:47.189 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:03:47.190 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.190 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:03:47.190 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:47.194 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:03:47.200 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:03:47.202 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:03:47.204 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:03:47.206 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a2445c2660399fc1,79c3ab20b0639b3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a2445c2660399fc1, duration=93 ms] 2024-09-11 18:03:47.209 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.209 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:47.210 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:03:47.211 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:03:47.215 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:03:47.254 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:03:47.260 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:03:47.262 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.265 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.266 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:03:47.267 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:03:47.267 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048922674000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:03:47.269 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:03:47.270 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:03:47.278 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.279 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.279 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:03:47.280 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:03:47.280 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.281 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:03:47.284 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:03:47.297 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:03:47.297 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:03:47.300 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:03:47.300 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:03:47.304 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.305 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:03:47.305 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.305 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:47.306 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:03:47.309 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.312 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:03:47.312 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:03:47.313 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:03:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,27b7be3f6350b833,6d179b886e6824f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:47.316 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:03:47.318 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:03:47.322 INFO 2981023 --- [prometheus-minute-schedule-1,27b7be3f6350b833,6d179b886e6824f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:47.330 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:03:47.331 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:03:47.337 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:03:47.337 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:03:47.352 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:03:47.357 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.361 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:03:47.367 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:03:47.367 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:03:47.368 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:03:47.375 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:03:47.377 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.378 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.379 INFO 2981023 --- [ocp-async-8,7f891a53480c6c14,0040dc3406eeb5d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:47.379 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:03:47.380 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:03:47.383 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.383 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:03:47.384 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:03:47.384 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.386 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.388 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:03:47.391 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:03:47.391 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.391 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:03:47.393 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:03:47.393 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:03:47.397 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.398 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.399 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:03:47.403 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:03:47.408 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:03:47.411 INFO 2981023 --- [host-check-host-status1,a4e95ac1c1287aef,4106e21e2e8a5de0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:03:47.414 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:03:47.415 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:03:47.415 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:03:47.422 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:03:47.424 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:03:47.424 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.424 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:03:47.425 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.425 INFO 2981023 --- [host-check-host-status4,a4e95ac1c1287aef,16607de5cee5f557] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:03:47.425 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:03:47.425 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:03:47.426 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:03:47.426 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.429 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:03:47.430 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:03:47.441 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:03:47.441 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:03:47.444 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:03:47.448 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.449 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.449 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:03:47.450 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:03:47.450 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.452 INFO 2981023 --- [host-check-host-status2,a4e95ac1c1287aef,27e59b9ca5c788ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:03:47.454 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:03:47.457 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:03:47.465 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:03:47.466 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:03:47.476 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.477 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.477 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:03:47.478 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:03:47.483 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:03:47.484 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:03:47.487 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:03:47.487 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:03:47.488 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:03:47.497 INFO 2981023 --- [host-check-host-status3,a4e95ac1c1287aef,da6b07166ea073f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:03:47.509 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.510 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.510 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:03:47.511 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:03:47.511 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:03:47.512 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.539 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:03:47.546 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.550 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.551 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:03:47.565 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:03:47.569 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:03:47.569 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:03:47.569 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:03:47.147639+08:00 2024-09-11 18:03:47.570 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:03:47.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:47.576 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:47.576 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:03:47.581 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:03:47.582 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:03:47.582 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:03:47.582 INFO 2981023 --- [ocp-schedules-11,caf2850910904fa3,287ddf6c4d052c51] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:03:47.587 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:03:47.591 INFO 2981023 --- [ocp-schedules-17,44b10dfce593b39f,8a03c6c3feea8920] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:03:47.778 INFO 2981023 --- [ocp-async-7,62d17d0e7a8557aa,54b41a479f80dea1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:03:47.833 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:47.834 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:49.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,bd14521c84602607,0a511656689ca7fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bd14521c84602607, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:49.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,bd14521c84602607,0a511656689ca7fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bd14521c84602607, duration=9 ms] 2024-09-11 18:03:49.378 INFO 2981023 --- [ocp-async-2,346a2dbe825f805a,f185cc4f3ba3bc89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:03:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:03:50.299 INFO 2981023 --- [ocp-schedules-7,dc30553c922971d2,dfe823084011e435] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:03:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,48ac7555e6a40ece,63ea6f771936b679] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:50.315 INFO 2981023 --- [prometheus-minute-schedule-1,48ac7555e6a40ece,63ea6f771936b679] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:50.834 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:50.834 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:51.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0b0a51f679e61b21,720e3369943dbfc2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0b0a51f679e61b21, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:51.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0b0a51f679e61b21,720e3369943dbfc2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0b0a51f679e61b21, duration=9 ms] 2024-09-11 18:03:51.378 INFO 2981023 --- [ocp-async-9,cfaf4f4702d237c6,1a3de8adf21f0d54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:51.595 INFO 2981023 --- [ocp-schedules-9,efcc0ba1df0625c9,9747466c77658379] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:03:51.596 INFO 2981023 --- [ocp-schedules-9,efcc0ba1df0625c9,9747466c77658379] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:03:51.597 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:03:51.597 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:03:51.597 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:03:51.597 INFO 2981023 --- [ocp-schedules-9,efcc0ba1df0625c9,9747466c77658379] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:03:52.138 INFO 2981023 --- [host-check-clockdiff4,89db8b3e8bbcdaa2,e5717e5311560864] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049032 0 0 , error= 2024-09-11 18:03:52.441 INFO 2981023 --- [ocp-schedules-17,75a06cc22cd18952,039dde16b19bbd68] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:03:53.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,339dc501f594f002,8f374615cb071827] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=339dc501f594f002, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:53.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,339dc501f594f002,8f374615cb071827] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=339dc501f594f002, duration=10 ms] 2024-09-11 18:03:53.244 INFO 2981023 --- [ocp-schedules-10,5ffe4f865d838f30,2147bd8f731de33a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:03:53.244 INFO 2981023 --- [ocp-schedules-10,5ffe4f865d838f30,2147bd8f731de33a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:03:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,f3ad506fc575e4d4,679ed409295b4dab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,f3ad506fc575e4d4,679ed409295b4dab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:53.378 INFO 2981023 --- [ocp-async-8,72c97db05c2fc53c,e05c51ee56a6e9a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:53.835 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:53.835 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:54.233 INFO 2981023 --- [ocp-schedules-24,71d734e6b8a03efe,76e8e3c4e26e8501] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:03:54.250 WARN 2981023 --- [ocp-schedules-24,71d734e6b8a03efe,76e8e3c4e26e8501] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:03:54.380 INFO 2981023 --- [ocp-schedules-5,7a6571114c0836b7,1f7d33859bf7e6cd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:03:54.801 INFO 2981023 --- [ocp-schedules-14,8900abba5a9539f6,a409ac7f89fb8457] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T18:03:54.801+08:00, tenantSize=6 2024-09-11 18:03:54.804 INFO 2981023 --- [ocp-schedules-14,8900abba5a9539f6,a409ac7f89fb8457] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726049034801000 2024-09-11 18:03:54.805 INFO 2981023 --- [ocp-schedules-14,8900abba5a9539f6,a409ac7f89fb8457] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726048734759000 2024-09-11 18:03:54.821 INFO 2981023 --- [ocp-schedules-14,8900abba5a9539f6,a409ac7f89fb8457] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 18:03:54.974 INFO 2981023 --- [ocp-schedules-11,fbda721446658f63,4bf59852a1037fd6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:03:54.974 INFO 2981023 --- [ocp-schedules-11,fbda721446658f63,4bf59852a1037fd6] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:03:54.978 INFO 2981023 --- [ocp-schedules-11,fbda721446658f63,4bf59852a1037fd6] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:03:54.978 INFO 2981023 --- [ocp-schedules-11,fbda721446658f63,4bf59852a1037fd6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:03:54.978 INFO 2981023 --- [ocp-schedules-11,fbda721446658f63,4bf59852a1037fd6] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:03:54.978 INFO 2981023 --- [ocp-schedules-23,da529c308c738999,f9ba22137a63c5d8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:03:54.981 INFO 2981023 --- [ocp-schedules-11,fbda721446658f63,4bf59852a1037fd6] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:03:54.981 INFO 2981023 --- [ocp-schedules-11,fbda721446658f63,4bf59852a1037fd6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:03:54.981 INFO 2981023 --- [ocp-schedules-11,fbda721446658f63,4bf59852a1037fd6] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:03:54.988 INFO 2981023 --- [ocp-schedules-23,da529c308c738999,f9ba22137a63c5d8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:03:54.988 INFO 2981023 --- [ocp-schedules-23,da529c308c738999,f9ba22137a63c5d8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:03:54.990 INFO 2981023 --- [ocp-schedules-11,fbda721446658f63,4bf59852a1037fd6] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:03:55.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,53d71986d6777d85,c91843b7b9ce5e1d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=53d71986d6777d85, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:55.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,53d71986d6777d85,c91843b7b9ce5e1d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=53d71986d6777d85, duration=9 ms] 2024-09-11 18:03:55.378 INFO 2981023 --- [ocp-async-10,8840d62320ef79e6,695d149d56bdb23d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:55.829 INFO 2981023 --- [ocp-schedules-18,63ed2537b1f38d9d,5cca6fca05e6d7fe] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:03:55.829 INFO 2981023 --- [ocp-schedules-18,63ed2537b1f38d9d,5cca6fca05e6d7fe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690420679617835008 2024-09-11 18:03:55.832 INFO 2981023 --- [ocp-schedules-18,63ed2537b1f38d9d,5cca6fca05e6d7fe] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:03:55.832 INFO 2981023 --- [ocp-schedules-18,63ed2537b1f38d9d,5cca6fca05e6d7fe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:03:55.834 INFO 2981023 --- [ocp-schedules-18,63ed2537b1f38d9d,5cca6fca05e6d7fe] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:03:55.834 INFO 2981023 --- [ocp-schedules-18,63ed2537b1f38d9d,5cca6fca05e6d7fe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:03:55.983 INFO 2981023 --- [ocp-schedules-30,1851771810957676,a6c5d0a8d5e4b59f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:03:55.990 INFO 2981023 --- [ocp-schedules-30,1851771810957676,a6c5d0a8d5e4b59f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:03:55.990 INFO 2981023 --- [ocp-schedules-30,1851771810957676,a6c5d0a8d5e4b59f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:03:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,ae5fec9a33ac9c57,f22d1919cc61583f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,ae5fec9a33ac9c57,f22d1919cc61583f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:56.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007106] will be resolved. 2024-09-11 18:03:56.346 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:03:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : resolveByTimeout, timeoutCount=1 2024-09-11 18:03:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:03:56.352525+08:00, lastProcessTime=2024-09-11T18:03:47.147639+08:00 2024-09-11 18:03:56.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:03:56.493 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:03:56.352525+08:00 2024-09-11 18:03:56.835 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:56.835 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:03:57.009 INFO 2981023 --- [ocp-schedules-6,62b48706c6020b5e,c2b7c20c55c82e67] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:03:57.009+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:03:57.013 INFO 2981023 --- [ocp-schedules-6,62b48706c6020b5e,c2b7c20c55c82e67] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:03:57.098 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,937c667590a70134,39bfdf3121a057a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=937c667590a70134, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:57.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,937c667590a70134,39bfdf3121a057a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=937c667590a70134, duration=11 ms] 2024-09-11 18:03:57.166 INFO 2981023 --- [ocp-schedules-6,62b48706c6020b5e,c2b7c20c55c82e67] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 47 to SqlAggregateData 2024-09-11 18:03:57.168 INFO 2981023 --- [ocp-schedules-6,62b48706c6020b5e,c2b7c20c55c82e67] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:03:57.216 INFO 2981023 --- [ocp-schedules-12,3abd96edccb00d4f,bbdb3301c51b1b25] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:03:57.250 WARN 2981023 --- [scanner-13598357-1,3abd96edccb00d4f,7219dd9108fec435] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:03:57.378 INFO 2981023 --- [ocp-async-9,fa501b921adbd0ac,be590c5072f3b543] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:57.428 INFO 2981023 --- [ocp-schedules-4,592413bab4359245,74ef3537858914e9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:03:57.428 INFO 2981023 --- [ocp-schedules-4,592413bab4359245,74ef3537858914e9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:03:57.442 INFO 2981023 --- [Collect all tenants major compactions1,592413bab4359245,8d099e2780582bea] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:03:57.443 INFO 2981023 --- [Collect all tenants major compactions3,592413bab4359245,964e0b3d7c3fc91d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:03:57.443 INFO 2981023 --- [Collect all tenants major compactions5,592413bab4359245,dadc799884623fd6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:03:57.444 INFO 2981023 --- [Collect all tenants major compactions2,592413bab4359245,452d54da90c05293] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:03:57.444 INFO 2981023 --- [Collect all tenants major compactions4,592413bab4359245,92754ae0979a853e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:03:57.445 INFO 2981023 --- [Collect all tenants major compactions6,592413bab4359245,07f138be47828ff4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:03:57.456 INFO 2981023 --- [Collect all tenants major compactions5,592413bab4359245,dadc799884623fd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.457 INFO 2981023 --- [Collect all tenants major compactions1,592413bab4359245,8d099e2780582bea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.457 INFO 2981023 --- [Collect all tenants major compactions4,592413bab4359245,92754ae0979a853e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.458 INFO 2981023 --- [Collect all tenants major compactions5,592413bab4359245,dadc799884623fd6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:03:57.458 INFO 2981023 --- [Collect all tenants major compactions2,592413bab4359245,452d54da90c05293] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.458 INFO 2981023 --- [Collect all tenants major compactions4,592413bab4359245,92754ae0979a853e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:03:57.458 INFO 2981023 --- [Collect all tenants major compactions3,592413bab4359245,964e0b3d7c3fc91d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.459 INFO 2981023 --- [Collect all tenants major compactions1,592413bab4359245,8d099e2780582bea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:03:57.460 INFO 2981023 --- [Collect all tenants major compactions3,592413bab4359245,964e0b3d7c3fc91d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:03:57.460 INFO 2981023 --- [Collect all tenants major compactions6,592413bab4359245,07f138be47828ff4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.460 INFO 2981023 --- [Collect all tenants major compactions2,592413bab4359245,452d54da90c05293] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:03:57.461 INFO 2981023 --- [Collect all tenants major compactions6,592413bab4359245,07f138be47828ff4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:03:57.462 INFO 2981023 --- [Collect all tenants major compactions4,592413bab4359245,92754ae0979a853e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:03:57.463 INFO 2981023 --- [Collect all tenants major compactions5,592413bab4359245,dadc799884623fd6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:03:57.463 INFO 2981023 --- [Collect all tenants major compactions1,592413bab4359245,8d099e2780582bea] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:03:57.463 INFO 2981023 --- [Collect all tenants major compactions3,592413bab4359245,964e0b3d7c3fc91d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:03:57.465 INFO 2981023 --- [Collect all tenants major compactions4,592413bab4359245,92754ae0979a853e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:03:57.466 INFO 2981023 --- [Collect all tenants major compactions2,592413bab4359245,452d54da90c05293] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:03:57.466 INFO 2981023 --- [Collect all tenants major compactions2,592413bab4359245,452d54da90c05293] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:03:57.467 INFO 2981023 --- [Collect all tenants major compactions6,592413bab4359245,07f138be47828ff4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:03:57.467 INFO 2981023 --- [Collect all tenants major compactions6,592413bab4359245,07f138be47828ff4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:03:57.468 INFO 2981023 --- [Collect all tenants major compactions5,592413bab4359245,dadc799884623fd6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:03:57.469 INFO 2981023 --- [Collect all tenants major compactions3,592413bab4359245,964e0b3d7c3fc91d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:03:57.470 INFO 2981023 --- [Collect all tenants major compactions1,592413bab4359245,8d099e2780582bea] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:03:57.472 INFO 2981023 --- [ocp-schedules-4,592413bab4359245,74ef3537858914e9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:03:57.778 INFO 2981023 --- [ocp-async-10,0b3dd17db1b10604,f5f4980713d2c77d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:03:57.920 INFO 2981023 --- [ocp-schedules-7,0513a6d13c7af2e1,360ccdcf0518e724] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:03:57.921 INFO 2981023 --- [ocp-schedules-7,0513a6d13c7af2e1,360ccdcf0518e724] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:03:57.949 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:03:57.950 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.951 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:03:57.951 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.951 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:03:57.952 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.953 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:03:57.954 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:03:57.954 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:03:57.961 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.961 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:57.963 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.963 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:57.964 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.964 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:57.967 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.967 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:57.972 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.973 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:57.976 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.977 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:57.985 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.986 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:57.986 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.986 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:03:57.986 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:03:57.986 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE46A1D6F5-0-0] 2024-09-11 18:03:57.987 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:03:57.988 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.989 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.989 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:03:57.989 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:57.995 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:57.996 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:03:58.001 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.001 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:03:58.003 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.004 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:03:58.016 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.016 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.016 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:03:58.016 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:03:58.017 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:03:58.018 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.018 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:03:58.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE46A1D6FF-0-0] 2024-09-11 18:03:58.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:03:58.018 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:03:58.024 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.024 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.025 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:03:58.025 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:03:58.025 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:03:58.026 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.026 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:03:58.027 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.027 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:03:58.028 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.029 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:03:58.029 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:03:58.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.031 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:03:58.031 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:03:58.034 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:03:58.034 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:03:58.035 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.036 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:03:58.036 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.038 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:03:58.039 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.039 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:03:58.040 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.041 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:03:58.042 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.042 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:03:58.043 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.043 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:03:58.045 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.046 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:03:58.047 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.047 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:03:58.048 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.048 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:03:58.049 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:58.049 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:03:58.050 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.051 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:03:58.051 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:03:58.051 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:58.053 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.054 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:58.063 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:03:58.065 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:03:58.066 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:03:58.067 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.067 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:03:58.067 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.067 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:58.067 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:03:58.068 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:03:58.068 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:03:58.068 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:03:58.074 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:03:58.075 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:03:58.076 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.076 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:03:58.079 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:03:58.081 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:03:58.081 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:03:58.081 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:03:58.082 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:03:58.084 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:03:58.084 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:03:58.084 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:03:58.090 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:03:58.092 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:03:58.092 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:03:58.092 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:03:58.092 INFO 2981023 --- [ocp-schedules-7,0513a6d13c7af2e1,360ccdcf0518e724] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:03:58.396 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,47d249628aec478e,40e81904ef5d8770] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=47d249628aec478e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:03:58.419 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,47d249628aec478e,40e81904ef5d8770] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=47d249628aec478e, duration=26 ms] 2024-09-11 18:03:59.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2836475e21efd618,495732e5478d3364] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2836475e21efd618, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:03:59.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2836475e21efd618,495732e5478d3364] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2836475e21efd618, duration=16 ms] 2024-09-11 18:03:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,7ff89a27316dcce4,696de8abd1cd1dd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049040, interval=60 2024-09-11 18:03:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,7ff89a27316dcce4,696de8abd1cd1dd7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:03:59.378 INFO 2981023 --- [ocp-async-1,cb7153118d5a48e7,9bbee6e8b7a5b97f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:03:59.836 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:03:59.837 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:00.173 INFO 2981023 --- [ocp-schedules-12,2502ddd14ff772a6,ead002529091fed2] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:04:01.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f526a9b85d2639cc,03722a9173b27404] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f526a9b85d2639cc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:01.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f526a9b85d2639cc,03722a9173b27404] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f526a9b85d2639cc, duration=10 ms] 2024-09-11 18:04:01.388 INFO 2981023 --- [ocp-async-10,1d29e584fec52fa8,44f4b35ff501cf7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,4394da45f376d264,c53225bb292c3b63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,4394da45f376d264,c53225bb292c3b63] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:02.444 INFO 2981023 --- [ocp-schedules-19,cbc17bb10c30ad58,ed8c098f82ec7d75] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:04:02.684 INFO 2981023 --- [ocp-schedules-4,6667642799ccfb6a,8d6a446bb754ec4c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:04:02.692 INFO 2981023 --- [ocp-schedules-4,6667642799ccfb6a,8d6a446bb754ec4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:04:02.728 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:02.729 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:04:02.734 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:02.736 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:04:02.737 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049042737000} 2024-09-11 18:04:02.743 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:04:02.746 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000226, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:04:02.746 INFO 2981023 --- [ocp-schedules-4,6667642799ccfb6a,8d6a446bb754ec4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:04:02.837 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:02.838 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:03.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,c4caf17eb18c7f46,b25a3bcbebf42396] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c4caf17eb18c7f46, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:03.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,c4caf17eb18c7f46,b25a3bcbebf42396] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c4caf17eb18c7f46, duration=9 ms] 2024-09-11 18:04:03.342 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:04:03.342 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:04:03.342 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:04:03.346 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000226, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:04:03.349 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:04:03.379 INFO 2981023 --- [ocp-async-7,77f8568f8d24de12,d2e65fd22423fdd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:04.008 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000226 2024-09-11 18:04:05.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,e40bc218ef818be9,79e47e484f5d6a70] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e40bc218ef818be9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:05.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,e40bc218ef818be9,79e47e484f5d6a70] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e40bc218ef818be9, duration=17 ms] 2024-09-11 18:04:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,20d5c9cfb9e26a13,6399d4484f215161] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,20d5c9cfb9e26a13,6399d4484f215161] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:05.379 INFO 2981023 --- [ocp-async-6,5a6d8355d47fb01a,0a2c9b03e979660a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:05.838 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:05.840 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:04:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:04:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:04:05.992 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:04:05.997 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:04:06.002 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:04:06.002 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:04:06.002 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:04:06.007 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:06.007 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:04:06.008 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:06.008 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:06.068 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:01:41, start=2024-09-11T17:57:41, end=2024-09-11T18:00:11, updates=297, granule=120000ms, delay=240000ms 2024-09-11 18:04:06.076 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:01:41, target=2024-09-11T18:00:11, granule=120000ms, delay=90000ms) 2024-09-11 18:04:06.076 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:00:11, target=2024-09-11T17:50:51, granule=600000ms, delay=560000ms) 2024-09-11 18:04:06.076 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=106ms, period=120000ms 2024-09-11 18:04:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:04:06.351880+08:00, lastProcessTime=2024-09-11T18:03:56.352525+08:00 2024-09-11 18:04:06.488 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:04:06.351880+08:00 2024-09-11 18:04:07.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,fe9ec1291721f922,4ef77d292a280ca2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fe9ec1291721f922, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:07.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,fe9ec1291721f922,4ef77d292a280ca2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fe9ec1291721f922, duration=11 ms] 2024-09-11 18:04:07.379 INFO 2981023 --- [ocp-async-1,c757eb79f7545873,2963801e41c459d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:07.400 INFO 2981023 --- [ocp-schedules-25,820243af722e138f,cbd2549d2986e730] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:04:07.454 WARN 2981023 --- [scanner-13598617-1,820243af722e138f,7cea9109240596e4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:04:07.473 INFO 2981023 --- [alarm-detect-2,820243af722e138f,540c72e79272b5aa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645283, data is empty. 2024-09-11 18:04:07.476 INFO 2981023 --- [alarm-detect-2,820243af722e138f,540c72e79272b5aa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645330, data is empty. 2024-09-11 18:04:07.480 INFO 2981023 --- [alarm-detect-2,820243af722e138f,540c72e79272b5aa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645330, data is empty. 2024-09-11 18:04:07.501 INFO 2981023 --- [alarm-detect-3,820243af722e138f,6f2411b221884709] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645283, data is empty. 2024-09-11 18:04:07.502 INFO 2981023 --- [alarm-detect-6,820243af722e138f,5f6919bd9190b913] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645283, data is empty. 2024-09-11 18:04:07.505 INFO 2981023 --- [alarm-detect-3,820243af722e138f,6f2411b221884709] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645330, data is empty. 2024-09-11 18:04:07.507 INFO 2981023 --- [alarm-detect-6,820243af722e138f,5f6919bd9190b913] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645330, data is empty. 2024-09-11 18:04:07.518 INFO 2981023 --- [alarm-detect-3,820243af722e138f,6f2411b221884709] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645330, data is empty. 2024-09-11 18:04:07.520 INFO 2981023 --- [alarm-detect-6,820243af722e138f,5f6919bd9190b913] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645330, data is empty. 2024-09-11 18:04:07.637 INFO 2981023 --- [alarm-detect-20,820243af722e138f,32692b73bf7ad19e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645283, data is empty. 2024-09-11 18:04:07.642 INFO 2981023 --- [alarm-detect-20,820243af722e138f,32692b73bf7ad19e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645330, data is empty. 2024-09-11 18:04:07.657 INFO 2981023 --- [alarm-detect-20,820243af722e138f,32692b73bf7ad19e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645330, data is empty. 2024-09-11 18:04:07.756 INFO 2981023 --- [ocp-schedules-19,0b5e1cda5fa257d4,3259fd7a5b4371a5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:04:07.778 INFO 2981023 --- [ocp-async-4,168ef6563f04f4bc,74988271c0df5265] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:04:07.884 INFO 2981023 --- [ocp-schedules-5,db1749b04175fa50,90fd16c874972ddf] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:04:07.884+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:04:07.885 INFO 2981023 --- [ocp-schedules-5,db1749b04175fa50,90fd16c874972ddf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:04:07.886 INFO 2981023 --- [ocp-schedules-5,db1749b04175fa50,90fd16c874972ddf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:04:07.887 INFO 2981023 --- [ocp-schedules-5,db1749b04175fa50,90fd16c874972ddf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:04:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,2c3b17fcf7af4cf6,158b9451d354473e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:08.315 INFO 2981023 --- [prometheus-minute-schedule-1,2c3b17fcf7af4cf6,158b9451d354473e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:08.840 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:08.840 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:09.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,143e6ac0da3ffa2a,2b1991163d43d06d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=143e6ac0da3ffa2a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:09.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,143e6ac0da3ffa2a,2b1991163d43d06d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=143e6ac0da3ffa2a, duration=9 ms] 2024-09-11 18:04:09.379 INFO 2981023 --- [ocp-async-9,efa7e5e5722b5c6c,1520cc68a495efe5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:10.003 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:04:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:04:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:04:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:04:11.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9e1c90c96583f511,2c4f61affbc4729e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9e1c90c96583f511, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:11.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9e1c90c96583f511,2c4f61affbc4729e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9e1c90c96583f511, duration=9 ms] 2024-09-11 18:04:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,d96811cc07b07c75,72a0f535a1276a2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,d96811cc07b07c75,72a0f535a1276a2e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:11.379 INFO 2981023 --- [ocp-async-4,c8623d556809a03c,93039442c28b755b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:11.578 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048980, nodeCount=194, affectRows=194 2024-09-11 18:04:11.840 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:11.841 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:12.447 INFO 2981023 --- [ocp-schedules-24,6f214007d38d6079,db9b606d5adc98f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:04:12.705 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048980, nodeCount=194, validCount=194 2024-09-11 18:04:12.751 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726048980, nodeCount=286, affectRows=286 2024-09-11 18:04:13.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,af1597d22309308c,561eeaa79edc4e92] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=af1597d22309308c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:13.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,af1597d22309308c,561eeaa79edc4e92] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=af1597d22309308c, duration=10 ms] 2024-09-11 18:04:13.379 INFO 2981023 --- [ocp-async-2,777c1b830ee46437,d3fbc1277b0541f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:13.999 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726048980, nodeCount=286, validCount=286 2024-09-11 18:04:14.029 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048980, nodeCount=1000, affectRows=1000 2024-09-11 18:04:14.029 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048980, nodeCount=1000, affectRows=1000 2024-09-11 18:04:14.032 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048980, nodeCount=1000, validCount=1000 2024-09-11 18:04:14.032 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048980, nodeCount=999, validCount=999 2024-09-11 18:04:14.037 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048980, nodeCount=10, affectRows=10 2024-09-11 18:04:14.037 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048980, nodeCount=9, validCount=9 2024-09-11 18:04:14.043 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726048980, nodeCount=42, affectRows=42 2024-09-11 18:04:14.043 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726048980, nodeCount=41, validCount=41 2024-09-11 18:04:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,f7a3a2112a13b1ca,5905690427b80ef1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,eb6c5c866463d528,3414fbd252cc0e37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:04:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,f7a3a2112a13b1ca,5905690427b80ef1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:14.841 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:14.841 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:14.991 INFO 2981023 --- [ocp-schedules-3,ae7ec96313809c9f,e831058cf3c60b5d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:04:15.000 INFO 2981023 --- [ocp-schedules-3,ae7ec96313809c9f,e831058cf3c60b5d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:04:15.000 INFO 2981023 --- [ocp-schedules-3,ae7ec96313809c9f,e831058cf3c60b5d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:04:15.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d93615cb637c22dc,178591001e335a78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d93615cb637c22dc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:15.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d93615cb637c22dc,178591001e335a78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d93615cb637c22dc, duration=10 ms] 2024-09-11 18:04:15.379 INFO 2981023 --- [ocp-async-10,3a94057c7c13a157,206dc7a23932b03e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:16.022 INFO 2981023 --- [ocp-schedules-21,a4a286ceb396dc0d,671f3d4fdf8d2289] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:04:16.022 INFO 2981023 --- [ocp-schedules-15,3ec44ae392814895,20e559ff8d33d713] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:04:16.023 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:04:16.024 INFO 2981023 --- [host-check-clockdiff1,3ec44ae392814895,f48ca1da82ded3a5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:04:16.024 INFO 2981023 --- [host-check-clockdiff2,3ec44ae392814895,06ce5ea59837d776] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:04:16.026 INFO 2981023 --- [host-check-clockdiff3,3ec44ae392814895,718769f2e54d78ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:04:16.026 INFO 2981023 --- [host-check-clockdiff4,3ec44ae392814895,77db75f080c5ad46] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:04:16.030 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:04:16.031 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:04:16.031 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:04:16.047 INFO 2981023 --- [ocp-schedules-24,6e3a7b968051b2f4,3ace7c936c32d44c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:04:16.047 INFO 2981023 --- [host-check-agent-status1,6e3a7b968051b2f4,aba4c9fa8d6a9949] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:04:16.047 INFO 2981023 --- [host-check-agent-status2,6e3a7b968051b2f4,94f417dfc53ae8f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:04:16.048 INFO 2981023 --- [host-check-agent-status3,6e3a7b968051b2f4,fd01a7d378f7adbe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:04:16.050 INFO 2981023 --- [host-check-clockdiff2,3ec44ae392814895,06ce5ea59837d776] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049056 5 5 , error= 2024-09-11 18:04:16.051 INFO 2981023 --- [host-check-clockdiff3,3ec44ae392814895,718769f2e54d78ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049056 0 0 , error= 2024-09-11 18:04:16.051 INFO 2981023 --- [host-check-agent-status4,6e3a7b968051b2f4,93352163933e3582] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:04:16.054 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:16.056 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:04:16.057 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:16.057 INFO 2981023 --- [host-check-agent-status1,6e3a7b968051b2f4,aba4c9fa8d6a9949] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:16.060 INFO 2981023 --- [host-check-agent-status2,6e3a7b968051b2f4,94f417dfc53ae8f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:04:16.060 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:04:16.061 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:04:16.061 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:16.061 INFO 2981023 --- [host-check-agent-status1,6e3a7b968051b2f4,aba4c9fa8d6a9949] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:16.062 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:04:16.062 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:16.061 INFO 2981023 --- [host-check-agent-status2,6e3a7b968051b2f4,94f417dfc53ae8f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:04:16.064 INFO 2981023 --- [host-check-agent-status4,6e3a7b968051b2f4,93352163933e3582] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:04:16.066 INFO 2981023 --- [host-check-agent-status4,6e3a7b968051b2f4,93352163933e3582] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:04:16.067 INFO 2981023 --- [host-check-agent-status3,6e3a7b968051b2f4,fd01a7d378f7adbe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:16.070 INFO 2981023 --- [host-check-agent-status3,6e3a7b968051b2f4,fd01a7d378f7adbe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:16.187 INFO 2981023 --- [ocp-schedules-16,b8f019cf5d0ba834,0d2a8cd6ebeb3a27] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:04:16.187 INFO 2981023 --- [ocp-schedules-27,e77a8744e226025a,de0113b37aa9ce8c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:04:16.188 INFO 2981023 --- [ocp-schedules-1,d191af5cab502da8,4301b6e769fcdc66] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:04:16.188 INFO 2981023 --- [ocp-schedules-18,a46ddb4f0eccce3b,6e9f1b640f925b8b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:04:16.190 INFO 2981023 --- [ocp-schedules-16,b8f019cf5d0ba834,0d2a8cd6ebeb3a27] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:04:16.191 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049042737000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:04:16.269 INFO 2981023 --- [ocp-schedules-27,e77a8744e226025a,de0113b37aa9ce8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:04:16.269 INFO 2981023 --- [ocp-schedules-1,d191af5cab502da8,4301b6e769fcdc66] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:04:16.299 INFO 2981023 --- [ocp-schedules-26,b64baf6d488a27cb,c20bb045c36afdef] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:04:16.301 INFO 2981023 --- [ocp-schedules-27,e77a8744e226025a,de0113b37aa9ce8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:04:16.304 INFO 2981023 --- [host-check-agent-status3,6e3a7b968051b2f4,fd01a7d378f7adbe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:04:16.305 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.305 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.306 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:04:16.306 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:04:16.306 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:04:16.306 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:04:16.307 INFO 2981023 --- [ocp-schedules-16,b8f019cf5d0ba834,0d2a8cd6ebeb3a27] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:04:16.309 INFO 2981023 --- [host-check-agent-status3,6e3a7b968051b2f4,fd01a7d378f7adbe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:04:16.309 INFO 2981023 --- [host-check-agent-status3,6e3a7b968051b2f4,fd01a7d378f7adbe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:04:16.309 INFO 2981023 --- [host-check-agent-status3,6e3a7b968051b2f4,fd01a7d378f7adbe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:04:16.370 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:04:16.369376+08:00, lastProcessTime=2024-09-11T18:04:06.351880+08:00 2024-09-11 18:04:16.382 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:04:16.386 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:04:16.385416654+08:00 2024-09-11 18:04:16.387 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:04:16.386+08:00 2024-09-11 18:04:16.389 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:04:16.386+08:00 and status AVAILABLE 2024-09-11 18:04:16.401 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:04:16.404 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:04:16.406 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:04:16.411249658+08:00 2024-09-11 18:04:16.407 INFO 2981023 --- [host-check-agent-status2,6e3a7b968051b2f4,94f417dfc53ae8f4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:04:16.407 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:04:16.406+08:00 2024-09-11 18:04:16.409 INFO 2981023 --- [host-check-agent-status2,6e3a7b968051b2f4,94f417dfc53ae8f4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:04:16.409 INFO 2981023 --- [host-check-agent-status2,6e3a7b968051b2f4,94f417dfc53ae8f4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:04:16.409 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:04:16.406+08:00 and status AVAILABLE 2024-09-11 18:04:16.409 INFO 2981023 --- [host-check-agent-status2,6e3a7b968051b2f4,94f417dfc53ae8f4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:04:16.412 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:04:16.412 INFO 2981023 --- [host-check-agent-status4,6e3a7b968051b2f4,93352163933e3582] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:04:16.412 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:04:16.412587280+08:00 2024-09-11 18:04:16.413 INFO 2981023 --- [host-check-agent-status1,6e3a7b968051b2f4,aba4c9fa8d6a9949] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:04:16.413 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:04:16.412+08:00 2024-09-11 18:04:16.415 INFO 2981023 --- [host-check-agent-status4,6e3a7b968051b2f4,93352163933e3582] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:04:16.415 INFO 2981023 --- [host-check-agent-status4,6e3a7b968051b2f4,93352163933e3582] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:04:16.415 INFO 2981023 --- [host-check-agent-status4,6e3a7b968051b2f4,93352163933e3582] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:04:16.416 INFO 2981023 --- [host-check-agent-status1,6e3a7b968051b2f4,aba4c9fa8d6a9949] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:04:16.416 INFO 2981023 --- [host-check-agent-status1,6e3a7b968051b2f4,aba4c9fa8d6a9949] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:04:16.416 INFO 2981023 --- [host-check-agent-status1,6e3a7b968051b2f4,aba4c9fa8d6a9949] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:04:16.418 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:04:16.420 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:04:16.412+08:00 and status AVAILABLE 2024-09-11 18:04:16.458 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:04:16.459 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.459 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:16.459 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:04:16.460 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:04:16.460 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:04:16.462 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:04:16.466443086+08:00 2024-09-11 18:04:16.464 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:04:16.462+08:00 2024-09-11 18:04:16.466 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:04:16.462+08:00 and status AVAILABLE 2024-09-11 18:04:16.473 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:04:16.475 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:04:16.475 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:04:16.475 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.476 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:16.478 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:16.480 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:16.482 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.482 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:16.485 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:04:16.489 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:16.491 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:16.497 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:04:16.507 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:04:16.508 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:04:16.514 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:04:16.523 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:04:16.532 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:04:16.547 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:04:16.555 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:04:16.565 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:04:16.572 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:04:16.572 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049042737000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:04:16.575 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:04:16.577 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:04:16.582 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:04:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:16.586 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:04:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:04:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:04:16.597 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:04:16.602 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:04:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:04:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:04:16.605 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:04:16.605 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:04:16.613 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:04:16.623 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:04:16.629 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:04:16.632 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:04:16.634 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:04:16.642 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:04:16.646 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:04:16.648 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:04:16.654 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:16.655 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.655 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:04:16.656 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:04:16.657 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:16.657 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:04:16.658 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.659 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:04:16.660 INFO 2981023 --- [host-check-host-status4,a4a286ceb396dc0d,a6d1f8c70da4afd5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:04:16.661 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:04:16.661 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:04:16.666 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:16.667 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:04:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:16.677 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:04:16.685 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:04:16.685 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:04:16.691 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:16.691 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.692 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:16.693 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:04:16.693 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.695 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:04:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:04:16.704 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:04:16.705 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:04:16.705 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:04:16.707 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:04:16.713 INFO 2981023 --- [host-check-host-status2,a4a286ceb396dc0d,ee8e1f15ad819d3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:04:16.719 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:04:16.719 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:04:16.722 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:04:16.369376+08:00 2024-09-11 18:04:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:16.727 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.729 INFO 2981023 --- [host-check-host-status3,a4a286ceb396dc0d,5a6b8b52c7043324] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:04:16.729 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:04:16.730 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:04:16.731 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:04:16.740 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:04:16.744 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:16.748 INFO 2981023 --- [host-check-host-status1,a4a286ceb396dc0d,5f8c656affb29af9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:04:16.772 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:04:16.782 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:04:16.789 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:16.789 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.789 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:04:16.795 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:04:16.795 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:04:16.804 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:04:16.810 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:04:16.818 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:04:16.818 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:04:16.819 INFO 2981023 --- [ocp-schedules-1,d191af5cab502da8,4301b6e769fcdc66] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:04:17.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,bb5a865948f50e9b,76fea4055a19dab6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bb5a865948f50e9b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:17.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,bb5a865948f50e9b,76fea4055a19dab6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bb5a865948f50e9b, duration=9 ms] 2024-09-11 18:04:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,220681e6fd3a3d80,8b8355d389cadd1c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,220681e6fd3a3d80,8b8355d389cadd1c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:17.380 INFO 2981023 --- [ocp-async-6,16f8ecf62214d506,60a123285ff81d48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:17.720 INFO 2981023 --- [ocp-schedules-29,38aedc1dee3798e1,423829fc222de7a5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:04:17.765 WARN 2981023 --- [scanner-13600445-1,38aedc1dee3798e1,6fe3ec564e2d8ac2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:04:17.779 INFO 2981023 --- [ocp-async-7,dbafcadf3481eef0,e1d56295508c113d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:04:17.846 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:17.847 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:17.849 INFO 2981023 --- [host-check-clockdiff1,3ec44ae392814895,f48ca1da82ded3a5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049057 5 5 , error= 2024-09-11 18:04:19.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,53e45d5b9da64c74,91fac34ae8738f3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=53e45d5b9da64c74, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:19.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,53e45d5b9da64c74,91fac34ae8738f3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=53e45d5b9da64c74, duration=9 ms] 2024-09-11 18:04:19.380 INFO 2981023 --- [ocp-async-3,12204f75ec3b5273,c808d12353586e3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:19.839 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:04:19.839 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000009 2024-09-11 18:04:19.839 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000007 2024-09-11 18:04:19.839 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2024-09-11 18:04:19.839 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000010 2024-09-11 18:04:19.871 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:04:19.876 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:04:19.886 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:04:19.887 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:04:19.905 WARN 2981023 --- [ForkJoinPool.commonPool-worker-2,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:04:19.906 WARN 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:04:19.912 WARN 2981023 --- [ForkJoinPool.commonPool-worker-3,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:04:19.917 WARN 2981023 --- [ForkJoinPool.commonPool-worker-0,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:04:19.927 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:04:19.927 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:04:19.928 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:04:19.928 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000007 agent sql collect property 2024-09-11 18:04:19.933 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:04:19.934 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:04:19.934 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:04:19.934 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2024-09-11 18:04:19.935 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:04:19.935 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:04:19.935 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:04:19.935 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000009 agent sql collect property 2024-09-11 18:04:19.943 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:04:19.945 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:04:19.946 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:19.948 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:19.949 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:04:19.949 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:04:19.949 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:04:19.950 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000010 agent sql collect property 2024-09-11 18:04:19.955 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2000009 to set sql collect property inactive 2024-09-11 18:04:19.957 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:19.959 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:19.960 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:04:19.962 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:04:19.971 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1718689631), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.5), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=rtdob), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"in 2024-09-11 18:04:19.972 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (monagent.ob.err.code.prefix=errcode=-), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.cgroup.status=active), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.run.user=admin), (ob.logcleaner.enabled=true), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.pipeline.awr.collect.disabled=true), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.ob.rpc.port=2882), (monagent.monitordb.user=root@ocp_monitor), (monagent.host.ip=10.31.35.6), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"includeKeywordsRelation":"Or"," 2024-09-11 18:04:19.983 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994437), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.10), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkprdocp), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"include 2024-09-11 18:04:19.986 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000007, do nothing 2024-09-11 18:04:19.987 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (monagent.ob.err.code.prefix=errcode=-), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.cgroup.status=active), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.run.user=admin), (ob.logcleaner.enabled=true), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.pipeline.awr.collect.disabled=true), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.ob.rpc.port=2882), (monagent.monitordb.user=root@ocp_monitor), (monagent.host.ip=10.31.35.7), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"includeKeywordsRelation":"Or"," 2024-09-11 18:04:19.992 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2024-09-11 18:04:20.001 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000009, do nothing 2024-09-11 18:04:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,d65c6c71c9682dc4,a48507b917eebc9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,d65c6c71c9682dc4,a48507b917eebc9b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:20.847 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:20.848 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:21.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,056d0b2043b34d0f,d42345641a898538] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=056d0b2043b34d0f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:21.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,056d0b2043b34d0f,d42345641a898538] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=056d0b2043b34d0f, duration=9 ms] 2024-09-11 18:04:21.209 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000010, do nothing 2024-09-11 18:04:21.210 INFO 2981023 --- [ocp-schedules-24,d769f8faf4f1111a,b1ea99e9631d79a9] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-09-11 18:04:21.380 INFO 2981023 --- [ocp-async-7,004adb5dd1c44a7c,d1cdec8203aa25b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:22.045 INFO 2981023 --- [host-check-clockdiff4,3ec44ae392814895,77db75f080c5ad46] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049062 0 0 , error= 2024-09-11 18:04:22.450 INFO 2981023 --- [ocp-schedules-21,07838c7382b2f020,a0f7318c99210224] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:04:23.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,bbc74a32979d4dfc,04d8038062a139e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bbc74a32979d4dfc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:23.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,bbc74a32979d4dfc,04d8038062a139e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bbc74a32979d4dfc, duration=11 ms] 2024-09-11 18:04:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,7864e0865d935367,e477a0d3ce5486c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,7864e0865d935367,e477a0d3ce5486c5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:23.380 INFO 2981023 --- [ocp-async-8,fc4e4ca77ca05b2b,d694158faac27c59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:23.848 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:23.848 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:24.385 INFO 2981023 --- [ocp-schedules-11,4c37bd767aca20be,e9c7498b48aa32d3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:04:24.780 INFO 2981023 --- [ocp-schedules-17,c8aa65f415cca705,245715bea76fe14a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:04:24.780 INFO 2981023 --- [ocp-schedules-17,c8aa65f415cca705,245715bea76fe14a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:04:24.780 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:04:24.780 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:04:24.780 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:04:24.784 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:04:24.785 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:04:24.785 INFO 2981023 --- [ocp-schedules-17,c8aa65f415cca705,245715bea76fe14a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:24.785 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:24.786 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:04:24.786 INFO 2981023 --- [ocp-schedules-17,c8aa65f415cca705,245715bea76fe14a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:24.787 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:24.787 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:04:24.806 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:04:24.808 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:04:24.808 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:04:24.811 INFO 2981023 --- [ocp-schedules-17,c8aa65f415cca705,245715bea76fe14a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:04:24.813 INFO 2981023 --- [ocp-schedules-17,c8aa65f415cca705,245715bea76fe14a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:04:25.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,4967af136b44ccaa,c4a121ff3b2c2731] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4967af136b44ccaa, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:25.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,4967af136b44ccaa,c4a121ff3b2c2731] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4967af136b44ccaa, duration=9 ms] 2024-09-11 18:04:25.380 INFO 2981023 --- [ocp-async-1,0dfeff3811594cca,9a9598a87d02411b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,3c095243a635c3b9,fb6d6f42a8656edb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,3c095243a635c3b9,fb6d6f42a8656edb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:04:26.337671+08:00, lastProcessTime=2024-09-11T18:04:16.369376+08:00 2024-09-11 18:04:26.465 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:04:26.337671+08:00 2024-09-11 18:04:26.848 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:26.849 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:27.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7c29d5097c6e0cd0,dd8baf3172da1003] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7c29d5097c6e0cd0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:27.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7c29d5097c6e0cd0,dd8baf3172da1003] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7c29d5097c6e0cd0, duration=9 ms] 2024-09-11 18:04:27.380 INFO 2981023 --- [ocp-async-3,738e9220573a4299,1abe554037fb5ace] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:27.673 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 18:04:27.673 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 18:04:27.674 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.674 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:04:27.675 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.675 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.676 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.676 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 18:04:27.687 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.687 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.688 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.688 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 18:04:27.698 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.699 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.699 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.700 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 18:04:27.708 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.709 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.709 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.709 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 18:04:27.719 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.719 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.719 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.720 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 18:04:27.724 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.724 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.724 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.725 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 18:04:27.728 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.728 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.729 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.729 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 18:04:27.732 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.732 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.733 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.733 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 18:04:27.737 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.737 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.738 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.738 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 18:04:27.741 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.741 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.742 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.742 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 18:04:27.745 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.746 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.746 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.746 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 18:04:27.752 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.752 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.753 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.753 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 18:04:27.762 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.762 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.763 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.763 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 18:04:27.766 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.766 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.767 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.767 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 18:04:27.771 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.771 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.771 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.772 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 18:04:27.775 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.775 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.775 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.776 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 18:04:27.779 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.779 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.779 INFO 2981023 --- [ocp-async-4,275f04e591afcbb3,1674a332cb7a100e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:04:27.779 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.780 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 18:04:27.783 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.783 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.783 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.784 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 18:04:27.787 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.787 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.788 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.788 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 18:04:27.791 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.792 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.792 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.793 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 18:04:27.795 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.796 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.796 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.797 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 18:04:27.800 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.800 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.800 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.801 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 18:04:27.803 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.804 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.804 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.804 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 18:04:27.807 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.808 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.808 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.809 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 18:04:27.811 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.811 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.812 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.812 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 18:04:27.815 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.815 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.815 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.816 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 18:04:27.818 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.819 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.820 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.820 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 18:04:27.823 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.823 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.824 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.824 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 18:04:27.826 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.827 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.827 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.827 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 18:04:27.830 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.830 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.831 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.831 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 18:04:27.834 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.834 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.834 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.835 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 18:04:27.837 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.838 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.838 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.838 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 18:04:27.841 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.841 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.842 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.842 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 18:04:27.845 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.846 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.846 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.846 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 18:04:27.851 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.851 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.852 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.852 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 18:04:27.856 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.856 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.857 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.857 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 18:04:27.860 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.861 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.861 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.861 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 18:04:27.865 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.865 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.865 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.866 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 18:04:27.868 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.868 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.868 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.869 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 18:04:27.871 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.871 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.872 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.873 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 18:04:27.875 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.875 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.876 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.876 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 18:04:27.879 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.879 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.879 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.880 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 18:04:27.886 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.886 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.887 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.887 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 18:04:27.903 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.903 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.903 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.904 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 18:04:27.908 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.908 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.909 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.909 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 18:04:27.913 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.913 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.913 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.914 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 18:04:27.918 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.918 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.919 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.919 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 18:04:27.922 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.922 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.922 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.923 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 18:04:27.926 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.926 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.926 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.927 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 18:04:27.929 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 18:04:27.929 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 18:04:27.930 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.931 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:04:27.933 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.933 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.934 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.934 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 18:04:27.939 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.939 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:04:27.940 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:27.940 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 18:04:27.943 INFO 2981023 --- [ocp-schedules-11,055dd18bbcf15ae1,7ef84ceefbd37cc6] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 18:04:27.962 INFO 2981023 --- [ocp-schedules-19,134da96fecb9da09,6dc46d577f154adf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:04:28.007 WARN 2981023 --- [scanner-13600656-1,134da96fecb9da09,4ab390f49b581505] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:04:28.026 INFO 2981023 --- [alarm-detect-18,134da96fecb9da09,917ff9f8372c73d4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645330, data is empty. 2024-09-11 18:04:28.031 INFO 2981023 --- [alarm-detect-18,134da96fecb9da09,917ff9f8372c73d4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645330, data is empty. 2024-09-11 18:04:28.036 INFO 2981023 --- [alarm-detect-18,134da96fecb9da09,917ff9f8372c73d4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645330, data is empty. 2024-09-11 18:04:28.060 INFO 2981023 --- [alarm-detect-10,134da96fecb9da09,8c73f3f7a874e7ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645330, data is empty. 2024-09-11 18:04:28.061 INFO 2981023 --- [alarm-detect-13,134da96fecb9da09,822f2fab5566c982] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645330, data is empty. 2024-09-11 18:04:28.065 INFO 2981023 --- [alarm-detect-10,134da96fecb9da09,8c73f3f7a874e7ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645330, data is empty. 2024-09-11 18:04:28.067 INFO 2981023 --- [alarm-detect-10,134da96fecb9da09,8c73f3f7a874e7ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645330, data is empty. 2024-09-11 18:04:28.068 INFO 2981023 --- [alarm-detect-13,134da96fecb9da09,822f2fab5566c982] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645330, data is empty. 2024-09-11 18:04:28.072 INFO 2981023 --- [alarm-detect-13,134da96fecb9da09,822f2fab5566c982] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645330, data is empty. 2024-09-11 18:04:28.174 INFO 2981023 --- [alarm-detect-11,134da96fecb9da09,10e1ef6b36647375] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645330, data is empty. 2024-09-11 18:04:28.180 INFO 2981023 --- [alarm-detect-11,134da96fecb9da09,10e1ef6b36647375] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645330, data is empty. 2024-09-11 18:04:28.190 INFO 2981023 --- [alarm-detect-11,134da96fecb9da09,10e1ef6b36647375] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645330, data is empty. 2024-09-11 18:04:28.371 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,85f9c4d7df4ff8ac,0703640ede3a21fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=85f9c4d7df4ff8ac, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:04:28.392 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,85f9c4d7df4ff8ac,0703640ede3a21fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=85f9c4d7df4ff8ac, duration=24 ms] 2024-09-11 18:04:29.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,95fffec4673f2c39,12be89053922c146] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=95fffec4673f2c39, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:29.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,95fffec4673f2c39,12be89053922c146] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=95fffec4673f2c39, duration=11 ms] 2024-09-11 18:04:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,072de543ee21f759,1c592e23d547e8b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,072de543ee21f759,1c592e23d547e8b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:29.380 INFO 2981023 --- [ocp-async-9,0ca40db55d9c45c5,24f41bfcc2b0c52c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:29.849 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:29.850 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:31.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f7b3c170217b4ba1,20294cbe1d3662b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f7b3c170217b4ba1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:31.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f7b3c170217b4ba1,20294cbe1d3662b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f7b3c170217b4ba1, duration=10 ms] 2024-09-11 18:04:31.380 INFO 2981023 --- [ocp-async-1,c78221a921fa7804,c603d5033ba8bcce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,99ae0bd86da43570,1243c61a08f0abba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,99ae0bd86da43570,1243c61a08f0abba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:32.453 INFO 2981023 --- [ocp-schedules-27,cd099939184e4ec8,c5e3a1bf33a5bb57] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:04:32.850 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:32.850 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:33.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,5630fbaa9fef00ad,4adcf7eb4a3bfc28] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5630fbaa9fef00ad, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:33.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,5630fbaa9fef00ad,4adcf7eb4a3bfc28] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5630fbaa9fef00ad, duration=12 ms] 2024-09-11 18:04:33.381 INFO 2981023 --- [ocp-async-5,9803e8eb21826715,dfc74451585c6784] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:35.001 INFO 2981023 --- [ocp-schedules-3,bfa51dd327482b13,35c676b50f14cadd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:04:35.006 INFO 2981023 --- [ocp-schedules-3,bfa51dd327482b13,35c676b50f14cadd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:04:35.006 INFO 2981023 --- [ocp-schedules-3,bfa51dd327482b13,35c676b50f14cadd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:04:35.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,f3440ee24588fa35,e13af992531bc89f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f3440ee24588fa35, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:35.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,f3440ee24588fa35,e13af992531bc89f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f3440ee24588fa35, duration=10 ms] 2024-09-11 18:04:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,353b2c2514942e65,dd99a49112332341] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,353b2c2514942e65,dd99a49112332341] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:35.381 INFO 2981023 --- [ocp-async-8,412a068713884da3,11ef7e66357a32c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:35.850 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:35.851 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:04:36.338408+08:00, lastProcessTime=2024-09-11T18:04:26.337671+08:00 2024-09-11 18:04:36.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:04:36.338408+08:00 2024-09-11 18:04:37.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6d44cf1e37de7831,4749cd3bc1d1e688] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6d44cf1e37de7831, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:37.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6d44cf1e37de7831,4749cd3bc1d1e688] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6d44cf1e37de7831, duration=9 ms] 2024-09-11 18:04:37.314 INFO 2981023 --- [prometheus-cache-monitor1,cabd6b27d0f8bfde,cb9b79477d6470ce] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8199,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2282,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":4914} 2024-09-11 18:04:37.315 INFO 2981023 --- [prometheus-cache-monitor1,cabd6b27d0f8bfde,cb9b79477d6470ce] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:04:37.381 INFO 2981023 --- [ocp-async-4,c3bee4bbb9198539,6dde5e410a957a8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:37.779 INFO 2981023 --- [ocp-async-10,0aedeb3c2d977823,5e33370f6939fd44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:04:38.227 INFO 2981023 --- [ocp-schedules-25,02a07f5b03c00d67,02658fe0d6734298] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:04:38.288 WARN 2981023 --- [scanner-13600958-1,02a07f5b03c00d67,bde3fee37bb98b54] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:04:38.314 INFO 2981023 --- [prometheus-minute-schedule-1,38fac8710d80a51f,ce5ed97495081854] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:38.324 INFO 2981023 --- [prometheus-minute-schedule-1,38fac8710d80a51f,ce5ed97495081854] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:38.369 INFO 2981023 --- [alarm-detect-5,02a07f5b03c00d67,826acc4b956cb38b] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007108] will be resolved. 2024-09-11 18:04:38.369 INFO 2981023 --- [alarm-detect-5,02a07f5b03c00d67,826acc4b956cb38b] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007108] will be resolved. 2024-09-11 18:04:38.424 INFO 2981023 --- [alarm-detect-5,02a07f5b03c00d67,826acc4b956cb38b] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:04:38.851 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:38.852 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:39.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,44e61470ea240daa,7907795578b89837] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=44e61470ea240daa, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:39.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,44e61470ea240daa,7907795578b89837] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=44e61470ea240daa, duration=9 ms] 2024-09-11 18:04:39.381 INFO 2981023 --- [ocp-async-5,294b71111ecb4853,fb9f4493bf0605bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:40.002 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:04:40.002 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:04:41.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4f48d62c68ac0f2b,983cf360a2b44ef3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4f48d62c68ac0f2b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:41.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4f48d62c68ac0f2b,983cf360a2b44ef3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4f48d62c68ac0f2b, duration=9 ms] 2024-09-11 18:04:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,dbcc2bed7e3d1a94,46dd8adcb88c23e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,dbcc2bed7e3d1a94,46dd8adcb88c23e8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:41.381 INFO 2981023 --- [ocp-async-8,453ed2fa44593169,e431a1f2621b08f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:41.852 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:41.852 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:42.455 INFO 2981023 --- [ocp-schedules-11,3f508e5262a3c4ef,6b3ef01b852f676b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:04:43.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,41fcb1b8aa79c6eb,fd17cd411e4ef4e7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=41fcb1b8aa79c6eb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:43.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,41fcb1b8aa79c6eb,fd17cd411e4ef4e7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=41fcb1b8aa79c6eb, duration=10 ms] 2024-09-11 18:04:43.381 INFO 2981023 --- [ocp-async-4,3154a88b624df76b,6d29eab82e11c6b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:44.152 INFO 2981023 --- [ocp-schedules-21,b469f4040541dd0a,67a9e5d6f48ae01b] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:04:44.169 WARN 2981023 --- [ocp-schedules-21,b469f4040541dd0a,67a9e5d6f48ae01b] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:04:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,2128403654b2a4ba,22f9b0b82fe3ea82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,2128403654b2a4ba,22f9b0b82fe3ea82] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:44.852 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:44.853 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:45.101 INFO 2981023 --- [prometheus-second-schedule-1,4ce3dfb33dc17046,dc2782a4a422f6f8] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:04:45.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ba2bf3dee9535e81,6c622a7e1b0ba02c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ba2bf3dee9535e81, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:45.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ba2bf3dee9535e81,6c622a7e1b0ba02c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ba2bf3dee9535e81, duration=11 ms] 2024-09-11 18:04:45.381 INFO 2981023 --- [ocp-async-2,e3bec1979bcfa1be,37491a5e7faeded7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:45.997 INFO 2981023 --- [ocp-schedules-14,270c140beb98a9c1,4dc0408625aa9f72] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:04:45.998 INFO 2981023 --- [host-check-clockdiff1,270c140beb98a9c1,f7f5b645d4df84ac] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:04:45.998 INFO 2981023 --- [host-check-clockdiff2,270c140beb98a9c1,9348d852ecfe1876] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:04:45.999 INFO 2981023 --- [host-check-clockdiff3,270c140beb98a9c1,30cd137efd811618] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:04:45.999 INFO 2981023 --- [host-check-clockdiff4,270c140beb98a9c1,18dcb40b3ea4452d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:04:46.004 INFO 2981023 --- [ocp-schedules-23,bfb5343798026ea0,cf637f8720bd8b3b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:04:46.006 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:04:46.016 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:04:46.016 INFO 2981023 --- [ocp-schedules-2,7045f2b10312f558,338987535d87f6d8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:04:46.017 INFO 2981023 --- [host-check-agent-status1,7045f2b10312f558,532704786dca15e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:04:46.017 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:46.017 INFO 2981023 --- [host-check-agent-status2,7045f2b10312f558,465b16b25d2afdc0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:04:46.017 INFO 2981023 --- [host-check-agent-status3,7045f2b10312f558,308ee9bce77f601d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:04:46.022 INFO 2981023 --- [host-check-clockdiff3,270c140beb98a9c1,30cd137efd811618] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049086 0 0 , error= 2024-09-11 18:04:46.022 INFO 2981023 --- [host-check-agent-status4,7045f2b10312f558,19917c71418e678d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:04:46.022 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:46.022 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:04:46.025 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:04:46.026 INFO 2981023 --- [host-check-agent-status1,7045f2b10312f558,532704786dca15e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:46.027 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:46.027 INFO 2981023 --- [host-check-agent-status1,7045f2b10312f558,532704786dca15e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:46.029 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:46.029 INFO 2981023 --- [host-check-agent-status4,7045f2b10312f558,19917c71418e678d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:04:46.030 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:04:46.030 INFO 2981023 --- [host-check-agent-status4,7045f2b10312f558,19917c71418e678d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:04:46.031 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:04:46.031 INFO 2981023 --- [host-check-agent-status2,7045f2b10312f558,465b16b25d2afdc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:04:46.032 INFO 2981023 --- [host-check-agent-status2,7045f2b10312f558,465b16b25d2afdc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:04:46.032 INFO 2981023 --- [host-check-agent-status3,7045f2b10312f558,308ee9bce77f601d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:46.034 INFO 2981023 --- [host-check-agent-status3,7045f2b10312f558,308ee9bce77f601d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:46.036 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:04:46.040 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:04:46.051 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:04:46.052 INFO 2981023 --- [host-check-agent-status4,7045f2b10312f558,19917c71418e678d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:04:46.053 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:04:46.053573691+08:00 2024-09-11 18:04:46.054 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:04:46.053+08:00 2024-09-11 18:04:46.056 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:04:46.053+08:00 and status AVAILABLE 2024-09-11 18:04:46.057 INFO 2981023 --- [host-check-agent-status4,7045f2b10312f558,19917c71418e678d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:04:46.057 INFO 2981023 --- [host-check-agent-status4,7045f2b10312f558,19917c71418e678d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:04:46.057 INFO 2981023 --- [host-check-agent-status4,7045f2b10312f558,19917c71418e678d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:04:46.058 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:04:46.060 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:04:46.065389619+08:00 2024-09-11 18:04:46.060 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:04:46.060 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:04:46.060+08:00 2024-09-11 18:04:46.061 INFO 2981023 --- [host-check-agent-status2,7045f2b10312f558,465b16b25d2afdc0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:04:46.062 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:04:46.067547088+08:00 2024-09-11 18:04:46.062 INFO 2981023 --- [host-check-agent-status1,7045f2b10312f558,532704786dca15e2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:04:46.062 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:04:46.062+08:00 2024-09-11 18:04:46.063 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:04:46.063 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:04:46.060+08:00 and status AVAILABLE 2024-09-11 18:04:46.064 INFO 2981023 --- [host-check-agent-status3,7045f2b10312f558,308ee9bce77f601d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:04:46.064 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:04:46.062+08:00 and status AVAILABLE 2024-09-11 18:04:46.065 INFO 2981023 --- [host-check-agent-status2,7045f2b10312f558,465b16b25d2afdc0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:04:46.065 INFO 2981023 --- [host-check-agent-status2,7045f2b10312f558,465b16b25d2afdc0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:04:46.065 INFO 2981023 --- [host-check-agent-status2,7045f2b10312f558,465b16b25d2afdc0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:04:46.065 INFO 2981023 --- [host-check-agent-status1,7045f2b10312f558,532704786dca15e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:04:46.065 INFO 2981023 --- [host-check-agent-status1,7045f2b10312f558,532704786dca15e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:04:46.065 INFO 2981023 --- [host-check-agent-status1,7045f2b10312f558,532704786dca15e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:04:46.067 INFO 2981023 --- [host-check-agent-status3,7045f2b10312f558,308ee9bce77f601d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:04:46.067 INFO 2981023 --- [host-check-agent-status3,7045f2b10312f558,308ee9bce77f601d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:04:46.067 INFO 2981023 --- [host-check-agent-status3,7045f2b10312f558,308ee9bce77f601d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:04:46.068 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:04:46.068 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:04:46.069 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:46.069 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:04:46.069591305+08:00 2024-09-11 18:04:46.070 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:04:46.069+08:00 2024-09-11 18:04:46.070 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:46.072 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:04:46.069+08:00 and status AVAILABLE 2024-09-11 18:04:46.073 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:04:46.077 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:46.078 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:04:46.078 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:46.083 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:04:46.084 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:04:46.088 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:04:46.101 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:04:46.103 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:04:46.103 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:04:46.110 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:04:46.119 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:04:46.127 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:04:46.134 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:04:46.141 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:04:46.142 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:04:46.157 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:04:46.158 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:04:46.166 INFO 2981023 --- [ocp-schedules-1,00e7fd32d63b25c8,c45259ad0f0e794d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:04:46.175 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:04:46.176 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:04:46.177 INFO 2981023 --- [ocp-schedules-1,00e7fd32d63b25c8,c45259ad0f0e794d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:04:46.183 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:04:46.187 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:04:46.187 INFO 2981023 --- [ocp-schedules-19,60815a84ad5f1d1d,725ece6ff68feedf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:04:46.189 INFO 2981023 --- [ocp-schedules-19,60815a84ad5f1d1d,725ece6ff68feedf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:04:46.191 INFO 2981023 --- [ocp-schedules-9,3ae25b6055556eec,f14e18a1283ba464] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:04:46.191 INFO 2981023 --- [ocp-schedules-4,e2195bcaa425dc9a,07363a59ed28a9cc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:04:46.191 INFO 2981023 --- [ocp-schedules-30,a64a576afe8739ae,c667ec1c2316b42d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:04:46.197 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:04:46.199 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:04:46.202 INFO 2981023 --- [ocp-schedules-4,e2195bcaa425dc9a,07363a59ed28a9cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:04:46.203 INFO 2981023 --- [ocp-schedules-9,3ae25b6055556eec,f14e18a1283ba464] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:04:46.218 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:04:46.219 INFO 2981023 --- [ocp-schedules-24,c57992422e54a5d1,6ca25c52b63a0383] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:04:46.222 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:04:46.223 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.224 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:04:46.224 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:04:46.224 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.239 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:04:46.240 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:04:46.246 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:04:46.247 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:04:46.256 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:04:46.260 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.262 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:04:46.263 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:04:46.267 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:04:46.268 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.268 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:04:46.270 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:04:46.301 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:04:46.304 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:04:46.305 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:04:46.306 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.312 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:04:46.313 INFO 2981023 --- [ocp-schedules-9,3ae25b6055556eec,f14e18a1283ba464] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:04:46.313 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049042737000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:04:46.314 INFO 2981023 --- [ocp-schedules-19,60815a84ad5f1d1d,725ece6ff68feedf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:04:46.317 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.320 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:04:46.322 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:04:46.322 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:04:46.322 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.329 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:04:46.341 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:04:46.342 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:04:46.341682+08:00, lastProcessTime=2024-09-11T18:04:36.338408+08:00 2024-09-11 18:04:46.346 INFO 2981023 --- [host-check-host-status1,bfb5343798026ea0,e7553dcd6ee6258d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:04:46.346 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:04:46.348 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.349 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:46.354 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:04:46.360 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:04:46.364 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:04:46.365 INFO 2981023 --- [host-check-host-status2,bfb5343798026ea0,66f0a6919b03b961] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:04:46.367 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.368 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:46.387 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:04:46.387 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:04:46.390 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.391 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:46.398 INFO 2981023 --- [host-check-host-status3,bfb5343798026ea0,042ab4f8cafe1ab7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:04:46.398 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.399 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.400 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:04:46.402 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.404 INFO 2981023 --- [host-check-host-status4,bfb5343798026ea0,d6dc2d4bf28799ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:04:46.417 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.419 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.419 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:04:46.420 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:04:46.421 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.421 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:04:46.422 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:04:46.422 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.429 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:04:46.432 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:04:46.432 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049042737000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:04:46.436 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.436 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.437 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:46.438 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:04:46.438 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.441 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:04:46.441 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:04:46.443 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.443 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.444 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:04:46.447 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:04:46.448 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:04:46.448 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:04:46.459 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:04:46.464 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.464 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.465 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:04:46.465 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.465 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.465 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:04:46.465 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.466 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:04:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.466 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:04:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:04:46.467 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.467 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:04:46.469 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:04:46.470 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:04:46.475 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:04:46.477 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:04:46.477 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:04:46.479 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:04:46.480 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:04:46.480 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:04:46.488 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:04:46.495 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.495 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:04:46.495 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.496 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:04:46.497 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:04:46.499 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:04:46.501 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:04:46.507 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:04:46.508 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:04:46.514 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.515 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.515 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:04:46.516 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:04:46.516 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.551 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:04:46.569 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:04:46.571 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:04:46.576 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:04:46.341682+08:00 2024-09-11 18:04:46.580 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:04:46.584 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:04:46.587 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.587 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.588 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:04:46.591 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:46.592 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:04:46.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:46.593 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:04:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:04:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:04:46.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:04:46.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:04:46.599 INFO 2981023 --- [ocp-schedules-4,e2195bcaa425dc9a,07363a59ed28a9cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:04:46.600 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:04:46.603 INFO 2981023 --- [ocp-schedules-3,be2bd52eb7fefcbe,da8fabd75d4c2b94] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:04:46.945 INFO 2981023 --- [host-check-clockdiff2,270c140beb98a9c1,9348d852ecfe1876] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049086 5 5 , error= 2024-09-11 18:04:47.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bd87e211a6f7d5c9,0940fe73333489f9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bd87e211a6f7d5c9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:47.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bd87e211a6f7d5c9,0940fe73333489f9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bd87e211a6f7d5c9, duration=9 ms] 2024-09-11 18:04:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,6417ffcc7aeca165,a4fc0b0d28d54cbd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,6417ffcc7aeca165,a4fc0b0d28d54cbd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:47.381 INFO 2981023 --- [ocp-async-8,7a334029d3e2cd64,47a5d9f8404c2dff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:47.779 INFO 2981023 --- [ocp-async-2,f652d8c3030607cf,a5986d681d66c9fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:04:47.853 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:47.853 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:48.463 INFO 2981023 --- [ocp-schedules-26,4a4508f09f255ce7,31f0d659c21c67a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:04:48.507 WARN 2981023 --- [scanner-13601197-1,4a4508f09f255ce7,91520e4f9233d682] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:04:48.534 INFO 2981023 --- [alarm-detect-17,4a4508f09f255ce7,25b6be528688c2b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645330, data is empty. 2024-09-11 18:04:48.539 INFO 2981023 --- [alarm-detect-17,4a4508f09f255ce7,25b6be528688c2b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645330, data is empty. 2024-09-11 18:04:48.542 INFO 2981023 --- [alarm-detect-17,4a4508f09f255ce7,25b6be528688c2b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645330, data is empty. 2024-09-11 18:04:48.558 INFO 2981023 --- [alarm-detect-4,4a4508f09f255ce7,df77d88ea229ae7d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645330, data is empty. 2024-09-11 18:04:48.564 INFO 2981023 --- [alarm-detect-15,4a4508f09f255ce7,6f16c33e1536610d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645330, data is empty. 2024-09-11 18:04:48.565 INFO 2981023 --- [alarm-detect-4,4a4508f09f255ce7,df77d88ea229ae7d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645330, data is empty. 2024-09-11 18:04:48.568 INFO 2981023 --- [alarm-detect-15,4a4508f09f255ce7,6f16c33e1536610d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645330, data is empty. 2024-09-11 18:04:48.572 INFO 2981023 --- [alarm-detect-4,4a4508f09f255ce7,df77d88ea229ae7d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645330, data is empty. 2024-09-11 18:04:48.574 INFO 2981023 --- [alarm-detect-15,4a4508f09f255ce7,6f16c33e1536610d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645330, data is empty. 2024-09-11 18:04:48.679 INFO 2981023 --- [alarm-detect-19,4a4508f09f255ce7,0f697eb9a4522405] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645330, data is empty. 2024-09-11 18:04:48.683 INFO 2981023 --- [alarm-detect-19,4a4508f09f255ce7,0f697eb9a4522405] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645330, data is empty. 2024-09-11 18:04:48.686 INFO 2981023 --- [alarm-detect-19,4a4508f09f255ce7,0f697eb9a4522405] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645330, data is empty. 2024-09-11 18:04:49.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,e4ebecf86cf247f0,ed75c0ffc4e72e29] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e4ebecf86cf247f0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:49.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,e4ebecf86cf247f0,ed75c0ffc4e72e29] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e4ebecf86cf247f0, duration=17 ms] 2024-09-11 18:04:49.381 INFO 2981023 --- [ocp-async-1,fd96dbe1edcd86fd,cce073c4513fe47a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:04:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:04:50.302 INFO 2981023 --- [ocp-schedules-26,8fdca197fa31f08b,cdcafa54a6528d0c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:04:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,6cc1f45ed8ed2174,b1f5d914d0a15183] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,6cc1f45ed8ed2174,b1f5d914d0a15183] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:50.853 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:50.855 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:51.051 INFO 2981023 --- [host-check-clockdiff1,270c140beb98a9c1,f7f5b645d4df84ac] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049091 6 6 , error= 2024-09-11 18:04:51.058 INFO 2981023 --- [host-check-clockdiff4,270c140beb98a9c1,18dcb40b3ea4452d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049091 0 0 , error= 2024-09-11 18:04:51.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4e7f53abc500fafe,d3461c126f5e022e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4e7f53abc500fafe, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:51.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4e7f53abc500fafe,d3461c126f5e022e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4e7f53abc500fafe, duration=10 ms] 2024-09-11 18:04:51.381 INFO 2981023 --- [ocp-async-2,f8b168dc88d8f5ca,4b078c8517c030d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:51.597 INFO 2981023 --- [ocp-schedules-9,06985fdead183841,11042a64baf42925] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:04:51.598 INFO 2981023 --- [ocp-schedules-9,06985fdead183841,11042a64baf42925] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:04:51.598 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:04:51.598 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:04:51.599 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:04:51.599 INFO 2981023 --- [ocp-schedules-9,06985fdead183841,11042a64baf42925] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:04:52.458 INFO 2981023 --- [ocp-schedules-7,92fbfb4e7fa4d7f7,898ab313924cd898] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:04:53.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d69597cf43a01f97,432a959233687ee6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d69597cf43a01f97, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:53.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d69597cf43a01f97,432a959233687ee6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d69597cf43a01f97, duration=10 ms] 2024-09-11 18:04:53.246 INFO 2981023 --- [ocp-schedules-10,cbefef93ab0a714d,6e097177c2660ced] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:04:53.246 INFO 2981023 --- [ocp-schedules-10,cbefef93ab0a714d,6e097177c2660ced] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:04:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,a776155b98edff0c,074f9b50f1a2171c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,a776155b98edff0c,074f9b50f1a2171c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:53.382 INFO 2981023 --- [ocp-async-8,183c8eb63ddd8f25,6133a7fd0e99dcc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:53.855 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:53.857 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:54.286 INFO 2981023 --- [ocp-schedules-14,6feb0ecf6447ef3e,c7c7e6eb766a48c3] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:04:54.305 WARN 2981023 --- [ocp-schedules-14,6feb0ecf6447ef3e,c7c7e6eb766a48c3] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:04:54.388 INFO 2981023 --- [ocp-schedules-9,4bc349075994e83e,58007aef15ccc473] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:04:55.010 INFO 2981023 --- [ocp-schedules-23,14a3ce6507ca5f5c,5628b5b4c0b32ebe] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:04:55.020 INFO 2981023 --- [ocp-schedules-23,14a3ce6507ca5f5c,5628b5b4c0b32ebe] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:04:55.020 INFO 2981023 --- [ocp-schedules-23,14a3ce6507ca5f5c,5628b5b4c0b32ebe] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:04:55.382 INFO 2981023 --- [ocp-async-4,d771e9e3d6f4c473,d6651aa359d0ad8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:55.836 INFO 2981023 --- [ocp-schedules-14,70430ca522b16b1f,c175c707775eb3a7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:04:55.836 INFO 2981023 --- [ocp-schedules-14,70430ca522b16b1f,c175c707775eb3a7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690420679617835008 2024-09-11 18:04:55.841 INFO 2981023 --- [ocp-schedules-14,70430ca522b16b1f,c175c707775eb3a7] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=110, maxSyncedRecordId=690422503666118656 2024-09-11 18:04:55.842 INFO 2981023 --- [ocp-schedules-14,70430ca522b16b1f,c175c707775eb3a7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:04:55.842 INFO 2981023 --- [ocp-schedules-14,70430ca522b16b1f,c175c707775eb3a7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:04:55.842 INFO 2981023 --- [ocp-schedules-14,70430ca522b16b1f,c175c707775eb3a7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:04:55.842 INFO 2981023 --- [ocp-schedules-14,70430ca522b16b1f,c175c707775eb3a7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:04:55.990 INFO 2981023 --- [ocp-schedules-9,129ec7270a4a2c7e,60f13b7de9c1cfb8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:04:55.996 INFO 2981023 --- [ocp-schedules-9,129ec7270a4a2c7e,60f13b7de9c1cfb8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:04:55.996 INFO 2981023 --- [ocp-schedules-9,129ec7270a4a2c7e,60f13b7de9c1cfb8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:04:56.209 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,45f89aa311c42842,9e8f672636285ad2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=45f89aa311c42842, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:04:56.233 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,45f89aa311c42842,9e8f672636285ad2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=45f89aa311c42842, duration=27 ms] 2024-09-11 18:04:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,a9960b624cd1ff1a,ab2055a39190047d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,a9960b624cd1ff1a,ab2055a39190047d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:56.353 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4173c94f70accc59,cca29086c7eecf8e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4173c94f70accc59, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:56.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:04:56.354236+08:00, lastProcessTime=2024-09-11T18:04:46.341682+08:00 2024-09-11 18:04:56.365 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4173c94f70accc59,cca29086c7eecf8e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4173c94f70accc59, duration=16 ms] 2024-09-11 18:04:56.501 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:04:56.354236+08:00 2024-09-11 18:04:56.857 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:56.858 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:04:57.198 INFO 2981023 --- [ocp-schedules-28,23d1ff41c3cffe9f,9922d765b9bccf73] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:04:57.198+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:04:57.200 INFO 2981023 --- [ocp-schedules-28,23d1ff41c3cffe9f,9922d765b9bccf73] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:04:57.239 INFO 2981023 --- [ocp-schedules-28,23d1ff41c3cffe9f,9922d765b9bccf73] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 51 to SqlAggregateData 2024-09-11 18:04:57.240 INFO 2981023 --- [ocp-schedules-28,23d1ff41c3cffe9f,9922d765b9bccf73] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:04:57.382 INFO 2981023 --- [ocp-async-5,601004a7f5420836,69997afbdf4c50c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:57.472 INFO 2981023 --- [ocp-schedules-8,173468d3236e574b,8a95295820fcffd3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:04:57.472 INFO 2981023 --- [ocp-schedules-8,173468d3236e574b,8a95295820fcffd3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:04:57.486 INFO 2981023 --- [Collect all tenants major compactions1,173468d3236e574b,aa60e587e0ebb2fc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:04:57.486 INFO 2981023 --- [Collect all tenants major compactions2,173468d3236e574b,c1e2c53425913dd4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:04:57.488 INFO 2981023 --- [Collect all tenants major compactions5,173468d3236e574b,1c7997c643135a3d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:04:57.493 INFO 2981023 --- [Collect all tenants major compactions3,173468d3236e574b,0eb75dbb4e7d3580] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:04:57.493 INFO 2981023 --- [Collect all tenants major compactions4,173468d3236e574b,84b0c00e66ddb5b5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:04:57.494 INFO 2981023 --- [Collect all tenants major compactions6,173468d3236e574b,cf9974f2bd783b9c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:04:57.509 INFO 2981023 --- [Collect all tenants major compactions1,173468d3236e574b,aa60e587e0ebb2fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:57.511 INFO 2981023 --- [Collect all tenants major compactions2,173468d3236e574b,c1e2c53425913dd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:57.511 INFO 2981023 --- [Collect all tenants major compactions5,173468d3236e574b,1c7997c643135a3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:57.512 INFO 2981023 --- [Collect all tenants major compactions1,173468d3236e574b,aa60e587e0ebb2fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:04:57.513 INFO 2981023 --- [Collect all tenants major compactions5,173468d3236e574b,1c7997c643135a3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:04:57.514 INFO 2981023 --- [Collect all tenants major compactions3,173468d3236e574b,0eb75dbb4e7d3580] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:57.514 INFO 2981023 --- [Collect all tenants major compactions2,173468d3236e574b,c1e2c53425913dd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:04:57.516 INFO 2981023 --- [Collect all tenants major compactions4,173468d3236e574b,84b0c00e66ddb5b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:57.516 INFO 2981023 --- [Collect all tenants major compactions1,173468d3236e574b,aa60e587e0ebb2fc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:04:57.516 INFO 2981023 --- [Collect all tenants major compactions1,173468d3236e574b,aa60e587e0ebb2fc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:04:57.517 INFO 2981023 --- [Collect all tenants major compactions3,173468d3236e574b,0eb75dbb4e7d3580] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:04:57.517 INFO 2981023 --- [Collect all tenants major compactions4,173468d3236e574b,84b0c00e66ddb5b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:04:57.519 INFO 2981023 --- [Collect all tenants major compactions4,173468d3236e574b,84b0c00e66ddb5b5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:04:57.519 INFO 2981023 --- [Collect all tenants major compactions4,173468d3236e574b,84b0c00e66ddb5b5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:04:57.519 INFO 2981023 --- [Collect all tenants major compactions5,173468d3236e574b,1c7997c643135a3d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:04:57.519 INFO 2981023 --- [Collect all tenants major compactions5,173468d3236e574b,1c7997c643135a3d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:04:57.519 INFO 2981023 --- [Collect all tenants major compactions2,173468d3236e574b,c1e2c53425913dd4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:04:57.519 INFO 2981023 --- [Collect all tenants major compactions2,173468d3236e574b,c1e2c53425913dd4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:04:57.519 INFO 2981023 --- [Collect all tenants major compactions3,173468d3236e574b,0eb75dbb4e7d3580] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:04:57.519 INFO 2981023 --- [Collect all tenants major compactions3,173468d3236e574b,0eb75dbb4e7d3580] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:04:57.520 INFO 2981023 --- [Collect all tenants major compactions6,173468d3236e574b,cf9974f2bd783b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:57.521 INFO 2981023 --- [Collect all tenants major compactions6,173468d3236e574b,cf9974f2bd783b9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:04:57.522 INFO 2981023 --- [Collect all tenants major compactions6,173468d3236e574b,cf9974f2bd783b9c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:04:57.522 INFO 2981023 --- [Collect all tenants major compactions6,173468d3236e574b,cf9974f2bd783b9c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:04:57.523 INFO 2981023 --- [ocp-schedules-8,173468d3236e574b,8a95295820fcffd3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:04:57.779 INFO 2981023 --- [ocp-async-4,19cb366ad254b1e2,1ea01fb43c39951f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:04:58.092 INFO 2981023 --- [ocp-schedules-11,510e56ec223cea83,b89edc33570f48c5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:04:58.093 INFO 2981023 --- [ocp-schedules-11,510e56ec223cea83,b89edc33570f48c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:04:58.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,8cff9368ad33982d,5390dee5ac8efdba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8cff9368ad33982d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:04:58.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,8cff9368ad33982d,5390dee5ac8efdba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8cff9368ad33982d, duration=14 ms] 2024-09-11 18:04:58.129 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:04:58.130 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.133 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:04:58.133 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:04:58.134 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.134 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:04:58.134 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.137 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:04:58.138 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.138 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.138 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:04:58.142 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.142 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.143 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.143 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.151 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.151 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.151 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.152 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.159 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.159 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:04:58.161 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.161 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.166 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.166 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:04:58.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:04:58.172 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.173 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.173 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:04:58.174 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE47F1D772-0-0] 2024-09-11 18:04:58.174 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:04:58.174 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.175 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:04:58.175 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.175 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.178 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.178 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:04:58.185 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.186 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:04:58.192 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.193 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:04:58.193 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.194 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:04:58.194 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.195 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:04:58.195 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.196 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:04:58.196 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.197 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:04:58.197 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.197 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:04:58.197 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE47F1D77C-0-0] 2024-09-11 18:04:58.197 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:04:58.197 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:04:58.198 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.199 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:04:58.200 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.201 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:04:58.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.202 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:04:58.203 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:04:58.203 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.204 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.204 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:04:58.204 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.204 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.205 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:04:58.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:04:58.205 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:04:58.205 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:04:58.205 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.205 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:04:58.206 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:04:58.206 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.207 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:04:58.208 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.208 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:04:58.209 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:04:58.209 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.210 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:04:58.210 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:04:58.210 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.211 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:04:58.211 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:04:58.214 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:04:58.222 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:58.224 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.224 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:04:58.227 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:58.228 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.229 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:04:58.229 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:04:58.232 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:04:58.232 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.233 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:04:58.239 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:04:58.239 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:04:58.240 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:04:58.241 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:04:58.241 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.241 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.241 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:04:58.242 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.242 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.242 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:04:58.243 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:04:58.248 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:04:58.249 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:04:58.249 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:04:58.249 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:04:58.252 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:04:58.254 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:04:58.254 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:04:58.254 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:04:58.256 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:04:58.258 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:04:58.258 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:04:58.258 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:04:58.258 INFO 2981023 --- [ocp-schedules-11,510e56ec223cea83,b89edc33570f48c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:04:58.728 INFO 2981023 --- [ocp-schedules-26,f64e08db340a768f,d6dcbbbfbfce7b54] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:04:58.772 WARN 2981023 --- [scanner-13601512-1,f64e08db340a768f,3dc45d5dfa5f8a43] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:04:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,d9cdd38fa2bd17d2,a83299234c3ba060] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049100, interval=60 2024-09-11 18:04:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,d9cdd38fa2bd17d2,a83299234c3ba060] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:04:59.382 INFO 2981023 --- [ocp-async-6,0c18ac635ce5b2e0,3d9b5f8f6c079eca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:04:59.858 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:04:59.859 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:00.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a3b3369e0438382e,3bb2d2b5f8096ca0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a3b3369e0438382e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:00.128 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a3b3369e0438382e,3bb2d2b5f8096ca0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a3b3369e0438382e, duration=17 ms] 2024-09-11 18:05:00.618 INFO 2981023 --- [ocp-schedules-4,853443f008d90376,14481f41b3a3f191] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:05:00.937 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:00:01.0, nextRun=2024-09-11 18:05:00.0, currentDate=Wed Sep 11 18:05:00 HKT 2024 2024-09-11 18:05:00.937 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:00:01.0, nextRunTime=2024-09-11 18:05:00.0} 2024-09-11 18:05:00.947 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 18:05:00.947 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:05:00.947 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:05:00.947+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:05:00.971 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:00:01.0, nextRun=2024-09-11 18:05:00.0, currentDate=Wed Sep 11 18:05:00 HKT 2024 2024-09-11 18:05:00.971 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:00:01.0, nextRunTime=2024-09-11 18:05:00.0} 2024-09-11 18:05:01.004 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 18:05:01.021 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 18:05:01.022 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 18:05:01.022 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:05:01.022 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,5718a469221bb0bd] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T18:05:01.022+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:05:01.382 INFO 2981023 --- [ocp-async-1,0ff2beabf0bf7d68,2ddda7bbe7c4539c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:01.952 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9480c61574c4dc80] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048240, operation:EXECUTE, state: RUNNING 2024-09-11 18:05:01.962 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9480c61574c4dc80] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048241, operation:EXECUTE, state: RUNNING 2024-09-11 18:05:01.975 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048240, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048210, task_operation=execute, latest_execution_start_time=2024-09-11T18:05:01.944+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048240}, listMap={}}, executor=10.31.35.10 2024-09-11 18:05:01.984 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048241, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048211, task_operation=execute, latest_execution_start_time=2024-09-11T18:05:01.961+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048241}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 18:05:01.986 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 18:05:01.992 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:05:02.010 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 18:05:02.014 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 18:05:02.026 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 18:05:02.027 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 18:05:02.027 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 18:05:02.040 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:05:02.047 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 18:05:02.067 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.perf.sql.internal.SqlParseService : parse 4 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 18:05:02.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,b585a3f8e617f779,0822f49fee1442d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b585a3f8e617f779, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:02.126 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,b585a3f8e617f779,0822f49fee1442d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b585a3f8e617f779, duration=16 ms] 2024-09-11 18:05:02.127 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:05:02.151 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:05:02.171 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:05:02.174 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,9650563a6e37bb94] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048241, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:05:02.188 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 18:05:02.188 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 18:05:02.195 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:05:02.292 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:05:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,7b3f301cef45b542,147d6949f90e5253] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,7b3f301cef45b542,147d6949f90e5253] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:02.354 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:05:02.373 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:05:02.387 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:05:02.424 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:05:02.438 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:05:02.452 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:05:02.457 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 18:05:02.457 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3395.459579986], b:[-5.807309276841022E12] 2024-09-11 18:05:02.460 INFO 2981023 --- [ocp-schedules-20,328bf4dcab587d51,2805b98074e76a27] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:05:02.468 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:05:02.535 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:05:02.575 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:05:02.587 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:05:02.594 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:05:02.603 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:05:02.609 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:05:02.615 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:05:02.616 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 18:05:02.617 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:05:02.630 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:05:02.639 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:05:02.651 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,6bd6b97ed639a7bf] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048240, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:05:02.859 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:02.860 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:02.959 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048210, status:SUCCESS 2024-09-11 18:05:02.961 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048210, status:SUCCESS 2024-09-11 18:05:02.967 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048211, status:SUCCESS 2024-09-11 18:05:02.967 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048211, status:SUCCESS 2024-09-11 18:05:03.382 INFO 2981023 --- [ocp-async-5,5427b5ebb92a3462,964b26ae85d14202] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:04.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,61bb71054bd8b00c,1633479db9cbf64e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=61bb71054bd8b00c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:04.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,61bb71054bd8b00c,1633479db9cbf64e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=61bb71054bd8b00c, duration=10 ms] 2024-09-11 18:05:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,96f80365c6a4e1a0,daa89c518fecc644] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,96f80365c6a4e1a0,daa89c518fecc644] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:05.382 INFO 2981023 --- [ocp-async-8,789fb1ad2e80531f,79766fd801e37afb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:05.860 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:05.860 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:06.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,72f61eb7cb4ca9c0,696a703d9ad51749] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=72f61eb7cb4ca9c0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:06.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,72f61eb7cb4ca9c0,696a703d9ad51749] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=72f61eb7cb4ca9c0, duration=9 ms] 2024-09-11 18:05:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:05:06.352136+08:00, lastProcessTime=2024-09-11T18:04:56.354236+08:00 2024-09-11 18:05:06.488 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:05:06.352136+08:00 2024-09-11 18:05:07.382 INFO 2981023 --- [ocp-async-4,8a14dc0af20781d2,ce3d3d046724caf4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:07.772 INFO 2981023 --- [ocp-schedules-28,0095e07ec8cb1d25,71d9237cb7ca3236] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:05:07.779 INFO 2981023 --- [ocp-async-9,a9c6001ecee9e974,7f64573a8e360eeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:05:07.901 INFO 2981023 --- [ocp-schedules-21,f7f02c0fe5cdd6ac,2e6aa8724d650c8c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:05:07.901+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:05:07.902 INFO 2981023 --- [ocp-schedules-21,f7f02c0fe5cdd6ac,2e6aa8724d650c8c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:05:07.903 INFO 2981023 --- [ocp-schedules-21,f7f02c0fe5cdd6ac,2e6aa8724d650c8c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:05:07.904 INFO 2981023 --- [ocp-schedules-21,f7f02c0fe5cdd6ac,2e6aa8724d650c8c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:05:08.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,22c987099b455547,e7fe484e4e004979] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=22c987099b455547, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:08.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,22c987099b455547,e7fe484e4e004979] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=22c987099b455547, duration=10 ms] 2024-09-11 18:05:08.179 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:01:07.884, end=2024-09-11T18:03:08.175} 2024-09-11 18:05:08.184 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clust 2024-09-11 18:05:08.184 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:05:08.185 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:07.884, end=2024-09-11T18:03:08.175},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:05:08.185 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:05:08.185 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:05:08.186 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:05:08.188 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:07.884, end=2024-09-11T18:03:08.175},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:05:08.188 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:05:08.188 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:05:08.201 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:05:08.201 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:05:08.202 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:07.884, end=2024-09-11T18:03:08.175},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:05:08.202 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:05:08.202 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:05:08.203 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:05:08.205 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:07.884, end=2024-09-11T18:03:08.175},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:05:08.205 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:05:08.205 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:05:08.253 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708253000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.253Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708256000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.256Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.261 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708261000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.261Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.263 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708263000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.263Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.265 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708265000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.265Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.266 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708266000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.266Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.267 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708267000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.267Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.267 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708267000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.267Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,73d894690a7ca931,3300606317b17d14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:08.315 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.315 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.324 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.324 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.324 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.324 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.325 INFO 2981023 --- [prometheus-minute-schedule-1,73d894690a7ca931,3300606317b17d14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:08.334 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.334 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.337 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.337 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.341 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.341 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.344 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 66 2024-09-11 18:05:08.344 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.349 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:05:08.349 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.351 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.351 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.351 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.351 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.352 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.352 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.354 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.354 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.356 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.356 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.361 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.361 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.363 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.363 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.364 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:05:08.364 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.367 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.367 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.367 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.367 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.370 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.370 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.370 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.371 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.371 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.374 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.374 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.374 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.374 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.374 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.376 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.376 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.376 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.376 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.376 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.377 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.377 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.378 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.378 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.379 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.379 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.381 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.381 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.383 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.256, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.383 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.384 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.253, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.386 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.386 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.388 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.390 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.267, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.391 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.263, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.393 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.393 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.393 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.393 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.395 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.395 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.396 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.261, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.396 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.397 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.398 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.403 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.266, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.404 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.265, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.409 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.267, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.426 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708426000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.426Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.427 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708427000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.427Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.434 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708434000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.434Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.434 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708434000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.434Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.436 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708436000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.436Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.437 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:05:08.437 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.446 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:05:08.446 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.447 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708447000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.447Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.447 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:05:08.447 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.450 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.450 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.453 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708453000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.453Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.458 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.458 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.460 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.460 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.461 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708461000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.461Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.463 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.463 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.464 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.464 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.465 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.465 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.466 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.466 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.467 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.467 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.472 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.472 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.473 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.473 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.476 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.476 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.477 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.477 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.479 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.480 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.480 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.483 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.483 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.483 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.483 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.484 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.484 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.486 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.434, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.486 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.486 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.486 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.486 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.486 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.486 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.489 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.489 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.492 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.492 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.492 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.492 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.494 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.494 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.494 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.501 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.427, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.503 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.503 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.503 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.503 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.504 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.504 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.504 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.504 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.505 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.505 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.507 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.507 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.507 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.508 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.508 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.509 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.509 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.509 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.513 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.426, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.513 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.447, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.514 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.434, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.514 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.436, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.515 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.461, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.516 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.453, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.518 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708518000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.518Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.527 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:05:08.527 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.532 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.532 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.533 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.533 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.539 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.539 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.540 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.544 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.518, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.544 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:05:08.545 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:07.884, end=2024-09-11T18:03:08.175},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:05:08.545 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:05:08.545 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:05:08.547 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:05:08.548 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:07.884, end=2024-09-11T18:03:08.175},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:05:08.548 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:05:08.548 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:05:08.559 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:05:08.577 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708577000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.577Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.577 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962708577000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=AD63995D4A6E8C3787D81BDA88E06774, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:05:08.577Z, endTime=2024-09-11T10:03:08.175Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:05:08.579 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:05:08.579 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.587 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.587 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.589 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.589 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.592 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 258 2024-09-11 18:05:08.592 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:05:08.597 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.597 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.598 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.598 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.598 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:05:08.598 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.577, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.601 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:05:08.601 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:05:08.606 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:05:08.606 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726048988175000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 18:05:08.607 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:05:08.611 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:05:08.577, end=2024-09-11T18:03:08.175}, clusterId=2, tenantId=4, sqlId=AD63995D4A6E8C3787D81BDA88E06774, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:05:08.611 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:05:08.860 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:08.861 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:08.922 INFO 2981023 --- [ocp-schedules-3,2fce0596d860aec5,a9467ec7cbb505b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:05:08.998 INFO 2981023 --- [alarm-detect-10,2fce0596d860aec5,70963186375bf8dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645330, data is empty. 2024-09-11 18:05:09.006 INFO 2981023 --- [alarm-detect-10,2fce0596d860aec5,70963186375bf8dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645381, data is empty. 2024-09-11 18:05:09.007 WARN 2981023 --- [scanner-13601928-1,2fce0596d860aec5,7aaa936e10b508c2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:05:09.009 INFO 2981023 --- [alarm-detect-10,2fce0596d860aec5,70963186375bf8dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645381, data is empty. 2024-09-11 18:05:09.029 INFO 2981023 --- [alarm-detect-5,2fce0596d860aec5,00711b920c2627f0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645330, data is empty. 2024-09-11 18:05:09.032 INFO 2981023 --- [alarm-detect-13,2fce0596d860aec5,055fe2b62037ccdd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645330, data is empty. 2024-09-11 18:05:09.038 INFO 2981023 --- [alarm-detect-13,2fce0596d860aec5,055fe2b62037ccdd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645381, data is empty. 2024-09-11 18:05:09.040 INFO 2981023 --- [alarm-detect-5,2fce0596d860aec5,00711b920c2627f0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645381, data is empty. 2024-09-11 18:05:09.043 INFO 2981023 --- [alarm-detect-13,2fce0596d860aec5,055fe2b62037ccdd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645381, data is empty. 2024-09-11 18:05:09.046 INFO 2981023 --- [alarm-detect-5,2fce0596d860aec5,00711b920c2627f0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645381, data is empty. 2024-09-11 18:05:09.132 INFO 2981023 --- [alarm-detect-17,2fce0596d860aec5,e4937db52fa8d4af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645330, data is empty. 2024-09-11 18:05:09.141 INFO 2981023 --- [alarm-detect-17,2fce0596d860aec5,e4937db52fa8d4af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645381, data is empty. 2024-09-11 18:05:09.147 INFO 2981023 --- [alarm-detect-17,2fce0596d860aec5,e4937db52fa8d4af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645381, data is empty. 2024-09-11 18:05:09.382 INFO 2981023 --- [ocp-async-5,a4eb730dcd13bce5,5a2ce96a52f774a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:10.003 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:05:10.004 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:05:10.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,67775c8236caab42,aa39842cdeef482a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=67775c8236caab42, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:10.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,67775c8236caab42,aa39842cdeef482a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=67775c8236caab42, duration=9 ms] 2024-09-11 18:05:10.918 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:05:10.920 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:05:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,6e2a93e978168c1b,2d34016e83ba0d12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,6e2a93e978168c1b,2d34016e83ba0d12] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:11.383 INFO 2981023 --- [ocp-async-9,948801491f2b2405,0c6d1e2f97cc77b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:11.861 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:11.862 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:12.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d28cffd369867e27,c53152761bbbc9d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d28cffd369867e27, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:12.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d28cffd369867e27,c53152761bbbc9d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d28cffd369867e27, duration=9 ms] 2024-09-11 18:05:12.463 INFO 2981023 --- [ocp-schedules-30,7779de301157cdd0,20f78372a876997d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:05:13.383 INFO 2981023 --- [ocp-async-6,97e590d9e7823c14,def725c544b598f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:14.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,17fa5bfe2f7b4d25,eb21ea77c3edc00b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=17fa5bfe2f7b4d25, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:14.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,17fa5bfe2f7b4d25,eb21ea77c3edc00b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=17fa5bfe2f7b4d25, duration=17 ms] 2024-09-11 18:05:14.128 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049040, nodeCount=480, affectRows=480 2024-09-11 18:05:14.161 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049040, nodeCount=1000, affectRows=1000 2024-09-11 18:05:14.164 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049040, nodeCount=999, validCount=999 2024-09-11 18:05:14.165 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049040, nodeCount=1000, affectRows=1000 2024-09-11 18:05:14.167 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049040, nodeCount=1000, validCount=1000 2024-09-11 18:05:14.181 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049040, nodeCount=282, affectRows=282 2024-09-11 18:05:14.182 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049040, nodeCount=280, validCount=280 2024-09-11 18:05:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,5f6e0c8c14b4f530,b86ba6d373afcf7f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,f4b9c42f66a5ab09,570b4d49c0c41906] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:05:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,5f6e0c8c14b4f530,b86ba6d373afcf7f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:14.862 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:14.862 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:15.023 INFO 2981023 --- [ocp-schedules-9,2902b1ba9a607904,dba2583fef5efa58] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:05:15.031 INFO 2981023 --- [ocp-schedules-9,2902b1ba9a607904,dba2583fef5efa58] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:05:15.031 INFO 2981023 --- [ocp-schedules-9,2902b1ba9a607904,dba2583fef5efa58] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:05:15.383 INFO 2981023 --- [ocp-async-7,7a82b2f6073e7f4d,ad88ec270e9950ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:15.882 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049040, nodeCount=480, validCount=480 2024-09-11 18:05:16.019 INFO 2981023 --- [ocp-schedules-26,5e50f7b42db5a3a4,969b59ab6cb3f73d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:05:16.019 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:05:16.020 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:05:16.020 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:05:16.021 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:05:16.022 INFO 2981023 --- [ocp-schedules-13,087d96b94ead1303,98698ba12f709ef0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:05:16.023 INFO 2981023 --- [host-check-clockdiff1,087d96b94ead1303,090951485008c10d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:05:16.023 INFO 2981023 --- [host-check-clockdiff2,087d96b94ead1303,ac4972c8474e1fce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:05:16.024 INFO 2981023 --- [host-check-clockdiff3,087d96b94ead1303,af46865008df42db] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:05:16.027 INFO 2981023 --- [host-check-clockdiff4,087d96b94ead1303,db1777480942d201] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:05:16.038 INFO 2981023 --- [host-check-clockdiff3,087d96b94ead1303,af46865008df42db] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049116 0 0 , error= 2024-09-11 18:05:16.050 INFO 2981023 --- [host-check-clockdiff2,087d96b94ead1303,ac4972c8474e1fce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049116 5 5 , error= 2024-09-11 18:05:16.064 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:05:16.064 INFO 2981023 --- [ocp-schedules-6,6c47a1e4053ab9d3,2023205ca9d6be56] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:05:16.064 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:05:16.065 INFO 2981023 --- [host-check-agent-status1,6c47a1e4053ab9d3,8333555804d9bbd1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:05:16.065 INFO 2981023 --- [host-check-agent-status2,6c47a1e4053ab9d3,7b1957b10b344e52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:05:16.065 INFO 2981023 --- [host-check-agent-status3,6c47a1e4053ab9d3,f28f648fe6d14ab6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:05:16.065 INFO 2981023 --- [host-check-agent-status4,6c47a1e4053ab9d3,ffc6275b95973ba5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:05:16.069 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:05:16.070 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:05:16.070 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:05:16.071 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:05:16.079 INFO 2981023 --- [host-check-agent-status3,6c47a1e4053ab9d3,f28f648fe6d14ab6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:05:16.080 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:05:16.080 INFO 2981023 --- [host-check-agent-status3,6c47a1e4053ab9d3,f28f648fe6d14ab6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:05:16.106 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:05:16.168 INFO 2981023 --- [host-check-agent-status2,6c47a1e4053ab9d3,7b1957b10b344e52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:05:16.180 INFO 2981023 --- [host-check-agent-status1,6c47a1e4053ab9d3,8333555804d9bbd1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:05:16.181 INFO 2981023 --- [host-check-agent-status4,6c47a1e4053ab9d3,ffc6275b95973ba5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:05:16.182 INFO 2981023 --- [host-check-agent-status2,6c47a1e4053ab9d3,7b1957b10b344e52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:05:16.184 INFO 2981023 --- [host-check-agent-status4,6c47a1e4053ab9d3,ffc6275b95973ba5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:05:16.192 INFO 2981023 --- [ocp-schedules-27,6cf049509b65dccb,82580f33d4225efd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:05:16.193 INFO 2981023 --- [ocp-schedules-15,ccdbfe95fd14a821,6bbc1e7f1cc5a93b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:05:16.194 INFO 2981023 --- [host-check-agent-status1,6c47a1e4053ab9d3,8333555804d9bbd1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:05:16.197 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0a7b7be659c78d49,893a70f91604479f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0a7b7be659c78d49, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:16.202 INFO 2981023 --- [ocp-schedules-28,45a3ff00ebcccbae,525e3607d07a741a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:05:16.208 INFO 2981023 --- [ocp-schedules-16,5d2b983675386741,c224423ba3c83fb1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:05:16.257 INFO 2981023 --- [ocp-schedules-27,6cf049509b65dccb,82580f33d4225efd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:05:16.257 INFO 2981023 --- [ocp-schedules-15,ccdbfe95fd14a821,6bbc1e7f1cc5a93b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:05:16.274 INFO 2981023 --- [ocp-schedules-28,45a3ff00ebcccbae,525e3607d07a741a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:05:16.279 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0a7b7be659c78d49,893a70f91604479f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0a7b7be659c78d49, duration=172 ms] 2024-09-11 18:05:16.307 INFO 2981023 --- [host-check-agent-status3,6c47a1e4053ab9d3,f28f648fe6d14ab6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:05:16.315 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.316 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:05:16.316 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:05:16.316 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.317 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:05:16.317 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:05:16.318 INFO 2981023 --- [ocp-schedules-15,ccdbfe95fd14a821,6bbc1e7f1cc5a93b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:05:16.318 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049042737000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:05:16.319 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:05:16.319078245+08:00 2024-09-11 18:05:16.319 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:05:16.319 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:05:16.319+08:00 2024-09-11 18:05:16.322 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:05:16.319+08:00 and status AVAILABLE 2024-09-11 18:05:16.324 INFO 2981023 --- [host-check-agent-status3,6c47a1e4053ab9d3,f28f648fe6d14ab6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:05:16.324 INFO 2981023 --- [host-check-agent-status3,6c47a1e4053ab9d3,f28f648fe6d14ab6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:05:16.324 INFO 2981023 --- [host-check-agent-status3,6c47a1e4053ab9d3,f28f648fe6d14ab6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:05:16.326 INFO 2981023 --- [ocp-schedules-27,6cf049509b65dccb,82580f33d4225efd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:05:16.339 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:05:16.340 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:05:16.347 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:05:16.352798922+08:00 2024-09-11 18:05:16.356 INFO 2981023 --- [ocp-schedules-18,3448b008caa0be78,eb746ba8a84fb784] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:05:16.357 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:05:16.347+08:00 2024-09-11 18:05:16.359 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:05:16.356783+08:00, lastProcessTime=2024-09-11T18:05:06.352136+08:00 2024-09-11 18:05:16.361 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:05:16.347+08:00 and status AVAILABLE 2024-09-11 18:05:16.362 INFO 2981023 --- [host-check-agent-status4,6c47a1e4053ab9d3,ffc6275b95973ba5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:05:16.362 INFO 2981023 --- [host-check-agent-status1,6c47a1e4053ab9d3,8333555804d9bbd1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:05:16.363 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:05:16.366 INFO 2981023 --- [host-check-agent-status1,6c47a1e4053ab9d3,8333555804d9bbd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:05:16.366 INFO 2981023 --- [host-check-agent-status1,6c47a1e4053ab9d3,8333555804d9bbd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:05:16.366 INFO 2981023 --- [host-check-agent-status1,6c47a1e4053ab9d3,8333555804d9bbd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:05:16.366 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:05:16.366063130+08:00 2024-09-11 18:05:16.367 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:05:16.366+08:00 2024-09-11 18:05:16.368 INFO 2981023 --- [host-check-agent-status4,6c47a1e4053ab9d3,ffc6275b95973ba5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:05:16.368 INFO 2981023 --- [host-check-agent-status4,6c47a1e4053ab9d3,ffc6275b95973ba5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:05:16.368 INFO 2981023 --- [host-check-agent-status4,6c47a1e4053ab9d3,ffc6275b95973ba5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:05:16.369 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.369 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:16.369 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:05:16.366+08:00 and status AVAILABLE 2024-09-11 18:05:16.371 INFO 2981023 --- [host-check-agent-status2,6c47a1e4053ab9d3,7b1957b10b344e52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:05:16.373 INFO 2981023 --- [host-check-agent-status2,6c47a1e4053ab9d3,7b1957b10b344e52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:05:16.373 INFO 2981023 --- [host-check-agent-status2,6c47a1e4053ab9d3,7b1957b10b344e52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:05:16.373 INFO 2981023 --- [host-check-agent-status2,6c47a1e4053ab9d3,7b1957b10b344e52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:05:16.374 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:05:16.374 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:05:16.377 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:05:16.382496987+08:00 2024-09-11 18:05:16.377 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:05:16.378 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.378 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:16.380 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:05:16.377+08:00 2024-09-11 18:05:16.381 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:05:16.382 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:05:16.377+08:00 and status AVAILABLE 2024-09-11 18:05:16.383 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:05:16.388 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:05:16.389 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.389 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:05:16.389 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:16.395 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:05:16.403 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:05:16.404 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:05:16.404 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:05:16.406 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:05:16.413 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:05:16.414 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:16.414 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.415 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:05:16.437 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:05:16.443 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:05:16.444 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:05:16.449 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:05:16.463 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:05:16.463 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049042737000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:05:16.464 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:05:16.469 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:05:16.478 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:05:16.485 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:05:16.488 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:05:16.491 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:16.491 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:05:16.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.496 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:16.499 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:05:16.499 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.503 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:05:16.504 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:05:16.507 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:05:16.562 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:05:16.562 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:05:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:05:16.573 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:05:16.575 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:05:16.591 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:05:16.596 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:05:16.599 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:05:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:05:16.604 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:16.605 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.606 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:16.606 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:05:16.607 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:05:16.607 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.610 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:05:16.610 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:05:16.614 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:05:16.615 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:16.616 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:05:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:05:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:05:16.628 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:05:16.647 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:05:16.658 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:05:16.664 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:05:16.664 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:05:16.669 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:05:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:05:16.672 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:05:16.673 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:05:16.688 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:05:16.704 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:05:16.708 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:05:16.717 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:05:16.719 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:05:16.731 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:05:16.749 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:05:16.757 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:05:16.760 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:05:16.764 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:05:16.765 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:05:16.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:16.770 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.770 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:05:16.772 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:05:16.773 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:05:16.783 INFO 2981023 --- [host-check-host-status4,5e50f7b42db5a3a4,f5de32267a6bcc93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:05:16.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:05:16.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:05:16.789 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:05:16.789 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:05:16.791 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:05:16.798 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:05:16.356783+08:00 2024-09-11 18:05:16.798 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:05:16.803 INFO 2981023 --- [host-check-host-status3,5e50f7b42db5a3a4,6f67751343f0e097] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:05:16.803 INFO 2981023 --- [host-check-host-status2,5e50f7b42db5a3a4,47853678a2867d10] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:05:16.807 INFO 2981023 --- [host-check-host-status1,5e50f7b42db5a3a4,80e89b499f7815f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:05:16.828 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:05:16.868 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:16.869 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:16.869 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:05:16.877 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:05:16.877 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:05:16.877 INFO 2981023 --- [ocp-schedules-28,45a3ff00ebcccbae,525e3607d07a741a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:05:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,778340d2967329aa,e591bc34f6a06954] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,778340d2967329aa,e591bc34f6a06954] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:17.383 INFO 2981023 --- [ocp-async-5,accabfcecdef199e,1d2b754a656d2d4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:17.780 INFO 2981023 --- [ocp-async-1,fe264130a6a60999,e64bebb66276e9d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:05:17.863 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:17.863 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:18.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,dfa27bb117c8b276,9b540646b6205944] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=dfa27bb117c8b276, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:18.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,dfa27bb117c8b276,9b540646b6205944] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=dfa27bb117c8b276, duration=9 ms] 2024-09-11 18:05:19.191 INFO 2981023 --- [ocp-schedules-30,bd092ff281025eea,a9e8b852d3d5f9da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:05:19.240 WARN 2981023 --- [scanner-13603737-1,bd092ff281025eea,4ab1373140130d64] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:05:19.383 INFO 2981023 --- [ocp-async-4,cda2365a6a0fc878,be52d0ee14fc9b52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:20.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,64114d868f1c11fa,74ea8c916b5a819a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=64114d868f1c11fa, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:20.127 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,64114d868f1c11fa,74ea8c916b5a819a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=64114d868f1c11fa, duration=23 ms] 2024-09-11 18:05:20.166 INFO 2981023 --- [host-check-clockdiff1,087d96b94ead1303,090951485008c10d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049120 6 6 , error= 2024-09-11 18:05:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,0c4ef59b20895741,da3711f49f682bc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,0c4ef59b20895741,da3711f49f682bc7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:20.863 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:20.865 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:21.383 INFO 2981023 --- [ocp-async-1,fc53eb0b463d2b1e,b238eef1fe4e3a1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:22.025 INFO 2981023 --- [host-check-clockdiff4,087d96b94ead1303,db1777480942d201] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049122 0 0 , error= 2024-09-11 18:05:22.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d8be01dd02325e3c,441e251a0ff262a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d8be01dd02325e3c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:22.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,d8be01dd02325e3c,441e251a0ff262a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d8be01dd02325e3c, duration=9 ms] 2024-09-11 18:05:22.466 INFO 2981023 --- [ocp-schedules-10,4d10f03ac9bb7992,03edb33d881d1696] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:05:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,7d40e922bd26f172,db7eebe48ccaaaa0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,7d40e922bd26f172,db7eebe48ccaaaa0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:23.383 INFO 2981023 --- [ocp-async-5,ca56bff4e64654ab,8928b4b03e06634a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:23.865 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:23.866 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:24.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,52e0ee59114f8867,7e0c85357f1d6ab4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=52e0ee59114f8867, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:24.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,52e0ee59114f8867,7e0c85357f1d6ab4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=52e0ee59114f8867, duration=10 ms] 2024-09-11 18:05:24.391 INFO 2981023 --- [ocp-schedules-20,53cac0e29f75e625,b6aac7698e5206f6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:05:24.816 INFO 2981023 --- [ocp-schedules-23,c03a0afee83f260d,8518435eaade48de] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:05:24.816 INFO 2981023 --- [ocp-schedules-23,c03a0afee83f260d,8518435eaade48de] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:05:24.816 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:05:24.816 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:05:24.816 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:05:24.819 INFO 2981023 --- [ocp-schedules-23,c03a0afee83f260d,8518435eaade48de] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:05:24.820 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:05:24.822 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:05:24.824 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:05:24.825 INFO 2981023 --- [ocp-schedules-23,c03a0afee83f260d,8518435eaade48de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:05:24.826 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:05:24.827 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:05:24.828 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:05:24.850 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:05:24.852 INFO 2981023 --- [ocp-schedules-23,c03a0afee83f260d,8518435eaade48de] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:05:24.852 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:05:24.852 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:05:24.857 INFO 2981023 --- [ocp-schedules-23,c03a0afee83f260d,8518435eaade48de] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:05:25.384 INFO 2981023 --- [ocp-async-9,782704ff500bf9d5,dabd2376d7a4e547] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:26.099 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,061c3074096dc053,5a6d3fcab24f3048] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=061c3074096dc053, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:26.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,061c3074096dc053,5a6d3fcab24f3048] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=061c3074096dc053, duration=9 ms] 2024-09-11 18:05:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,df5f8b1ba33334f9,9195d030f8590ec1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,df5f8b1ba33334f9,9195d030f8590ec1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:26.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:05:26.353606+08:00, lastProcessTime=2024-09-11T18:05:16.356783+08:00 2024-09-11 18:05:26.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:05:26.353606+08:00 2024-09-11 18:05:26.866 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:26.866 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:27.384 INFO 2981023 --- [ocp-async-6,3d53d6b3679b1f98,874c7c5a6c59ef02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:27.780 INFO 2981023 --- [ocp-async-3,1140f3df317d809a,f15f2ab1a079a658] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:05:28.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,fbeec6685e982fa0,bfc8d896191903c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fbeec6685e982fa0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:28.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,fbeec6685e982fa0,bfc8d896191903c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fbeec6685e982fa0, duration=9 ms] 2024-09-11 18:05:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,d637ddc6da8e5568,05369b1ec4bc76cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,d637ddc6da8e5568,05369b1ec4bc76cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:29.378 INFO 2981023 --- [ocp-schedules-22,12d886ba83e58325,0b6127604451d288] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:05:29.384 INFO 2981023 --- [ocp-async-5,56ed545dcdc6f21b,c42b2e9cf77ff8ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:29.413 WARN 2981023 --- [scanner-13603912-1,12d886ba83e58325,5cbb5b805fc18fa6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:05:29.446 INFO 2981023 --- [alarm-detect-2,12d886ba83e58325,2d1ba09915518dd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645381, data is empty. 2024-09-11 18:05:29.450 INFO 2981023 --- [alarm-detect-2,12d886ba83e58325,2d1ba09915518dd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645381, data is empty. 2024-09-11 18:05:29.454 INFO 2981023 --- [alarm-detect-2,12d886ba83e58325,2d1ba09915518dd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645381, data is empty. 2024-09-11 18:05:29.472 INFO 2981023 --- [alarm-detect-20,12d886ba83e58325,054012109d26ebfb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645381, data is empty. 2024-09-11 18:05:29.476 INFO 2981023 --- [alarm-detect-17,12d886ba83e58325,e5e910a0b7255604] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645381, data is empty. 2024-09-11 18:05:29.477 INFO 2981023 --- [alarm-detect-20,12d886ba83e58325,054012109d26ebfb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645381, data is empty. 2024-09-11 18:05:29.481 INFO 2981023 --- [alarm-detect-20,12d886ba83e58325,054012109d26ebfb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645381, data is empty. 2024-09-11 18:05:29.482 INFO 2981023 --- [alarm-detect-17,12d886ba83e58325,e5e910a0b7255604] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645381, data is empty. 2024-09-11 18:05:29.487 INFO 2981023 --- [alarm-detect-17,12d886ba83e58325,e5e910a0b7255604] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645381, data is empty. 2024-09-11 18:05:29.620 INFO 2981023 --- [alarm-detect-2,12d886ba83e58325,41cbb1b547f07bd3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645381, data is empty. 2024-09-11 18:05:29.624 INFO 2981023 --- [alarm-detect-2,12d886ba83e58325,41cbb1b547f07bd3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645381, data is empty. 2024-09-11 18:05:29.629 INFO 2981023 --- [alarm-detect-2,12d886ba83e58325,41cbb1b547f07bd3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645381, data is empty. 2024-09-11 18:05:29.867 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:29.867 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:30.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,641160999296728e,224f32f76d590ca9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=641160999296728e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:30.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,641160999296728e,224f32f76d590ca9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=641160999296728e, duration=12 ms] 2024-09-11 18:05:31.384 INFO 2981023 --- [ocp-async-3,4a61679041c0bc80,121f2b8fbeff340b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:32.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,075a96f80354078f,00509d039efac9b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=075a96f80354078f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:32.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,075a96f80354078f,00509d039efac9b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=075a96f80354078f, duration=9 ms] 2024-09-11 18:05:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,0d716ee9ee29ff95,e9ffba6d04e6db7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,0d716ee9ee29ff95,e9ffba6d04e6db7c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:32.468 INFO 2981023 --- [ocp-schedules-25,51f8388ef67303f2,9bbe6dec551d36fd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:05:32.867 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:32.868 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:33.384 INFO 2981023 --- [ocp-async-6,4f702df8aaa7ce87,410a9658b84af878] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:34.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,8fcca61d6fa58a42,4406875d66fe9ac1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8fcca61d6fa58a42, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:34.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,8fcca61d6fa58a42,4406875d66fe9ac1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8fcca61d6fa58a42, duration=10 ms] 2024-09-11 18:05:35.033 INFO 2981023 --- [ocp-schedules-22,6b3c5fa3e0d1dba8,cffd65b087145d63] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:05:35.045 INFO 2981023 --- [ocp-schedules-22,6b3c5fa3e0d1dba8,cffd65b087145d63] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:05:35.045 INFO 2981023 --- [ocp-schedules-22,6b3c5fa3e0d1dba8,cffd65b087145d63] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:05:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,0d3d9e483f9305b4,81174a8f295d980d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,0d3d9e483f9305b4,81174a8f295d980d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:35.384 INFO 2981023 --- [ocp-async-7,5ab68314d9f61848,3c5f53be4efaa724] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:35.868 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:35.868 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:36.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,40f33c6640646cbb,2ccbd7e928d47fe6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=40f33c6640646cbb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:36.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,40f33c6640646cbb,2ccbd7e928d47fe6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=40f33c6640646cbb, duration=9 ms] 2024-09-11 18:05:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:05:36.337388+08:00, lastProcessTime=2024-09-11T18:05:26.353606+08:00 2024-09-11 18:05:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:05:36.337388+08:00 2024-09-11 18:05:37.314 INFO 2981023 --- [prometheus-cache-monitor1,af4c6bde7966489c,587222acb4cea316] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8432,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5148} 2024-09-11 18:05:37.315 INFO 2981023 --- [prometheus-cache-monitor1,af4c6bde7966489c,587222acb4cea316] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:05:37.384 INFO 2981023 --- [ocp-async-10,71bf1745e690c1f0,f5798434c0b70a88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:37.780 INFO 2981023 --- [ocp-async-8,58954a3060faa4ae,7bb9977b4f1717bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:05:38.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,b55d29d07e912fd2,5ca0c9b05709ab06] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b55d29d07e912fd2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:38.130 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,b55d29d07e912fd2,5ca0c9b05709ab06] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b55d29d07e912fd2, duration=9 ms] 2024-09-11 18:05:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,1dd392c1769bd0c2,d7651e85679121d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,1dd392c1769bd0c2,d7651e85679121d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:38.869 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:38.870 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:39.385 INFO 2981023 --- [ocp-async-6,b4a4c04b0a2a955a,dfca471599265a5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:39.681 INFO 2981023 --- [ocp-schedules-29,2983c8dfc5e95dbb,e55a47a703d561ec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:05:39.722 WARN 2981023 --- [scanner-13604241-1,2983c8dfc5e95dbb,6d1e746871e75ce0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:05:40.003 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:05:40.003 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:05:40.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e8bf6ed51f544c34,0e9f644d2b2b462c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e8bf6ed51f544c34, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:40.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,e8bf6ed51f544c34,0e9f644d2b2b462c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e8bf6ed51f544c34, duration=9 ms] 2024-09-11 18:05:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,f96693e88e89665f,65398de4477d34ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,f96693e88e89665f,65398de4477d34ef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:41.385 INFO 2981023 --- [ocp-async-8,ae89e920ceb89e37,adaf6f3f763d0d03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:41.875 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:41.876 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:42.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,6f55378648b3d7eb,db8a17b072e73810] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6f55378648b3d7eb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:42.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,6f55378648b3d7eb,db8a17b072e73810] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6f55378648b3d7eb, duration=10 ms] 2024-09-11 18:05:42.471 INFO 2981023 --- [ocp-schedules-21,88a42b525f7dc11b,9473465d4b71d0fb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:05:43.385 INFO 2981023 --- [ocp-async-9,ab23dcef018aa2dd,a6ff6284e74caf98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:44.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ced5fcaa80bf0a34,131a854784c540ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ced5fcaa80bf0a34, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:44.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ced5fcaa80bf0a34,131a854784c540ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ced5fcaa80bf0a34, duration=9 ms] 2024-09-11 18:05:44.300 INFO 2981023 --- [ocp-schedules-3,f7384e2ee8a43ce4,573da829d7c132e3] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:05:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,147aadad1546d0ce,fedcd277c82db466] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,147aadad1546d0ce,fedcd277c82db466] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:44.324 WARN 2981023 --- [ocp-schedules-3,f7384e2ee8a43ce4,573da829d7c132e3] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:05:44.876 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:44.877 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:45.101 INFO 2981023 --- [prometheus-second-schedule-1,585869f048ae18cf,db0979531c4e8d8d] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:05:45.385 INFO 2981023 --- [ocp-async-4,c23e623f4aef12df,1ec81aef70f43b13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:46.016 INFO 2981023 --- [ocp-schedules-23,7e74d7ac8a038ddb,54ec3c7f28b25f95] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:05:46.016 INFO 2981023 --- [host-check-clockdiff1,7e74d7ac8a038ddb,06b646e2fe4fdd74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:05:46.016 INFO 2981023 --- [host-check-clockdiff2,7e74d7ac8a038ddb,14e5f1cfda8c626f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:05:46.017 INFO 2981023 --- [host-check-clockdiff3,7e74d7ac8a038ddb,d732155bc705e604] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:05:46.017 INFO 2981023 --- [ocp-schedules-19,009d24f18bacf750,19a9b5cbb36e778b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:05:46.018 INFO 2981023 --- [host-check-clockdiff4,7e74d7ac8a038ddb,ddc3c0aeed87ae37] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:05:46.019 INFO 2981023 --- [host-check-agent-status1,009d24f18bacf750,82e83601c9efdd27] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:05:46.024 INFO 2981023 --- [host-check-agent-status2,009d24f18bacf750,73c05c9646da17a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:05:46.024 INFO 2981023 --- [host-check-agent-status4,009d24f18bacf750,f3cfb4bbbcb1b208] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:05:46.025 INFO 2981023 --- [host-check-agent-status3,009d24f18bacf750,eafc72c08dd2e07c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:05:46.025 INFO 2981023 --- [ocp-schedules-30,24ea792c35f7ce76,f8dd8a159d351cbd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:05:46.027 INFO 2981023 --- [host-check-agent-status1,009d24f18bacf750,82e83601c9efdd27] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:05:46.029 INFO 2981023 --- [host-check-agent-status1,009d24f18bacf750,82e83601c9efdd27] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:05:46.030 INFO 2981023 --- [host-check-agent-status4,009d24f18bacf750,f3cfb4bbbcb1b208] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:05:46.031 INFO 2981023 --- [host-check-agent-status4,009d24f18bacf750,f3cfb4bbbcb1b208] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:05:46.032 INFO 2981023 --- [host-check-agent-status2,009d24f18bacf750,73c05c9646da17a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:05:46.032 INFO 2981023 --- [host-check-agent-status3,009d24f18bacf750,eafc72c08dd2e07c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:05:46.033 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:05:46.033 INFO 2981023 --- [host-check-agent-status3,009d24f18bacf750,eafc72c08dd2e07c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:05:46.034 INFO 2981023 --- [host-check-agent-status2,009d24f18bacf750,73c05c9646da17a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:05:46.034 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:05:46.034 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:05:46.039 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:05:46.045 INFO 2981023 --- [host-check-clockdiff3,7e74d7ac8a038ddb,d732155bc705e604] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049146 0 0 , error= 2024-09-11 18:05:46.050 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:05:46.051 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:05:46.051 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:05:46.053 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:05:46.054 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:05:46.054 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:05:46.055 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:05:46.056 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:05:46.063 INFO 2981023 --- [host-check-agent-status1,009d24f18bacf750,82e83601c9efdd27] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:05:46.067 INFO 2981023 --- [host-check-agent-status4,009d24f18bacf750,f3cfb4bbbcb1b208] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:05:46.068 INFO 2981023 --- [host-check-agent-status1,009d24f18bacf750,82e83601c9efdd27] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:05:46.068 INFO 2981023 --- [host-check-agent-status1,009d24f18bacf750,82e83601c9efdd27] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:05:46.068 INFO 2981023 --- [host-check-agent-status1,009d24f18bacf750,82e83601c9efdd27] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:05:46.071 INFO 2981023 --- [host-check-agent-status4,009d24f18bacf750,f3cfb4bbbcb1b208] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:05:46.071 INFO 2981023 --- [host-check-agent-status4,009d24f18bacf750,f3cfb4bbbcb1b208] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:05:46.071 INFO 2981023 --- [host-check-agent-status4,009d24f18bacf750,f3cfb4bbbcb1b208] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:05:46.073 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:05:46.075 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:05:46.080872299+08:00 2024-09-11 18:05:46.076 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:05:46.075+08:00 2024-09-11 18:05:46.079 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:05:46.075+08:00 and status AVAILABLE 2024-09-11 18:05:46.082 INFO 2981023 --- [host-check-agent-status3,009d24f18bacf750,eafc72c08dd2e07c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:05:46.083 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:05:46.085 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:05:46.085136865+08:00 2024-09-11 18:05:46.085 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:05:46.085 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:05:46.085+08:00 2024-09-11 18:05:46.086 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:05:46.086492244+08:00 2024-09-11 18:05:46.087 INFO 2981023 --- [host-check-agent-status3,009d24f18bacf750,eafc72c08dd2e07c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:05:46.087 INFO 2981023 --- [host-check-agent-status3,009d24f18bacf750,eafc72c08dd2e07c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:05:46.087 INFO 2981023 --- [host-check-agent-status3,009d24f18bacf750,eafc72c08dd2e07c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:05:46.088 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:05:46.086+08:00 2024-09-11 18:05:46.089 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:05:46.085+08:00 and status AVAILABLE 2024-09-11 18:05:46.090 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:05:46.086+08:00 and status AVAILABLE 2024-09-11 18:05:46.093 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:05:46.093 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:05:46.099177151+08:00 2024-09-11 18:05:46.094 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:05:46.093+08:00 2024-09-11 18:05:46.095 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:05:46.093+08:00 and status AVAILABLE 2024-09-11 18:05:46.096 INFO 2981023 --- [host-check-agent-status2,009d24f18bacf750,73c05c9646da17a8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:05:46.097 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:05:46.098 INFO 2981023 --- [host-check-agent-status2,009d24f18bacf750,73c05c9646da17a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:05:46.098 INFO 2981023 --- [host-check-agent-status2,009d24f18bacf750,73c05c9646da17a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:05:46.098 INFO 2981023 --- [host-check-agent-status2,009d24f18bacf750,73c05c9646da17a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:05:46.118 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:05:46.118 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:05:46.121 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:05:46.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2d08f74f9e1d0144,21e2841c26cf5a58] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2d08f74f9e1d0144, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:46.125 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:05:46.127 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:05:46.132 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:05:46.134 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:05:46.136 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:05:46.138 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2d08f74f9e1d0144,21e2841c26cf5a58] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2d08f74f9e1d0144, duration=25 ms] 2024-09-11 18:05:46.139 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:05:46.141 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:05:46.143 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:05:46.164 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:05:46.174 INFO 2981023 --- [ocp-schedules-26,9b30e3195ae59666,cac64e19fe76897c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:05:46.187 INFO 2981023 --- [ocp-schedules-20,69de34bc3b957775,1b38a3081e5b77a4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:05:46.188 INFO 2981023 --- [ocp-schedules-25,e8f944cfd27d6a33,f0c8a99a1a65431d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:05:46.188 INFO 2981023 --- [ocp-schedules-27,a6c05bdc2769f3d9,dd3bfd0b0bb2648d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:05:46.188 INFO 2981023 --- [ocp-schedules-5,c4a0ff990af3b9d0,9e2e7526b17a3e99] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:05:46.192 INFO 2981023 --- [ocp-schedules-26,9b30e3195ae59666,cac64e19fe76897c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:05:46.192 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:05:46.197 INFO 2981023 --- [ocp-schedules-20,69de34bc3b957775,1b38a3081e5b77a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:05:46.197 INFO 2981023 --- [ocp-schedules-25,e8f944cfd27d6a33,f0c8a99a1a65431d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:05:46.198 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:05:46.202 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:05:46.207 INFO 2981023 --- [ocp-schedules-27,a6c05bdc2769f3d9,dd3bfd0b0bb2648d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:05:46.211 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:05:46.215 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049042737000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:05:46.225 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:05:46.225 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:05:46.227 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:05:46.248 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:05:46.252 INFO 2981023 --- [ocp-schedules-11,7e3467d114745f94,5a1ad665ec4cab9f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:05:46.253 INFO 2981023 --- [ocp-schedules-25,e8f944cfd27d6a33,f0c8a99a1a65431d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:05:46.258 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:05:46.260 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:05:46.269 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:05:46.276 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.277 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.277 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:05:46.279 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:05:46.281 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:05:46.285 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:05:46.285 INFO 2981023 --- [ocp-schedules-20,69de34bc3b957775,1b38a3081e5b77a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:05:46.305 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:05:46.315 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:05:46.318 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:05:46.323 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.323 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:46.325 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:05:46.326 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:05:46.337 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.337 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.338 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:46.338 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:05:46.338 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:05:46.339 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:05:46.339 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.344 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:05:46.345 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:05:46.339474+08:00, lastProcessTime=2024-09-11T18:05:36.337388+08:00 2024-09-11 18:05:46.351 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:05:46.351 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.352 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:05:46.357 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:05:46.359 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:46.376 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:05:46.377 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.378 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.379 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:05:46.388 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:05:46.419 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:05:46.423 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:05:46.425 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:05:46.430 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.430 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:46.431 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:05:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.434 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:05:46.435 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:05:46.441 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:05:46.446 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:05:46.449 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:05:46.449 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049042737000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:05:46.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:05:46.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:05:46.456 INFO 2981023 --- [host-check-host-status4,24ea792c35f7ce76,b3011aa6041bf740] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:05:46.458 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:05:46.458 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:05:46.458 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.459 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.460 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:05:46.462 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:05:46.462 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.467 INFO 2981023 --- [host-check-host-status2,24ea792c35f7ce76,d6d097f7e4cb13c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:05:46.470 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:05:46.477 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:05:46.488 INFO 2981023 --- [host-check-host-status1,24ea792c35f7ce76,ca7f95bcbe381f3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:05:46.490 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:05:46.493 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:05:46.501 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:05:46.504 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.504 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.505 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:05:46.506 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:05:46.511 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:05:46.515 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:05:46.520 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:05:46.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:05:46.531 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:05:46.537 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.559 INFO 2981023 --- [host-check-host-status3,24ea792c35f7ce76,2dc9366d0264b875] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:05:46.560 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:05:46.561 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.562 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.562 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:05:46.563 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:05:46.563 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.564 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:05:46.567 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:05:46.567 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.571 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.571 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:05:46.573 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:05:46.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:05:46.613 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:05:46.613 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:05:46.613 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.616 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:05:46.617 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:05:46.633 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.633 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.634 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:05:46.634 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:05:46.635 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.644 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:05:46.644 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:05:46.651 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:05:46.653 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:05:46.663 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.663 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.664 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:05:46.667 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:05:46.670 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:05:46.339474+08:00 2024-09-11 18:05:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:05:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:05:46.680 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.681 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.682 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:05:46.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:05:46.685 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:05:46.688 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:05:46.689 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:05:46.693 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:05:46.702 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:05:46.707 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.707 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.708 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:05:46.708 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:05:46.708 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.732 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:05:46.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:46.739 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:46.739 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:05:46.744 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:05:46.744 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:05:46.745 INFO 2981023 --- [ocp-schedules-27,a6c05bdc2769f3d9,dd3bfd0b0bb2648d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:05:46.749 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:05:46.779 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:05:46.785 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:05:46.787 INFO 2981023 --- [ocp-schedules-1,2856e49a86247db2,c5bbb4dd7f080e63] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:05:46.844 INFO 2981023 --- [ocp-schedules-3,ac2c39a417bb9a27,59fbfa40f2732ae7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8432,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5148} 2024-09-11 18:05:46.844 INFO 2981023 --- [ocp-schedules-3,ac2c39a417bb9a27,59fbfa40f2732ae7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:05:46.962 INFO 2981023 --- [host-check-clockdiff2,7e74d7ac8a038ddb,14e5f1cfda8c626f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049146 5 5 , error= 2024-09-11 18:05:46.983 INFO 2981023 --- [host-check-clockdiff1,7e74d7ac8a038ddb,06b646e2fe4fdd74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049146 5 5 , error= 2024-09-11 18:05:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,8e287ddef8958fee,56bcee55fe7c9ebf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,8e287ddef8958fee,56bcee55fe7c9ebf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:47.385 INFO 2981023 --- [ocp-async-5,5e0d0a4514b6e461,688ef323611ff134] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:47.780 INFO 2981023 --- [ocp-async-4,a468b1b0972263b3,ccdebb20d571b36d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:05:47.877 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:47.877 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:48.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,af62de965ce61bfe,803f0261817acfd8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=af62de965ce61bfe, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:48.121 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,af62de965ce61bfe,803f0261817acfd8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=af62de965ce61bfe, duration=9 ms] 2024-09-11 18:05:49.385 INFO 2981023 --- [ocp-async-3,94f75c69a4e5184a,2e993fc8f7c0762e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:49.902 INFO 2981023 --- [ocp-schedules-20,58b4908cf7a119a5,db9b4f9256abceec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:05:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:05:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:05:49.956 WARN 2981023 --- [scanner-13604496-1,58b4908cf7a119a5,7cf387d8a8807275] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:05:49.981 INFO 2981023 --- [alarm-detect-18,58b4908cf7a119a5,f895666cddedefac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645381, data is empty. 2024-09-11 18:05:49.985 INFO 2981023 --- [alarm-detect-18,58b4908cf7a119a5,f895666cddedefac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645381, data is empty. 2024-09-11 18:05:49.990 INFO 2981023 --- [alarm-detect-18,58b4908cf7a119a5,f895666cddedefac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645381, data is empty. 2024-09-11 18:05:50.008 INFO 2981023 --- [alarm-detect-10,58b4908cf7a119a5,d2882a91a57b2044] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645381, data is empty. 2024-09-11 18:05:50.009 INFO 2981023 --- [alarm-detect-16,58b4908cf7a119a5,962c29c2ced026b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645381, data is empty. 2024-09-11 18:05:50.014 INFO 2981023 --- [alarm-detect-10,58b4908cf7a119a5,d2882a91a57b2044] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645381, data is empty. 2024-09-11 18:05:50.019 INFO 2981023 --- [alarm-detect-16,58b4908cf7a119a5,962c29c2ced026b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645381, data is empty. 2024-09-11 18:05:50.021 INFO 2981023 --- [alarm-detect-10,58b4908cf7a119a5,d2882a91a57b2044] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645381, data is empty. 2024-09-11 18:05:50.028 INFO 2981023 --- [alarm-detect-16,58b4908cf7a119a5,962c29c2ced026b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645381, data is empty. 2024-09-11 18:05:50.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,5c07a2a452e33100,138add5142e93fef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5c07a2a452e33100, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:50.129 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,5c07a2a452e33100,138add5142e93fef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5c07a2a452e33100, duration=29 ms] 2024-09-11 18:05:50.141 INFO 2981023 --- [alarm-detect-10,58b4908cf7a119a5,3daf1359961f8d99] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645381, data is empty. 2024-09-11 18:05:50.148 INFO 2981023 --- [alarm-detect-10,58b4908cf7a119a5,3daf1359961f8d99] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645381, data is empty. 2024-09-11 18:05:50.152 INFO 2981023 --- [alarm-detect-10,58b4908cf7a119a5,3daf1359961f8d99] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645381, data is empty. 2024-09-11 18:05:50.300 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:05:50.304 INFO 2981023 --- [ocp-schedules-6,c88269c5239ab61a,59d77411f36d49b7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.305 INFO 2981023 --- [ocp-schedules-9,10038e7cd075fdd0,f3c23d12818d18d7] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:05:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,82949e0110578548,96b7f31b07a469b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,82949e0110578548,96b7f31b07a469b3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:50.877 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:50.879 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:51.385 INFO 2981023 --- [ocp-async-1,6283198dc23bce42,c0dec873867f8822] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:51.599 INFO 2981023 --- [ocp-schedules-21,8faa068493304ac6,e87e7dc1015b7493] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:05:51.600 INFO 2981023 --- [ocp-schedules-21,8faa068493304ac6,e87e7dc1015b7493] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:05:51.600 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:05:51.600 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:05:51.600 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:05:51.600 INFO 2981023 --- [ocp-schedules-21,8faa068493304ac6,e87e7dc1015b7493] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:05:51.997 INFO 2981023 --- [host-check-clockdiff4,7e74d7ac8a038ddb,ddc3c0aeed87ae37] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049151 0 0 , error= 2024-09-11 18:05:52.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,501d7d3786a46738,a9311fa7a33ccce9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=501d7d3786a46738, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:52.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,501d7d3786a46738,a9311fa7a33ccce9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=501d7d3786a46738, duration=10 ms] 2024-09-11 18:05:52.473 INFO 2981023 --- [ocp-schedules-28,b59ed3c8467d69ab,80b0a2a21a3aaaa6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:05:53.253 INFO 2981023 --- [ocp-schedules-5,e05dfc0dc6740979,33504e820aceb0e4] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:05:53.253 INFO 2981023 --- [ocp-schedules-5,e05dfc0dc6740979,33504e820aceb0e4] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:05:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,04c9f1e22095ff56,ae7d1bde639b53d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,04c9f1e22095ff56,ae7d1bde639b53d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:53.385 INFO 2981023 --- [ocp-async-8,626c50745b1637a7,8a5124eaafa10631] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:53.879 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:53.881 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:54.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,a45c979f43811124,b7e0be047852d846] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a45c979f43811124, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:54.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,a45c979f43811124,b7e0be047852d846] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a45c979f43811124, duration=14 ms] 2024-09-11 18:05:54.342 INFO 2981023 --- [ocp-schedules-9,7ed870fd841d6d54,2c050c412654aeee] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:05:54.361 WARN 2981023 --- [ocp-schedules-9,7ed870fd841d6d54,2c050c412654aeee] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:05:54.394 INFO 2981023 --- [ocp-schedules-8,39522934fb9a06f5,fc3a963af7242ed1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:05:55.047 INFO 2981023 --- [ocp-schedules-5,f4b58af523c1f7fd,d158373dd1f9bb21] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:05:55.056 INFO 2981023 --- [ocp-schedules-5,f4b58af523c1f7fd,d158373dd1f9bb21] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:05:55.056 INFO 2981023 --- [ocp-schedules-5,f4b58af523c1f7fd,d158373dd1f9bb21] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:05:55.385 INFO 2981023 --- [ocp-async-2,2277d4474a4dd211,64b9651045354738] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:55.842 INFO 2981023 --- [ocp-schedules-29,ec34b912d646171a,69ccf81998c404d1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:05:55.842 INFO 2981023 --- [ocp-schedules-29,ec34b912d646171a,69ccf81998c404d1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:05:55.843 INFO 2981023 --- [ocp-schedules-29,ec34b912d646171a,69ccf81998c404d1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:05:55.843 INFO 2981023 --- [ocp-schedules-29,ec34b912d646171a,69ccf81998c404d1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:05:55.843 INFO 2981023 --- [ocp-schedules-29,ec34b912d646171a,69ccf81998c404d1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:05:55.843 INFO 2981023 --- [ocp-schedules-29,ec34b912d646171a,69ccf81998c404d1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:05:55.997 INFO 2981023 --- [ocp-schedules-24,3db3728f7925d689,fafee2bf60183e30] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:05:56.003 INFO 2981023 --- [ocp-schedules-24,3db3728f7925d689,fafee2bf60183e30] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:05:56.003 INFO 2981023 --- [ocp-schedules-24,3db3728f7925d689,fafee2bf60183e30] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:05:56.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,cf313c2036112a40,69271a591c77c955] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cf313c2036112a40, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:56.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,cf313c2036112a40,69271a591c77c955] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cf313c2036112a40, duration=9 ms] 2024-09-11 18:05:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,9838f87dd02de371,573d86f5609808e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,9838f87dd02de371,573d86f5609808e2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:05:56.352803+08:00, lastProcessTime=2024-09-11T18:05:46.339474+08:00 2024-09-11 18:05:56.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:05:56.352803+08:00 2024-09-11 18:05:56.882 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:56.882 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:05:57.270 INFO 2981023 --- [ocp-schedules-7,8cb7b68198a0186a,4b33ead255b24a34] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:05:57.270+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:05:57.274 INFO 2981023 --- [ocp-schedules-7,8cb7b68198a0186a,4b33ead255b24a34] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:05:57.385 INFO 2981023 --- [ocp-async-4,50a01d603688659a,e1316c554b798779] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:57.418 INFO 2981023 --- [ocp-schedules-7,8cb7b68198a0186a,4b33ead255b24a34] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 51 to SqlAggregateData 2024-09-11 18:05:57.419 INFO 2981023 --- [ocp-schedules-7,8cb7b68198a0186a,4b33ead255b24a34] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:05:57.523 INFO 2981023 --- [ocp-schedules-18,eb3fe91b53e41322,4a47dc71d6f54f89] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:05:57.523 INFO 2981023 --- [ocp-schedules-18,eb3fe91b53e41322,4a47dc71d6f54f89] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:05:57.540 INFO 2981023 --- [Collect all tenants major compactions1,eb3fe91b53e41322,c6fcf1d0755e0989] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:05:57.541 INFO 2981023 --- [Collect all tenants major compactions2,eb3fe91b53e41322,2583988bb33b8e17] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:05:57.541 INFO 2981023 --- [Collect all tenants major compactions4,eb3fe91b53e41322,365188009632c1f8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:05:57.547 INFO 2981023 --- [Collect all tenants major compactions5,eb3fe91b53e41322,f966993090bfffa8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:05:57.548 INFO 2981023 --- [Collect all tenants major compactions3,eb3fe91b53e41322,6e95d048eef2b91c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:05:57.549 INFO 2981023 --- [Collect all tenants major compactions6,eb3fe91b53e41322,624dac8d5aba660f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:05:57.565 INFO 2981023 --- [Collect all tenants major compactions2,eb3fe91b53e41322,2583988bb33b8e17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:57.565 INFO 2981023 --- [Collect all tenants major compactions1,eb3fe91b53e41322,c6fcf1d0755e0989] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:57.566 INFO 2981023 --- [Collect all tenants major compactions2,eb3fe91b53e41322,2583988bb33b8e17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:05:57.566 INFO 2981023 --- [Collect all tenants major compactions5,eb3fe91b53e41322,f966993090bfffa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:57.567 INFO 2981023 --- [Collect all tenants major compactions1,eb3fe91b53e41322,c6fcf1d0755e0989] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:05:57.567 INFO 2981023 --- [Collect all tenants major compactions4,eb3fe91b53e41322,365188009632c1f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:57.567 INFO 2981023 --- [Collect all tenants major compactions3,eb3fe91b53e41322,6e95d048eef2b91c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:57.567 INFO 2981023 --- [Collect all tenants major compactions6,eb3fe91b53e41322,624dac8d5aba660f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:57.568 INFO 2981023 --- [Collect all tenants major compactions5,eb3fe91b53e41322,f966993090bfffa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:05:57.568 INFO 2981023 --- [Collect all tenants major compactions3,eb3fe91b53e41322,6e95d048eef2b91c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:05:57.569 INFO 2981023 --- [Collect all tenants major compactions4,eb3fe91b53e41322,365188009632c1f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:05:57.570 INFO 2981023 --- [Collect all tenants major compactions2,eb3fe91b53e41322,2583988bb33b8e17] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:05:57.570 INFO 2981023 --- [Collect all tenants major compactions2,eb3fe91b53e41322,2583988bb33b8e17] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:05:57.570 INFO 2981023 --- [Collect all tenants major compactions1,eb3fe91b53e41322,c6fcf1d0755e0989] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:05:57.570 INFO 2981023 --- [Collect all tenants major compactions1,eb3fe91b53e41322,c6fcf1d0755e0989] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:05:57.570 INFO 2981023 --- [Collect all tenants major compactions6,eb3fe91b53e41322,624dac8d5aba660f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:05:57.570 INFO 2981023 --- [Collect all tenants major compactions3,eb3fe91b53e41322,6e95d048eef2b91c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:05:57.570 INFO 2981023 --- [Collect all tenants major compactions3,eb3fe91b53e41322,6e95d048eef2b91c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:05:57.571 INFO 2981023 --- [Collect all tenants major compactions5,eb3fe91b53e41322,f966993090bfffa8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:05:57.571 INFO 2981023 --- [Collect all tenants major compactions5,eb3fe91b53e41322,f966993090bfffa8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:05:57.572 INFO 2981023 --- [Collect all tenants major compactions4,eb3fe91b53e41322,365188009632c1f8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:05:57.572 INFO 2981023 --- [Collect all tenants major compactions4,eb3fe91b53e41322,365188009632c1f8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:05:57.572 INFO 2981023 --- [Collect all tenants major compactions6,eb3fe91b53e41322,624dac8d5aba660f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:05:57.572 INFO 2981023 --- [Collect all tenants major compactions6,eb3fe91b53e41322,624dac8d5aba660f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:05:57.573 INFO 2981023 --- [ocp-schedules-18,eb3fe91b53e41322,4a47dc71d6f54f89] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:05:57.780 INFO 2981023 --- [ocp-async-2,59525448c7e2c7a5,8b0b9f7174985892] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:05:58.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,0e03001c24dd6c3e,f95f9d983c87ff08] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0e03001c24dd6c3e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:05:58.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,0e03001c24dd6c3e,f95f9d983c87ff08] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0e03001c24dd6c3e, duration=10 ms] 2024-09-11 18:05:58.258 INFO 2981023 --- [ocp-schedules-17,1cef4f443aae9740,48783202b9dcc290] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:05:58.259 INFO 2981023 --- [ocp-schedules-17,1cef4f443aae9740,48783202b9dcc290] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:05:58.289 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:05:58.289 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.292 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:05:58.295 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:05:58.296 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.296 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.296 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.297 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:05:58.297 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.299 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:05:58.301 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:05:58.301 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.301 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.304 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.305 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.306 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.307 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:05:58.314 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.315 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:05:58.319 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.319 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.319 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.320 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.320 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.321 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:05:58.327 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.327 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:05:58.331 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.332 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.332 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:05:58.332 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE48F1D730-0-0] 2024-09-11 18:05:58.333 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:05:58.333 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.333 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:05:58.333 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.334 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.342 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.343 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:05:58.344 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.344 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:05:58.345 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.346 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:05:58.346 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.346 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.347 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:05:58.347 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:05:58.348 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.348 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:05:58.349 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.349 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:05:58.350 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.350 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.351 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:05:58.351 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:05:58.351 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.352 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:05:58.352 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.353 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:05:58.353 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:05:58.357 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:05:58.363 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.363 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.364 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:05:58.364 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:05:58.364 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.365 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:05:58.365 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE48F1D73A-0-0] 2024-09-11 18:05:58.365 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:05:58.366 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:05:58.366 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:05:58.369 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:05:58.372 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:58.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.374 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:05:58.375 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:05:58.376 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.377 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:05:58.381 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:05:58.381 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.384 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:05:58.384 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:05:58.384 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:05:58.385 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.386 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:58.386 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:05:58.387 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.388 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.388 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:05:58.388 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.390 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.390 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:05:58.392 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:05:58.392 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:05:58.392 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.392 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:05:58.393 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.395 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:05:58.399 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:05:58.401 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:05:58.401 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:05:58.401 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:05:58.401 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:05:58.404 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:05:58.404 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:05:58.404 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:05:58.408 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:05:58.410 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.411 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:05:58.423 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:05:58.425 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:05:58.425 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:05:58.426 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:05:58.438 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:05:58.440 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:05:58.440 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:05:58.440 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:05:58.440 INFO 2981023 --- [ocp-schedules-17,1cef4f443aae9740,48783202b9dcc290] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:05:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,b2d7781e895a85d5,8d41a15a08f64245] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049160, interval=60 2024-09-11 18:05:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,b2d7781e895a85d5,8d41a15a08f64245] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:05:59.386 INFO 2981023 --- [ocp-async-8,104cef49f39e1fcd,2ac1b132fdf2fa5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:05:59.882 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:05:59.883 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:00.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,075c6480fce79276,5f9f274590845dd4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=075c6480fce79276, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:00.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,075c6480fce79276,5f9f274590845dd4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=075c6480fce79276, duration=10 ms] 2024-09-11 18:06:00.195 INFO 2981023 --- [ocp-schedules-19,547d977e2a0a4c5e,3e94a093beaf7622] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:06:00.237 WARN 2981023 --- [scanner-13604794-1,547d977e2a0a4c5e,51e731b7ee264e30] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:06:01.123 INFO 2981023 --- [ocp-schedules-7,c4e2887170c9cc55,e413843821ed3340] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:06:01.391 INFO 2981023 --- [ocp-async-2,057fae3e928effc0,548304895ecc4d15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:02.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,cc60f5dc3852c3c7,87ecf82601e7e8b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cc60f5dc3852c3c7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:02.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,cc60f5dc3852c3c7,87ecf82601e7e8b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cc60f5dc3852c3c7, duration=9 ms] 2024-09-11 18:06:02.182 INFO 2981023 --- [ocp-schedules-1,35faf963568b0f4f,aae4ef78342fdd0b] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:06:02.182 INFO 2981023 --- [ocp-schedules-1,35faf963568b0f4f,aae4ef78342fdd0b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:06:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,e4b52d813edef965,4cee89a1d8c8bd53] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,e4b52d813edef965,4cee89a1d8c8bd53] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:02.476 INFO 2981023 --- [ocp-schedules-25,fe6125b4e777feae,95a730bfae649ead] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:06:02.746 INFO 2981023 --- [ocp-schedules-4,07f982cdd2bce8db,82cb7b5b579c898b] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:06:02.748 INFO 2981023 --- [ocp-schedules-4,07f982cdd2bce8db,82cb7b5b579c898b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:02.769 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:02.769 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:06:02.774 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:02.776 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:06:02.777 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049162777000} 2024-09-11 18:06:02.782 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:06:02.796 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000227, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:06:02.797 INFO 2981023 --- [ocp-schedules-4,07f982cdd2bce8db,82cb7b5b579c898b] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:06:02.883 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:02.884 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:03.386 INFO 2981023 --- [ocp-async-1,2b8a8ff09666065b,4e055c663cea90c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:03.501 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:06:03.501 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:06:03.501 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:06:03.506 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000227, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:06:03.508 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:06:04.057 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000227 2024-09-11 18:06:04.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,49effa73d4ea9036,2f69da4585c1b4d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=49effa73d4ea9036, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:04.125 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,49effa73d4ea9036,2f69da4585c1b4d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=49effa73d4ea9036, duration=16 ms] 2024-09-11 18:06:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,750849c345cd2f89,0f83547ca4345b52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,750849c345cd2f89,0f83547ca4345b52] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:05.386 INFO 2981023 --- [ocp-async-5,4f0061ba7f139ed8,b82ac08f9c8eac76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:05.884 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:05.886 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:06:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:06:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:06:06.001 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:06:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:06:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:06:06.004 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:06.004 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:06:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:06:06.066 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:03:41, start=2024-09-11T18:00:11, end=2024-09-11T18:02:41, updates=322, granule=120000ms, delay=210000ms 2024-09-11 18:06:06.072 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:03:41, target=2024-09-11T18:02:41, granule=120000ms, delay=60000ms) 2024-09-11 18:06:06.072 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:02:41, target=2024-09-11T17:50:51, granule=600000ms, delay=710000ms) 2024-09-11 18:06:06.072 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=105ms, period=120000ms 2024-09-11 18:06:06.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,482ff6c71b4d286b,245096f7f46eb18d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=482ff6c71b4d286b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:06.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,482ff6c71b4d286b,245096f7f46eb18d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=482ff6c71b4d286b, duration=10 ms] 2024-09-11 18:06:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:06:06.353356+08:00, lastProcessTime=2024-09-11T18:05:56.352803+08:00 2024-09-11 18:06:06.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:06:06.353356+08:00 2024-09-11 18:06:07.386 INFO 2981023 --- [ocp-async-3,14d5d4150adaeb8a,f794779b4bedb35c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:07.780 INFO 2981023 --- [ocp-async-7,45876d4de857cde1,c2910872cb97f70a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:06:07.787 INFO 2981023 --- [ocp-schedules-8,3b0b78084bb54c99,b88c2e25fdd4ffee] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:06:07.918 INFO 2981023 --- [ocp-schedules-2,1696b2576a5e7f32,ec46ff86e9e470e5] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:06:07.918+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:06:07.918 INFO 2981023 --- [ocp-schedules-2,1696b2576a5e7f32,ec46ff86e9e470e5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:06:07.920 INFO 2981023 --- [ocp-schedules-2,1696b2576a5e7f32,ec46ff86e9e470e5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:06:07.920 INFO 2981023 --- [ocp-schedules-2,1696b2576a5e7f32,ec46ff86e9e470e5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:06:08.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2ec20257e3061f8e,e15c8ad165a43c70] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2ec20257e3061f8e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:08.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2ec20257e3061f8e,e15c8ad165a43c70] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2ec20257e3061f8e, duration=9 ms] 2024-09-11 18:06:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,a4ef590465fe4739,1b14d5ce8d6ae75b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,a4ef590465fe4739,1b14d5ce8d6ae75b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:08.886 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:08.888 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:09.386 INFO 2981023 --- [ocp-async-6,3bf6a85c176da378,f49bf0db5b4e1402] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:10.003 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:06:10.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:06:10.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,776fde08a49fbeb4,17a3dad33265b088] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=776fde08a49fbeb4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:10.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,776fde08a49fbeb4,17a3dad33265b088] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=776fde08a49fbeb4, duration=17 ms] 2024-09-11 18:06:10.401 INFO 2981023 --- [ocp-schedules-21,7b436cebf55bc82e,bef4c68319d4c856] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:06:10.455 WARN 2981023 --- [scanner-13604996-1,7b436cebf55bc82e,0c33785f743e816e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:06:10.481 INFO 2981023 --- [alarm-detect-20,7b436cebf55bc82e,4373797e5bd84142] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645381, data is empty. 2024-09-11 18:06:10.485 INFO 2981023 --- [alarm-detect-20,7b436cebf55bc82e,4373797e5bd84142] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645432, data is empty. 2024-09-11 18:06:10.489 INFO 2981023 --- [alarm-detect-20,7b436cebf55bc82e,4373797e5bd84142] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645432, data is empty. 2024-09-11 18:06:10.497 INFO 2981023 --- [alarm-detect-8,7b436cebf55bc82e,f130c05d73952254] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645381, data is empty. 2024-09-11 18:06:10.499 INFO 2981023 --- [alarm-detect-20,7b436cebf55bc82e,1ad1265286551fcf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645381, data is empty. 2024-09-11 18:06:10.502 INFO 2981023 --- [alarm-detect-20,7b436cebf55bc82e,1ad1265286551fcf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645432, data is empty. 2024-09-11 18:06:10.503 INFO 2981023 --- [alarm-detect-8,7b436cebf55bc82e,f130c05d73952254] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645432, data is empty. 2024-09-11 18:06:10.505 INFO 2981023 --- [alarm-detect-20,7b436cebf55bc82e,1ad1265286551fcf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645432, data is empty. 2024-09-11 18:06:10.507 INFO 2981023 --- [alarm-detect-8,7b436cebf55bc82e,f130c05d73952254] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645432, data is empty. 2024-09-11 18:06:10.640 INFO 2981023 --- [alarm-detect-17,7b436cebf55bc82e,74632aeb1c669c62] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645381, data is empty. 2024-09-11 18:06:10.646 INFO 2981023 --- [alarm-detect-17,7b436cebf55bc82e,74632aeb1c669c62] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645432, data is empty. 2024-09-11 18:06:10.653 INFO 2981023 --- [alarm-detect-17,7b436cebf55bc82e,74632aeb1c669c62] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645432, data is empty. 2024-09-11 18:06:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:06:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:06:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,a5a32a95d7095f4f,5720fe9a4991b53d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:11.320 INFO 2981023 --- [prometheus-minute-schedule-1,a5a32a95d7095f4f,5720fe9a4991b53d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:11.386 INFO 2981023 --- [ocp-async-5,f121e19daebeb08a,d5680aad853d6c4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:11.888 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:11.889 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:12.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,de23dea98f150f84,a195e74e7cfc95a7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=de23dea98f150f84, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:12.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,de23dea98f150f84,a195e74e7cfc95a7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=de23dea98f150f84, duration=10 ms] 2024-09-11 18:06:12.479 INFO 2981023 --- [ocp-schedules-22,85577d00d2e68a31,15257e6536e180af] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:06:13.387 INFO 2981023 --- [ocp-async-3,d433f087ccde4748,1fb7ab78e77905d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:14.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,c03ad56eb634b97e,cdc3dfdfa09674cc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c03ad56eb634b97e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:14.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,c03ad56eb634b97e,cdc3dfdfa09674cc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c03ad56eb634b97e, duration=10 ms] 2024-09-11 18:06:14.263 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049100, nodeCount=1000, affectRows=1000 2024-09-11 18:06:14.265 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049100, nodeCount=999, validCount=999 2024-09-11 18:06:14.273 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049100, nodeCount=48, affectRows=48 2024-09-11 18:06:14.273 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049100, nodeCount=46, validCount=46 2024-09-11 18:06:14.276 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049100, nodeCount=1000, affectRows=1000 2024-09-11 18:06:14.278 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049100, nodeCount=1000, validCount=1000 2024-09-11 18:06:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,d229ce7e246bfb48,85eb8e76686b6956] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,5ff6159a9772177b,11d6dc158dbab81f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:06:14.320 INFO 2981023 --- [prometheus-minute-schedule-1,d229ce7e246bfb48,85eb8e76686b6956] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:14.409 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049100, nodeCount=480, affectRows=480 2024-09-11 18:06:14.889 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:14.890 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:15.059 INFO 2981023 --- [ocp-schedules-7,0ad4ce777d4f9aa3,b67ca95480eabbce] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:06:15.069 INFO 2981023 --- [ocp-schedules-7,0ad4ce777d4f9aa3,b67ca95480eabbce] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:06:15.069 INFO 2981023 --- [ocp-schedules-7,0ad4ce777d4f9aa3,b67ca95480eabbce] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:06:15.387 INFO 2981023 --- [ocp-async-2,d720e2db2982beff,2545a0515d2a98a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:16.025 INFO 2981023 --- [ocp-schedules-25,e48ccb7c6f7e4a8b,cd667635da7875ac] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:06:16.025 INFO 2981023 --- [ocp-schedules-13,3b3658aa9423a25e,f1328abd14196711] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:06:16.025 INFO 2981023 --- [host-check-clockdiff1,e48ccb7c6f7e4a8b,d46e2113abdf81d6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:06:16.026 INFO 2981023 --- [host-check-clockdiff2,e48ccb7c6f7e4a8b,c9ee5229d7c62cab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:06:16.026 INFO 2981023 --- [host-check-clockdiff3,e48ccb7c6f7e4a8b,64060ac02bcbc80b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:06:16.027 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:06:16.028 INFO 2981023 --- [ocp-schedules-11,77dedf6715a4294d,bc7fc6488731b45c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:06:16.028 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:06:16.028 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:06:16.028 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:06:16.029 INFO 2981023 --- [host-check-clockdiff4,e48ccb7c6f7e4a8b,7fb9dfe82c3f5039] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:06:16.048 INFO 2981023 --- [host-check-agent-status1,77dedf6715a4294d,8fee569d49b61446] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:06:16.048 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:06:16.049 INFO 2981023 --- [host-check-agent-status2,77dedf6715a4294d,c3ad0af9c468e99b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:06:16.049 INFO 2981023 --- [host-check-agent-status3,77dedf6715a4294d,bda2caff39a0a796] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:06:16.049 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:06:16.049 INFO 2981023 --- [host-check-agent-status4,77dedf6715a4294d,264285fb5e90f197] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:06:16.058 INFO 2981023 --- [host-check-agent-status1,77dedf6715a4294d,8fee569d49b61446] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:06:16.060 INFO 2981023 --- [host-check-agent-status1,77dedf6715a4294d,8fee569d49b61446] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:06:16.060 INFO 2981023 --- [host-check-clockdiff3,e48ccb7c6f7e4a8b,64060ac02bcbc80b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049176 0 0 , error= 2024-09-11 18:06:16.061 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:06:16.062 INFO 2981023 --- [host-check-agent-status4,77dedf6715a4294d,264285fb5e90f197] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:06:16.065 INFO 2981023 --- [host-check-agent-status4,77dedf6715a4294d,264285fb5e90f197] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:06:16.082 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:06:16.093 INFO 2981023 --- [host-check-agent-status3,77dedf6715a4294d,bda2caff39a0a796] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:06:16.164 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:06:16.172 INFO 2981023 --- [host-check-agent-status3,77dedf6715a4294d,bda2caff39a0a796] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:06:16.179 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:06:16.193 INFO 2981023 --- [ocp-schedules-19,878a85ff1930c2bf,75a580ee34a4d068] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:06:16.193 INFO 2981023 --- [ocp-schedules-12,89095d2e51a13968,550bfa96c43a588b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:06:16.195 INFO 2981023 --- [ocp-schedules-29,6c3876609e6cf44c,f50fc3df0b8d8a0c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:06:16.195 INFO 2981023 --- [ocp-schedules-3,f4fccda2cd3deaf4,6fba0ce053ce1005] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:06:16.256 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:06:16.257 INFO 2981023 --- [ocp-schedules-19,878a85ff1930c2bf,75a580ee34a4d068] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:16.259 INFO 2981023 --- [ocp-schedules-29,6c3876609e6cf44c,f50fc3df0b8d8a0c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:16.261 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:06:16.261 INFO 2981023 --- [host-check-agent-status2,77dedf6715a4294d,c3ad0af9c468e99b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:06:16.264 INFO 2981023 --- [ocp-schedules-12,89095d2e51a13968,550bfa96c43a588b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:16.270 INFO 2981023 --- [host-check-agent-status2,77dedf6715a4294d,c3ad0af9c468e99b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:06:16.291 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049162777000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:06:16.292 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2750d57d0dde2d42,57c8036d638fb194] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2750d57d0dde2d42, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:16.292 INFO 2981023 --- [ocp-schedules-22,57214682bba420ad,47c1981408e247db] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:06:16.300 INFO 2981023 --- [ocp-schedules-12,89095d2e51a13968,550bfa96c43a588b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:06:16.301 INFO 2981023 --- [host-check-agent-status3,77dedf6715a4294d,bda2caff39a0a796] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:06:16.306 INFO 2981023 --- [host-check-agent-status3,77dedf6715a4294d,bda2caff39a0a796] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:06:16.306 INFO 2981023 --- [host-check-agent-status3,77dedf6715a4294d,bda2caff39a0a796] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:06:16.306 INFO 2981023 --- [host-check-agent-status3,77dedf6715a4294d,bda2caff39a0a796] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:06:16.313 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:06:16.314 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:06:16.313868265+08:00 2024-09-11 18:06:16.368 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:06:16.370 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:06:16.314+08:00 2024-09-11 18:06:16.375 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:06:16.375274405+08:00 2024-09-11 18:06:16.383 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:06:16.375+08:00 2024-09-11 18:06:16.384 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:06:16.314+08:00 and status AVAILABLE 2024-09-11 18:06:16.385 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:06:16.375+08:00 and status AVAILABLE 2024-09-11 18:06:16.391 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2750d57d0dde2d42,57c8036d638fb194] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2750d57d0dde2d42, duration=275 ms] 2024-09-11 18:06:16.395 INFO 2981023 --- [host-check-agent-status4,77dedf6715a4294d,264285fb5e90f197] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:06:16.400 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:06:16.397872+08:00, lastProcessTime=2024-09-11T18:06:06.353356+08:00 2024-09-11 18:06:16.400 INFO 2981023 --- [host-check-agent-status4,77dedf6715a4294d,264285fb5e90f197] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:06:16.400 INFO 2981023 --- [host-check-agent-status4,77dedf6715a4294d,264285fb5e90f197] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:06:16.400 INFO 2981023 --- [host-check-agent-status4,77dedf6715a4294d,264285fb5e90f197] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:06:16.402 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.402 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:06:16.403 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:06:16.406 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:06:16.408 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:06:16.409 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:06:16.410 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:06:16.410341567+08:00 2024-09-11 18:06:16.410 INFO 2981023 --- [host-check-agent-status1,77dedf6715a4294d,8fee569d49b61446] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:06:16.412 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:06:16.410+08:00 2024-09-11 18:06:16.412 INFO 2981023 --- [host-check-agent-status2,77dedf6715a4294d,c3ad0af9c468e99b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:06:16.415 INFO 2981023 --- [host-check-agent-status2,77dedf6715a4294d,c3ad0af9c468e99b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:06:16.415 INFO 2981023 --- [host-check-agent-status2,77dedf6715a4294d,c3ad0af9c468e99b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:06:16.415 INFO 2981023 --- [host-check-agent-status2,77dedf6715a4294d,c3ad0af9c468e99b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:06:16.416 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:06:16.410+08:00 and status AVAILABLE 2024-09-11 18:06:16.420 INFO 2981023 --- [host-check-agent-status1,77dedf6715a4294d,8fee569d49b61446] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:06:16.420 INFO 2981023 --- [host-check-agent-status1,77dedf6715a4294d,8fee569d49b61446] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:06:16.420 INFO 2981023 --- [host-check-agent-status1,77dedf6715a4294d,8fee569d49b61446] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:06:16.421 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.421 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:06:16.422 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:06:16.422 INFO 2981023 --- [ocp-schedules-19,878a85ff1930c2bf,75a580ee34a4d068] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:06:16.454 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:06:16.473 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:06:16.475 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:06:16.479833144+08:00 2024-09-11 18:06:16.476 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:06:16.475+08:00 2024-09-11 18:06:16.482 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:06:16.475+08:00 and status AVAILABLE 2024-09-11 18:06:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.483 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:06:16.484 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:16.485 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:06:16.485 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:06:16.485 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:06:16.486 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:06:16.486 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:06:16.490 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.492 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:16.498 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:06:16.507 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.509 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:16.510 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:06:16.512 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:06:16.516 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:06:16.519 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:06:16.537 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:06:16.556 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:06:16.560 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:06:16.572 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:06:16.578 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:06:16.581 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:06:16.589 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:06:16.594 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:16.595 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.595 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:16.595 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:06:16.596 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.596 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:06:16.598 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:06:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:06:16.605 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:06:16.605 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049162777000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:06:16.606 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:06:16.610 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:06:16.620 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049100, nodeCount=480, validCount=480 2024-09-11 18:06:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:06:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:06:16.623 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:06:16.633 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:06:16.636 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:06:16.642 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:06:16.653 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:06:16.653 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:06:16.656 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:06:16.661 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:06:16.663 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:06:16.669 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:16.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:06:16.670 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:06:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:06:16.673 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:06:16.673 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:06:16.675 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:06:16.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:06:16.681 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:06:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:16.684 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:06:16.693 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:06:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.698 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:06:16.699 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:06:16.711 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:06:16.719 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:06:16.727 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:06:16.733 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:06:16.741 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:06:16.744 INFO 2981023 --- [host-check-host-status3,3b3658aa9423a25e,a4fa9bb2eb804988] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:06:16.753 INFO 2981023 --- [host-check-host-status2,3b3658aa9423a25e,2f84a63e5dbf5f1f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:06:16.754 INFO 2981023 --- [host-check-host-status4,3b3658aa9423a25e,ca80525bba0ea78c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:06:16.759 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:06:16.761 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:16.763 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.764 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:16.766 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:06:16.768 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:06:16.773 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:06:16.783 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:06:16.783 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:06:16.784 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:06:16.792 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:06:16.796 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:06:16.397872+08:00 2024-09-11 18:06:16.799 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:06:16.799 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:16.799 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.800 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:06:16.805 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:16.807 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:06:16.807 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:06:16.808 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.809 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:06:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:06:16.817 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:06:16.820 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:06:16.821 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:06:16.829 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:06:16.833 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:06:16.834 INFO 2981023 --- [host-check-host-status1,3b3658aa9423a25e,16ee702337348a29] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:06:16.879 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:06:16.886 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:16.887 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:16.889 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:06:16.895 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:06:16.895 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:06:16.895 INFO 2981023 --- [ocp-schedules-29,6c3876609e6cf44c,f50fc3df0b8d8a0c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:06:16.975 INFO 2981023 --- [host-check-clockdiff1,e48ccb7c6f7e4a8b,d46e2113abdf81d6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049176 6 6 , error= 2024-09-11 18:06:16.975 INFO 2981023 --- [host-check-clockdiff2,e48ccb7c6f7e4a8b,c9ee5229d7c62cab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049176 5 5 , error= 2024-09-11 18:06:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,a592b5e1ddac19a9,ff89e4bf782503a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,a592b5e1ddac19a9,ff89e4bf782503a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:17.387 INFO 2981023 --- [ocp-async-7,3934a56104a0652a,6cf3471d8572353a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:17.780 INFO 2981023 --- [ocp-async-9,a84949de925a9f7a,60714d963e3eb4ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:06:17.891 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:17.892 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:18.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,fed979ecf0bc264e,54266b3d730f4591] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fed979ecf0bc264e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:18.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,fed979ecf0bc264e,54266b3d730f4591] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fed979ecf0bc264e, duration=9 ms] 2024-09-11 18:06:19.387 INFO 2981023 --- [ocp-async-10,9d6c4dd49669eea1,29e3bf808e03273d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:20.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,92f3810b1308bad0,4f56cb6f405f0aaa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=92f3810b1308bad0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:20.131 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,92f3810b1308bad0,4f56cb6f405f0aaa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=92f3810b1308bad0, duration=17 ms] 2024-09-11 18:06:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,60fa8a9c3c731c0b,f2ef8eb4fd8c8a54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,60fa8a9c3c731c0b,f2ef8eb4fd8c8a54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:20.699 INFO 2981023 --- [ocp-schedules-12,d6a7f6d06ba0d216,6aead2838f757536] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:06:20.725 WARN 2981023 --- [scanner-13606818-1,d6a7f6d06ba0d216,47432b1764ecb609] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:06:20.892 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:20.895 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:21.387 INFO 2981023 --- [ocp-async-9,837aa49b69cf6939,6cbd832709f0b906] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:22.037 INFO 2981023 --- [host-check-clockdiff4,e48ccb7c6f7e4a8b,7fb9dfe82c3f5039] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049182 0 0 , error= 2024-09-11 18:06:22.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,18ba866293642f71,8de1778576ce751a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=18ba866293642f71, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:22.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,18ba866293642f71,8de1778576ce751a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=18ba866293642f71, duration=9 ms] 2024-09-11 18:06:22.482 INFO 2981023 --- [ocp-schedules-29,c3b24b44e1660d40,b7acee7a63c9ca24] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:06:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,1edd223f4eb36668,a91e91f5a03ba187] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,1edd223f4eb36668,a91e91f5a03ba187] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:23.387 INFO 2981023 --- [ocp-async-4,8602e02486b18568,64d7a1deb17d3acd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:23.896 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:23.897 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:24.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,e655708849b9f490,fbee5ebe28fc0f9c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e655708849b9f490, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:24.132 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,e655708849b9f490,fbee5ebe28fc0f9c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e655708849b9f490, duration=15 ms] 2024-09-11 18:06:24.396 INFO 2981023 --- [ocp-schedules-28,c96e3e789b7148b4,d0c728b4d24b2014] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:06:24.868 INFO 2981023 --- [ocp-schedules-11,6f26504ad3391472,4d69deee68bb9924] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:06:24.868 INFO 2981023 --- [ocp-schedules-11,6f26504ad3391472,4d69deee68bb9924] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:06:24.868 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:06:24.868 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:06:24.868 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:06:24.889 INFO 2981023 --- [ocp-schedules-11,6f26504ad3391472,4d69deee68bb9924] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:06:24.892 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:06:24.892 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:06:24.893 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:06:24.894 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:06:24.896 INFO 2981023 --- [ocp-schedules-11,6f26504ad3391472,4d69deee68bb9924] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:06:24.896 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:06:24.897 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:06:25.170 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:06:25.171 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:06:25.171 INFO 2981023 --- [ocp-schedules-11,6f26504ad3391472,4d69deee68bb9924] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:06:25.173 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:06:25.190 INFO 2981023 --- [ocp-schedules-11,6f26504ad3391472,4d69deee68bb9924] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:06:25.387 INFO 2981023 --- [ocp-async-5,03ed6223b2a88a41,1d0e4c378fd9be96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:26.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ff745fd82c60ec02,40a271322bfcd39f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ff745fd82c60ec02, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:26.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ff745fd82c60ec02,40a271322bfcd39f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ff745fd82c60ec02, duration=11 ms] 2024-09-11 18:06:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,0823709d116bbf33,84f398cac26f0b9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,0823709d116bbf33,84f398cac26f0b9e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:06:26.352553+08:00, lastProcessTime=2024-09-11T18:06:16.397872+08:00 2024-09-11 18:06:26.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:06:26.352553+08:00 2024-09-11 18:06:26.897 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:26.899 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:27.387 INFO 2981023 --- [ocp-async-10,ad505c361a1af1a0,3ccfd49ce53f241f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:27.780 INFO 2981023 --- [ocp-async-8,c3ee7c1102f62b77,e4fda1cd9c265fcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:06:28.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5de3b3032eea2ebc,6db27731941c9646] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5de3b3032eea2ebc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:28.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5de3b3032eea2ebc,6db27731941c9646] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5de3b3032eea2ebc, duration=10 ms] 2024-09-11 18:06:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,3581cabd4934fbaf,60ad06fa15d38fec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:29.319 INFO 2981023 --- [prometheus-minute-schedule-1,3581cabd4934fbaf,60ad06fa15d38fec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:29.387 INFO 2981023 --- [ocp-async-6,28279aea44c5e47e,2799d4f4241f4453] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:29.899 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:29.900 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:30.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b3b4bc3fa5fb900c,184474be9301fa54] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b3b4bc3fa5fb900c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:30.126 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b3b4bc3fa5fb900c,184474be9301fa54] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b3b4bc3fa5fb900c, duration=16 ms] 2024-09-11 18:06:30.918 INFO 2981023 --- [ocp-schedules-25,937755cfab7f139c,5b1c34a59ff5e1f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:06:30.977 WARN 2981023 --- [scanner-13607072-1,937755cfab7f139c,38391b3c3f04b0fd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:06:31.022 INFO 2981023 --- [alarm-detect-5,937755cfab7f139c,d11e7c043e51d94c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645432, data is empty. 2024-09-11 18:06:31.029 INFO 2981023 --- [alarm-detect-5,937755cfab7f139c,d11e7c043e51d94c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645432, data is empty. 2024-09-11 18:06:31.035 INFO 2981023 --- [alarm-detect-5,937755cfab7f139c,d11e7c043e51d94c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645432, data is empty. 2024-09-11 18:06:31.050 INFO 2981023 --- [alarm-detect-9,937755cfab7f139c,335c6c018392beec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645432, data is empty. 2024-09-11 18:06:31.055 INFO 2981023 --- [alarm-detect-5,937755cfab7f139c,cafc7e5ee741aa98] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645432, data is empty. 2024-09-11 18:06:31.056 INFO 2981023 --- [alarm-detect-9,937755cfab7f139c,335c6c018392beec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645432, data is empty. 2024-09-11 18:06:31.059 INFO 2981023 --- [alarm-detect-5,937755cfab7f139c,cafc7e5ee741aa98] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645432, data is empty. 2024-09-11 18:06:31.061 INFO 2981023 --- [alarm-detect-9,937755cfab7f139c,335c6c018392beec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645432, data is empty. 2024-09-11 18:06:31.063 INFO 2981023 --- [alarm-detect-5,937755cfab7f139c,cafc7e5ee741aa98] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645432, data is empty. 2024-09-11 18:06:31.387 INFO 2981023 --- [alarm-detect-1,937755cfab7f139c,76a554c362898859] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645432, data is empty. 2024-09-11 18:06:31.388 INFO 2981023 --- [ocp-async-3,9519446fa6e24c14,184a4a27d34faf9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:31.392 INFO 2981023 --- [alarm-detect-1,937755cfab7f139c,76a554c362898859] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645432, data is empty. 2024-09-11 18:06:31.396 INFO 2981023 --- [alarm-detect-1,937755cfab7f139c,76a554c362898859] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645432, data is empty. 2024-09-11 18:06:32.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2d1bf372b0803849,d228f64da3044623] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2d1bf372b0803849, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:32.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2d1bf372b0803849,d228f64da3044623] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2d1bf372b0803849, duration=10 ms] 2024-09-11 18:06:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,9f4147a25d922f94,7a70d229b91367d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,9f4147a25d922f94,7a70d229b91367d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:32.484 INFO 2981023 --- [ocp-schedules-6,3afa3eac94a32105,fca1aa6116ebccbc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:06:32.900 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:32.901 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:33.388 INFO 2981023 --- [ocp-async-1,c541073e46097e92,8ab8cf2ffb153594] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:34.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,068b0f65c510aae6,f2207b663f5bdcfa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=068b0f65c510aae6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:34.135 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,068b0f65c510aae6,f2207b663f5bdcfa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=068b0f65c510aae6, duration=31 ms] 2024-09-11 18:06:35.072 INFO 2981023 --- [ocp-schedules-8,5d5952f97b3da3f4,ad5efa4f8053f7d1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:06:35.083 INFO 2981023 --- [ocp-schedules-8,5d5952f97b3da3f4,ad5efa4f8053f7d1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:06:35.083 INFO 2981023 --- [ocp-schedules-8,5d5952f97b3da3f4,ad5efa4f8053f7d1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:06:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,f4d2d5cb7ad74edb,55541768e6d351ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:35.319 INFO 2981023 --- [prometheus-minute-schedule-1,f4d2d5cb7ad74edb,55541768e6d351ec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:35.388 INFO 2981023 --- [ocp-async-7,d0b4bd0367773bec,b3261e596dcf6e64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:35.901 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:35.903 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:36.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,b1c2f67a7690bc42,2598c41af3449201] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b1c2f67a7690bc42, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:36.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,b1c2f67a7690bc42,2598c41af3449201] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b1c2f67a7690bc42, duration=10 ms] 2024-09-11 18:06:36.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:06:36.354809+08:00, lastProcessTime=2024-09-11T18:06:26.352553+08:00 2024-09-11 18:06:36.504 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:06:36.354809+08:00 2024-09-11 18:06:37.314 INFO 2981023 --- [prometheus-cache-monitor1,4e3677195ff7a9a9,382ffc65571d86d5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8431,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5148} 2024-09-11 18:06:37.315 INFO 2981023 --- [prometheus-cache-monitor1,4e3677195ff7a9a9,382ffc65571d86d5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:06:37.388 INFO 2981023 --- [ocp-async-6,f8899397564cb04a,53767a8f46435aa6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:37.780 INFO 2981023 --- [ocp-async-9,bc93dd6be2a624dc,79d3b7e62d9c5a9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:06:38.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,8d16e0877b5c6a9d,e3556469bec56428] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8d16e0877b5c6a9d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:38.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,8d16e0877b5c6a9d,e3556469bec56428] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8d16e0877b5c6a9d, duration=9 ms] 2024-09-11 18:06:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,5856fcd36c6b701c,dd91665f84f1dd40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,5856fcd36c6b701c,dd91665f84f1dd40] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:38.903 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:38.905 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:39.051 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:06:39.051 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:06:39.053 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:06:39.053 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:06:39.053 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:06:39.053 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:06:39.056 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:06:39.056 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:06:39.056 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:06:39.056 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:06:39.077 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:06:39.081 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:06:39.081 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:06:39.081 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:06:39.081 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:06:39.081 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:06:39.085 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:06:39.334 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:06:39.388 INFO 2981023 --- [ocp-async-2,51a477812715fbcd,01fbb07e040a2ab4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:39.616 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.560632248S 2024-09-11 18:06:39.617 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:06:39.663 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:06:39.663 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:06:39.678 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:06:39.713 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:06:39.713 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:06:39.713 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:06:39.713 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:06:39.713 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:06:39.713 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:06:39.721 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:06:39.808 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] o.s.boot.SpringApplication : Started application in 0.755 seconds (JVM running for 84999.681) 2024-09-11 18:06:39.809 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.755161371S 2024-09-11 18:06:40.003 INFO 2981023 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:06:40.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:06:40.006 INFO 2981023 --- [ocp-schedules-23,d732a75242c799b5,2808aeabc96835bd] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:06:40.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,827b2c4f23ea9c44,ce731156caf00783] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=827b2c4f23ea9c44, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:40.127 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,827b2c4f23ea9c44,ce731156caf00783] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=827b2c4f23ea9c44, duration=21 ms] 2024-09-11 18:06:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,490303e616ae0a32,cec1f7d6724321c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,490303e616ae0a32,cec1f7d6724321c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:41.388 INFO 2981023 --- [ocp-async-7,1def7ccce36a46b0,656a057eeb8bedf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:41.444 INFO 2981023 --- [ocp-schedules-23,c99e772fc3791e30,603bd40bcb29aad9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:06:41.474 WARN 2981023 --- [scanner-13607337-1,c99e772fc3791e30,30cd37c22f69853a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:06:41.905 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:41.905 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:42.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,904bbbf9133edfdd,5c66e99b3df3c4da] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=904bbbf9133edfdd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:42.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,904bbbf9133edfdd,5c66e99b3df3c4da] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=904bbbf9133edfdd, duration=9 ms] 2024-09-11 18:06:42.488 INFO 2981023 --- [ocp-schedules-3,a49081ab45419480,56f3c576d6a390f7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:06:43.388 INFO 2981023 --- [ocp-async-8,fcae684a66e01116,3a0948660d5e2a65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:44.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,6e3e6dbf842d08dc,82a6feba980d1768] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6e3e6dbf842d08dc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:44.129 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,6e3e6dbf842d08dc,82a6feba980d1768] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6e3e6dbf842d08dc, duration=23 ms] 2024-09-11 18:06:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,e0b76f1eff8e6141,ad2bea1a6cc7e726] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:44.318 INFO 2981023 --- [prometheus-minute-schedule-1,e0b76f1eff8e6141,ad2bea1a6cc7e726] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:44.483 INFO 2981023 --- [ocp-schedules-24,0e7ef5de1b9feba9,2e81abc716023d0f] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:06:44.503 WARN 2981023 --- [ocp-schedules-24,0e7ef5de1b9feba9,2e81abc716023d0f] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:06:44.906 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:44.907 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:45.101 INFO 2981023 --- [prometheus-second-schedule-1,b04b23d4ce8ca28f,334c027e5fd7477b] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:06:45.388 INFO 2981023 --- [ocp-async-1,3f0fcf8c7c8c2d10,c45f693555faa7cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:46.018 INFO 2981023 --- [ocp-schedules-1,affb687c68f314de,236c310d5c7b1d2b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:06:46.019 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:06:46.019 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:06:46.020 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:06:46.021 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:06:46.021 INFO 2981023 --- [ocp-schedules-11,1c2d77e6382ce0d2,155bbef1ac348cdf] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:06:46.022 INFO 2981023 --- [host-check-clockdiff1,1c2d77e6382ce0d2,9bb0a5f26277d1c3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:06:46.022 INFO 2981023 --- [host-check-clockdiff2,1c2d77e6382ce0d2,39ae7b5dd93af3b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:06:46.025 INFO 2981023 --- [host-check-clockdiff3,1c2d77e6382ce0d2,ad038fbc24022d52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:06:46.026 INFO 2981023 --- [host-check-clockdiff4,1c2d77e6382ce0d2,18ed559eac793fb0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:06:46.028 INFO 2981023 --- [ocp-schedules-24,50df7bd023f8a05c,2de2b0fefd087d0f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:06:46.029 INFO 2981023 --- [host-check-agent-status1,50df7bd023f8a05c,00257709045c62f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:06:46.029 INFO 2981023 --- [host-check-agent-status2,50df7bd023f8a05c,360cfa8c129ead39] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:06:46.029 INFO 2981023 --- [host-check-agent-status3,50df7bd023f8a05c,5acc720bd92ad5fe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:06:46.029 INFO 2981023 --- [host-check-agent-status4,50df7bd023f8a05c,36fd3528989cb16d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:06:46.064 INFO 2981023 --- [host-check-clockdiff3,1c2d77e6382ce0d2,ad038fbc24022d52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049206 0 0 , error= 2024-09-11 18:06:46.068 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:06:46.070 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:06:46.074 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:06:46.074 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:06:46.080 INFO 2981023 --- [host-check-agent-status3,50df7bd023f8a05c,5acc720bd92ad5fe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:06:46.081 INFO 2981023 --- [host-check-agent-status2,50df7bd023f8a05c,360cfa8c129ead39] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:06:46.081 INFO 2981023 --- [host-check-agent-status3,50df7bd023f8a05c,5acc720bd92ad5fe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:06:46.082 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:06:46.083 INFO 2981023 --- [host-check-agent-status2,50df7bd023f8a05c,360cfa8c129ead39] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:06:46.083 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:06:46.098 INFO 2981023 --- [host-check-agent-status4,50df7bd023f8a05c,36fd3528989cb16d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:06:46.101 INFO 2981023 --- [host-check-agent-status4,50df7bd023f8a05c,36fd3528989cb16d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:06:46.119 INFO 2981023 --- [host-check-agent-status1,50df7bd023f8a05c,00257709045c62f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:06:46.172 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:06:46.173 INFO 2981023 --- [host-check-agent-status1,50df7bd023f8a05c,00257709045c62f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:06:46.187 INFO 2981023 --- [ocp-schedules-20,571738fcaf1cec9c,ff18f247dc3f10ee] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:06:46.189 INFO 2981023 --- [ocp-schedules-3,b3cde13b78c32f5a,3d5dbfe84356e100] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:06:46.192 INFO 2981023 --- [ocp-schedules-2,b657da1b3e370a05,2b3d5d3813021f4b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:06:46.192 INFO 2981023 --- [ocp-schedules-25,f2c1acedf5727b79,ef72dd06fc64cc77] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:06:46.193 INFO 2981023 --- [ocp-schedules-19,adad6fcd76f02b45,341cd3284c4cb641] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:06:46.235 INFO 2981023 --- [ocp-schedules-23,7183ab99ea50bf93,290638fd56641382] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 18:06:46.235 INFO 2981023 --- [ocp-schedules-23,7183ab99ea50bf93,290638fd56641382] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 18:06:46.286 INFO 2981023 --- [ocp-schedules-20,571738fcaf1cec9c,ff18f247dc3f10ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:46.287 INFO 2981023 --- [ocp-schedules-3,b3cde13b78c32f5a,3d5dbfe84356e100] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:46.292 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049162777000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:06:46.294 INFO 2981023 --- [ocp-schedules-3,b3cde13b78c32f5a,3d5dbfe84356e100] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:06:46.353 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:06:46.355 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:06:46.359 INFO 2981023 --- [ocp-schedules-2,b657da1b3e370a05,2b3d5d3813021f4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:46.379 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2b5747ef0442b5ad,87e7c89c70ed9ad0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=2b5747ef0442b5ad, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:06:46.392 INFO 2981023 --- [ocp-schedules-19,adad6fcd76f02b45,341cd3284c4cb641] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:06:46.404 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:06:46.464 INFO 2981023 --- [ocp-schedules-27,57930aafaf5045f1,986199fe02d431d5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 18:06:46.471 INFO 2981023 --- [ocp-schedules-30,60fe89ad9656a520,5fdb2c2e2e343a3d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:06:46.477 INFO 2981023 --- [ocp-schedules-27,57930aafaf5045f1,986199fe02d431d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:46.498 INFO 2981023 --- [host-check-agent-status3,50df7bd023f8a05c,5acc720bd92ad5fe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:06:46.501 INFO 2981023 --- [host-check-agent-status3,50df7bd023f8a05c,5acc720bd92ad5fe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:06:46.501 INFO 2981023 --- [host-check-agent-status3,50df7bd023f8a05c,5acc720bd92ad5fe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:06:46.501 INFO 2981023 --- [host-check-agent-status3,50df7bd023f8a05c,5acc720bd92ad5fe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:06:46.556 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:06:46.558 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:06:46.563771244+08:00 2024-09-11 18:06:46.560 INFO 2981023 --- [host-check-agent-status2,50df7bd023f8a05c,360cfa8c129ead39] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:06:46.561 INFO 2981023 --- [host-check-agent-status1,50df7bd023f8a05c,00257709045c62f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:06:46.563 INFO 2981023 --- [host-check-agent-status1,50df7bd023f8a05c,00257709045c62f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:06:46.563 INFO 2981023 --- [host-check-agent-status1,50df7bd023f8a05c,00257709045c62f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:06:46.563 INFO 2981023 --- [host-check-agent-status1,50df7bd023f8a05c,00257709045c62f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:06:46.563 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:06:46.558+08:00 2024-09-11 18:06:46.564 INFO 2981023 --- [host-check-agent-status2,50df7bd023f8a05c,360cfa8c129ead39] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:06:46.564 INFO 2981023 --- [host-check-agent-status2,50df7bd023f8a05c,360cfa8c129ead39] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:06:46.564 INFO 2981023 --- [host-check-agent-status2,50df7bd023f8a05c,360cfa8c129ead39] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:06:46.566 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:06:46.563122+08:00, lastProcessTime=2024-09-11T18:06:36.354809+08:00 2024-09-11 18:06:46.570 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:06:46.558+08:00 and status AVAILABLE 2024-09-11 18:06:46.571 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2b5747ef0442b5ad,87e7c89c70ed9ad0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=2b5747ef0442b5ad, duration=196 ms] 2024-09-11 18:06:46.573 INFO 2981023 --- [host-check-agent-status4,50df7bd023f8a05c,36fd3528989cb16d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:06:46.579 INFO 2981023 --- [host-check-agent-status4,50df7bd023f8a05c,36fd3528989cb16d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:06:46.579 INFO 2981023 --- [host-check-agent-status4,50df7bd023f8a05c,36fd3528989cb16d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:06:46.579 INFO 2981023 --- [host-check-agent-status4,50df7bd023f8a05c,36fd3528989cb16d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:06:46.585 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:06:46.586 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:06:46.586169193+08:00 2024-09-11 18:06:46.588 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:06:46.588 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:06:46.588613329+08:00 2024-09-11 18:06:46.589 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:06:46.588+08:00 2024-09-11 18:06:46.590 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:06:46.586+08:00 2024-09-11 18:06:46.592 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:06:46.594 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:06:46.586+08:00 and status AVAILABLE 2024-09-11 18:06:46.595 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:06:46.588+08:00 and status AVAILABLE 2024-09-11 18:06:46.600 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.600 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:06:46.604 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:06:46.604 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.607 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:06:46.610 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.610 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:06:46.610 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:06:46.611 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:06:46.616518313+08:00 2024-09-11 18:06:46.611 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:06:46.615 INFO 2981023 --- [ocp-schedules-20,571738fcaf1cec9c,ff18f247dc3f10ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:06:46.615 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:06:46.616 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:06:46.611+08:00 2024-09-11 18:06:46.616 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f69f6311624bbefa,7c69f76b54f9f4c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=f69f6311624bbefa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 18:06:46.616 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.616 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,f69f6311624bbefa,7c69f76b54f9f4c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=f69f6311624bbefa, duration=1 ms] 2024-09-11 18:06:46.616 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:46.620 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:06:46.611+08:00 and status AVAILABLE 2024-09-11 18:06:46.621 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.621 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:06:46.631 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.634 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 18:06:46.655 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:06:46.659 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.659 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:06:46.661 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:06:46.663 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.664 INFO 2981023 --- [pool-10-thread-14,3b242f040dc0e184,c9f1585c64f7480d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 18:06:46.674 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.674 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:46.674 INFO 2981023 --- [pool-10-thread-13,3b242f040dc0e184,09747cf6b9950834] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 18:06:46.675 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:06:46.677 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,7d6f796c971cb102] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 18:06:46.677 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,7d6f796c971cb102] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 18:06:46.677 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:06:46.679 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:06:46.681 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.681 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 18:06:46.681 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 18:06:46.682 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:06:46.691 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:06:46.703 INFO 2981023 --- [pool-10-thread-13,3b242f040dc0e184,09747cf6b9950834] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 18:06:46.703 INFO 2981023 --- [pool-10-thread-13,3b242f040dc0e184,09747cf6b9950834] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 18:06:46.704 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,daa94b845f4e6c34,5c3b4784381b324c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=daa94b845f4e6c34, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 18:06:46.704 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,daa94b845f4e6c34,5c3b4784381b324c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=daa94b845f4e6c34, duration=1 ms] 2024-09-11 18:06:46.705 INFO 2981023 --- [pool-10-thread-15,3b242f040dc0e184,e7edf626ef58f4ba] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 18:06:46.706 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.706 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:46.709 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.710 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:06:46.713 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 18:06:46.713 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 18:06:46.715 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.717 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:06:46.717 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 18:06:46.719 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:06:46.719 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:06:46.721 INFO 2981023 --- [pool-10-thread-16,3b242f040dc0e184,ec4ff475930f87cb] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 18:06:46.722 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:06:46.725 INFO 2981023 --- [ocp-schedules-27,57930aafaf5045f1,986199fe02d431d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 18:06:46.730 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:06:46.732 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:06:46.760 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:46.760 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.761 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:06:46.774 WARN 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,7d6f796c971cb102] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 18:06:46.775 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:06:46.778 INFO 2981023 --- [pool-10-thread-16,3b242f040dc0e184,ec4ff475930f87cb] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 18:06:46.782 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.782 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:06:46.783 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:06:46.784 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.784 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:06:46.785 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 18:06:46.790 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.791 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:06:46.794 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f1a4198edb6a30bd,97d463ad3c13cca5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f1a4198edb6a30bd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:46.797 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.798 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:06:46.800 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:06:46.821 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:46.827 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.828 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:06:46.829 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:06:46.830 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.835 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:06:46.848 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:06:46.854 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:06:46.854 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049162777000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:06:46.859 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:46.859 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:06:46.863 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,f1a4198edb6a30bd,97d463ad3c13cca5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f1a4198edb6a30bd, duration=80 ms] 2024-09-11 18:06:46.864 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:06:46.864 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.865 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:46.867 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:06:46.867 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.867 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:06:46.870 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.870 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:06:46.870 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:06:46.871 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:06:46, 2024-09-11 18:06:46] 2024-09-11 18:06:46.872 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 18:06:46.872 INFO 2981023 --- [ForkJoinPool-6-worker-6,3b242f040dc0e184,4e64ad70104a61d0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 18:06:46.874 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:06:46.876 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:06:46.879 INFO 2981023 --- [ForkJoinPool-6-worker-15,3b242f040dc0e184,14a1228368f2b744] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.884 INFO 2981023 --- [ForkJoinPool-6-worker-15,3b242f040dc0e184,14a1228368f2b744] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:06:46.890 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:06:46.891 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:06:46.904 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:06:46.911 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.913 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:06:46.925 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:06:46.925 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:06:46.929 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.929 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:06:46.930 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.931 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:06:46.932 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 18:06:46.939 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:06:46.940 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:06:46.941 INFO 2981023 --- [ForkJoinPool-6-worker-15,3b242f040dc0e184,14a1228368f2b744] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:06:46.941 INFO 2981023 --- [ForkJoinPool-6-worker-15,3b242f040dc0e184,14a1228368f2b744] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.942 INFO 2981023 --- [ForkJoinPool-6-worker-15,3b242f040dc0e184,14a1228368f2b744] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:06:46.943 INFO 2981023 --- [ForkJoinPool-6-worker-15,3b242f040dc0e184,14a1228368f2b744] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.944 INFO 2981023 --- [ForkJoinPool-6-worker-15,3b242f040dc0e184,14a1228368f2b744] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:06:46.948 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:06:46.948 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.949 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:06:46.953 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.953 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:06:46.954 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:06:46.957 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:06:46.958 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.962 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:06:46, 2024-09-11 18:06:46] 2024-09-11 18:06:46.963 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:46.966 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 18:06:46.980 INFO 2981023 --- [host-check-clockdiff2,1c2d77e6382ce0d2,39ae7b5dd93af3b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049206 5 5 , error= 2024-09-11 18:06:46.982 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:06:46.987 INFO 2981023 --- [host-check-clockdiff1,1c2d77e6382ce0d2,9bb0a5f26277d1c3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049206 5 5 , error= 2024-09-11 18:06:46.988 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:46.998 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:06:47.000 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:06:47.000 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:06:47.001 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:06:47.002 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.003 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.003 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:06:47.004 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:06:47.006 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.007 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.007 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:06:47, 2024-09-11 18:06:47] 2024-09-11 18:06:47.007 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.007 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:06:47.008 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.008 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:06:47.008 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 18:06:47.008 INFO 2981023 --- [ForkJoinPool-6-worker-4,3b242f040dc0e184,4aec83608b3d6cfa] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 18:06:47.010 INFO 2981023 --- [pool-10-thread-14,3b242f040dc0e184,c9f1585c64f7480d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 18:06:47.011 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:06:47.011 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.011 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:06:47.017 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:06:47.017 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:06:47.021 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.023 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:06:47.023 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:06:47.025 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:06:47.026 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:06:47.037 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:06:47.038 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:06:47.053 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:06:47.054 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:06:47.055 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:06:47.055 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.057 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:06:47.058 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:06:47.059 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.059 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.059 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.060 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:06:47.060 INFO 2981023 --- [ForkJoinPool-6-worker-13,3b242f040dc0e184,52e376d8871b3d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:06:47.061 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.061 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:06:47.064 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.065 INFO 2981023 --- [pool-10-thread-15,3b242f040dc0e184,e7edf626ef58f4ba] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 18:06:47.068 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:06:47.076 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 18:06:47.093 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:06:47.094 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:06:47.095 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:06:47.099 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.100 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.100 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:06:47.101 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:06:47.101 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.106 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:06:47.123 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:06:47.124 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:06:47.125 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:06:47.127 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:06:47.136 INFO 2981023 --- [host-check-host-status4,affb687c68f314de,00a0e5d880d89f4d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:06:47.143 INFO 2981023 --- [host-check-host-status1,affb687c68f314de,1c2c15b67649a41b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:06:47.150 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:06:47.152 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:06:47.166 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.167 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.168 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:06:47.168 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:06:47.169 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.170 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:06:47.171 INFO 2981023 --- [host-check-host-status2,affb687c68f314de,3e128dcba1da8611] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:06:47.171 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:06:47.173 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:06:47.177 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.178 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.179 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:06:47.179 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:06:47.179 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.188 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:06:47.188 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:06:47.191 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:06:47.200 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.204 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.204 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:06:47.222 INFO 2981023 --- [pool-10-thread-16,3b242f040dc0e184,ec4ff475930f87cb] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 18:06:47.225 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:06:47.225 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:06:47.227 INFO 2981023 --- [pool-10-thread-16,3b242f040dc0e184,ec4ff475930f87cb] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 18:06:47.236 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:06:47.236 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:06:47.243 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:06:47.245 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.246 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.246 INFO 2981023 --- [host-check-host-status3,affb687c68f314de,8b59fabadfccb7a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:06:47.250 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:06:47.250 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:06:47.254 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:06:46.563122+08:00 2024-09-11 18:06:47.254 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:06:47.258 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:06:47.259 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:06:47.261 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:06:47.269 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:06:47.272 INFO 2981023 --- [ocp-schedules-18,83f950b52db690a4,7b6cfb3c272c5fe9] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:06:47.275 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:06:47.303 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:06:47.309 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:47.309 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:47.310 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:06:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,d3c8231458be5800,50910bfa4cf80a1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:47.315 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:06:47.315 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:06:47.315 INFO 2981023 --- [ocp-schedules-2,b657da1b3e370a05,2b3d5d3813021f4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:06:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,d3c8231458be5800,50910bfa4cf80a1a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:47.388 INFO 2981023 --- [ocp-async-1,2223bd88b47efe81,dfc163b63f863d4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:47.781 INFO 2981023 --- [ocp-async-9,f82495d2c01ad80a,5a5da36a42cebbc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:06:47.907 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:47.908 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:49.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,739c8c1e4880b017,5580aa6a109baea6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=739c8c1e4880b017, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:49.132 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,739c8c1e4880b017,5580aa6a109baea6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=739c8c1e4880b017, duration=25 ms] 2024-09-11 18:06:49.389 INFO 2981023 --- [ocp-async-3,ddef9629e2901d41,0674d2413c894361] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:06:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:06:50.307 INFO 2981023 --- [ocp-schedules-5,08dfae20a478381c,54554dc63b69e0aa] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:06:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,043bebdb82c432ff,6614a41b111bd548] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,043bebdb82c432ff,6614a41b111bd548] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:50.908 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:50.910 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:51.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ba0a69d04a6660d8,4a2d39d7bcd9242e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ba0a69d04a6660d8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:51.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ba0a69d04a6660d8,4a2d39d7bcd9242e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ba0a69d04a6660d8, duration=9 ms] 2024-09-11 18:06:51.389 INFO 2981023 --- [ocp-async-9,8c78fba52c5f0281,ecbb3df012efaad5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:51.600 INFO 2981023 --- [ocp-schedules-6,fe693d5037a99c57,c98c7a669ec8a376] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:06:51.601 INFO 2981023 --- [ocp-schedules-6,fe693d5037a99c57,c98c7a669ec8a376] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:51.602 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:06:51.603 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:06:51.603 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:06:51.603 INFO 2981023 --- [ocp-schedules-6,fe693d5037a99c57,c98c7a669ec8a376] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:06:51.643 INFO 2981023 --- [ocp-schedules-12,b62f7e28f3315b3b,cabce4f9dbf869d3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:06:51.721 WARN 2981023 --- [scanner-13607588-1,b62f7e28f3315b3b,af51faf318db019b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:06:51.747 INFO 2981023 --- [alarm-detect-13,b62f7e28f3315b3b,a01b5b0556d4d47d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645432, data is empty. 2024-09-11 18:06:51.752 INFO 2981023 --- [alarm-detect-13,b62f7e28f3315b3b,a01b5b0556d4d47d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645432, data is empty. 2024-09-11 18:06:51.756 INFO 2981023 --- [alarm-detect-13,b62f7e28f3315b3b,a01b5b0556d4d47d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645432, data is empty. 2024-09-11 18:06:51.762 INFO 2981023 --- [alarm-detect-8,b62f7e28f3315b3b,bed678bc99a7f86e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645432, data is empty. 2024-09-11 18:06:51.762 INFO 2981023 --- [alarm-detect-19,b62f7e28f3315b3b,414a405b966d08b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645432, data is empty. 2024-09-11 18:06:51.767 INFO 2981023 --- [alarm-detect-19,b62f7e28f3315b3b,414a405b966d08b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645432, data is empty. 2024-09-11 18:06:51.767 INFO 2981023 --- [alarm-detect-8,b62f7e28f3315b3b,bed678bc99a7f86e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645432, data is empty. 2024-09-11 18:06:51.775 INFO 2981023 --- [alarm-detect-19,b62f7e28f3315b3b,414a405b966d08b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645432, data is empty. 2024-09-11 18:06:51.776 INFO 2981023 --- [alarm-detect-8,b62f7e28f3315b3b,bed678bc99a7f86e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645432, data is empty. 2024-09-11 18:06:51.875 INFO 2981023 --- [alarm-detect-20,b62f7e28f3315b3b,328d0c86f4029c0d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645432, data is empty. 2024-09-11 18:06:51.879 INFO 2981023 --- [alarm-detect-20,b62f7e28f3315b3b,328d0c86f4029c0d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645432, data is empty. 2024-09-11 18:06:51.884 INFO 2981023 --- [alarm-detect-20,b62f7e28f3315b3b,328d0c86f4029c0d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645432, data is empty. 2024-09-11 18:06:52.028 INFO 2981023 --- [host-check-clockdiff4,1c2d77e6382ce0d2,18ed559eac793fb0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049212 0 0 , error= 2024-09-11 18:06:52.490 INFO 2981023 --- [ocp-schedules-15,c6d7a9a8370a5229,8e9202d971661823] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:06:53.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,200e810c92358d7e,c4bb2c402410e33d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=200e810c92358d7e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:53.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,200e810c92358d7e,c4bb2c402410e33d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=200e810c92358d7e, duration=10 ms] 2024-09-11 18:06:53.261 INFO 2981023 --- [ocp-schedules-25,3b749b0e2101a9af,928bad9b14d05ec6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:06:53.261 INFO 2981023 --- [ocp-schedules-25,3b749b0e2101a9af,928bad9b14d05ec6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:06:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,1a693b672a1db9c4,27c401214c805e0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,1a693b672a1db9c4,27c401214c805e0c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:53.389 INFO 2981023 --- [ocp-async-1,e9f701d03d72bd19,f3bbf6d6437b3241] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:53.910 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:53.912 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:54.399 INFO 2981023 --- [ocp-schedules-5,987088fff3964d4f,bb3aae81049e78cb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:06:54.400 INFO 2981023 --- [ocp-schedules-4,dacb76e9e4527fa6,3a394c738726fb68] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:06:54.419 WARN 2981023 --- [ocp-schedules-4,dacb76e9e4527fa6,3a394c738726fb68] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:06:54.990 INFO 2981023 --- [ocp-schedules-25,ccda45ed25b86c39,48f76212e674fb21] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:06:54.990 INFO 2981023 --- [ocp-schedules-25,ccda45ed25b86c39,48f76212e674fb21] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:06:54.995 INFO 2981023 --- [ocp-schedules-25,ccda45ed25b86c39,48f76212e674fb21] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:06:54.995 INFO 2981023 --- [ocp-schedules-25,ccda45ed25b86c39,48f76212e674fb21] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:06:54.995 INFO 2981023 --- [ocp-schedules-25,ccda45ed25b86c39,48f76212e674fb21] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:06:55.005 INFO 2981023 --- [ocp-schedules-25,ccda45ed25b86c39,48f76212e674fb21] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:06:55.005 INFO 2981023 --- [ocp-schedules-25,ccda45ed25b86c39,48f76212e674fb21] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:06:55.005 INFO 2981023 --- [ocp-schedules-25,ccda45ed25b86c39,48f76212e674fb21] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:06:55.018 INFO 2981023 --- [ocp-schedules-25,ccda45ed25b86c39,48f76212e674fb21] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:06:55.085 INFO 2981023 --- [ocp-schedules-27,00207502b409268a,11d05d4c94e6fe70] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:06:55.093 INFO 2981023 --- [ocp-schedules-27,00207502b409268a,11d05d4c94e6fe70] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:06:55.093 INFO 2981023 --- [ocp-schedules-27,00207502b409268a,11d05d4c94e6fe70] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:06:55.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,107242fecf4f228e,daa63d1005097f36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=107242fecf4f228e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:55.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,107242fecf4f228e,daa63d1005097f36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=107242fecf4f228e, duration=15 ms] 2024-09-11 18:06:55.389 INFO 2981023 --- [ocp-async-5,9f4ed65a3358319d,69dca37a81e13f0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:55.844 INFO 2981023 --- [ocp-schedules-20,8799c175243fa0bb,c4a69d0c5eb3ecc3] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:06:55.844 INFO 2981023 --- [ocp-schedules-20,8799c175243fa0bb,c4a69d0c5eb3ecc3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:06:55.846 INFO 2981023 --- [ocp-schedules-20,8799c175243fa0bb,c4a69d0c5eb3ecc3] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:06:55.846 INFO 2981023 --- [ocp-schedules-20,8799c175243fa0bb,c4a69d0c5eb3ecc3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:06:55.849 INFO 2981023 --- [ocp-schedules-20,8799c175243fa0bb,c4a69d0c5eb3ecc3] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:06:55.849 INFO 2981023 --- [ocp-schedules-20,8799c175243fa0bb,c4a69d0c5eb3ecc3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:06:56.004 INFO 2981023 --- [ocp-schedules-4,2615cb322aac6440,c0bcd97e179f776c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:06:56.010 INFO 2981023 --- [ocp-schedules-4,2615cb322aac6440,c0bcd97e179f776c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:06:56.010 INFO 2981023 --- [ocp-schedules-4,2615cb322aac6440,c0bcd97e179f776c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:06:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,b8c7c86d010da012,7a6552208885bfd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,b8c7c86d010da012,7a6552208885bfd6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:06:56.352343+08:00, lastProcessTime=2024-09-11T18:06:46.563122+08:00 2024-09-11 18:06:56.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:06:56.352343+08:00 2024-09-11 18:06:56.912 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:56.913 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:06:57.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,14146929908943af,4a167b954537ce46] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=14146929908943af, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:57.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,14146929908943af,4a167b954537ce46] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=14146929908943af, duration=13 ms] 2024-09-11 18:06:57.389 INFO 2981023 --- [ocp-async-2,1fce8c6f3294cf37,df8ed3a4a8aedc77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:57.451 INFO 2981023 --- [ocp-schedules-5,f7c3a316090e0cda,23f97e1d8e3bba05] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:06:57.451+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:06:57.455 INFO 2981023 --- [ocp-schedules-5,f7c3a316090e0cda,23f97e1d8e3bba05] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:06:57.573 INFO 2981023 --- [ocp-schedules-21,23d3cceb88a3752d,67f17fa6ea607bd5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:06:57.573 INFO 2981023 --- [ocp-schedules-21,23d3cceb88a3752d,67f17fa6ea607bd5] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:06:57.594 INFO 2981023 --- [Collect all tenants major compactions1,23d3cceb88a3752d,93fb620d014bbbc6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:06:57.599 INFO 2981023 --- [Collect all tenants major compactions2,23d3cceb88a3752d,1ca64df0fc942ff5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:06:57.603 INFO 2981023 --- [Collect all tenants major compactions4,23d3cceb88a3752d,0c4141695a749269] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:06:57.603 INFO 2981023 --- [Collect all tenants major compactions6,23d3cceb88a3752d,960f18a6aa6c61d0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:06:57.603 INFO 2981023 --- [Collect all tenants major compactions3,23d3cceb88a3752d,66a924d7c2c9395d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:06:57.605 INFO 2981023 --- [Collect all tenants major compactions5,23d3cceb88a3752d,449ca86ca2126337] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:06:57.622 INFO 2981023 --- [ocp-schedules-5,f7c3a316090e0cda,23f97e1d8e3bba05] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 49 to SqlAggregateData 2024-09-11 18:06:57.625 INFO 2981023 --- [Collect all tenants major compactions4,23d3cceb88a3752d,0c4141695a749269] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:57.627 INFO 2981023 --- [ocp-schedules-5,f7c3a316090e0cda,23f97e1d8e3bba05] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:06:57.627 INFO 2981023 --- [Collect all tenants major compactions4,23d3cceb88a3752d,0c4141695a749269] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:06:57.627 INFO 2981023 --- [Collect all tenants major compactions5,23d3cceb88a3752d,449ca86ca2126337] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:57.629 INFO 2981023 --- [Collect all tenants major compactions5,23d3cceb88a3752d,449ca86ca2126337] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:06:57.630 INFO 2981023 --- [Collect all tenants major compactions3,23d3cceb88a3752d,66a924d7c2c9395d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:57.630 INFO 2981023 --- [Collect all tenants major compactions4,23d3cceb88a3752d,0c4141695a749269] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:06:57.631 INFO 2981023 --- [Collect all tenants major compactions3,23d3cceb88a3752d,66a924d7c2c9395d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:06:57.631 INFO 2981023 --- [Collect all tenants major compactions6,23d3cceb88a3752d,960f18a6aa6c61d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:57.631 INFO 2981023 --- [Collect all tenants major compactions1,23d3cceb88a3752d,93fb620d014bbbc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:57.632 INFO 2981023 --- [Collect all tenants major compactions6,23d3cceb88a3752d,960f18a6aa6c61d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:06:57.633 INFO 2981023 --- [Collect all tenants major compactions3,23d3cceb88a3752d,66a924d7c2c9395d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:06:57.633 INFO 2981023 --- [Collect all tenants major compactions2,23d3cceb88a3752d,1ca64df0fc942ff5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:57.634 INFO 2981023 --- [Collect all tenants major compactions4,23d3cceb88a3752d,0c4141695a749269] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:06:57.634 INFO 2981023 --- [Collect all tenants major compactions6,23d3cceb88a3752d,960f18a6aa6c61d0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:06:57.634 INFO 2981023 --- [Collect all tenants major compactions6,23d3cceb88a3752d,960f18a6aa6c61d0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:06:57.636 INFO 2981023 --- [Collect all tenants major compactions3,23d3cceb88a3752d,66a924d7c2c9395d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:06:57.637 INFO 2981023 --- [Collect all tenants major compactions5,23d3cceb88a3752d,449ca86ca2126337] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:06:57.637 INFO 2981023 --- [Collect all tenants major compactions5,23d3cceb88a3752d,449ca86ca2126337] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:06:57.637 INFO 2981023 --- [Collect all tenants major compactions1,23d3cceb88a3752d,93fb620d014bbbc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:06:57.637 INFO 2981023 --- [Collect all tenants major compactions2,23d3cceb88a3752d,1ca64df0fc942ff5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:06:57.638 INFO 2981023 --- [Collect all tenants major compactions1,23d3cceb88a3752d,93fb620d014bbbc6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:06:57.638 INFO 2981023 --- [Collect all tenants major compactions1,23d3cceb88a3752d,93fb620d014bbbc6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:06:57.638 INFO 2981023 --- [Collect all tenants major compactions2,23d3cceb88a3752d,1ca64df0fc942ff5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:06:57.638 INFO 2981023 --- [Collect all tenants major compactions2,23d3cceb88a3752d,1ca64df0fc942ff5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:06:57.639 INFO 2981023 --- [ocp-schedules-21,23d3cceb88a3752d,67f17fa6ea607bd5] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:06:57.781 INFO 2981023 --- [ocp-async-5,991daeeb30bd8b2a,0a55085c335c1117] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:06:58.441 INFO 2981023 --- [ocp-schedules-26,c47eb6c99c051238,11b5cbd433811d9a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:06:58.442 INFO 2981023 --- [ocp-schedules-26,c47eb6c99c051238,11b5cbd433811d9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:06:58.474 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:06:58.474 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.475 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:06:58.475 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.476 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:06:58.476 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.478 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:06:58.479 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:06:58.479 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:06:58.484 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.485 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.486 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.486 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.487 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.487 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.490 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.491 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.497 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.497 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.498 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.498 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:06:58.499 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.500 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.506 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.507 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:06:58.514 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.514 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.515 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.515 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:06:58.515 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:06:58.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4B21D6DA-0-0] 2024-09-11 18:06:58.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:06:58.516 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.516 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.516 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:06:58.516 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.525 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.526 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:06:58.529 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.529 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:06:58.530 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.530 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:06:58.531 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.532 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:06:58.532 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.533 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:06:58.533 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:06:58.540 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.540 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:06:58.541 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.542 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:06:58.542 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.542 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:06:58.543 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.543 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:06:58.543 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:06:58.543 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4B21D6E4-0-0] 2024-09-11 18:06:58.543 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:06:58.543 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:06:58.544 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.545 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:06:58.545 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.546 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:06:58.547 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.548 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:06:58.548 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.549 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:06:58.550 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.550 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.551 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:06:58.551 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:06:58.551 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.552 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.552 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:06:58.552 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:06:58.553 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:06:58.553 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.553 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:06:58.554 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.555 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:06:58.555 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:06:58.555 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.556 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:06:58.556 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.557 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:06:58.557 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:06:58.559 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:06:58.561 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:58.564 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.565 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.569 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:58.571 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.572 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:06:58.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:06:58.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:06:58.579 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:06:58.579 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:06:58.581 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:06:58.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:06:58.585 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:06:58.586 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:06:58.586 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:06:58.586 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:06:58.587 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:06:58.587 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.587 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.588 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:06:58.589 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:06:58.590 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:06:58.590 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:06:58.596 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:06:58.597 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:06:58.597 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:06:58.597 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:06:58.598 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:06:58.599 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:06:58.599 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:06:58.599 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:06:58.599 INFO 2981023 --- [ocp-schedules-26,c47eb6c99c051238,11b5cbd433811d9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:06:59.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,3b3e617de8c6e39e,1a28ae443cc04018] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=3b3e617de8c6e39e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:06:59.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,3b3e617de8c6e39e,1a28ae443cc04018] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=3b3e617de8c6e39e, duration=23 ms] 2024-09-11 18:06:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,a212ed8b3eb7daf0,916798464a00851a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049220, interval=60 2024-09-11 18:06:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,a212ed8b3eb7daf0,916798464a00851a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:06:59.389 INFO 2981023 --- [ocp-async-6,3843c4a1ac7ebd97,5b069b3f6e0f4808] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:06:59.913 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:06:59.914 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:01.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,77be07c6ac5d3a7f,1b93708a9ebf990c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=77be07c6ac5d3a7f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:01.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,77be07c6ac5d3a7f,1b93708a9ebf990c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=77be07c6ac5d3a7f, duration=12 ms] 2024-09-11 18:07:01.391 INFO 2981023 --- [ocp-async-3,df6e6bf6c4f06f6d,496276ae653589de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:01.741 INFO 2981023 --- [ocp-schedules-9,e3f94cd0b09950fe,ffb23c9193111d8b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:07:01.934 INFO 2981023 --- [ocp-schedules-8,8cc418a3035f217d,1166c353c72cd68c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:07:02.001 WARN 2981023 --- [scanner-13607909-1,8cc418a3035f217d,1023f876fbac1cf8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:07:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,8b9fcf84a7ae8a40,ec10322c31c03cba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,8b9fcf84a7ae8a40,ec10322c31c03cba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:02.493 INFO 2981023 --- [ocp-schedules-29,d0317a187beed7ae,80d2f76ac010a03e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:07:02.914 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:02.915 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:03.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,9e9cf33bcef4aeba,6148e8026d299da4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9e9cf33bcef4aeba, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:03.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,9e9cf33bcef4aeba,6148e8026d299da4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9e9cf33bcef4aeba, duration=9 ms] 2024-09-11 18:07:03.389 INFO 2981023 --- [ocp-async-9,08e2994924212548,09ddb7a0781ee2e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:05.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9ba2115f3e2144c0,4bb9a90b9c446227] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9ba2115f3e2144c0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:05.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,9ba2115f3e2144c0,4bb9a90b9c446227] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9ba2115f3e2144c0, duration=15 ms] 2024-09-11 18:07:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,eaa16d787e42bc77,124f6c21b92295b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:05.318 INFO 2981023 --- [prometheus-minute-schedule-1,eaa16d787e42bc77,124f6c21b92295b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:05.390 INFO 2981023 --- [ocp-async-1,642ff920ddd4d29e,5fac7cffa9164f56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:05.915 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:05.916 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:07:06.352628+08:00, lastProcessTime=2024-09-11T18:06:56.352343+08:00 2024-09-11 18:07:06.495 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:07:06.352628+08:00 2024-09-11 18:07:07.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ca8272580936722d,f91bb0667f4b9b34] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ca8272580936722d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:07.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ca8272580936722d,f91bb0667f4b9b34] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ca8272580936722d, duration=9 ms] 2024-09-11 18:07:07.390 INFO 2981023 --- [ocp-async-5,a70c5068a0bcfcde,4f34584179cfdde3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:07.781 INFO 2981023 --- [ocp-async-4,58331d9fb0b01d0e,e0ff9be2637bef5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:07:07.819 INFO 2981023 --- [ocp-schedules-24,287d3f81e63838c6,a1060a2db66adc48] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:07:07.934 INFO 2981023 --- [ocp-schedules-19,04494791c6ffbb31,f643452555987dcf] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:07:07.934+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:07:07.935 INFO 2981023 --- [ocp-schedules-19,04494791c6ffbb31,f643452555987dcf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:07:07.938 INFO 2981023 --- [ocp-schedules-19,04494791c6ffbb31,f643452555987dcf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:07:07.938 INFO 2981023 --- [ocp-schedules-19,04494791c6ffbb31,f643452555987dcf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:07:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,4b55ef043b9952ab,70efd2810309b9f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,4b55ef043b9952ab,70efd2810309b9f4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:08.639 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:03:08.175, end=2024-09-11T18:05:08.635} 2024-09-11 18:07:08.642 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, 2024-09-11 18:07:08.642 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:07:08.644 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:03:08.175, end=2024-09-11T18:05:08.635},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:07:08.644 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:07:08.644 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:07:08.659 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:07:08.682 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828682000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.682Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.686 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:07:08.686 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.691 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.691 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.695 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.695 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.700 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.700 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.701 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:07:08.701 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.682, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.701 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:07:08.703 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:03:08.175, end=2024-09-11T18:05:08.635},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:07:08.709 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:07:08.709 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:07:08.752 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828752000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.752Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.757 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828757000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.757Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.758 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828758000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.758Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.758 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828758000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.758Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.761 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828761000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.761Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.763 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828763000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.763Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.763 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828763000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.763Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.764 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828764000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.764Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.778 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 66 2024-09-11 18:07:08.778 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.787 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.787 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.814 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.814 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.815 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.815 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.829 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.829 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.832 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:07:08.832 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.835 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:07:08.835 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.837 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:07:08.837 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.837 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.837 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.839 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.839 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.842 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.842 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.843 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.843 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.847 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.847 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.849 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.849 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.850 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.850 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.850 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.850 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.852 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.852 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.853 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.853 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.853 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.853 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.853 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.853 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.855 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.855 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.855 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.855 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.855 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.855 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.856 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.857 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.857 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.857 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.857 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.858 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.858 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.860 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.863 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.863 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.867 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.869 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.869 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.869 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.869 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.870 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.870 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.871 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.752, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.871 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.871 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.872 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.872 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.872 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.873 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.873 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.758, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.874 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.874 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.874 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.875 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.757, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.880 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.761, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.880 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.763, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.882 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.764, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.882 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.763, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.883 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.758, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.905 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828905000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.905Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.909 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828909000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.909Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.917 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:08.918 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.918 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.920 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:08.922 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828922000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.922Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.924 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828924000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.924Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.925 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.925 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.927 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828927000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=9FCAA02A2535F5AD551595F0912DDFC5, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.927Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.929 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.929 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.932 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.932 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.934 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828934000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.934Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.934 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.934 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.936 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:07:08.936 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.939 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828939000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.939Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.942 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828942000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=9562F1BE552A476225AECD175FEB3062, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.942Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.946 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.946 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.946 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.946 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.947 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.947 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.948 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.948 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.951 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.951 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.951 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.952 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.952 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.953 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.953 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.957 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.957 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.958 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.958 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:08.958 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.958 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.958 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.958 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.959 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.959 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9FCAA02A2535F5AD551595F0912DDFC5, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.964 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.964 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.964 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.905, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.970 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.970 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.971 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.971 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.971 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.971 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.972 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.972 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9562F1BE552A476225AECD175FEB3062, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.972 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.972 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.973 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.973 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.973 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.973 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.974 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.974 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.974 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.975 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.976 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:08.976 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:08.977 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.980 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.909, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.980 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.980 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9FCAA02A2535F5AD551595F0912DDFC5, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.981 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.924, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.982 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.983 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.922, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.985 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.927, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=9FCAA02A2535F5AD551595F0912DDFC5, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.987 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.987 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.988 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.988 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.989 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:08.989 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9562F1BE552A476225AECD175FEB3062, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:08.989 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.990 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.990 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:08.991 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962828991000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:07:08.991Z, endTime=2024-09-11T10:05:08.635Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:07:08.993 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.934, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.994 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.939, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.995 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.942, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=9562F1BE552A476225AECD175FEB3062, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:08.998 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:07:08.998 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:07:09.003 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:09.003 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:09.004 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:07:09.004 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:07:09.009 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:07:09.009 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049108635000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:07:09.010 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:07:09.013 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:07:08.991, end=2024-09-11T18:05:08.635}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:07:09.013 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:07:09.015 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:03:08.175, end=2024-09-11T18:05:08.635},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:07:09.015 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:07:09.015 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:07:09.025 INFO 2981023 --- [plan_change_detector18,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:07:09.025 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:07:09.025 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:03:08.175, end=2024-09-11T18:05:08.635},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:07:09.025 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:07:09.025 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:07:09.026 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:07:09.027 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:03:08.175, end=2024-09-11T18:05:08.635},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:07:09.027 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:07:09.027 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:07:09.028 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:07:09.028 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:03:08.175, end=2024-09-11T18:05:08.635},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:07:09.028 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:07:09.028 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:07:09.029 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:07:09.098 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,99bcdf4be7493989,36c0049a6417952d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=99bcdf4be7493989, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:09.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,99bcdf4be7493989,36c0049a6417952d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=99bcdf4be7493989, duration=22 ms] 2024-09-11 18:07:09.390 INFO 2981023 --- [ocp-async-2,b1241563985395a2,336da898cffcaac5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:10.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:07:10.003 INFO 2981023 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:07:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:07:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:07:11.098 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,52391d21f8b83061,5d38401717a9d10e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=52391d21f8b83061, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:11.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,52391d21f8b83061,5d38401717a9d10e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=52391d21f8b83061, duration=10 ms] 2024-09-11 18:07:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,f7a227c7604b8094,d77a3b7501f9c232] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,f7a227c7604b8094,d77a3b7501f9c232] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:11.390 INFO 2981023 --- [ocp-async-1,27b8550e5323c845,408c97e37bd2060e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:11.920 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:11.922 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:12.159 INFO 2981023 --- [ocp-schedules-1,5f750d48983b0473,f663918ee924838b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:07:12.224 WARN 2981023 --- [scanner-13609627-1,5f750d48983b0473,da6f2c905acd2d80] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:07:12.235 INFO 2981023 --- [alarm-detect-10,5f750d48983b0473,62efe3b1afb4f9b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645432, data is empty. 2024-09-11 18:07:12.248 INFO 2981023 --- [alarm-detect-10,5f750d48983b0473,62efe3b1afb4f9b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645481, data is empty. 2024-09-11 18:07:12.254 INFO 2981023 --- [alarm-detect-10,5f750d48983b0473,62efe3b1afb4f9b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645481, data is empty. 2024-09-11 18:07:12.263 INFO 2981023 --- [alarm-detect-16,5f750d48983b0473,20b6ddf740fda747] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645432, data is empty. 2024-09-11 18:07:12.267 INFO 2981023 --- [alarm-detect-17,5f750d48983b0473,0cde6237d3e02c8b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645432, data is empty. 2024-09-11 18:07:12.272 INFO 2981023 --- [alarm-detect-16,5f750d48983b0473,20b6ddf740fda747] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645481, data is empty. 2024-09-11 18:07:12.276 INFO 2981023 --- [alarm-detect-17,5f750d48983b0473,0cde6237d3e02c8b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645481, data is empty. 2024-09-11 18:07:12.277 INFO 2981023 --- [alarm-detect-16,5f750d48983b0473,20b6ddf740fda747] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645481, data is empty. 2024-09-11 18:07:12.283 INFO 2981023 --- [alarm-detect-17,5f750d48983b0473,0cde6237d3e02c8b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645481, data is empty. 2024-09-11 18:07:12.397 INFO 2981023 --- [alarm-detect-12,5f750d48983b0473,72f9b0aac8406ce9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645432, data is empty. 2024-09-11 18:07:12.402 INFO 2981023 --- [alarm-detect-12,5f750d48983b0473,72f9b0aac8406ce9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645481, data is empty. 2024-09-11 18:07:12.406 INFO 2981023 --- [alarm-detect-12,5f750d48983b0473,72f9b0aac8406ce9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645481, data is empty. 2024-09-11 18:07:12.495 INFO 2981023 --- [ocp-schedules-20,977d22580c005968,798a9c3d0b6e5519] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:07:13.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,c4c53a3a26a138bd,b3ba129d612f2e19] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c4c53a3a26a138bd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:13.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,c4c53a3a26a138bd,b3ba129d612f2e19] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c4c53a3a26a138bd, duration=9 ms] 2024-09-11 18:07:13.197 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 18:07:13.210 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:13.213 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:13.267 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:07:13.267 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:07:13.278 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:13.279 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:13.290 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:07:13.290 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:07:13.301 INFO 2981023 --- [ocp-schedules-2,a56cc6f07d92bb8a,f82f94785aabceda] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2024-09-11 18:07:13.390 INFO 2981023 --- [ocp-async-5,248879221dbf1af1,41aad9b6731a1ce4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,cedb1f7284f917ce,bfe5c762d7237f74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,7198cb5a90dd5b3d,0378aa92b5fe2a6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:07:14.316 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,39b5819c2af84fa3,17bc773c51c8149b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=39b5819c2af84fa3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:07:14.324 INFO 2981023 --- [prometheus-minute-schedule-1,cedb1f7284f917ce,bfe5c762d7237f74] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:14.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,39b5819c2af84fa3,17bc773c51c8149b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=39b5819c2af84fa3, duration=33 ms] 2024-09-11 18:07:14.445 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049160, nodeCount=1000, affectRows=1000 2024-09-11 18:07:14.447 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049160, nodeCount=999, validCount=999 2024-09-11 18:07:14.450 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049160, nodeCount=1000, affectRows=1000 2024-09-11 18:07:14.452 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049160, nodeCount=1000, validCount=1000 2024-09-11 18:07:14.458 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049160, nodeCount=48, affectRows=48 2024-09-11 18:07:14.458 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049160, nodeCount=46, validCount=46 2024-09-11 18:07:14.922 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:14.925 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:15.096 INFO 2981023 --- [ocp-schedules-19,11e3656fc46e9f3c,baa347e498086bfc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:07:15.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6d15ad7f4c5b1243,75c85ca1c5073d28] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6d15ad7f4c5b1243, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:15.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6d15ad7f4c5b1243,75c85ca1c5073d28] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6d15ad7f4c5b1243, duration=10 ms] 2024-09-11 18:07:15.117 INFO 2981023 --- [ocp-schedules-19,11e3656fc46e9f3c,baa347e498086bfc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:07:15.117 INFO 2981023 --- [ocp-schedules-19,11e3656fc46e9f3c,baa347e498086bfc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:07:15.390 INFO 2981023 --- [ocp-async-9,447eae145dd75a37,007fe829171cf114] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:15.950 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049160, nodeCount=480, affectRows=480 2024-09-11 18:07:16.031 INFO 2981023 --- [ocp-schedules-21,251d01ccb44b2d90,ff791229ac51e8ed] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:07:16.031 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:07:16.031 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:07:16.032 INFO 2981023 --- [ocp-schedules-15,13b83c549321d879,ff8ced5bacb0bcb9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:07:16.032 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:07:16.032 INFO 2981023 --- [ocp-schedules-14,3f5b7bb8bf18f0cd,db3c0fc7fc8b9f8e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:07:16.032 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:07:16.032 INFO 2981023 --- [host-check-agent-status1,3f5b7bb8bf18f0cd,3d983c60544e286e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:07:16.032 INFO 2981023 --- [host-check-clockdiff1,13b83c549321d879,630faa8b2d64a46d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:07:16.034 INFO 2981023 --- [host-check-agent-status2,3f5b7bb8bf18f0cd,85acd8d48c485a30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:07:16.035 INFO 2981023 --- [host-check-agent-status3,3f5b7bb8bf18f0cd,e544e30c7e1f9590] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:07:16.042 INFO 2981023 --- [host-check-agent-status4,3f5b7bb8bf18f0cd,bc9f842fcf6910b7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:07:16.045 INFO 2981023 --- [host-check-clockdiff2,13b83c549321d879,e176b928f951c432] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:07:16.045 INFO 2981023 --- [host-check-clockdiff3,13b83c549321d879,9f8aec35efa87450] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:07:16.046 INFO 2981023 --- [host-check-clockdiff4,13b83c549321d879,396bfc5bed42c7da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:07:16.092 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:07:16.100 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:07:16.100 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:07:16.103 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:16.104 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:16.108 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:07:16.114 INFO 2981023 --- [host-check-agent-status4,3f5b7bb8bf18f0cd,bc9f842fcf6910b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:07:16.116 INFO 2981023 --- [host-check-agent-status1,3f5b7bb8bf18f0cd,3d983c60544e286e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:16.117 INFO 2981023 --- [host-check-agent-status3,3f5b7bb8bf18f0cd,e544e30c7e1f9590] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:16.117 INFO 2981023 --- [host-check-agent-status1,3f5b7bb8bf18f0cd,3d983c60544e286e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:16.118 INFO 2981023 --- [host-check-agent-status4,3f5b7bb8bf18f0cd,bc9f842fcf6910b7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:07:16.119 INFO 2981023 --- [host-check-agent-status3,3f5b7bb8bf18f0cd,e544e30c7e1f9590] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:16.130 INFO 2981023 --- [host-check-agent-status2,3f5b7bb8bf18f0cd,85acd8d48c485a30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:07:16.133 INFO 2981023 --- [host-check-agent-status2,3f5b7bb8bf18f0cd,85acd8d48c485a30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:07:16.135 INFO 2981023 --- [host-check-clockdiff2,13b83c549321d879,e176b928f951c432] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049236 5 5 , error= 2024-09-11 18:07:16.140 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:07:16.143 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:07:16.148766940+08:00 2024-09-11 18:07:16.146 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:07:16.143+08:00 2024-09-11 18:07:16.151 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:07:16.143+08:00 and status AVAILABLE 2024-09-11 18:07:16.151 INFO 2981023 --- [host-check-clockdiff3,13b83c549321d879,9f8aec35efa87450] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049236 0 0 , error= 2024-09-11 18:07:16.161 INFO 2981023 --- [host-check-agent-status3,3f5b7bb8bf18f0cd,e544e30c7e1f9590] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:07:16.166 INFO 2981023 --- [host-check-agent-status1,3f5b7bb8bf18f0cd,3d983c60544e286e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:07:16.166 INFO 2981023 --- [host-check-agent-status4,3f5b7bb8bf18f0cd,bc9f842fcf6910b7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:07:16.169 INFO 2981023 --- [host-check-agent-status3,3f5b7bb8bf18f0cd,e544e30c7e1f9590] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:07:16.169 INFO 2981023 --- [host-check-agent-status3,3f5b7bb8bf18f0cd,e544e30c7e1f9590] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:07:16.169 INFO 2981023 --- [host-check-agent-status3,3f5b7bb8bf18f0cd,e544e30c7e1f9590] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:07:16.176 INFO 2981023 --- [host-check-agent-status1,3f5b7bb8bf18f0cd,3d983c60544e286e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:07:16.176 INFO 2981023 --- [host-check-agent-status1,3f5b7bb8bf18f0cd,3d983c60544e286e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:07:16.176 INFO 2981023 --- [host-check-agent-status1,3f5b7bb8bf18f0cd,3d983c60544e286e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:07:16.180 INFO 2981023 --- [host-check-agent-status4,3f5b7bb8bf18f0cd,bc9f842fcf6910b7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:07:16.180 INFO 2981023 --- [host-check-agent-status4,3f5b7bb8bf18f0cd,bc9f842fcf6910b7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:07:16.180 INFO 2981023 --- [host-check-agent-status4,3f5b7bb8bf18f0cd,bc9f842fcf6910b7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:07:16.182 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:16.187 INFO 2981023 --- [ocp-schedules-16,f3b3b794c56cc191,0106aea8a374f696] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:07:16.187 INFO 2981023 --- [ocp-schedules-12,b4acc1a7aa2609f7,27496d55e73c3242] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:07:16.191 INFO 2981023 --- [ocp-schedules-30,b86160b48a96a95a,c05b86224dc643f8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:07:16.191 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:07:16.191 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:16.192 INFO 2981023 --- [ocp-schedules-11,ecf71f19e7ca3c00,5b721a66db2e7875] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:07:16.193 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:07:16.192477098+08:00 2024-09-11 18:07:16.194 INFO 2981023 --- [host-check-agent-status2,3f5b7bb8bf18f0cd,85acd8d48c485a30] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:07:16.196 INFO 2981023 --- [ocp-schedules-16,f3b3b794c56cc191,0106aea8a374f696] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:07:16.199 INFO 2981023 --- [host-check-agent-status2,3f5b7bb8bf18f0cd,85acd8d48c485a30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:07:16.199 INFO 2981023 --- [host-check-agent-status2,3f5b7bb8bf18f0cd,85acd8d48c485a30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:07:16.199 INFO 2981023 --- [host-check-agent-status2,3f5b7bb8bf18f0cd,85acd8d48c485a30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:07:16.199 INFO 2981023 --- [ocp-schedules-12,b4acc1a7aa2609f7,27496d55e73c3242] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:07:16.200 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:07:16.193+08:00 2024-09-11 18:07:16.202 INFO 2981023 --- [ocp-schedules-30,b86160b48a96a95a,c05b86224dc643f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:07:16.207 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:07:16.193+08:00 and status AVAILABLE 2024-09-11 18:07:16.207 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:07:16.227 INFO 2981023 --- [ocp-schedules-26,1d066f56c6c1a1bc,039580cb8a8382b3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:07:16.231 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049162777000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:07:16.237 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:07:16.257 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:07:16.260 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:07:16.263220863+08:00 2024-09-11 18:07:16.260 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:07:16.261 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:07:16.260752926+08:00 2024-09-11 18:07:16.261 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:16.264 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:07:16.261+08:00 2024-09-11 18:07:16.265 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:07:16.265 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:16.267 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:07:16.261+08:00 and status AVAILABLE 2024-09-11 18:07:16.267 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:07:16.268 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:07:16.260+08:00 2024-09-11 18:07:16.270 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:07:16.260+08:00 and status AVAILABLE 2024-09-11 18:07:16.277 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:07:16.277 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.278 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:07:16.279 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:07:16.282 INFO 2981023 --- [ocp-schedules-12,b4acc1a7aa2609f7,27496d55e73c3242] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:07:16.285 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.286 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:07:16.286 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:07:16.287 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:07:16.288 INFO 2981023 --- [ocp-schedules-16,f3b3b794c56cc191,0106aea8a374f696] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:07:16.302 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:07:16.303 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:16.304 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:07:16.305 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:16.311 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:07:16.319 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:07:16.324 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.325 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:16.330 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.330 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:16.338 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:07:16.338 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:07:16.339 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:07:16.340 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:07:16.349 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.350 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:16.361 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:07:16.367 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:07:16.373 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:07:16.373 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:07:16.372473+08:00, lastProcessTime=2024-09-11T18:07:06.352628+08:00 2024-09-11 18:07:16.375 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:16.386 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.387 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:07:16.393 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:07:16.400 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:07:16.401 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:07:16.412 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:07:16.412 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049162777000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:07:16.415 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:07:16.417 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:07:16.426 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:07:16.434 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:07:16.439 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:07:16.442 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:16.444 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.444 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:16.445 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:07:16.446 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:07:16.446 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.447 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:07:16.450 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:07:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:07:16.459 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:07:16.478 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:07:16.478 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:07:16.479 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:07:16.480 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:07:16.485 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:07:16.491 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:07:16.493 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:07:16.495 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:07:16.507 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:07:16.512 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:07:16.516 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:07:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.519 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:07:16.520 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:16.520 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:07:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:07:16.521 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.521 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:07:16.523 INFO 2981023 --- [host-check-host-status1,251d01ccb44b2d90,1ddfabc5e9260831] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:07:16.524 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:07:16.531 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:07:16.534 INFO 2981023 --- [host-check-host-status4,251d01ccb44b2d90,9372c536c18fe2cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:07:16.538 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:16.540 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:16.549 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:07:16.553 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:07:16.560 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:16.563 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.563 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:16.564 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:07:16.564 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.568 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:07:16.569 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:07:16.569 INFO 2981023 --- [host-check-host-status2,251d01ccb44b2d90,80ddfc5bcc4aee6e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:07:16.571 INFO 2981023 --- [host-check-host-status3,251d01ccb44b2d90,baf445eee468056e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:07:16.572 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:07:16.593 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:07:16.593 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:07:16.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:07:16.622 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:16.626 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.627 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:07:16.627 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:07:16.632 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:07:16.636 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:16.637 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:16.649 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:07:16.651 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:07:16.372473+08:00 2024-09-11 18:07:16.655 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:07:16.655 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:16.655 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.656 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:07:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:07:16.661 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:07:16.684 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:07:16.690 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:16.690 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:16.691 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:07:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:07:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:07:16.696 INFO 2981023 --- [ocp-schedules-30,b86160b48a96a95a,c05b86224dc643f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:07:17.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,234bab4fb591547e,91f99e73bc93e356] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=234bab4fb591547e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:17.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,234bab4fb591547e,91f99e73bc93e356] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=234bab4fb591547e, duration=10 ms] 2024-09-11 18:07:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,c44c9c4dbfa1d092,54a31e392ca7c058] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,c44c9c4dbfa1d092,54a31e392ca7c058] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:17.390 INFO 2981023 --- [ocp-async-7,a15fdcf4ea58e143,c753f31532f8d326] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:17.781 INFO 2981023 --- [ocp-async-3,a726d27fcb72f023,3bab9504b1ff404b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:07:17.925 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:17.925 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:18.203 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049160, nodeCount=480, validCount=480 2024-09-11 18:07:19.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,0cd2b0b929c3254e,6c8075f410dd0f75] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0cd2b0b929c3254e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:19.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,0cd2b0b929c3254e,6c8075f410dd0f75] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0cd2b0b929c3254e, duration=19 ms] 2024-09-11 18:07:19.390 INFO 2981023 --- [ocp-async-8,830e88fbcd4130b0,dbc3fddce4793bbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,f6041816031c39dd,ab404246027126e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,f6041816031c39dd,ab404246027126e1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:20.837 INFO 2981023 --- [host-check-clockdiff1,13b83c549321d879,630faa8b2d64a46d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049240 6 6 , error= 2024-09-11 18:07:20.925 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:20.927 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:21.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,51f3377adac70750,f6e579e988cde136] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=51f3377adac70750, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:21.121 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,51f3377adac70750,f6e579e988cde136] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=51f3377adac70750, duration=9 ms] 2024-09-11 18:07:21.390 INFO 2981023 --- [ocp-async-10,419811dc317015d5,55bebff651f66cc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:22.004 INFO 2981023 --- [host-check-clockdiff4,13b83c549321d879,396bfc5bed42c7da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049242 0 0 , error= 2024-09-11 18:07:22.449 INFO 2981023 --- [ocp-schedules-21,0aa5c5649cbc8eca,156eb42a2119e221] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:07:22.497 INFO 2981023 --- [ocp-schedules-18,698c273522a6840f,2699b747cb572adb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:07:22.516 WARN 2981023 --- [scanner-13609936-1,0aa5c5649cbc8eca,3c10340da76825a5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:07:22.547 INFO 2981023 --- [alarm-detect-10,0aa5c5649cbc8eca,0af85cb6027e99d2] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726049205 ~ 1726049232 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726049220, value=1.662)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.662, values={1726049220=1.662}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:07:22.548 INFO 2981023 --- [alarm-detect-10,0aa5c5649cbc8eca,0af85cb6027e99d2] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Pending 2024-09-11 18:07:23.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,45b3590fe411d93c,8c3484040bb3f1a5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=45b3590fe411d93c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:23.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,45b3590fe411d93c,8c3484040bb3f1a5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=45b3590fe411d93c, duration=10 ms] 2024-09-11 18:07:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,0a6828e3c09d8683,4a0a20259d8b6588] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,0a6828e3c09d8683,4a0a20259d8b6588] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:23.390 INFO 2981023 --- [ocp-async-2,c2306d35109f3ba6,a851a77d70630ac7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:23.927 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:23.929 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:24.402 INFO 2981023 --- [ocp-schedules-5,c9d677662eb1ae50,a16b5bddda89a2f5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:07:25.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,abcb18497a7404dc,549c0683082db0e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=abcb18497a7404dc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:25.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,abcb18497a7404dc,549c0683082db0e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=abcb18497a7404dc, duration=17 ms] 2024-09-11 18:07:25.201 INFO 2981023 --- [ocp-schedules-23,f7c39bdfd17cb60c,afdb6bc31556a87f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:07:25.201 INFO 2981023 --- [ocp-schedules-23,f7c39bdfd17cb60c,afdb6bc31556a87f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:07:25.201 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:07:25.201 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:07:25.206 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:07:25.229 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:07:25.229 INFO 2981023 --- [ocp-schedules-23,f7c39bdfd17cb60c,afdb6bc31556a87f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:25.229 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:07:25.232 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:07:25.233 INFO 2981023 --- [ocp-schedules-23,f7c39bdfd17cb60c,afdb6bc31556a87f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:25.233 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:07:25.235 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:25.237 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:25.391 INFO 2981023 --- [ocp-async-6,b191381c4297a74a,9f4a6a849faa276a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:25.466 INFO 2981023 --- [ocp-schedules-23,f7c39bdfd17cb60c,afdb6bc31556a87f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:07:25.474 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:07:25.477 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:07:25.481 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:07:25.486 INFO 2981023 --- [ocp-schedules-23,f7c39bdfd17cb60c,afdb6bc31556a87f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:07:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,4372f001febd10b7,713d52bda192a9b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,4372f001febd10b7,713d52bda192a9b8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:07:26.352510+08:00, lastProcessTime=2024-09-11T18:07:16.372473+08:00 2024-09-11 18:07:26.488 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:07:26.352510+08:00 2024-09-11 18:07:26.929 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:26.930 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:27.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,903274113f605ab3,7c8d0c66496309fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=903274113f605ab3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:27.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,903274113f605ab3,7c8d0c66496309fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=903274113f605ab3, duration=10 ms] 2024-09-11 18:07:27.391 INFO 2981023 --- [ocp-async-3,86e21a959269edd9,a54358f3690a7f01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:27.781 INFO 2981023 --- [ocp-async-5,ecc2c42634331190,7a3009c20eaf61af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:07:29.099 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,4633fa06dc2eaa93,9a83c6f20cbf4d2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4633fa06dc2eaa93, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:29.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,4633fa06dc2eaa93,9a83c6f20cbf4d2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4633fa06dc2eaa93, duration=23 ms] 2024-09-11 18:07:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,efcb99851a5870d3,e5ffc5537ad388e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:29.318 INFO 2981023 --- [prometheus-minute-schedule-1,efcb99851a5870d3,e5ffc5537ad388e1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:29.391 INFO 2981023 --- [ocp-async-4,f825552a53508eff,7d2b6dadeba2973d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:29.930 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:29.931 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:31.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,d332ba78c8342567,0b937efc2135e44e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d332ba78c8342567, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:31.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,d332ba78c8342567,0b937efc2135e44e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d332ba78c8342567, duration=11 ms] 2024-09-11 18:07:31.391 INFO 2981023 --- [ocp-async-1,2e2040eddd573d94,d013ad5a0a7fe811] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,9d22b577a15cfdaa,5459eee39d58b555] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,9d22b577a15cfdaa,5459eee39d58b555] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:32.503 INFO 2981023 --- [ocp-schedules-30,b0fa5bdb2fac5b06,ecf341d7a9a68fd2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:07:32.655 INFO 2981023 --- [ocp-schedules-1,ab5d2f3cd204654a,c54f0a0bbd666f59] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:07:32.702 WARN 2981023 --- [scanner-13610165-1,ab5d2f3cd204654a,6b025e4b8298b357] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:07:32.724 INFO 2981023 --- [alarm-detect-5,ab5d2f3cd204654a,79a3b87f096cce2b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645481, data is empty. 2024-09-11 18:07:32.728 INFO 2981023 --- [alarm-detect-5,ab5d2f3cd204654a,79a3b87f096cce2b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645481, data is empty. 2024-09-11 18:07:32.735 INFO 2981023 --- [alarm-detect-5,ab5d2f3cd204654a,79a3b87f096cce2b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645481, data is empty. 2024-09-11 18:07:32.792 INFO 2981023 --- [alarm-detect-3,ab5d2f3cd204654a,4ec69f96cfc36189] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645481, data is empty. 2024-09-11 18:07:32.794 INFO 2981023 --- [alarm-detect-14,ab5d2f3cd204654a,d0c9d925b2bc4e88] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645481, data is empty. 2024-09-11 18:07:32.797 INFO 2981023 --- [alarm-detect-14,ab5d2f3cd204654a,d0c9d925b2bc4e88] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645481, data is empty. 2024-09-11 18:07:32.799 INFO 2981023 --- [alarm-detect-3,ab5d2f3cd204654a,4ec69f96cfc36189] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645481, data is empty. 2024-09-11 18:07:32.805 INFO 2981023 --- [alarm-detect-14,ab5d2f3cd204654a,d0c9d925b2bc4e88] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645481, data is empty. 2024-09-11 18:07:32.810 INFO 2981023 --- [alarm-detect-3,ab5d2f3cd204654a,4ec69f96cfc36189] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645481, data is empty. 2024-09-11 18:07:32.914 INFO 2981023 --- [alarm-detect-5,ab5d2f3cd204654a,8ca8ec8544c3bb40] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645481, data is empty. 2024-09-11 18:07:32.918 INFO 2981023 --- [alarm-detect-5,ab5d2f3cd204654a,8ca8ec8544c3bb40] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645481, data is empty. 2024-09-11 18:07:32.923 INFO 2981023 --- [alarm-detect-5,ab5d2f3cd204654a,8ca8ec8544c3bb40] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645481, data is empty. 2024-09-11 18:07:32.931 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:32.933 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:33.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,cd1b1a7e985eb0e7,ae02ddbba3315c38] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cd1b1a7e985eb0e7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:33.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,cd1b1a7e985eb0e7,ae02ddbba3315c38] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cd1b1a7e985eb0e7, duration=9 ms] 2024-09-11 18:07:33.391 INFO 2981023 --- [ocp-async-8,3544ec056bbae573,f15c07c19ce0bb0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:35.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d618408c5b032b72,f9162a5e0414f17b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d618408c5b032b72, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:35.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d618408c5b032b72,f9162a5e0414f17b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d618408c5b032b72, duration=9 ms] 2024-09-11 18:07:35.120 INFO 2981023 --- [ocp-schedules-29,037904937681caee,34f6a5c4aac5be6b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:07:35.140 INFO 2981023 --- [ocp-schedules-29,037904937681caee,34f6a5c4aac5be6b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:07:35.140 INFO 2981023 --- [ocp-schedules-29,037904937681caee,34f6a5c4aac5be6b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:07:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,385f6e5de1607081,bfa1f798d084afb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,385f6e5de1607081,bfa1f798d084afb8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:35.391 INFO 2981023 --- [ocp-async-7,1d93a301bfa597e3,55f30078c77eb933] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:35.933 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:35.933 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:36.327 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a76ac86f93aadba9,8b39f2888b074086] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=a76ac86f93aadba9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:07:36.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:07:36.354284+08:00, lastProcessTime=2024-09-11T18:07:26.352510+08:00 2024-09-11 18:07:36.356 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a76ac86f93aadba9,8b39f2888b074086] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=a76ac86f93aadba9, duration=32 ms] 2024-09-11 18:07:36.478 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:07:36.354284+08:00 2024-09-11 18:07:37.314 INFO 2981023 --- [prometheus-cache-monitor1,27a6f1c183282fd5,510153d9a66b102e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8430,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2051,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5148} 2024-09-11 18:07:37.315 INFO 2981023 --- [prometheus-cache-monitor1,27a6f1c183282fd5,510153d9a66b102e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:07:37.391 INFO 2981023 --- [ocp-async-5,a0cedbf4deaac11b,cb1c5458b75e8071] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:37.540 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,a8242b694070e516,0e4a9f3fd436f1ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a8242b694070e516, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:37.546 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,a8242b694070e516,0e4a9f3fd436f1ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a8242b694070e516, duration=9 ms] 2024-09-11 18:07:37.781 INFO 2981023 --- [ocp-async-10,911a2d4d431f585b,65c3f74c344cfa00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:07:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,d9c46cd9bca4e9aa,9d9a526dc05765f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,d9c46cd9bca4e9aa,9d9a526dc05765f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:38.934 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:38.935 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:39.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,cd8db8b455dda3d6,516f1fb2890e0410] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cd8db8b455dda3d6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:39.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,cd8db8b455dda3d6,516f1fb2890e0410] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cd8db8b455dda3d6, duration=17 ms] 2024-09-11 18:07:39.391 INFO 2981023 --- [ocp-async-3,b5f1a77341624cf1,5c2a406be8167761] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:40.004 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:07:40.004 INFO 2981023 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:07:41.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,01b4367090ba910e,63812b652894e2ab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=01b4367090ba910e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:41.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,01b4367090ba910e,63812b652894e2ab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=01b4367090ba910e, duration=9 ms] 2024-09-11 18:07:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,f8675895949363d3,de980f5af9cf4878] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:41.319 INFO 2981023 --- [prometheus-minute-schedule-1,f8675895949363d3,de980f5af9cf4878] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:41.391 INFO 2981023 --- [ocp-async-10,7cf6fd3e4386f259,cfa8c54fcec37811] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:41.935 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:41.936 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:42.505 INFO 2981023 --- [ocp-schedules-11,10c288666126f73b,6a2c7a9b31f5c796] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:07:42.965 INFO 2981023 --- [ocp-schedules-7,65d019454f7e0da0,fd6403e934027147] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:07:43.021 WARN 2981023 --- [scanner-13610473-1,65d019454f7e0da0,f4dd8b0136c849d7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:07:43.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b06f60b80ee5568d,811895a875512569] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b06f60b80ee5568d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:43.126 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b06f60b80ee5568d,811895a875512569] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b06f60b80ee5568d, duration=23 ms] 2024-09-11 18:07:43.392 INFO 2981023 --- [ocp-async-2,625635de1c6d2bd9,4fc35e1db6456e1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,9495f3f420a546a5,8b5c86a7202c0e80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,9495f3f420a546a5,8b5c86a7202c0e80] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:44.639 INFO 2981023 --- [ocp-schedules-20,e227e124f999fe57,29f47e83417d6371] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:07:44.659 WARN 2981023 --- [ocp-schedules-20,e227e124f999fe57,29f47e83417d6371] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:07:44.936 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:44.936 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:45.101 INFO 2981023 --- [prometheus-second-schedule-1,c32350ec2eb77fcb,5d5bba52b07a2678] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:07:45.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,722e206f7936b324,6e59a8cc2b4741cb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=722e206f7936b324, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:45.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,722e206f7936b324,6e59a8cc2b4741cb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=722e206f7936b324, duration=12 ms] 2024-09-11 18:07:45.392 INFO 2981023 --- [ocp-async-6,69b4ecd10767aacd,f89f3d5ca44cf41f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:46.016 INFO 2981023 --- [ocp-schedules-14,4f81f73153ba013d,56769e80b0e34865] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:07:46.016 INFO 2981023 --- [host-check-clockdiff1,4f81f73153ba013d,190d064fdb216576] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:07:46.016 INFO 2981023 --- [host-check-clockdiff2,4f81f73153ba013d,0793f6a63ad29219] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:07:46.017 INFO 2981023 --- [host-check-clockdiff3,4f81f73153ba013d,3a83814e6b24f120] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:07:46.018 INFO 2981023 --- [host-check-clockdiff4,4f81f73153ba013d,7c68fa645557ae9a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:07:46.029 INFO 2981023 --- [host-check-clockdiff3,4f81f73153ba013d,3a83814e6b24f120] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049266 0 0 , error= 2024-09-11 18:07:46.029 INFO 2981023 --- [ocp-schedules-26,ed3a7457d4a7f45f,79584a12b69a33e4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:07:46.029 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:07:46.029 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:07:46.029 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:07:46.030 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:07:46.030 INFO 2981023 --- [ocp-schedules-18,1da9c276d95b2cb5,dba3acfffab76fb1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:07:46.030 INFO 2981023 --- [host-check-agent-status1,1da9c276d95b2cb5,24ad2bbe61f23d25] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:07:46.030 INFO 2981023 --- [host-check-agent-status2,1da9c276d95b2cb5,d043d20559f435db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:07:46.031 INFO 2981023 --- [host-check-agent-status3,1da9c276d95b2cb5,ee35430f8347fbe6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:07:46.031 INFO 2981023 --- [host-check-agent-status4,1da9c276d95b2cb5,193055efe7b2b014] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:07:46.063 INFO 2981023 --- [host-check-agent-status3,1da9c276d95b2cb5,ee35430f8347fbe6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:46.065 INFO 2981023 --- [host-check-agent-status2,1da9c276d95b2cb5,d043d20559f435db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:07:46.067 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:46.067 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:46.071 INFO 2981023 --- [host-check-agent-status3,1da9c276d95b2cb5,ee35430f8347fbe6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:46.075 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:07:46.076 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:07:46.082 INFO 2981023 --- [host-check-agent-status2,1da9c276d95b2cb5,d043d20559f435db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:07:46.098 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:07:46.101 INFO 2981023 --- [host-check-agent-status4,1da9c276d95b2cb5,193055efe7b2b014] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:07:46.101 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:07:46.169 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:07:46.186 INFO 2981023 --- [host-check-agent-status4,1da9c276d95b2cb5,193055efe7b2b014] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:07:46.187 INFO 2981023 --- [ocp-schedules-24,66d22e592bf64d87,b5d2b1c81812c4aa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:07:46.187 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:46.188 INFO 2981023 --- [ocp-schedules-19,743b0f58b47a8579,6d4f02ccac02b56e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:07:46.188 INFO 2981023 --- [ocp-schedules-21,b0d1f53d520dd663,609eb8f1aaf9161a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:07:46.188 INFO 2981023 --- [ocp-schedules-13,5997a2b791aef3b6,f3b51ddaede1d393] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:07:46.202 INFO 2981023 --- [ocp-schedules-8,48e201a5a2892d4f,a58354a9a00c7468] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:07:46.202 INFO 2981023 --- [host-check-agent-status1,1da9c276d95b2cb5,24ad2bbe61f23d25] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:46.205 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:46.208 INFO 2981023 --- [ocp-schedules-19,743b0f58b47a8579,6d4f02ccac02b56e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:07:46.214 INFO 2981023 --- [ocp-schedules-24,66d22e592bf64d87,b5d2b1c81812c4aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:07:46.220 INFO 2981023 --- [host-check-agent-status1,1da9c276d95b2cb5,24ad2bbe61f23d25] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:46.223 INFO 2981023 --- [ocp-schedules-21,b0d1f53d520dd663,609eb8f1aaf9161a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:07:46.249 INFO 2981023 --- [ocp-schedules-19,743b0f58b47a8579,6d4f02ccac02b56e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:07:46.249 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049162777000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:07:46.268 INFO 2981023 --- [ocp-schedules-8,48e201a5a2892d4f,a58354a9a00c7468] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:07:46.308 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:07:46.309 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:07:46.322 INFO 2981023 --- [host-check-agent-status1,1da9c276d95b2cb5,24ad2bbe61f23d25] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:07:46.322 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:07:46.326 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:07:46.327 INFO 2981023 --- [host-check-agent-status1,1da9c276d95b2cb5,24ad2bbe61f23d25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:07:46.327 INFO 2981023 --- [host-check-agent-status1,1da9c276d95b2cb5,24ad2bbe61f23d25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:07:46.327 INFO 2981023 --- [host-check-agent-status1,1da9c276d95b2cb5,24ad2bbe61f23d25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:07:46.332 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:07:46.334034572+08:00 2024-09-11 18:07:46.334 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:07:46.333882944+08:00 2024-09-11 18:07:46.337 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:07:46.337 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:07:46.343096999+08:00 2024-09-11 18:07:46.339 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:07:46.339256658+08:00 2024-09-11 18:07:46.357 INFO 2981023 --- [ocp-schedules-10,da5fd7c5eb347b68,a88514922ee8b7b1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:07:46.360 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:07:46.337+08:00 2024-09-11 18:07:46.360 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:07:46.334+08:00 2024-09-11 18:07:46.363 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:07:46.339+08:00 2024-09-11 18:07:46.364 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:07:46.332+08:00 2024-09-11 18:07:46.370 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:07:46.334+08:00 and status AVAILABLE 2024-09-11 18:07:46.371 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:07:46.337+08:00 and status AVAILABLE 2024-09-11 18:07:46.371 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:07:46.339+08:00 and status AVAILABLE 2024-09-11 18:07:46.372 INFO 2981023 --- [host-check-agent-status4,1da9c276d95b2cb5,193055efe7b2b014] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:07:46.372 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:07:46.332+08:00 and status AVAILABLE 2024-09-11 18:07:46.382 INFO 2981023 --- [host-check-agent-status4,1da9c276d95b2cb5,193055efe7b2b014] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:07:46.382 INFO 2981023 --- [host-check-agent-status4,1da9c276d95b2cb5,193055efe7b2b014] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:07:46.382 INFO 2981023 --- [host-check-agent-status4,1da9c276d95b2cb5,193055efe7b2b014] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:07:46.388 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:07:46.394 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:07:46.394 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:07:46.396 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:07:46.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:07:46.452358+08:00, lastProcessTime=2024-09-11T18:07:36.354284+08:00 2024-09-11 18:07:46.461 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.462 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:07:46.462 INFO 2981023 --- [host-check-agent-status2,1da9c276d95b2cb5,d043d20559f435db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:07:46.463 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:07:46.463 INFO 2981023 --- [host-check-agent-status3,1da9c276d95b2cb5,ee35430f8347fbe6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:07:46.465 INFO 2981023 --- [host-check-agent-status2,1da9c276d95b2cb5,d043d20559f435db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:07:46.465 INFO 2981023 --- [host-check-agent-status2,1da9c276d95b2cb5,d043d20559f435db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:07:46.465 INFO 2981023 --- [host-check-agent-status2,1da9c276d95b2cb5,d043d20559f435db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:07:46.469 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:07:46.470 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:07:46.473 INFO 2981023 --- [host-check-agent-status3,1da9c276d95b2cb5,ee35430f8347fbe6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:07:46.473 INFO 2981023 --- [host-check-agent-status3,1da9c276d95b2cb5,ee35430f8347fbe6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:07:46.473 INFO 2981023 --- [host-check-agent-status3,1da9c276d95b2cb5,ee35430f8347fbe6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:07:46.480 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.480 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:07:46.481 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:46.481 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:07:46.482 INFO 2981023 --- [ocp-schedules-24,66d22e592bf64d87,b5d2b1c81812c4aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:07:46.484 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:46.485 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:46.487 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:07:46.488 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:46.490 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:07:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:46.496 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.497 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:46.506 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:07:46.516 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.517 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:07:46.517 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:07:46.518 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.527 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:07:46.528 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:07:46.565 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:07:46.566 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:07:46.569 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.569 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:46.571 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:07:46.574 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:07:46.593 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:07:46.595 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:07:46.595 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:07:46.603 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:07:46.603 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049162777000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:07:46.614 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:07:46.615 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.615 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:07:46.615 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:07:46.616 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.616 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:07:46.660 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:07:46.662 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.666 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:07:46.667 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.668 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:07:46.668 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:07:46.668 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:46.680 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:07:46.680 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.682 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:07:46.683 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:07:46.684 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:07:46.693 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:07:46.695 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:07:46.697 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:07:46.699 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:07:46.701 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:07:46.701 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:07:46.705 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:07:46.716 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:07:46.720 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.721 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.721 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:07:46.722 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:07:46.726 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:07:46.727 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:07:46.731 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:07:46.732 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.732 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:07:46.733 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:07:46.740 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.742 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.743 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:07:46.743 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:07:46.743 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:07:46.744 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.745 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.745 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:07:46.746 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:07:46.747 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.747 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:07:46.748 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:07:46.748 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.748 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:07:46.750 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.751 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.751 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:07:46.752 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:07:46.752 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.752 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:07:46.754 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:07:46.754 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:07:46.755 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:07:46.758 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:07:46.762 INFO 2981023 --- [host-check-host-status4,ed3a7457d4a7f45f,84e70c87c9ba44ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:07:46.764 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:07:46.764 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:07:46.764 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:07:46.766 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:07:46.774 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:07:46.774 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:07:46.779 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:07:46.786 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:07:46.787 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:07:46.792 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:07:46.793 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:07:46.795 INFO 2981023 --- [host-check-host-status3,ed3a7457d4a7f45f,2c80c99ed1827006] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:07:46.801 INFO 2981023 --- [host-check-host-status1,ed3a7457d4a7f45f,0db32a0726925324] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:07:46.801 INFO 2981023 --- [host-check-host-status2,ed3a7457d4a7f45f,b38c5c78da66b599] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:07:46.805 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:07:46.817 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.820 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.821 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.821 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.821 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:07:46.821 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:07:46.822 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:07:46.822 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:07:46.822 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.836 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:07:46.836 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:07:46.844 INFO 2981023 --- [ocp-schedules-22,9151df5b61c3a502,1a3c48a9340dfc4d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8430,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2051,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5148} 2024-09-11 18:07:46.844 INFO 2981023 --- [ocp-schedules-22,9151df5b61c3a502,1a3c48a9340dfc4d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:07:46.844 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:07:46.846 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:07:46.861 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:07:46.868 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:07:46.872 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:07:46.876 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.877 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.877 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:07:46.878 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.879 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.879 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:07:46.880 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:07:46.880 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.883 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:07:46.452358+08:00 2024-09-11 18:07:46.883 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:07:46.883 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:07:46.901 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:07:46.907 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:46.907 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:46.908 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:07:46.913 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:07:46.914 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:07:46.914 INFO 2981023 --- [ocp-schedules-21,b0d1f53d520dd663,609eb8f1aaf9161a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:07:46.921 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:07:46.940 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:07:46.951 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:07:46.954 INFO 2981023 --- [ocp-schedules-6,b8e52ba302d2a2d7,c7148417e0eae388] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:07:46.967 INFO 2981023 --- [host-check-clockdiff4,4f81f73153ba013d,7c68fa645557ae9a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049266 0 0 , error= 2024-09-11 18:07:47.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,91813590bd8a3638,56ac637df9dc1350] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=91813590bd8a3638, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:47.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,91813590bd8a3638,56ac637df9dc1350] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=91813590bd8a3638, duration=9 ms] 2024-09-11 18:07:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,1cec833cb7cbb83a,22327559b6be5ecf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,1cec833cb7cbb83a,22327559b6be5ecf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:47.392 INFO 2981023 --- [ocp-async-4,d9c90201943abd0f,912a824847d137eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:47.781 INFO 2981023 --- [ocp-async-3,b272273d86dc7bdf,a09f095ce0731e6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:07:47.808 INFO 2981023 --- [host-check-clockdiff2,4f81f73153ba013d,0793f6a63ad29219] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049267 5 5 , error= 2024-09-11 18:07:47.936 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:47.937 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:49.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1d480ad9a961f85d,0ad4813b9881cb09] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1d480ad9a961f85d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:49.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1d480ad9a961f85d,0ad4813b9881cb09] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1d480ad9a961f85d, duration=17 ms] 2024-09-11 18:07:49.392 INFO 2981023 --- [ocp-async-8,e0c3f4178a626273,84bb1c4b8e6afc19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:07:49.912 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:07:50.180 INFO 2981023 --- [host-check-clockdiff1,4f81f73153ba013d,190d064fdb216576] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049270 5 5 , error= 2024-09-11 18:07:50.310 INFO 2981023 --- [ocp-schedules-10,aad84683bc785f24,eceb35d4091501a7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:07:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,01e474dd418b458e,00fa70b3d6c4bdad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,01e474dd418b458e,00fa70b3d6c4bdad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:50.937 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:50.939 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:51.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,cf0ecd32cb03f66c,f6eb6e8e84533b28] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cf0ecd32cb03f66c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:51.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,cf0ecd32cb03f66c,f6eb6e8e84533b28] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cf0ecd32cb03f66c, duration=9 ms] 2024-09-11 18:07:51.392 INFO 2981023 --- [ocp-async-3,34beca73a5d56ea5,c94677396f3b699f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:51.603 INFO 2981023 --- [ocp-schedules-14,c123d043ec076cc3,ddd509814c4a3cff] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:07:51.604 INFO 2981023 --- [ocp-schedules-14,c123d043ec076cc3,ddd509814c4a3cff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:07:51.604 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:07:51.605 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:07:51.605 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:07:51.605 INFO 2981023 --- [ocp-schedules-14,c123d043ec076cc3,ddd509814c4a3cff] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:07:52.510 INFO 2981023 --- [ocp-schedules-16,3e502c15989d92e0,b90ae72d89d0523a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:07:53.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,2869174a12bb5e66,c2c56123cdbf7788] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2869174a12bb5e66, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:53.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,2869174a12bb5e66,c2c56123cdbf7788] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2869174a12bb5e66, duration=10 ms] 2024-09-11 18:07:53.173 INFO 2981023 --- [ocp-schedules-15,c2448a97be802b7c,42d226e19c108f2a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:07:53.236 WARN 2981023 --- [scanner-13610707-1,c2448a97be802b7c,9e958f913a1b62dc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:07:53.250 INFO 2981023 --- [alarm-detect-12,c2448a97be802b7c,f0d4612d2e714dce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645481, data is empty. 2024-09-11 18:07:53.255 INFO 2981023 --- [alarm-detect-12,c2448a97be802b7c,f0d4612d2e714dce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645481, data is empty. 2024-09-11 18:07:53.259 INFO 2981023 --- [alarm-detect-12,c2448a97be802b7c,f0d4612d2e714dce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645481, data is empty. 2024-09-11 18:07:53.268 INFO 2981023 --- [ocp-schedules-14,d73e3957b89715b8,149f0847e873c7d5] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:07:53.268 INFO 2981023 --- [ocp-schedules-14,d73e3957b89715b8,149f0847e873c7d5] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:07:53.271 INFO 2981023 --- [alarm-detect-11,c2448a97be802b7c,f96a12b300c4f8a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645481, data is empty. 2024-09-11 18:07:53.271 INFO 2981023 --- [alarm-detect-12,c2448a97be802b7c,207ddbbd8c8b838b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645481, data is empty. 2024-09-11 18:07:53.276 INFO 2981023 --- [alarm-detect-11,c2448a97be802b7c,f96a12b300c4f8a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645481, data is empty. 2024-09-11 18:07:53.280 INFO 2981023 --- [alarm-detect-12,c2448a97be802b7c,207ddbbd8c8b838b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645481, data is empty. 2024-09-11 18:07:53.282 INFO 2981023 --- [alarm-detect-11,c2448a97be802b7c,f96a12b300c4f8a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645481, data is empty. 2024-09-11 18:07:53.286 INFO 2981023 --- [alarm-detect-12,c2448a97be802b7c,207ddbbd8c8b838b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645481, data is empty. 2024-09-11 18:07:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,52eeeb204567fbf4,d0e28607ff96102c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:53.323 INFO 2981023 --- [prometheus-minute-schedule-1,52eeeb204567fbf4,d0e28607ff96102c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:53.392 INFO 2981023 --- [ocp-async-10,5d725225af540b2d,12d7657643372fae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:53.404 INFO 2981023 --- [alarm-detect-10,c2448a97be802b7c,89763f1641fc6f10] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645481, data is empty. 2024-09-11 18:07:53.411 INFO 2981023 --- [alarm-detect-10,c2448a97be802b7c,89763f1641fc6f10] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645481, data is empty. 2024-09-11 18:07:53.417 INFO 2981023 --- [alarm-detect-10,c2448a97be802b7c,89763f1641fc6f10] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645481, data is empty. 2024-09-11 18:07:53.939 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:53.939 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:54.404 INFO 2981023 --- [ocp-schedules-26,fdbd94e6d0241e3e,2e6c39b7c769e85c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:07:54.448 INFO 2981023 --- [ocp-schedules-7,32154b133a28ab92,72291ec7bc785c70] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:07:54.466 WARN 2981023 --- [ocp-schedules-7,32154b133a28ab92,72291ec7bc785c70] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:07:55.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,18d3508915d880ed,d42fd818ebdf5e0d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=18d3508915d880ed, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:55.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,18d3508915d880ed,d42fd818ebdf5e0d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=18d3508915d880ed, duration=10 ms] 2024-09-11 18:07:55.143 INFO 2981023 --- [ocp-schedules-4,a0607f0082520592,caf26bebd3ffc84d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:07:55.151 INFO 2981023 --- [ocp-schedules-4,a0607f0082520592,caf26bebd3ffc84d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:07:55.151 INFO 2981023 --- [ocp-schedules-4,a0607f0082520592,caf26bebd3ffc84d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:07:55.392 INFO 2981023 --- [ocp-async-2,ddbeb3b2f0d0e286,0a5624c1be662f88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:55.851 INFO 2981023 --- [ocp-schedules-27,a696e55105f358ba,77491519e2a69dd9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:07:55.851 INFO 2981023 --- [ocp-schedules-27,a696e55105f358ba,77491519e2a69dd9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:07:55.852 INFO 2981023 --- [ocp-schedules-27,a696e55105f358ba,77491519e2a69dd9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:07:55.852 INFO 2981023 --- [ocp-schedules-27,a696e55105f358ba,77491519e2a69dd9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:07:55.852 INFO 2981023 --- [ocp-schedules-27,a696e55105f358ba,77491519e2a69dd9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:07:55.852 INFO 2981023 --- [ocp-schedules-27,a696e55105f358ba,77491519e2a69dd9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:07:56.010 INFO 2981023 --- [ocp-schedules-12,082be6148026cf11,650c8a721ec22c39] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:07:56.018 INFO 2981023 --- [ocp-schedules-12,082be6148026cf11,650c8a721ec22c39] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:07:56.018 INFO 2981023 --- [ocp-schedules-12,082be6148026cf11,650c8a721ec22c39] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:07:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,a03293c33a8503a5,f81e8ec6916711a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,a03293c33a8503a5,f81e8ec6916711a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:07:56.351686+08:00, lastProcessTime=2024-09-11T18:07:46.452358+08:00 2024-09-11 18:07:56.479 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:07:56.351686+08:00 2024-09-11 18:07:56.939 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:56.940 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:07:57.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,c789a7676482256d,d6e724b1bb239c8a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c789a7676482256d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:57.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,c789a7676482256d,d6e724b1bb239c8a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c789a7676482256d, duration=10 ms] 2024-09-11 18:07:57.392 INFO 2981023 --- [ocp-async-3,9eea5f5c1482ce83,1ebc02f9f15c0a4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:57.639 INFO 2981023 --- [ocp-schedules-18,fa8e2912564d6273,88a72104496e578c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:07:57.639 INFO 2981023 --- [ocp-schedules-18,fa8e2912564d6273,88a72104496e578c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:07:57.652 INFO 2981023 --- [ocp-schedules-12,55de7b07a1082052,d9933050df6f3328] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:07:57.652+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:07:57.656 INFO 2981023 --- [ocp-schedules-12,55de7b07a1082052,d9933050df6f3328] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:07:57.657 INFO 2981023 --- [Collect all tenants major compactions2,fa8e2912564d6273,81a6a642f5712753] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:07:57.658 INFO 2981023 --- [Collect all tenants major compactions5,fa8e2912564d6273,21e9dc1847a5374f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:07:57.660 INFO 2981023 --- [Collect all tenants major compactions1,fa8e2912564d6273,b7997e12fa175131] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:07:57.660 INFO 2981023 --- [Collect all tenants major compactions4,fa8e2912564d6273,a3939550a5d40128] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:07:57.660 INFO 2981023 --- [Collect all tenants major compactions3,fa8e2912564d6273,f386a7f30702f879] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:07:57.663 INFO 2981023 --- [Collect all tenants major compactions6,fa8e2912564d6273,41ef85d00496517e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:07:57.681 INFO 2981023 --- [Collect all tenants major compactions5,fa8e2912564d6273,21e9dc1847a5374f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:57.682 INFO 2981023 --- [Collect all tenants major compactions5,fa8e2912564d6273,21e9dc1847a5374f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:07:57.682 INFO 2981023 --- [Collect all tenants major compactions2,fa8e2912564d6273,81a6a642f5712753] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:57.683 INFO 2981023 --- [Collect all tenants major compactions4,fa8e2912564d6273,a3939550a5d40128] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:57.683 INFO 2981023 --- [Collect all tenants major compactions1,fa8e2912564d6273,b7997e12fa175131] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:57.684 INFO 2981023 --- [Collect all tenants major compactions2,fa8e2912564d6273,81a6a642f5712753] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:07:57.684 INFO 2981023 --- [Collect all tenants major compactions1,fa8e2912564d6273,b7997e12fa175131] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:07:57.684 INFO 2981023 --- [Collect all tenants major compactions4,fa8e2912564d6273,a3939550a5d40128] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:07:57.686 INFO 2981023 --- [Collect all tenants major compactions5,fa8e2912564d6273,21e9dc1847a5374f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:07:57.686 INFO 2981023 --- [Collect all tenants major compactions5,fa8e2912564d6273,21e9dc1847a5374f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:07:57.687 INFO 2981023 --- [Collect all tenants major compactions1,fa8e2912564d6273,b7997e12fa175131] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:07:57.687 INFO 2981023 --- [Collect all tenants major compactions1,fa8e2912564d6273,b7997e12fa175131] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:07:57.687 INFO 2981023 --- [Collect all tenants major compactions4,fa8e2912564d6273,a3939550a5d40128] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:07:57.687 INFO 2981023 --- [Collect all tenants major compactions4,fa8e2912564d6273,a3939550a5d40128] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:07:57.687 INFO 2981023 --- [Collect all tenants major compactions3,fa8e2912564d6273,f386a7f30702f879] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:57.688 INFO 2981023 --- [Collect all tenants major compactions3,fa8e2912564d6273,f386a7f30702f879] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:07:57.688 INFO 2981023 --- [Collect all tenants major compactions2,fa8e2912564d6273,81a6a642f5712753] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:07:57.688 INFO 2981023 --- [Collect all tenants major compactions2,fa8e2912564d6273,81a6a642f5712753] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:07:57.690 INFO 2981023 --- [Collect all tenants major compactions3,fa8e2912564d6273,f386a7f30702f879] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:07:57.690 INFO 2981023 --- [Collect all tenants major compactions3,fa8e2912564d6273,f386a7f30702f879] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:07:57.693 INFO 2981023 --- [Collect all tenants major compactions6,fa8e2912564d6273,41ef85d00496517e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:57.694 INFO 2981023 --- [Collect all tenants major compactions6,fa8e2912564d6273,41ef85d00496517e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:07:57.696 INFO 2981023 --- [Collect all tenants major compactions6,fa8e2912564d6273,41ef85d00496517e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:07:57.697 INFO 2981023 --- [Collect all tenants major compactions6,fa8e2912564d6273,41ef85d00496517e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:07:57.697 INFO 2981023 --- [ocp-schedules-18,fa8e2912564d6273,88a72104496e578c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:07:57.781 INFO 2981023 --- [ocp-async-2,93ceb6819e3ce27d,e85e7ea7a6ebe752] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:07:57.805 INFO 2981023 --- [ocp-schedules-12,55de7b07a1082052,d9933050df6f3328] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 56 to SqlAggregateData 2024-09-11 18:07:57.806 INFO 2981023 --- [ocp-schedules-12,55de7b07a1082052,d9933050df6f3328] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:07:58.600 INFO 2981023 --- [ocp-schedules-10,a0d66034245dc3d7,37e2ea718101f745] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:07:58.600 INFO 2981023 --- [ocp-schedules-10,a0d66034245dc3d7,37e2ea718101f745] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:07:58.632 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:07:58.632 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.634 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:07:58.634 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.636 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:07:58.637 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:07:58.637 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.637 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:07:58.640 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:07:58.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.641 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.641 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.641 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.646 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.646 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.646 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.646 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.651 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.651 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.653 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.654 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:07:58.660 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.660 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.662 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.662 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.662 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.663 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:07:58.663 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:07:58.663 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4C41D6E9-0-0] 2024-09-11 18:07:58.663 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:07:58.663 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.664 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:07:58.668 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.668 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:07:58.670 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.671 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.676 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.676 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:07:58.680 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.680 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.680 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:07:58.681 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:07:58.681 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.682 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:07:58.682 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.683 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:07:58.683 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:07:58.687 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.688 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:07:58.688 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:07:58.689 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.689 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:07:58.689 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.690 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:07:58.691 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.691 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:07:58.692 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.693 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:07:58.693 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:07:58.694 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:07:58.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4C41D6F5-0-0] 2024-09-11 18:07:58.694 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:07:58.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:07:58.699 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.700 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.700 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:07:58.700 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:07:58.701 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.701 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.701 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:07:58.702 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:07:58.702 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.702 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.703 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:07:58.703 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:07:58.703 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:07:58.704 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.704 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:07:58.705 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.705 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:07:58.706 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:07:58.706 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.707 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:07:58.707 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:58.707 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:07:58.709 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.709 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:07:58.710 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.719 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:58.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:07:58.724 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:07:58.727 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:07:58.727 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:07:58.727 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:07:58.727 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:07:58.727 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:07:58.728 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:07:58.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:07:58.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:07:58.736 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:07:58.738 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:07:58.738 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.738 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.739 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:07:58.741 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:07:58.741 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:07:58.762 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:07:58.765 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:07:58.766 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:07:58.766 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:07:58.766 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:07:58.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:07:58.773 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:07:58.773 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:07:58.773 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:07:58.773 INFO 2981023 --- [ocp-schedules-10,a0d66034245dc3d7,37e2ea718101f745] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:07:58.887 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.890 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obCl 2024-09-11 18:07:58.893 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.896 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.898 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.901 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.901 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.902 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.903 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.903 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.907 INFO 2981023 --- [pool-17-thread-1,,] .o.o.p.s.d.i.d.AbstractSqlIndexDiagnoser : INEFFECTIVE_HINT, Found out 1 targets of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), between :Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872},QuerySqlAuditRawStatList(timeout=30000000, level=0, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, startTimeUs=1726048918786000, endTimeUs=1726049218872000, sqlId=null, obServerId=null, obDbId=null, dbName=null, sqlIds=null, cpuTimeUs=20000, maxCpuTimeUs=null, execPs=5.0, tableScans=null, affectedRows=null, partitionCount=null, logicalReads=null, retryCount=null, retryCountRatio=null, maxRetryCount=null, rpcCount=null, elapsedTimeUs=null, cpuTimePercent=null, sqlTypes=null, obServerIdList=null, duration=null, limit=1000, batchSize=null, batchTime=null, dynamicSql=false) 2024-09-11 18:07:58.907 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.911 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.914 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.917 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.918 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.919 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.920 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.920 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.923 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.926 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.928 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.931 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.931 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.932 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.933 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.933 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:01:58.786, end=2024-09-11T18:06:58.872} 2024-09-11 18:07:58.933 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 18:07:59.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,85cf825f2198e585,955fd1b906636a5d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=85cf825f2198e585, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:07:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,d1e11e1a6a0cd2dd,20a49c72368c84b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049280, interval=60 2024-09-11 18:07:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,d1e11e1a6a0cd2dd,20a49c72368c84b8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:07:59.329 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,85cf825f2198e585,955fd1b906636a5d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=85cf825f2198e585, duration=29 ms] 2024-09-11 18:07:59.392 INFO 2981023 --- [ocp-async-10,6aa51a5f65832e64,8f5e47b2c764436b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:07:59.446 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2acfdb5f2419a4b0,1ffa24ea0ee27def] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2acfdb5f2419a4b0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:07:59.452 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2acfdb5f2419a4b0,1ffa24ea0ee27def] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2acfdb5f2419a4b0, duration=9 ms] 2024-09-11 18:07:59.940 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:07:59.940 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:01.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e764389bd0d23b09,2031a11e8b782bdb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e764389bd0d23b09, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:01.126 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e764389bd0d23b09,2031a11e8b782bdb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e764389bd0d23b09, duration=16 ms] 2024-09-11 18:08:01.398 INFO 2981023 --- [ocp-async-2,b2795d93e4fc51a5,fbaa4260d85d9ec0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,d51062b11dcef9dd,0cc0953689c91094] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,d51062b11dcef9dd,0cc0953689c91094] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:02.333 INFO 2981023 --- [ocp-schedules-28,0155f90a43429394,f1e5845fe7c9756d] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:08:02.513 INFO 2981023 --- [ocp-schedules-22,26c71445e00c5ef4,d66fdc29fe617789] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:08:02.797 INFO 2981023 --- [ocp-schedules-5,37877f4ab71c8e66,9c8e2d538cd35911] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:08:02.798 INFO 2981023 --- [ocp-schedules-5,37877f4ab71c8e66,9c8e2d538cd35911] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:08:02.817 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:02.818 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:08:02.823 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:02.825 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:08:02.825 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049282825000} 2024-09-11 18:08:02.830 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:08:02.834 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000228, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:08:02.834 INFO 2981023 --- [ocp-schedules-5,37877f4ab71c8e66,9c8e2d538cd35911] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:08:02.940 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:02.941 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:03.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,9c81e7d669fb7163,d139facc794407a1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9c81e7d669fb7163, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:03.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,9c81e7d669fb7163,d139facc794407a1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9c81e7d669fb7163, duration=9 ms] 2024-09-11 18:08:03.393 INFO 2981023 --- [ocp-async-6,1610d2dc55b6061b,7966c1beab8c287f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:03.444 INFO 2981023 --- [ocp-schedules-19,174fd8e8ecc742e0,888d2764c5958f94] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:08:03.490 WARN 2981023 --- [scanner-13610996-1,174fd8e8ecc742e0,d7520edf76655f69] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:08:03.691 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:08:03.691 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:08:03.691 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:08:03.695 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000228, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:08:03.696 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:08:04.106 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000228 2024-09-11 18:08:05.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,3ea39d31945f8b7d,94ad7828bfb9fd46] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=3ea39d31945f8b7d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:05.127 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,3ea39d31945f8b7d,94ad7828bfb9fd46] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=3ea39d31945f8b7d, duration=18 ms] 2024-09-11 18:08:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,8b10c7705da8c490,1171ea93e7c0e7f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,8b10c7705da8c490,1171ea93e7c0e7f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:05.393 INFO 2981023 --- [ocp-async-5,0f30a2b5d9c9eba4,ede731a497a47416] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:05.941 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:05.943 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:08:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:08:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:08:05.993 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:08:05.998 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:08:06.003 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:08:06.003 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:08:06.003 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:08:06.005 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:06.006 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:08:06.006 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:06.007 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:08:06.048 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:05:41, start=2024-09-11T18:02:41, end=2024-09-11T18:05:11, updates=293, granule=120000ms, delay=180000ms 2024-09-11 18:08:06.054 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:05:41, target=2024-09-11T18:05:11, granule=120000ms, delay=30000ms) 2024-09-11 18:08:06.141 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T18:05:11, start=2024-09-11T17:50:51, end=2024-09-11T18:00:51, updates=407, granule=600000ms, delay=860000ms 2024-09-11 18:08:06.145 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T18:05:11, target=2024-09-11T18:00:51, granule=600000ms, delay=260000ms) 2024-09-11 18:08:06.145 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=175ms, period=120000ms 2024-09-11 18:08:06.356 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:08:06.355040+08:00, lastProcessTime=2024-09-11T18:07:56.351686+08:00 2024-09-11 18:08:06.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:08:06.355040+08:00 2024-09-11 18:08:07.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,eccf12fc5f77f182,b85248706c84f841] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=eccf12fc5f77f182, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:07.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,eccf12fc5f77f182,b85248706c84f841] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=eccf12fc5f77f182, duration=9 ms] 2024-09-11 18:08:07.393 INFO 2981023 --- [ocp-async-8,a8650028bb71652f,f25fb06e5fb3c594] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:07.781 INFO 2981023 --- [ocp-async-7,6ce8dbbc28316d55,0ff125f84d275a79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:08:07.833 INFO 2981023 --- [ocp-schedules-18,d698eb6a16337c76,ac6132b948f1888e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:08:07.953 INFO 2981023 --- [ocp-schedules-20,29aa0e3964a916e5,c32603b2a2b91c41] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:08:07.953+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:08:07.954 INFO 2981023 --- [ocp-schedules-20,29aa0e3964a916e5,c32603b2a2b91c41] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:08:07.956 INFO 2981023 --- [ocp-schedules-20,29aa0e3964a916e5,c32603b2a2b91c41] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:08:07.956 INFO 2981023 --- [ocp-schedules-20,29aa0e3964a916e5,c32603b2a2b91c41] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:08:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,7b299956cad8b36f,3cc34bd44d395b83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,7b299956cad8b36f,3cc34bd44d395b83] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:08.943 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:08.943 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:09.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1f5c70bb05069351,8e585c447a753716] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1f5c70bb05069351, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:09.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1f5c70bb05069351,8e585c447a753716] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1f5c70bb05069351, duration=10 ms] 2024-09-11 18:08:09.393 INFO 2981023 --- [ocp-async-6,47b4e158d28e1d61,4a8970e4920cf781] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:10.004 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:08:10.004 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:08:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:08:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:08:11.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b654f6bd3fc21627,7cf9d768e6318065] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b654f6bd3fc21627, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:11.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b654f6bd3fc21627,7cf9d768e6318065] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b654f6bd3fc21627, duration=17 ms] 2024-09-11 18:08:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,dae25204e6c8deaa,fb7c9ac1cd623b50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,dae25204e6c8deaa,fb7c9ac1cd623b50] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:11.393 INFO 2981023 --- [ocp-async-7,763d16a21617332a,9e7d0623e4b74471] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:11.720 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049220, nodeCount=480, affectRows=480 2024-09-11 18:08:11.943 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:11.944 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:12.516 INFO 2981023 --- [ocp-schedules-28,43423f1306b05033,eeda285693e43f2d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:08:13.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,4a963708e7b36092,588f244c7e9f9a0d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4a963708e7b36092, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:13.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,4a963708e7b36092,588f244c7e9f9a0d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4a963708e7b36092, duration=9 ms] 2024-09-11 18:08:13.393 INFO 2981023 --- [ocp-async-10,aac931c05c7aee7e,3656b100a3eaf1d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:13.631 INFO 2981023 --- [ocp-schedules-26,98632719456b9cf3,5e241a4182ffb0bc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:08:13.684 WARN 2981023 --- [scanner-13612740-1,98632719456b9cf3,e8eb27de5926c0f5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:08:13.852 INFO 2981023 --- [alarm-detect-2,98632719456b9cf3,c51c4c4d4f136941] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645481, data is empty. 2024-09-11 18:08:13.860 INFO 2981023 --- [alarm-detect-16,98632719456b9cf3,ccf4ed1ee33d52dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645481, data is empty. 2024-09-11 18:08:13.869 INFO 2981023 --- [alarm-detect-8,98632719456b9cf3,7567df59af43b04a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645481, data is empty. 2024-09-11 18:08:13.871 INFO 2981023 --- [alarm-detect-2,98632719456b9cf3,c51c4c4d4f136941] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645537, data is empty. 2024-09-11 18:08:13.876 INFO 2981023 --- [alarm-detect-16,98632719456b9cf3,ccf4ed1ee33d52dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645537, data is empty. 2024-09-11 18:08:13.876 INFO 2981023 --- [alarm-detect-2,98632719456b9cf3,c51c4c4d4f136941] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645537, data is empty. 2024-09-11 18:08:13.876 INFO 2981023 --- [alarm-detect-8,98632719456b9cf3,7567df59af43b04a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645537, data is empty. 2024-09-11 18:08:13.882 INFO 2981023 --- [alarm-detect-8,98632719456b9cf3,7567df59af43b04a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645537, data is empty. 2024-09-11 18:08:13.883 INFO 2981023 --- [alarm-detect-16,98632719456b9cf3,ccf4ed1ee33d52dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645537, data is empty. 2024-09-11 18:08:13.913 INFO 2981023 --- [alarm-detect-4,98632719456b9cf3,92d9f2be93f8bf2d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645481, data is empty. 2024-09-11 18:08:13.916 INFO 2981023 --- [alarm-detect-4,98632719456b9cf3,92d9f2be93f8bf2d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645537, data is empty. 2024-09-11 18:08:13.921 INFO 2981023 --- [alarm-detect-4,98632719456b9cf3,92d9f2be93f8bf2d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645537, data is empty. 2024-09-11 18:08:14.192 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049220, nodeCount=480, validCount=480 2024-09-11 18:08:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,6047ec405875d273,4ed1595dad3aaffe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,1e143919c840d8fb,9ec4a027eef133ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:08:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,6047ec405875d273,4ed1595dad3aaffe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:14.528 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049220, nodeCount=23, affectRows=23 2024-09-11 18:08:14.528 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049220, nodeCount=23, validCount=23 2024-09-11 18:08:14.530 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049220, nodeCount=28, affectRows=28 2024-09-11 18:08:14.530 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049220, nodeCount=26, validCount=26 2024-09-11 18:08:14.586 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049220, nodeCount=1000, affectRows=1000 2024-09-11 18:08:14.589 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049220, nodeCount=1000, validCount=1000 2024-09-11 18:08:14.589 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049220, nodeCount=1000, affectRows=1000 2024-09-11 18:08:14.591 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049220, nodeCount=999, validCount=999 2024-09-11 18:08:14.944 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:14.945 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:15.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,69de5aa9a8cfe4f1,aab337551056b2fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=69de5aa9a8cfe4f1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:15.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,69de5aa9a8cfe4f1,aab337551056b2fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=69de5aa9a8cfe4f1, duration=12 ms] 2024-09-11 18:08:15.154 INFO 2981023 --- [ocp-schedules-4,3bd4a72599befc22,ccca9f9c2d11e261] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:08:15.163 INFO 2981023 --- [ocp-schedules-4,3bd4a72599befc22,ccca9f9c2d11e261] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:08:15.163 INFO 2981023 --- [ocp-schedules-4,3bd4a72599befc22,ccca9f9c2d11e261] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:08:15.393 INFO 2981023 --- [ocp-async-2,8f894c28cdabf984,7baf7e4b92f015d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:16.020 INFO 2981023 --- [ocp-schedules-27,dca3008af982954a,901e77a188034373] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:08:16.020 INFO 2981023 --- [host-check-agent-status1,dca3008af982954a,f35fa344e22eebb0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:08:16.020 INFO 2981023 --- [host-check-agent-status2,dca3008af982954a,e8871caa20b25d92] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:08:16.021 INFO 2981023 --- [host-check-agent-status3,dca3008af982954a,952b0854560cd689] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:08:16.021 INFO 2981023 --- [host-check-agent-status4,dca3008af982954a,e3db37640c89978f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:08:16.023 INFO 2981023 --- [ocp-schedules-15,7445e6ad4588d8a7,cad4db63811aeb6f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:08:16.024 INFO 2981023 --- [host-check-clockdiff1,7445e6ad4588d8a7,053c31b358191ac6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:08:16.024 INFO 2981023 --- [host-check-clockdiff2,7445e6ad4588d8a7,51b8466c260c7a3e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:08:16.027 INFO 2981023 --- [ocp-schedules-24,0f60a9609f2841e2,9c6db8a9b4e2d9cd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:08:16.027 INFO 2981023 --- [host-check-clockdiff3,7445e6ad4588d8a7,ca41460c713549d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:08:16.029 INFO 2981023 --- [host-check-clockdiff4,7445e6ad4588d8a7,0a2c0ad6138fc750] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:08:16.030 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:08:16.032 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:08:16.032 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:08:16.032 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:08:16.046 INFO 2981023 --- [host-check-clockdiff3,7445e6ad4588d8a7,ca41460c713549d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049296 0 0 , error= 2024-09-11 18:08:16.061 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:08:16.066 INFO 2981023 --- [host-check-agent-status3,dca3008af982954a,952b0854560cd689] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:08:16.066 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:08:16.067 INFO 2981023 --- [host-check-agent-status2,dca3008af982954a,e8871caa20b25d92] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:08:16.067 INFO 2981023 --- [host-check-agent-status4,dca3008af982954a,e3db37640c89978f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:08:16.067 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:08:16.068 INFO 2981023 --- [host-check-agent-status4,dca3008af982954a,e3db37640c89978f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:08:16.070 INFO 2981023 --- [host-check-agent-status2,dca3008af982954a,e8871caa20b25d92] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:08:16.071 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:08:16.071 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:08:16.072 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:08:16.081 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:08:16.081 INFO 2981023 --- [host-check-agent-status3,dca3008af982954a,952b0854560cd689] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:08:16.104 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:08:16.176 INFO 2981023 --- [host-check-agent-status1,dca3008af982954a,f35fa344e22eebb0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:08:16.191 INFO 2981023 --- [ocp-schedules-22,6b18011634b8c739,b5dbb2949a28630d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:08:16.192 INFO 2981023 --- [ocp-schedules-16,cc050678dd0a44a8,a9d0879a885f620d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:08:16.192 INFO 2981023 --- [ocp-schedules-14,3b73f8f0a831629b,4e743f78fbbf6d93] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:08:16.192 INFO 2981023 --- [ocp-schedules-9,d1c0a7729c672cf5,80a3fe46d42f0a96] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:08:16.202 INFO 2981023 --- [host-check-agent-status1,dca3008af982954a,f35fa344e22eebb0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:08:16.211 INFO 2981023 --- [ocp-schedules-14,3b73f8f0a831629b,4e743f78fbbf6d93] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:08:16.263 INFO 2981023 --- [ocp-schedules-16,cc050678dd0a44a8,a9d0879a885f620d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:08:16.264 INFO 2981023 --- [ocp-schedules-22,6b18011634b8c739,b5dbb2949a28630d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:08:16.299 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:08:16.301 INFO 2981023 --- [ocp-schedules-17,fa526bd376a64063,bc0a3644dca42dc3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:08:16.302 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:08:16.301521309+08:00 2024-09-11 18:08:16.304 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:08:16.302+08:00 2024-09-11 18:08:16.307 INFO 2981023 --- [ocp-schedules-16,cc050678dd0a44a8,a9d0879a885f620d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:08:16.307 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049282825000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:08:16.307 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:08:16.302+08:00 and status AVAILABLE 2024-09-11 18:08:16.321 INFO 2981023 --- [host-check-agent-status4,dca3008af982954a,e3db37640c89978f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:08:16.321 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:08:16.328 INFO 2981023 --- [host-check-agent-status2,dca3008af982954a,e8871caa20b25d92] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:08:16.328 INFO 2981023 --- [host-check-agent-status4,dca3008af982954a,e3db37640c89978f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:08:16.328 INFO 2981023 --- [host-check-agent-status4,dca3008af982954a,e3db37640c89978f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:08:16.328 INFO 2981023 --- [host-check-agent-status4,dca3008af982954a,e3db37640c89978f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:08:16.329 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.330 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:16.332 INFO 2981023 --- [host-check-agent-status2,dca3008af982954a,e8871caa20b25d92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:08:16.332 INFO 2981023 --- [host-check-agent-status2,dca3008af982954a,e8871caa20b25d92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:08:16.332 INFO 2981023 --- [host-check-agent-status2,dca3008af982954a,e8871caa20b25d92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:08:16.333 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:08:16.333 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:08:16.338970245+08:00 2024-09-11 18:08:16.334 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:08:16.333+08:00 2024-09-11 18:08:16.337 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:08:16.333+08:00 and status AVAILABLE 2024-09-11 18:08:16.338 INFO 2981023 --- [host-check-agent-status3,dca3008af982954a,952b0854560cd689] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:08:16.340 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:08:16.341 INFO 2981023 --- [host-check-agent-status1,dca3008af982954a,f35fa344e22eebb0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:08:16.341 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:08:16.342 INFO 2981023 --- [host-check-agent-status3,dca3008af982954a,952b0854560cd689] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:08:16.342 INFO 2981023 --- [host-check-agent-status3,dca3008af982954a,952b0854560cd689] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:08:16.342 INFO 2981023 --- [host-check-agent-status3,dca3008af982954a,952b0854560cd689] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:08:16.342 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:08:16.347886216+08:00 2024-09-11 18:08:16.343 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:08:16.342+08:00 2024-09-11 18:08:16.344 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:08:16.342718148+08:00 2024-09-11 18:08:16.344 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:08:16.347 INFO 2981023 --- [host-check-agent-status1,dca3008af982954a,f35fa344e22eebb0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:08:16.347 INFO 2981023 --- [host-check-agent-status1,dca3008af982954a,f35fa344e22eebb0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:08:16.347 INFO 2981023 --- [host-check-agent-status1,dca3008af982954a,f35fa344e22eebb0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:08:16.353 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:08:16.344+08:00 2024-09-11 18:08:16.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:08:16.352592+08:00, lastProcessTime=2024-09-11T18:08:06.355040+08:00 2024-09-11 18:08:16.355 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:08:16.344+08:00 and status AVAILABLE 2024-09-11 18:08:16.356 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:08:16.342+08:00 and status AVAILABLE 2024-09-11 18:08:16.357 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.358 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:08:16.359 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:08:16.360 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.360 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:16.360 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.361 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:08:16.362 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:08:16.363 INFO 2981023 --- [ocp-schedules-22,6b18011634b8c739,b5dbb2949a28630d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:08:16.363 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:08:16.363 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:08:16.364 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:08:16.365 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:08:16.369 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:08:16.370 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:08:16.386 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:16.387 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.387 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:08:16.388 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.388 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:08:16.389 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:16.389 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:08:16.396 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:08:16.398 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:08:16.399 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:08:16.400 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:08:16.419 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:08:16.423 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:16.424 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:08:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.425 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:16.425 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:08:16.426 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:08:16.428 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:08:16.432 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:08:16.439 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:08:16.439 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:08:16.442 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:08:16.461 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:08:16.462 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:08:16.473 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:08:16.473 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:08:16.507 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:08:16.507 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:08:16.507 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049282825000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:08:16.510 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:08:16.511 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:08:16.522 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:08:16.522 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:08:16.530 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:08:16.530 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:08:16.531 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:08:16.539 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:08:16.541 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:08:16.544 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:16.544 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.545 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:08:16.546 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:08:16.550 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:08:16.555 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:08:16.556 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:08:16.556 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:08:16.559 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:08:16.559 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:08:16.562 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:08:16.563 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:16.563 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.564 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:08:16.567 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:08:16.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:08:16.581 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:08:16.583 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:08:16.593 INFO 2981023 --- [host-check-host-status4,0f60a9609f2841e2,5cb9200779934a42] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:08:16.594 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:08:16.597 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:08:16.598 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:16.599 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.600 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:16.601 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:08:16.601 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.601 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:08:16.603 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:08:16.604 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:08:16.608 INFO 2981023 --- [host-check-host-status3,0f60a9609f2841e2,a6b1a49843fbd6c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:08:16.613 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:08:16.613 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:08:16.621 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:08:16.627 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:08:16.631 INFO 2981023 --- [host-check-host-status2,0f60a9609f2841e2,ea498ce43952e2a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:08:16.634 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:08:16.641 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:16.641 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.642 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:08:16.642 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:08:16.646 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:08:16.650 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:08:16.654 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:08:16.660 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:08:16.668 INFO 2981023 --- [host-check-host-status1,0f60a9609f2841e2,eda364b9cdc4e618] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:08:16.670 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:08:16.671 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:08:16.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:16.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:08:16.687 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:08:16.687 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:08:16.700 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:08:16.702 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:08:16.352592+08:00 2024-09-11 18:08:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:16.708 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:08:16.714 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:08:16.714 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:08:16.714 INFO 2981023 --- [ocp-schedules-14,3b73f8f0a831629b,4e743f78fbbf6d93] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:08:16.974 INFO 2981023 --- [host-check-clockdiff1,7445e6ad4588d8a7,053c31b358191ac6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049296 5 5 , error= 2024-09-11 18:08:16.984 INFO 2981023 --- [host-check-clockdiff2,7445e6ad4588d8a7,51b8466c260c7a3e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049296 5 5 , error= 2024-09-11 18:08:17.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f4aff507602b17b2,68b38086c85c4102] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f4aff507602b17b2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:17.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f4aff507602b17b2,68b38086c85c4102] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f4aff507602b17b2, duration=8 ms] 2024-09-11 18:08:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,552e3135bdfcd8e6,c620dae19b2c4dc0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,552e3135bdfcd8e6,c620dae19b2c4dc0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:17.393 INFO 2981023 --- [ocp-async-4,0ea77c4f9876065b,692a197edfa186ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:17.782 INFO 2981023 --- [ocp-async-9,6f4077075b6020bb,6e7431404adbfe45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:08:17.945 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:17.946 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:19.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e7edb7ae3e7cefa9,712dd9c5e9fe2454] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e7edb7ae3e7cefa9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:19.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e7edb7ae3e7cefa9,712dd9c5e9fe2454] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e7edb7ae3e7cefa9, duration=9 ms] 2024-09-11 18:08:19.394 INFO 2981023 --- [ocp-async-1,d1e18fae3c142b97,2164e365a1e6906b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,689032b9522ba3fc,e02794b7ad1425b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:20.315 INFO 2981023 --- [prometheus-minute-schedule-1,689032b9522ba3fc,e02794b7ad1425b8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:20.946 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:20.947 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:21.309 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b21b108573edbf34,4669ec40b198e02a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=b21b108573edbf34, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:08:21.331 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b21b108573edbf34,4669ec40b198e02a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=b21b108573edbf34, duration=24 ms] 2024-09-11 18:08:21.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,0e42712bba907a53,09235530751824c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0e42712bba907a53, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:21.344 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,0e42712bba907a53,09235530751824c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0e42712bba907a53, duration=10 ms] 2024-09-11 18:08:21.394 INFO 2981023 --- [ocp-async-9,42d223b63a15ff37,c95ae68cd89e096b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:22.012 INFO 2981023 --- [host-check-clockdiff4,7445e6ad4588d8a7,0a2c0ad6138fc750] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049302 0 0 , error= 2024-09-11 18:08:22.518 INFO 2981023 --- [ocp-schedules-13,8ed3d010ac78926d,3460d904a2d148bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:08:23.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,cd0f43850a0f5940,bb101ffe778fb559] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cd0f43850a0f5940, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:23.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,cd0f43850a0f5940,bb101ffe778fb559] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cd0f43850a0f5940, duration=10 ms] 2024-09-11 18:08:23.200 INFO 2981023 --- [ocp-schedules-17,7f49cf725447be52,6fa89710f0582df9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 18:08:23.200 INFO 2981023 --- [ocp-schedules-17,7f49cf725447be52,6fa89710f0582df9] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 18:08:23.217 INFO 2981023 --- [pool-564-thread-2,7f49cf725447be52,943b26efca0d9828] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:08:23.218 INFO 2981023 --- [pool-564-thread-1,7f49cf725447be52,e8c7b05d9327bb43] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:08:23.218 INFO 2981023 --- [pool-564-thread-3,7f49cf725447be52,0859a98242bceae7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:08:23.237 INFO 2981023 --- [pool-564-thread-1,7f49cf725447be52,e8c7b05d9327bb43] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:23.239 INFO 2981023 --- [pool-564-thread-2,7f49cf725447be52,943b26efca0d9828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:23.239 INFO 2981023 --- [pool-564-thread-1,7f49cf725447be52,e8c7b05d9327bb43] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049003000000, 1726049303000000] 2024-09-11 18:08:23.243 INFO 2981023 --- [pool-564-thread-2,7f49cf725447be52,943b26efca0d9828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049003000000, 1726049303000000] 2024-09-11 18:08:23.243 INFO 2981023 --- [pool-564-thread-3,7f49cf725447be52,0859a98242bceae7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:23.246 INFO 2981023 --- [pool-564-thread-3,7f49cf725447be52,0859a98242bceae7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049003000000, 1726049303000000] 2024-09-11 18:08:23.250 INFO 2981023 --- [pool-564-thread-3,7f49cf725447be52,0859a98242bceae7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:08:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,7e22bb2c437c32e8,b4bc666bcab06d5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,7e22bb2c437c32e8,b4bc666bcab06d5c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:23.324 INFO 2981023 --- [pool-564-thread-1,7f49cf725447be52,e8c7b05d9327bb43] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:08:23.394 INFO 2981023 --- [ocp-async-6,d97bae7f2318b8d3,bb0bd63c480d75aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:23.419 INFO 2981023 --- [pool-564-thread-2,7f49cf725447be52,943b26efca0d9828] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:08:23.419 INFO 2981023 --- [ocp-schedules-17,7f49cf725447be52,6fa89710f0582df9] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 18:08:23.947 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:23.947 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:23.971 INFO 2981023 --- [ocp-schedules-1,ba3d040fe3d65176,d2d3ebf59abe0ced] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:08:24.021 WARN 2981023 --- [scanner-13613033-1,ba3d040fe3d65176,eacaa95fa16c0cc3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:08:24.407 INFO 2981023 --- [ocp-schedules-9,2c06d6db8668753d,fa9c546ba20f3b33] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:08:25.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,7b86cfdb67ae59a5,848797800a468625] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7b86cfdb67ae59a5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:25.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,7b86cfdb67ae59a5,848797800a468625] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7b86cfdb67ae59a5, duration=9 ms] 2024-09-11 18:08:25.394 INFO 2981023 --- [ocp-async-7,76867f7406480636,35e94eb93cad5796] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:25.489 INFO 2981023 --- [ocp-schedules-2,000368a4a3e18bbd,031915e390355b4c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:08:25.489 INFO 2981023 --- [ocp-schedules-2,000368a4a3e18bbd,031915e390355b4c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:08:25.489 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:08:25.489 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:08:25.489 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:08:25.514 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:08:25.517 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:08:25.518 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:08:25.519 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:08:25.519 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:08:25.520 INFO 2981023 --- [ocp-schedules-2,000368a4a3e18bbd,031915e390355b4c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:08:25.520 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:08:25.522 INFO 2981023 --- [ocp-schedules-2,000368a4a3e18bbd,031915e390355b4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:08:25.543 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:08:25.553 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:08:25.553 INFO 2981023 --- [ocp-schedules-2,000368a4a3e18bbd,031915e390355b4c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:08:25.553 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:08:25.557 INFO 2981023 --- [ocp-schedules-2,000368a4a3e18bbd,031915e390355b4c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:08:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,35c9b7eda8d239ac,a0c1632f28c1e549] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,35c9b7eda8d239ac,a0c1632f28c1e549] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:08:26.351980+08:00, lastProcessTime=2024-09-11T18:08:16.352592+08:00 2024-09-11 18:08:26.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:08:26.351980+08:00 2024-09-11 18:08:26.947 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:26.948 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:27.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,305297bc5e331a49,20add4b3728e99c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=305297bc5e331a49, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:27.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,305297bc5e331a49,20add4b3728e99c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=305297bc5e331a49, duration=11 ms] 2024-09-11 18:08:27.394 INFO 2981023 --- [ocp-async-3,9113badf826cdee4,1e1b3cbe57fa4101] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:27.782 INFO 2981023 --- [ocp-async-8,9241d15b7d9056fe,c93daab4e4017fa7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:08:28.080 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.082 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterI 2024-09-11 18:08:28.089 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.095 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.100 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.104 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.110 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.114 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.114 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:08:28.117 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.121 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.130 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.135 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.156 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 33 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.229 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 13 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.260 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 162 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.281 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.283 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.283 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:08:28.284 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.285 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.285 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.286 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.287 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.287 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.288 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.289 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.292 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.292 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:08:28.294 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.297 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.300 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:02:27.853, end=2024-09-11T18:07:28.076} 2024-09-11 18:08:28.300 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 5 dubious sqls 2024-09-11 18:08:29.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,1d833d1904cad2dc,903e8fc4a0133153] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1d833d1904cad2dc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:29.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,1d833d1904cad2dc,903e8fc4a0133153] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1d833d1904cad2dc, duration=9 ms] 2024-09-11 18:08:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,066d1e055594fe25,7fb08abdf00744c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:29.315 INFO 2981023 --- [prometheus-minute-schedule-1,066d1e055594fe25,7fb08abdf00744c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:29.394 INFO 2981023 --- [ocp-async-4,82d259ce13075bef,11e7457638474d97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:29.948 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:29.949 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:31.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2acc7631a69338de,433f84c9fc19ac57] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2acc7631a69338de, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:31.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2acc7631a69338de,433f84c9fc19ac57] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2acc7631a69338de, duration=10 ms] 2024-09-11 18:08:31.394 INFO 2981023 --- [ocp-async-8,8428f31061a42396,8bb8666dfb0c148f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,0991b7f59219bdec,63781a0421134d53] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,0991b7f59219bdec,63781a0421134d53] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:32.522 INFO 2981023 --- [ocp-schedules-1,037c43dbcb2600c8,afa676000fb08f88] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:08:32.949 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:32.950 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:33.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,503747ec980b2220,d9d5c87f9bb23ce2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=503747ec980b2220, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:33.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,503747ec980b2220,d9d5c87f9bb23ce2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=503747ec980b2220, duration=9 ms] 2024-09-11 18:08:33.394 INFO 2981023 --- [ocp-async-1,1b9d78c6729614d9,dbf0d0286fcd4ebc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:34.171 INFO 2981023 --- [ocp-schedules-19,559670c4d3baa479,75df06b624b8e8a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:08:34.234 WARN 2981023 --- [scanner-13613273-1,559670c4d3baa479,d2932aee6b3d5f69] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:08:34.255 INFO 2981023 --- [alarm-detect-3,559670c4d3baa479,3cf79e40bd11c4dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645537, data is empty. 2024-09-11 18:08:34.260 INFO 2981023 --- [alarm-detect-3,559670c4d3baa479,3cf79e40bd11c4dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645537, data is empty. 2024-09-11 18:08:34.266 INFO 2981023 --- [alarm-detect-3,559670c4d3baa479,3cf79e40bd11c4dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645537, data is empty. 2024-09-11 18:08:34.286 INFO 2981023 --- [alarm-detect-5,559670c4d3baa479,3870493c1daa7773] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645537, data is empty. 2024-09-11 18:08:34.286 INFO 2981023 --- [alarm-detect-4,559670c4d3baa479,0380c7f6963322c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645537, data is empty. 2024-09-11 18:08:34.295 INFO 2981023 --- [alarm-detect-5,559670c4d3baa479,3870493c1daa7773] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645537, data is empty. 2024-09-11 18:08:34.296 INFO 2981023 --- [alarm-detect-4,559670c4d3baa479,0380c7f6963322c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645537, data is empty. 2024-09-11 18:08:34.299 INFO 2981023 --- [alarm-detect-4,559670c4d3baa479,0380c7f6963322c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645537, data is empty. 2024-09-11 18:08:34.300 INFO 2981023 --- [alarm-detect-5,559670c4d3baa479,3870493c1daa7773] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645537, data is empty. 2024-09-11 18:08:34.403 INFO 2981023 --- [alarm-detect-4,559670c4d3baa479,469a58a95f18b44e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645537, data is empty. 2024-09-11 18:08:34.407 INFO 2981023 --- [alarm-detect-4,559670c4d3baa479,469a58a95f18b44e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645537, data is empty. 2024-09-11 18:08:34.411 INFO 2981023 --- [alarm-detect-4,559670c4d3baa479,469a58a95f18b44e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645537, data is empty. 2024-09-11 18:08:35.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,7e4def647c4eb2ab,a64e2cf8fcc50a6c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7e4def647c4eb2ab, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:35.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,7e4def647c4eb2ab,a64e2cf8fcc50a6c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7e4def647c4eb2ab, duration=9 ms] 2024-09-11 18:08:35.166 INFO 2981023 --- [ocp-schedules-26,fa8a4b0838f68fbb,0c10519235998006] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:08:35.180 INFO 2981023 --- [ocp-schedules-26,fa8a4b0838f68fbb,0c10519235998006] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:08:35.180 INFO 2981023 --- [ocp-schedules-26,fa8a4b0838f68fbb,0c10519235998006] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:08:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,e3cb94c8c98d3596,57c127dd2909bb28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,e3cb94c8c98d3596,57c127dd2909bb28] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:35.395 INFO 2981023 --- [ocp-async-6,0f2f86cc3bab615f,0ae437ed5931e671] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:35.950 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:35.951 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:08:36.337452+08:00, lastProcessTime=2024-09-11T18:08:26.351980+08:00 2024-09-11 18:08:36.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:08:36.337452+08:00 2024-09-11 18:08:37.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,01392374c636d732,8be96860dec83478] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=01392374c636d732, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:37.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,01392374c636d732,8be96860dec83478] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=01392374c636d732, duration=9 ms] 2024-09-11 18:08:37.314 INFO 2981023 --- [prometheus-cache-monitor1,8d934e76e4d7122e,032cdcf6c75f9ca1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8429,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5149} 2024-09-11 18:08:37.315 INFO 2981023 --- [prometheus-cache-monitor1,8d934e76e4d7122e,032cdcf6c75f9ca1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:08:37.395 INFO 2981023 --- [ocp-async-7,680bac2bdfe61dc3,74dcda3c19b3fd08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:37.782 INFO 2981023 --- [ocp-async-2,ef812fd4c2bcc184,cf19954cd09bd8ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:08:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,38ea69b62a079c01,fcc13b2fc56683ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,38ea69b62a079c01,fcc13b2fc56683ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:38.951 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:38.953 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:39.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d2a789a30e2ddcc3,8a59c85555d21a89] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d2a789a30e2ddcc3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:39.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,d2a789a30e2ddcc3,8a59c85555d21a89] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d2a789a30e2ddcc3, duration=16 ms] 2024-09-11 18:08:39.395 INFO 2981023 --- [ocp-async-1,82fd8f026c3cec2c,eca3be6218db9fea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:40.003 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:08:40.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:08:41.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2d781c68ab0e370c,ce179f61517c6ae5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2d781c68ab0e370c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:41.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2d781c68ab0e370c,ce179f61517c6ae5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2d781c68ab0e370c, duration=9 ms] 2024-09-11 18:08:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,b2a41e260c19bcf2,1f8a0d32e8a07c45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,b2a41e260c19bcf2,1f8a0d32e8a07c45] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:41.395 INFO 2981023 --- [ocp-async-6,3d789d9671d40ee7,25d6191a7bf0b731] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:41.953 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:41.954 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:42.524 INFO 2981023 --- [ocp-schedules-30,2ded75a27cb9e3f1,66e804dab4ed22d5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:08:43.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,24583ab936c0de8d,42de74b8a8eb5827] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=24583ab936c0de8d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:43.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,24583ab936c0de8d,42de74b8a8eb5827] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=24583ab936c0de8d, duration=9 ms] 2024-09-11 18:08:43.395 INFO 2981023 --- [ocp-async-7,fc82b59323157ec3,5bc404fd884a31fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,737f855c5cc4b73d,70843d0995c0f566] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,737f855c5cc4b73d,70843d0995c0f566] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:44.464 INFO 2981023 --- [ocp-schedules-14,7b6467e580e4059b,927a0bafe5973561] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:08:44.523 WARN 2981023 --- [scanner-13613571-1,7b6467e580e4059b,b453788fce9306f0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:08:44.789 INFO 2981023 --- [ocp-schedules-5,312e419ebdf58de1,074345f19f3d5d3a] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:08:44.807 WARN 2981023 --- [ocp-schedules-5,312e419ebdf58de1,074345f19f3d5d3a] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:08:44.954 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:44.954 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:45.101 INFO 2981023 --- [prometheus-second-schedule-1,7be0a84e4dab677d,b77105355d3f55f7] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:08:45.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4fc1c8cffc308596,218e400f13ac5eed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4fc1c8cffc308596, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:45.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4fc1c8cffc308596,218e400f13ac5eed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4fc1c8cffc308596, duration=10 ms] 2024-09-11 18:08:45.395 INFO 2981023 --- [ocp-async-10,8fc17d3c60c50ba0,638b2e3f664eba40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:46.017 INFO 2981023 --- [ocp-schedules-25,fa890d61807f309a,df388bfdcfa3610b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:08:46.018 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:08:46.018 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:08:46.019 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:08:46.020 INFO 2981023 --- [ocp-schedules-21,97e6b30576d0ea87,fc12807be674e38a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:08:46.037 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:08:46.038 INFO 2981023 --- [ocp-schedules-6,36020558c6153e2b,78951e00f5c10236] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:08:46.038 INFO 2981023 --- [host-check-agent-status1,36020558c6153e2b,e44ae5c0b4594649] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:08:46.038 INFO 2981023 --- [host-check-agent-status2,36020558c6153e2b,4ec1f4c051d99fe4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:08:46.039 INFO 2981023 --- [host-check-agent-status3,36020558c6153e2b,66e83ee24a3e393b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:08:46.039 INFO 2981023 --- [host-check-agent-status4,36020558c6153e2b,10a51bec227bf603] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:08:46.039 INFO 2981023 --- [host-check-clockdiff1,97e6b30576d0ea87,5ace7bfca9e18fa1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:08:46.040 INFO 2981023 --- [host-check-clockdiff2,97e6b30576d0ea87,790c00fd46cb1e47] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:08:46.041 INFO 2981023 --- [host-check-clockdiff3,97e6b30576d0ea87,7b4efb1c7ded1214] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:08:46.041 INFO 2981023 --- [host-check-clockdiff4,97e6b30576d0ea87,68a1c202aff6294c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:08:46.047 INFO 2981023 --- [host-check-clockdiff3,97e6b30576d0ea87,7b4efb1c7ded1214] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049326 0 0 , error= 2024-09-11 18:08:46.057 INFO 2981023 --- [host-check-agent-status3,36020558c6153e2b,66e83ee24a3e393b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:08:46.061 INFO 2981023 --- [host-check-agent-status2,36020558c6153e2b,4ec1f4c051d99fe4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:08:46.061 INFO 2981023 --- [host-check-agent-status3,36020558c6153e2b,66e83ee24a3e393b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:08:46.063 INFO 2981023 --- [host-check-agent-status1,36020558c6153e2b,e44ae5c0b4594649] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:08:46.065 INFO 2981023 --- [host-check-agent-status2,36020558c6153e2b,4ec1f4c051d99fe4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:08:46.065 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:08:46.066 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:08:46.066 INFO 2981023 --- [host-check-agent-status1,36020558c6153e2b,e44ae5c0b4594649] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:08:46.066 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:08:46.067 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:08:46.069 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:08:46.069 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:08:46.070 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:08:46.070 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:08:46.170 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:08:46.187 INFO 2981023 --- [ocp-schedules-15,bebf988ee5f88c3e,423ad085d6eb70ad] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:08:46.191 INFO 2981023 --- [ocp-schedules-1,0012a7a265e83fc1,5b045a3b6feaa86b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:08:46.191 INFO 2981023 --- [ocp-schedules-19,45f0162b0fba986e,2d63caf8d7f60e28] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:08:46.191 INFO 2981023 --- [ocp-schedules-12,ea92e88b877496cf,552cf6c4285f4e1d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:08:46.271 INFO 2981023 --- [ocp-schedules-2,2bbd3830be289328,5836ace876bc9b4b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:08:46.273 INFO 2981023 --- [ocp-schedules-1,0012a7a265e83fc1,5b045a3b6feaa86b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:08:46.273 INFO 2981023 --- [ocp-schedules-19,45f0162b0fba986e,2d63caf8d7f60e28] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:08:46.273 INFO 2981023 --- [host-check-agent-status4,36020558c6153e2b,10a51bec227bf603] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:08:46.285 INFO 2981023 --- [ocp-schedules-15,bebf988ee5f88c3e,423ad085d6eb70ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:08:46.286 INFO 2981023 --- [host-check-agent-status4,36020558c6153e2b,10a51bec227bf603] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:08:46.297 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049282825000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:08:46.303 INFO 2981023 --- [ocp-schedules-1,0012a7a265e83fc1,5b045a3b6feaa86b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:08:46.357 INFO 2981023 --- [ocp-schedules-2,2bbd3830be289328,5836ace876bc9b4b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:08:46.374 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:08:46.376 INFO 2981023 --- [ocp-schedules-17,cedcbb854aedc82d,c071fc198d11431c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:08:46.455 INFO 2981023 --- [host-check-agent-status2,36020558c6153e2b,4ec1f4c051d99fe4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:08:46.462 INFO 2981023 --- [host-check-agent-status2,36020558c6153e2b,4ec1f4c051d99fe4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:08:46.462 INFO 2981023 --- [host-check-agent-status2,36020558c6153e2b,4ec1f4c051d99fe4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:08:46.462 INFO 2981023 --- [host-check-agent-status2,36020558c6153e2b,4ec1f4c051d99fe4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:08:46.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:08:46.461747+08:00, lastProcessTime=2024-09-11T18:08:36.337452+08:00 2024-09-11 18:08:46.477 INFO 2981023 --- [host-check-agent-status1,36020558c6153e2b,e44ae5c0b4594649] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:08:46.478 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:08:46.480 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:08:46.480023814+08:00 2024-09-11 18:08:46.483 INFO 2981023 --- [host-check-agent-status1,36020558c6153e2b,e44ae5c0b4594649] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:08:46.483 INFO 2981023 --- [host-check-agent-status1,36020558c6153e2b,e44ae5c0b4594649] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:08:46.483 INFO 2981023 --- [host-check-agent-status1,36020558c6153e2b,e44ae5c0b4594649] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:08:46.484 INFO 2981023 --- [host-check-agent-status3,36020558c6153e2b,66e83ee24a3e393b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:08:46.486 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:08:46.480+08:00 2024-09-11 18:08:46.487 INFO 2981023 --- [host-check-agent-status3,36020558c6153e2b,66e83ee24a3e393b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:08:46.487 INFO 2981023 --- [host-check-agent-status3,36020558c6153e2b,66e83ee24a3e393b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:08:46.487 INFO 2981023 --- [host-check-agent-status3,36020558c6153e2b,66e83ee24a3e393b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:08:46.488 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:08:46.488 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:08:46.488 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:08:46.494012136+08:00 2024-09-11 18:08:46.489 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:08:46.494628765+08:00 2024-09-11 18:08:46.489 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:08:46.480+08:00 and status AVAILABLE 2024-09-11 18:08:46.490 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:08:46.488+08:00 2024-09-11 18:08:46.490 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:08:46.489+08:00 2024-09-11 18:08:46.492 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:08:46.496 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:08:46.495087262+08:00 2024-09-11 18:08:46.498 INFO 2981023 --- [host-check-agent-status4,36020558c6153e2b,10a51bec227bf603] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:08:46.499 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:08:46.496+08:00 2024-09-11 18:08:46.500 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:08:46.488+08:00 and status AVAILABLE 2024-09-11 18:08:46.500 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.501 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:08:46.489+08:00 and status AVAILABLE 2024-09-11 18:08:46.501 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:08:46.502 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:08:46.496+08:00 and status AVAILABLE 2024-09-11 18:08:46.502 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:08:46.502 INFO 2981023 --- [host-check-agent-status4,36020558c6153e2b,10a51bec227bf603] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:08:46.502 INFO 2981023 --- [host-check-agent-status4,36020558c6153e2b,10a51bec227bf603] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:08:46.502 INFO 2981023 --- [host-check-agent-status4,36020558c6153e2b,10a51bec227bf603] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:08:46.504 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:08:46.512 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:08:46.512 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:08:46.514 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:08:46.569 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.570 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.571 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:46.571 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:08:46.572 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:08:46.572 INFO 2981023 --- [ocp-schedules-15,bebf988ee5f88c3e,423ad085d6eb70ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:08:46.573 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.573 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:46.576 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:08:46.576 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:08:46.578 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:08:46.578 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:08:46.578 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:08:46.578 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:08:46.579 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:08:46.580 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:08:46.580 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.580 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:08:46.581 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:08:46.582 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.596 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.597 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:46.600 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:08:46.605 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:08:46.607 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:08:46.628 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:08:46.634 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:08:46.654 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:08:46.660 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:08:46.666 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:08:46.667 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:08:46.673 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.674 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.674 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:08:46.682 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:08:46.695 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:08:46.709 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.713 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:08:46.716 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:08:46.720 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:08:46.722 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:08:46.731 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.732 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:08:46.732 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:08:46.732 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.743 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:08:46.749 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:08:46.754 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:08:46.755 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:08:46.757 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:08:46.757 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049282825000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:08:46.761 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:08:46.764 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:08:46.764 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:08:46.766 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:08:46.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.770 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.770 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:46.771 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:08:46.771 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.773 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:08:46.774 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:08:46.780 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:08:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:08:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:08:46.794 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:08:46.796 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:08:46.801 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:08:46.803 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.807 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:08:46.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.810 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.810 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:08:46.810 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:08:46.810 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:08:46.812 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.812 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.812 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:08:46.813 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:08:46.813 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.813 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:08:46.814 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:08:46.814 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.818 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.818 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.818 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:08:46.819 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:08:46.820 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:08:46.823 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:08:46.823 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:08:46.827 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:08:46.828 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:08:46.828 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:08:46.829 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:08:46.830 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:08:46.833 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.833 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.834 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:08:46.834 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:08:46.834 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.835 INFO 2981023 --- [host-check-host-status4,fa890d61807f309a,f9fd4296a152d355] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:08:46.837 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:08:46.838 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:08:46.848 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:08:46.853 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:08:46.856 INFO 2981023 --- [host-check-host-status2,fa890d61807f309a,8ffb368f420a169b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:08:46.858 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:08:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:08:46.861 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:08:46.863 INFO 2981023 --- [host-check-host-status1,fa890d61807f309a,f3eb2a3571ac7f70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:08:46.866 INFO 2981023 --- [host-check-host-status3,fa890d61807f309a,e18dfbd655a12b23] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:08:46.867 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:08:46.879 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:08:46.888 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.889 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.890 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:08:46.890 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:08:46.891 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:08:46.895 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:08:46.902 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:08:46.907 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:08:46.908 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:08:46.909 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:08:46.925 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:08:46.929 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:08:46.948 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.949 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.950 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:08:46.951 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:08:46.951 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.951 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:08:46.461747+08:00 2024-09-11 18:08:46.956 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:08:46.962 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:08:46.964 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.965 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.965 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:08:46.972 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:46.973 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:46.973 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:08:46.974 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:08:46.974 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:08:46.980 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:08:46.980 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:08:46.981 INFO 2981023 --- [ocp-schedules-19,45f0162b0fba986e,2d63caf8d7f60e28] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:08:46.985 INFO 2981023 --- [host-check-clockdiff2,97e6b30576d0ea87,790c00fd46cb1e47] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049326 5 5 , error= 2024-09-11 18:08:46.986 INFO 2981023 --- [host-check-clockdiff1,97e6b30576d0ea87,5ace7bfca9e18fa1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049326 5 5 , error= 2024-09-11 18:08:47.005 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:08:47.020 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:08:47.026 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:08:47.029 INFO 2981023 --- [ocp-schedules-24,24f8bd54be0be278,81d39e509f6595e8] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:08:47.095 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,909a18615780f3ac,b16ef5693d871027] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=909a18615780f3ac, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:47.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,909a18615780f3ac,b16ef5693d871027] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=909a18615780f3ac, duration=10 ms] 2024-09-11 18:08:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,82af51a6a1a7f11a,5fa453f7c52b6c19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,82af51a6a1a7f11a,5fa453f7c52b6c19] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:47.395 INFO 2981023 --- [ocp-async-3,2f2a8fefc1f1ea6e,d4fed1d2d3e26d09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:47.782 INFO 2981023 --- [ocp-async-2,534586def657132c,c2d199b33fdeb952] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:08:47.955 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:47.955 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:49.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,aeabc630ab93eb93,5b1ff7dceaa16e2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=aeabc630ab93eb93, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:49.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,aeabc630ab93eb93,5b1ff7dceaa16e2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=aeabc630ab93eb93, duration=9 ms] 2024-09-11 18:08:49.395 INFO 2981023 --- [ocp-async-8,f9a2f7afc7b9c279,a04a54a9ef94fa5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:08:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:08:50.308 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:08:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,38103982f702c873,d5340bddb922e585] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.313 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-8,d917430955ae8a2c,6c6e3404e6202796] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:08:50.314 INFO 2981023 --- [ocp-schedules-25,7bc0686c6f19be47,3b24d45cca4c6380] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:08:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,38103982f702c873,d5340bddb922e585] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:50.955 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:50.956 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:51.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,eee32e4007e35000,ebba0e5969e9bd7a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=eee32e4007e35000, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:51.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,eee32e4007e35000,ebba0e5969e9bd7a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=eee32e4007e35000, duration=9 ms] 2024-09-11 18:08:51.395 INFO 2981023 --- [ocp-async-2,d50a8b5b75fd846b,42b874a2b41dda45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:51.605 INFO 2981023 --- [ocp-schedules-1,553e19428fbc1d25,fdef738cff15b940] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:08:51.606 INFO 2981023 --- [ocp-schedules-1,553e19428fbc1d25,fdef738cff15b940] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:08:51.606 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:08:51.606 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:08:51.607 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:08:51.607 INFO 2981023 --- [ocp-schedules-1,553e19428fbc1d25,fdef738cff15b940] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:08:52.071 INFO 2981023 --- [host-check-clockdiff4,97e6b30576d0ea87,68a1c202aff6294c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049332 0 0 , error= 2024-09-11 18:08:52.527 INFO 2981023 --- [ocp-schedules-5,e8d632205586be8f,61b965a6a4439430] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:08:53.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,c906ed8cba905109,fe96d596628615c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c906ed8cba905109, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:53.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,c906ed8cba905109,fe96d596628615c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c906ed8cba905109, duration=10 ms] 2024-09-11 18:08:53.281 INFO 2981023 --- [ocp-schedules-6,46e519267d72783c,c1c070d18da0d8ea] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:08:53.281 INFO 2981023 --- [ocp-schedules-6,46e519267d72783c,c1c070d18da0d8ea] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:08:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,3e7bfeccd6539bfb,aa5ade25acb3ddf4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,3e7bfeccd6539bfb,aa5ade25acb3ddf4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:53.395 INFO 2981023 --- [ocp-async-4,f31538cee3d2e072,8435e758af7801dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:53.956 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:53.957 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:54.408 INFO 2981023 --- [ocp-schedules-26,970040dd88ac9bf8,ee7b08df1df661e9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:08:54.491 INFO 2981023 --- [ocp-schedules-18,b09e9e943b8c5f6f,4900ec26323bca4e] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:08:54.509 WARN 2981023 --- [ocp-schedules-18,b09e9e943b8c5f6f,4900ec26323bca4e] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:08:54.658 INFO 2981023 --- [ocp-schedules-22,4dd564052e2ebd45,545902abae1445bf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:08:54.708 WARN 2981023 --- [scanner-13613802-1,4dd564052e2ebd45,6460c34fada9dda2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:08:54.720 INFO 2981023 --- [alarm-detect-11,4dd564052e2ebd45,31a8a2a67f25e783] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645537, data is empty. 2024-09-11 18:08:54.723 INFO 2981023 --- [alarm-detect-11,4dd564052e2ebd45,31a8a2a67f25e783] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645537, data is empty. 2024-09-11 18:08:54.729 INFO 2981023 --- [alarm-detect-11,4dd564052e2ebd45,31a8a2a67f25e783] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645537, data is empty. 2024-09-11 18:08:54.746 INFO 2981023 --- [alarm-detect-20,4dd564052e2ebd45,cda5f4ca6d965db1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645537, data is empty. 2024-09-11 18:08:54.746 INFO 2981023 --- [alarm-detect-19,4dd564052e2ebd45,5c01aac265ecce90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645537, data is empty. 2024-09-11 18:08:54.750 INFO 2981023 --- [alarm-detect-19,4dd564052e2ebd45,5c01aac265ecce90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645537, data is empty. 2024-09-11 18:08:54.754 INFO 2981023 --- [alarm-detect-20,4dd564052e2ebd45,cda5f4ca6d965db1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645537, data is empty. 2024-09-11 18:08:54.754 INFO 2981023 --- [alarm-detect-19,4dd564052e2ebd45,5c01aac265ecce90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645537, data is empty. 2024-09-11 18:08:54.758 INFO 2981023 --- [alarm-detect-20,4dd564052e2ebd45,cda5f4ca6d965db1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645537, data is empty. 2024-09-11 18:08:54.834 INFO 2981023 --- [ocp-schedules-15,10c0b134bb4ad8fa,c152aee866f4a5c7] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T18:08:54.834+08:00, tenantSize=6 2024-09-11 18:08:54.847 INFO 2981023 --- [ocp-schedules-15,10c0b134bb4ad8fa,c152aee866f4a5c7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726049334834000 2024-09-11 18:08:54.847 INFO 2981023 --- [ocp-schedules-15,10c0b134bb4ad8fa,c152aee866f4a5c7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726049034801000 2024-09-11 18:08:54.876 INFO 2981023 --- [alarm-detect-2,4dd564052e2ebd45,2a4a035928176163] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645537, data is empty. 2024-09-11 18:08:54.885 INFO 2981023 --- [alarm-detect-2,4dd564052e2ebd45,2a4a035928176163] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645537, data is empty. 2024-09-11 18:08:54.888 INFO 2981023 --- [ocp-schedules-15,10c0b134bb4ad8fa,c152aee866f4a5c7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 18:08:54.890 INFO 2981023 --- [alarm-detect-2,4dd564052e2ebd45,2a4a035928176163] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645537, data is empty. 2024-09-11 18:08:55.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bea9f83ab24c97c5,4dc3056915f02401] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bea9f83ab24c97c5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:55.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bea9f83ab24c97c5,4dc3056915f02401] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bea9f83ab24c97c5, duration=11 ms] 2024-09-11 18:08:55.183 INFO 2981023 --- [ocp-schedules-13,163eaae62e3f401c,cc08c6b4ff75804d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:08:55.191 INFO 2981023 --- [ocp-schedules-13,163eaae62e3f401c,cc08c6b4ff75804d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:08:55.191 INFO 2981023 --- [ocp-schedules-13,163eaae62e3f401c,cc08c6b4ff75804d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:08:55.396 INFO 2981023 --- [ocp-async-1,df3c3013bffb5f67,032b97db9e126465] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:55.852 INFO 2981023 --- [ocp-schedules-28,ea798dd29d8ae58c,c4494f860800b00f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:08:55.852 INFO 2981023 --- [ocp-schedules-28,ea798dd29d8ae58c,c4494f860800b00f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:08:55.853 INFO 2981023 --- [ocp-schedules-28,ea798dd29d8ae58c,c4494f860800b00f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:08:55.853 INFO 2981023 --- [ocp-schedules-28,ea798dd29d8ae58c,c4494f860800b00f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:08:55.853 INFO 2981023 --- [ocp-schedules-28,ea798dd29d8ae58c,c4494f860800b00f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:08:55.853 INFO 2981023 --- [ocp-schedules-28,ea798dd29d8ae58c,c4494f860800b00f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:08:56.018 INFO 2981023 --- [ocp-schedules-18,4e2c4845feff2af9,debf33334881753d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:08:56.024 INFO 2981023 --- [ocp-schedules-18,4e2c4845feff2af9,debf33334881753d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:08:56.024 INFO 2981023 --- [ocp-schedules-18,4e2c4845feff2af9,debf33334881753d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:08:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,be8e0d511d19939c,b61b5c80fdd86c4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,be8e0d511d19939c,b61b5c80fdd86c4c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:56.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:08:56.337898+08:00, lastProcessTime=2024-09-11T18:08:46.461747+08:00 2024-09-11 18:08:56.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:08:56.337898+08:00 2024-09-11 18:08:56.957 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:56.958 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:08:57.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,ad3d27a625e09f14,6a9665e7fdea80a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ad3d27a625e09f14, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:57.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,ad3d27a625e09f14,6a9665e7fdea80a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ad3d27a625e09f14, duration=10 ms] 2024-09-11 18:08:57.396 INFO 2981023 --- [ocp-async-10,792341ee5a4afa51,6f94b7db4b456be7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:57.697 INFO 2981023 --- [ocp-schedules-22,5c0ba5df6fa2d6eb,c0f0a7148be1e874] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:08:57.697 INFO 2981023 --- [ocp-schedules-22,5c0ba5df6fa2d6eb,c0f0a7148be1e874] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:08:57.713 INFO 2981023 --- [Collect all tenants major compactions1,5c0ba5df6fa2d6eb,4d45c212a91be3df] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:08:57.713 INFO 2981023 --- [Collect all tenants major compactions2,5c0ba5df6fa2d6eb,faf85a121911a647] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:08:57.713 INFO 2981023 --- [Collect all tenants major compactions3,5c0ba5df6fa2d6eb,2c30acbbfd423ca4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:08:57.714 INFO 2981023 --- [Collect all tenants major compactions5,5c0ba5df6fa2d6eb,a4801c28b274ed7d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:08:57.715 INFO 2981023 --- [Collect all tenants major compactions4,5c0ba5df6fa2d6eb,85dd56e1e6140301] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:08:57.716 INFO 2981023 --- [Collect all tenants major compactions6,5c0ba5df6fa2d6eb,17df759c126beb83] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:08:57.726 INFO 2981023 --- [Collect all tenants major compactions1,5c0ba5df6fa2d6eb,4d45c212a91be3df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:57.727 INFO 2981023 --- [Collect all tenants major compactions3,5c0ba5df6fa2d6eb,2c30acbbfd423ca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:57.727 INFO 2981023 --- [Collect all tenants major compactions1,5c0ba5df6fa2d6eb,4d45c212a91be3df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:08:57.727 INFO 2981023 --- [Collect all tenants major compactions2,5c0ba5df6fa2d6eb,faf85a121911a647] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:57.728 INFO 2981023 --- [Collect all tenants major compactions3,5c0ba5df6fa2d6eb,2c30acbbfd423ca4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:08:57.728 INFO 2981023 --- [Collect all tenants major compactions4,5c0ba5df6fa2d6eb,85dd56e1e6140301] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:57.729 INFO 2981023 --- [Collect all tenants major compactions5,5c0ba5df6fa2d6eb,a4801c28b274ed7d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:57.729 INFO 2981023 --- [Collect all tenants major compactions2,5c0ba5df6fa2d6eb,faf85a121911a647] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:08:57.730 INFO 2981023 --- [Collect all tenants major compactions4,5c0ba5df6fa2d6eb,85dd56e1e6140301] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:08:57.731 INFO 2981023 --- [Collect all tenants major compactions5,5c0ba5df6fa2d6eb,a4801c28b274ed7d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:08:57.731 INFO 2981023 --- [Collect all tenants major compactions6,5c0ba5df6fa2d6eb,17df759c126beb83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:57.731 INFO 2981023 --- [Collect all tenants major compactions1,5c0ba5df6fa2d6eb,4d45c212a91be3df] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:08:57.731 INFO 2981023 --- [Collect all tenants major compactions1,5c0ba5df6fa2d6eb,4d45c212a91be3df] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:08:57.732 INFO 2981023 --- [Collect all tenants major compactions4,5c0ba5df6fa2d6eb,85dd56e1e6140301] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:08:57.732 INFO 2981023 --- [Collect all tenants major compactions4,5c0ba5df6fa2d6eb,85dd56e1e6140301] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:08:57.732 INFO 2981023 --- [Collect all tenants major compactions6,5c0ba5df6fa2d6eb,17df759c126beb83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:08:57.732 INFO 2981023 --- [Collect all tenants major compactions2,5c0ba5df6fa2d6eb,faf85a121911a647] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:08:57.732 INFO 2981023 --- [Collect all tenants major compactions2,5c0ba5df6fa2d6eb,faf85a121911a647] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:08:57.732 INFO 2981023 --- [Collect all tenants major compactions3,5c0ba5df6fa2d6eb,2c30acbbfd423ca4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:08:57.733 INFO 2981023 --- [Collect all tenants major compactions3,5c0ba5df6fa2d6eb,2c30acbbfd423ca4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:08:57.733 INFO 2981023 --- [Collect all tenants major compactions5,5c0ba5df6fa2d6eb,a4801c28b274ed7d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:08:57.733 INFO 2981023 --- [Collect all tenants major compactions5,5c0ba5df6fa2d6eb,a4801c28b274ed7d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:08:57.734 INFO 2981023 --- [Collect all tenants major compactions6,5c0ba5df6fa2d6eb,17df759c126beb83] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:08:57.735 INFO 2981023 --- [Collect all tenants major compactions6,5c0ba5df6fa2d6eb,17df759c126beb83] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:08:57.735 INFO 2981023 --- [ocp-schedules-22,5c0ba5df6fa2d6eb,c0f0a7148be1e874] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:08:57.782 INFO 2981023 --- [ocp-async-1,be6df4222ba1e4ac,fb18f70e7ecd49eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:08:57.821 INFO 2981023 --- [ocp-schedules-4,6a5d2ad5a80511a4,3c3659adb4a1e0df] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:08:57.821+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:08:57.824 INFO 2981023 --- [ocp-schedules-4,6a5d2ad5a80511a4,3c3659adb4a1e0df] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:08:57.964 INFO 2981023 --- [ocp-schedules-4,6a5d2ad5a80511a4,3c3659adb4a1e0df] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 38 to SqlAggregateData 2024-09-11 18:08:57.965 INFO 2981023 --- [ocp-schedules-4,6a5d2ad5a80511a4,3c3659adb4a1e0df] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:08:58.773 INFO 2981023 --- [ocp-schedules-14,0b6419319e7bd363,db39598923126af8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:08:58.775 INFO 2981023 --- [ocp-schedules-14,0b6419319e7bd363,db39598923126af8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:08:58.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:08:58.801 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.802 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:08:58.802 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.804 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:08:58.806 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:08:58.807 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:08:58.807 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.809 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.809 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.810 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.810 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.810 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:08:58.814 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.814 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.815 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.815 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.820 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.821 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.821 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.821 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:08:58.828 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.828 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.829 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.829 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:08:58.832 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.832 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.833 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:08:58.833 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4D81D746-0-0] 2024-09-11 18:08:58.833 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:08:58.833 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.833 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:08:58.834 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.835 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:08:58.843 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.843 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.844 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:08:58.844 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:08:58.845 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.845 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.856 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.856 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:08:58.857 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:08:58.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4D81D750-0-0] 2024-09-11 18:08:58.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:08:58.867 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.868 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:08:58.869 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.869 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.870 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:08:58.870 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.870 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:08:58.870 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.871 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:08:58.871 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:08:58.871 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.871 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.872 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.872 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:08:58.872 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.872 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:08:58.872 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:08:58.873 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:08:58.873 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.873 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.873 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.874 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:08:58.874 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:08:58.874 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:08:58.874 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.874 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:08:58.875 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:08:58.875 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.875 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.875 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:08:58.876 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:08:58.876 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.876 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.877 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:08:58.878 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.878 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:08:58.878 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:08:58.878 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:08:58.878 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:08:58.878 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:08:58.880 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:08:58.880 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:08:58.893 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:58.895 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.895 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:58.896 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:08:58.896 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:08:58.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:08:58.898 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.898 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.903 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:08:58.906 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:08:58.906 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:08:58.906 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.906 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:08:58.912 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:08:58.912 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.913 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:08:58.914 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:08:58.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:08:58.915 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:08:58.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:08:58.918 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:08:58.918 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:08:58.918 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:08:58.920 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:08:58.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:08:58.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:08:58.921 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:08:58.922 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:08:58.923 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:08:58.923 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:08:58.923 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:08:58.924 INFO 2981023 --- [ocp-schedules-14,0b6419319e7bd363,db39598923126af8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:08:59.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4aff53e7a813abdd,5daff99f47f0a89e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4aff53e7a813abdd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:08:59.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4aff53e7a813abdd,5daff99f47f0a89e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4aff53e7a813abdd, duration=16 ms] 2024-09-11 18:08:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,89ed5894fc487a8e,03fc42564ca48160] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049340, interval=60 2024-09-11 18:08:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,89ed5894fc487a8e,03fc42564ca48160] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:08:59.396 INFO 2981023 --- [ocp-async-5,1a6b9e27f04a531f,ee3c86d29d491f31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:08:59.958 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:08:59.960 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:01.099 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,6edd2123189b2863,69a119d8b250ab10] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6edd2123189b2863, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:01.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,6edd2123189b2863,69a119d8b250ab10] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6edd2123189b2863, duration=10 ms] 2024-09-11 18:09:01.397 INFO 2981023 --- [ocp-async-7,14f113518b88e93e,92fe9324295cf0a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,e4cd96c5c1e02ce4,1c264b58646846cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,e4cd96c5c1e02ce4,1c264b58646846cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:02.529 INFO 2981023 --- [ocp-schedules-9,bcfe720760485212,25828026b3c5065b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:09:02.788 INFO 2981023 --- [ocp-schedules-5,fab6e90af34a2df6,fdcb371c59a6eb35] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:09:02.960 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:02.961 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:03.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,6499bc570e80e5e4,8c451ddd1d9b16d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6499bc570e80e5e4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:03.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,6499bc570e80e5e4,8c451ddd1d9b16d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6499bc570e80e5e4, duration=10 ms] 2024-09-11 18:09:03.396 INFO 2981023 --- [ocp-async-2,7cb34a99cd3473c8,9714b5b3bcb58f6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:04.915 INFO 2981023 --- [ocp-schedules-18,860d7b7a38c0ac78,e84b547742873f94] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:09:04.971 WARN 2981023 --- [scanner-13614109-1,860d7b7a38c0ac78,a07d3d1c661ae96c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:09:05.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,62933d3d4b6edaf5,0cc8951ffb0bf037] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=62933d3d4b6edaf5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:05.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,62933d3d4b6edaf5,0cc8951ffb0bf037] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=62933d3d4b6edaf5, duration=11 ms] 2024-09-11 18:09:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,02b312c5a035d745,ed89f4a3632fc3b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,02b312c5a035d745,ed89f4a3632fc3b8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:05.396 INFO 2981023 --- [ocp-async-6,3e1010e946ba3e17,f665ef95fecb8abc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:05.961 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:05.962 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:09:06.352568+08:00, lastProcessTime=2024-09-11T18:08:56.337898+08:00 2024-09-11 18:09:06.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:09:06.352568+08:00 2024-09-11 18:09:07.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,feaa8f646daf7a21,ee96a83094ba6cfb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=feaa8f646daf7a21, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:07.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,feaa8f646daf7a21,ee96a83094ba6cfb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=feaa8f646daf7a21, duration=10 ms] 2024-09-11 18:09:07.396 INFO 2981023 --- [ocp-async-5,95b81746ef0d607b,948d36a6b46c1731] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:07.782 INFO 2981023 --- [ocp-async-4,fe5699b443bb12f0,34f31de8826eb30f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:09:07.864 INFO 2981023 --- [ocp-schedules-30,e144c1a52b55bffe,e9caac1138399a2a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:09:07.970 INFO 2981023 --- [ocp-schedules-17,bca531abaae3219d,46c37eaf5efeaf46] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:09:07.970+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:09:07.970 INFO 2981023 --- [ocp-schedules-17,bca531abaae3219d,46c37eaf5efeaf46] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:09:07.972 INFO 2981023 --- [ocp-schedules-17,bca531abaae3219d,46c37eaf5efeaf46] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:09:07.972 INFO 2981023 --- [ocp-schedules-17,bca531abaae3219d,46c37eaf5efeaf46] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:09:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,ed7de602581345f6,e4c161ba57a712a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,ed7de602581345f6,e4c161ba57a712a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:08.962 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:08.962 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:09.043 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:05:08.635, end=2024-09-11T18:07:09.039} 2024-09-11 18:09:09.046 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkpr 2024-09-11 18:09:09.047 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:09:09.050 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:05:08.635, end=2024-09-11T18:07:09.039},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:09:09.050 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:09:09.050 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:09:09.055 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:09:09.057 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:05:08.635, end=2024-09-11T18:07:09.039},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:09:09.057 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:09:09.057 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:09:09.074 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:09:09.090 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949090000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.090Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.094 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:09:09.094 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.099 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.099 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.099 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05f029da1807e7bd,9dda3b76e3b14dea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=05f029da1807e7bd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:09.103 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.103 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,05f029da1807e7bd,9dda3b76e3b14dea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=05f029da1807e7bd, duration=10 ms] 2024-09-11 18:09:09.109 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.109 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:09:09.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.09, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.111 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:09:09.115 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:05:08.635, end=2024-09-11T18:07:09.039},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:09:09.115 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:09:09.115 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:09:09.127 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:09:09.145 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949145000, endTimeUs=1726049229039000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=5264F920B43F519F6537946203A89DFC, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T10:09:09.145Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000029, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000010, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 18:09:09.148 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 2 2024-09-11 18:09:09.148 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.154 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.154 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=5264F920B43F519F6537946203A89DFC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.176 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.176 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.183 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.183 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=5264F920B43F519F6537946203A89DFC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.184 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.189 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.145, end=2024-09-11T18:07:09.039}, clusterId=1000006, tenantId=1000006, sqlId=5264F920B43F519F6537946203A89DFC, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.190 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:09:09.190 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:05:08.635, end=2024-09-11T18:07:09.039},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:09:09.190 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:09:09.190 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:09:09.191 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:09:09.193 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:05:08.635, end=2024-09-11T18:07:09.039},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:09:09.193 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:09:09.193 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:09:09.233 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949233000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.233Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.240 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949240000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.240Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.242 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949242000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.242Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.243 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949243000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.243Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.246 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 66 2024-09-11 18:09:09.246 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.246 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949246000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.246Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.249 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949249000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.249Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.254 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949254000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.254Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.256 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949256000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.256Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.262 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.263 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.267 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:09:09.267 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.270 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.270 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.271 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.271 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.278 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:09:09.278 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.279 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.279 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.283 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.283 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.284 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:09:09.284 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.285 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.285 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.286 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.286 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.287 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.287 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.288 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.288 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.290 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.290 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.291 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.291 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.293 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.293 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.293 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.293 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.293 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.293 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.295 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.295 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.295 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.297 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.297 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.299 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.299 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.299 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.299 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.301 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.302 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.302 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.302 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.302 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.305 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.305 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.305 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.305 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.306 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.233, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.306 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.306 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.307 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.307 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.307 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.308 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.24, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.309 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.309 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.314 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.314 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.316 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.246, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.316 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.243, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.316 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.242, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.317 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.318 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.318 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.319 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.319 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.322 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.323 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.326 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.249, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.328 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.254, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.341 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949341000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.341Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.341 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.256, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.350 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.350 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.355 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949355000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.355Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.359 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949359000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.359Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.362 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.362 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949362000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.362Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.362 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.363 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949363000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.363Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.364 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:09:09.364 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.366 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.366 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.367 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949367000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.367Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.370 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.370 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.372 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.372 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.373 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725962949373000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:09:09.373Z, endTime=2024-09-11T10:07:09.039Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:09:09.378 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.378 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.378 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.378 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.379 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.379 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.381 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.381 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.382 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.382 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.387 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.387 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.387 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.387 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.388 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.388 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.391 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.391 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.391 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.391 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.391 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.391 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.392 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.392 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.395 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:09:09.395 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:09:09.395 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.395 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.397 INFO 2981023 --- [ocp-async-9,7f7b4ca7f5ff8895,5e24ce01ef70cd55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:09.399 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.341, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.402 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.355, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.405 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.405 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.406 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.406 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.408 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.408 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.408 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.408 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.408 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.408 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.408 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.409 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.409 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.410 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:09:09.410 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:09:09.411 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.412 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.413 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.359, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.416 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.362, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.418 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.363, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=3A91F462ACE7A9A67A9669D151BE10B1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.418 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.418 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.418 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:09:09.418 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049229039000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:09:09.419 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.420 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:09:09.423 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.373, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.423 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:09:09.367, end=2024-09-11T18:07:09.039}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:09:09.423 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:09:09.423 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:05:08.635, end=2024-09-11T18:07:09.039},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:09:09.424 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:09:09.424 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:09:09.424 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:09:10.003 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:09:10.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:09:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:09:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:09:11.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,9668b9831b3ac9a7,c995b7eb68b57470] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9668b9831b3ac9a7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:11.125 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,9668b9831b3ac9a7,c995b7eb68b57470] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9668b9831b3ac9a7, duration=9 ms] 2024-09-11 18:09:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,60c718561bd74652,2083b58cf6b4d10a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,60c718561bd74652,2083b58cf6b4d10a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:11.397 INFO 2981023 --- [ocp-async-3,e648842ba5a8778c,c6ee3bb446da4d60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:11.962 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:11.963 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:12.533 INFO 2981023 --- [ocp-schedules-23,d7b369b24533723f,1138df1d26577836] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:09:13.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,6090544e6535ccee,80f42feb1a578103] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6090544e6535ccee, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:13.130 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,6090544e6535ccee,80f42feb1a578103] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6090544e6535ccee, duration=11 ms] 2024-09-11 18:09:13.297 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049280, nodeCount=480, affectRows=480 2024-09-11 18:09:13.397 INFO 2981023 --- [ocp-async-8,ed92288a43565dac,7c1ce84e877ff943] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,5657f6d4a7fc6261,bc6a855cbf0fc20b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,de105c2cc14152a9,551d637eff6c5923] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:09:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,5657f6d4a7fc6261,bc6a855cbf0fc20b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:14.622 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049280, nodeCount=48, affectRows=48 2024-09-11 18:09:14.622 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049280, nodeCount=46, validCount=46 2024-09-11 18:09:14.742 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049280, nodeCount=1000, affectRows=1000 2024-09-11 18:09:14.744 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049280, nodeCount=1000, affectRows=1000 2024-09-11 18:09:14.745 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049280, nodeCount=1000, validCount=1000 2024-09-11 18:09:14.746 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049280, nodeCount=999, validCount=999 2024-09-11 18:09:14.963 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:14.964 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:15.102 INFO 2981023 --- [ocp-schedules-23,fba6271c8b7c8283,5b76d10496fb21c9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:09:15.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0357f560c1c73c73,a5ecf0477023571a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0357f560c1c73c73, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:15.129 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0357f560c1c73c73,a5ecf0477023571a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0357f560c1c73c73, duration=27 ms] 2024-09-11 18:09:15.179 WARN 2981023 --- [scanner-13615839-1,fba6271c8b7c8283,5121af5f999dd252] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:09:15.186 INFO 2981023 --- [alarm-detect-8,fba6271c8b7c8283,a07217201e9ed5b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645537, data is empty. 2024-09-11 18:09:15.190 INFO 2981023 --- [alarm-detect-8,fba6271c8b7c8283,a07217201e9ed5b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645575, data is empty. 2024-09-11 18:09:15.194 INFO 2981023 --- [alarm-detect-8,fba6271c8b7c8283,a07217201e9ed5b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645575, data is empty. 2024-09-11 18:09:15.196 INFO 2981023 --- [ocp-schedules-7,3a677d6dc4e584ad,9046887b638769c7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:09:15.208 INFO 2981023 --- [ocp-schedules-7,3a677d6dc4e584ad,9046887b638769c7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:09:15.208 INFO 2981023 --- [ocp-schedules-7,3a677d6dc4e584ad,9046887b638769c7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:09:15.215 INFO 2981023 --- [alarm-detect-15,fba6271c8b7c8283,fc1c131d17132332] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645537, data is empty. 2024-09-11 18:09:15.223 INFO 2981023 --- [alarm-detect-7,fba6271c8b7c8283,f0236a62451269fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645537, data is empty. 2024-09-11 18:09:15.223 INFO 2981023 --- [alarm-detect-15,fba6271c8b7c8283,fc1c131d17132332] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645575, data is empty. 2024-09-11 18:09:15.228 INFO 2981023 --- [alarm-detect-15,fba6271c8b7c8283,fc1c131d17132332] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645575, data is empty. 2024-09-11 18:09:15.228 INFO 2981023 --- [alarm-detect-7,fba6271c8b7c8283,f0236a62451269fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645575, data is empty. 2024-09-11 18:09:15.235 INFO 2981023 --- [alarm-detect-7,fba6271c8b7c8283,f0236a62451269fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645575, data is empty. 2024-09-11 18:09:15.244 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049280, nodeCount=480, validCount=480 2024-09-11 18:09:15.342 INFO 2981023 --- [alarm-detect-13,fba6271c8b7c8283,06ade1e6f8811dd0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645537, data is empty. 2024-09-11 18:09:15.345 INFO 2981023 --- [alarm-detect-13,fba6271c8b7c8283,06ade1e6f8811dd0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645575, data is empty. 2024-09-11 18:09:15.349 INFO 2981023 --- [alarm-detect-13,fba6271c8b7c8283,06ade1e6f8811dd0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645575, data is empty. 2024-09-11 18:09:15.397 INFO 2981023 --- [ocp-async-10,d11a285ad37b39ff,89e10d4bbbfec0dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:16.015 INFO 2981023 --- [ocp-schedules-11,3560ea2c5b6f5566,743fbf507419bff8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:09:16.015 INFO 2981023 --- [host-check-clockdiff1,3560ea2c5b6f5566,6238c4e3a0688dd0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:09:16.016 INFO 2981023 --- [host-check-clockdiff2,3560ea2c5b6f5566,1cfe03737d500810] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:09:16.018 INFO 2981023 --- [host-check-clockdiff3,3560ea2c5b6f5566,b7fa1efbe0ce9537] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:09:16.018 INFO 2981023 --- [host-check-clockdiff4,3560ea2c5b6f5566,f68afcff3ea60035] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:09:16.028 INFO 2981023 --- [ocp-schedules-6,43c63a8fbcc49c39,9b9d4e01b7b044e8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:09:16.029 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:09:16.029 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:09:16.030 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:09:16.030 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:09:16.031 INFO 2981023 --- [ocp-schedules-12,b983931236c191c8,32d8621db9332907] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:09:16.031 INFO 2981023 --- [host-check-agent-status1,b983931236c191c8,64dee60af9901734] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:09:16.032 INFO 2981023 --- [host-check-agent-status2,b983931236c191c8,688a487746010b83] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:09:16.032 INFO 2981023 --- [host-check-agent-status3,b983931236c191c8,8c40a07bf99fe0ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:09:16.033 INFO 2981023 --- [host-check-agent-status4,b983931236c191c8,6651d2c4bf723967] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:09:16.048 INFO 2981023 --- [host-check-clockdiff3,3560ea2c5b6f5566,b7fa1efbe0ce9537] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049356 0 0 , error= 2024-09-11 18:09:16.064 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:09:16.065 INFO 2981023 --- [host-check-agent-status1,b983931236c191c8,64dee60af9901734] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:09:16.066 INFO 2981023 --- [host-check-agent-status2,b983931236c191c8,688a487746010b83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:09:16.068 INFO 2981023 --- [host-check-agent-status2,b983931236c191c8,688a487746010b83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:09:16.068 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:09:16.068 INFO 2981023 --- [host-check-agent-status4,b983931236c191c8,6651d2c4bf723967] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:09:16.069 INFO 2981023 --- [host-check-agent-status1,b983931236c191c8,64dee60af9901734] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:09:16.069 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:09:16.072 INFO 2981023 --- [host-check-agent-status4,b983931236c191c8,6651d2c4bf723967] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:09:16.081 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:09:16.082 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:09:16.103 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:09:16.107 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:09:16.187 INFO 2981023 --- [ocp-schedules-22,9cbd33c1e94be6cd,487cad427bcff4a8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:09:16.187 INFO 2981023 --- [ocp-schedules-17,5c982d1831f5da26,6e7bc3a1666fb5ff] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:09:16.188 INFO 2981023 --- [ocp-schedules-28,833d8235062ed167,f3460f0752a92981] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:09:16.188 INFO 2981023 --- [ocp-schedules-29,72bd857544ad4db8,ed30c1b8a2483f2c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:09:16.188 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:09:16.191 INFO 2981023 --- [host-check-agent-status3,b983931236c191c8,8c40a07bf99fe0ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:09:16.193 INFO 2981023 --- [ocp-schedules-28,833d8235062ed167,f3460f0752a92981] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:09:16.193 INFO 2981023 --- [ocp-schedules-17,5c982d1831f5da26,6e7bc3a1666fb5ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:09:16.193 INFO 2981023 --- [host-check-agent-status3,b983931236c191c8,8c40a07bf99fe0ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:09:16.195 INFO 2981023 --- [ocp-schedules-17,5c982d1831f5da26,6e7bc3a1666fb5ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:09:16.203 INFO 2981023 --- [ocp-schedules-22,9cbd33c1e94be6cd,487cad427bcff4a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:09:16.230 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049282825000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:09:16.257 INFO 2981023 --- [ocp-schedules-15,39cce47ca80f0957,4990fa536c5663f5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:09:16.270 INFO 2981023 --- [host-check-agent-status4,b983931236c191c8,6651d2c4bf723967] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:09:16.275 INFO 2981023 --- [host-check-agent-status4,b983931236c191c8,6651d2c4bf723967] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:09:16.275 INFO 2981023 --- [host-check-agent-status4,b983931236c191c8,6651d2c4bf723967] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:09:16.275 INFO 2981023 --- [host-check-agent-status4,b983931236c191c8,6651d2c4bf723967] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:09:16.294 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:09:16.294 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:09:16.295 INFO 2981023 --- [host-check-agent-status1,b983931236c191c8,64dee60af9901734] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:09:16.297 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:09:16.302496021+08:00 2024-09-11 18:09:16.297 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:09:16.302880968+08:00 2024-09-11 18:09:16.299 INFO 2981023 --- [host-check-agent-status2,b983931236c191c8,688a487746010b83] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:09:16.299 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:09:16.297+08:00 2024-09-11 18:09:16.300 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:09:16.297+08:00 2024-09-11 18:09:16.302 INFO 2981023 --- [host-check-agent-status2,b983931236c191c8,688a487746010b83] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:09:16.302 INFO 2981023 --- [host-check-agent-status2,b983931236c191c8,688a487746010b83] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:09:16.302 INFO 2981023 --- [host-check-agent-status2,b983931236c191c8,688a487746010b83] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:09:16.302 INFO 2981023 --- [host-check-agent-status1,b983931236c191c8,64dee60af9901734] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:09:16.302 INFO 2981023 --- [host-check-agent-status1,b983931236c191c8,64dee60af9901734] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:09:16.302 INFO 2981023 --- [host-check-agent-status1,b983931236c191c8,64dee60af9901734] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:09:16.303 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:09:16.297+08:00 and status AVAILABLE 2024-09-11 18:09:16.308 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:09:16.297+08:00 and status AVAILABLE 2024-09-11 18:09:16.312 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:09:16.314 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:09:16.315 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:09:16.315389253+08:00 2024-09-11 18:09:16.316 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:09:16.316+08:00 2024-09-11 18:09:16.318 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.318 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:16.327 INFO 2981023 --- [host-check-agent-status3,b983931236c191c8,8c40a07bf99fe0ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:09:16.330 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:09:16.331 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.331 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:09:16.331 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:09:16.331180378+08:00 2024-09-11 18:09:16.332 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:09:16.332 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:09:16.331+08:00 2024-09-11 18:09:16.334 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:09:16.331+08:00 and status AVAILABLE 2024-09-11 18:09:16.334 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:09:16.316+08:00 and status AVAILABLE 2024-09-11 18:09:16.335 INFO 2981023 --- [host-check-agent-status3,b983931236c191c8,8c40a07bf99fe0ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:09:16.335 INFO 2981023 --- [host-check-agent-status3,b983931236c191c8,8c40a07bf99fe0ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:09:16.335 INFO 2981023 --- [host-check-agent-status3,b983931236c191c8,8c40a07bf99fe0ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:09:16.338 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:09:16.340 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:09:16.340086+08:00, lastProcessTime=2024-09-11T18:09:06.352568+08:00 2024-09-11 18:09:16.340 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:09:16.344 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:09:16.345 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:09:16.347 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.348 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:09:16.349 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:09:16.349 INFO 2981023 --- [ocp-schedules-22,9cbd33c1e94be6cd,487cad427bcff4a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:09:16.355 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.356 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:16.356 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:09:16.358 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:09:16.361 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:09:16.362 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.363 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:16.364 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:09:16.368 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:09:16.369 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:09:16.369 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:09:16.390 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:09:16.394 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:09:16.396 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:09:16.409 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:09:16.415 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:16.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:09:16.419 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:09:16.420 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:09:16.424 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:09:16.427 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:09:16.427 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049282825000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:09:16.429 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:09:16.440 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:09:16.440 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:09:16.449 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:09:16.461 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:16.461 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:09:16.461 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.463 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:16.464 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:09:16.465 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.468 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:09:16.468 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:09:16.475 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:09:16.478 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:09:16.483 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:09:16.489 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:09:16.489 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:09:16.497 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:09:16.498 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:16.500 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.501 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:09:16.502 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:09:16.504 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:09:16.509 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:09:16.517 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:09:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:16.518 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:09:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.519 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:09:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:09:16.525 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:09:16.528 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:09:16.531 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:09:16.531 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:09:16.532 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:09:16.532 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:09:16.536 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:16.536 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.537 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:16.537 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:09:16.538 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.538 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:09:16.540 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:09:16.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:09:16.541 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:09:16.550 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:09:16.550 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:09:16.554 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:09:16.556 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:09:16.561 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:09:16.562 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:09:16.563 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:09:16.565 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:09:16.579 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:16.579 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.580 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:09:16.580 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:09:16.581 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:09:16.586 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:09:16.586 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:09:16.591 INFO 2981023 --- [host-check-host-status1,43c63a8fbcc49c39,4411feacc5fa2d84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:09:16.595 INFO 2981023 --- [host-check-host-status4,43c63a8fbcc49c39,2bcee2aa04aa1303] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:09:16.598 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:09:16.601 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:09:16.611 INFO 2981023 --- [host-check-host-status2,43c63a8fbcc49c39,2af4cc348a7aeebf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:09:16.613 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:09:16.635 INFO 2981023 --- [host-check-host-status3,43c63a8fbcc49c39,89ffc8cfc57f67f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:09:16.636 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:09:16.340086+08:00 2024-09-11 18:09:16.639 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:09:16.643 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:09:16.648 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:16.648 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.649 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:09:16.654 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:09:16.654 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:09:16.669 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:09:16.674 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:16.675 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:16.675 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:09:16.680 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:09:16.680 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:09:16.680 INFO 2981023 --- [ocp-schedules-28,833d8235062ed167,f3460f0752a92981] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:09:16.966 INFO 2981023 --- [host-check-clockdiff1,3560ea2c5b6f5566,6238c4e3a0688dd0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049356 5 5 , error= 2024-09-11 18:09:17.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,89f979e88dddd10f,b5dd1224aa87da1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=89f979e88dddd10f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:17.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,89f979e88dddd10f,b5dd1224aa87da1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=89f979e88dddd10f, duration=9 ms] 2024-09-11 18:09:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,3f75b16fea123cd1,994d436c0b27572a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,3f75b16fea123cd1,994d436c0b27572a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:17.397 INFO 2981023 --- [ocp-async-4,94c129f4ca2b8907,bb711c9841aed0c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:17.782 INFO 2981023 --- [ocp-async-7,16f6453fee61e81b,d58d2da4261466cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:09:17.807 INFO 2981023 --- [host-check-clockdiff2,3560ea2c5b6f5566,1cfe03737d500810] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049357 5 5 , error= 2024-09-11 18:09:17.964 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:17.965 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:19.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,baf4ba98511d4859,81bd395e13dd3a2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=baf4ba98511d4859, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:19.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,baf4ba98511d4859,81bd395e13dd3a2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=baf4ba98511d4859, duration=9 ms] 2024-09-11 18:09:19.397 INFO 2981023 --- [ocp-async-5,9308e4e846d0b23d,76c9ddafacf0f3c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,f78bf5e347523a7a,0f7b9990583ed0d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,f78bf5e347523a7a,0f7b9990583ed0d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:20.965 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:20.966 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:21.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a9b62be43f493af7,0512c5b4bcdc985a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a9b62be43f493af7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:21.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a9b62be43f493af7,0512c5b4bcdc985a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a9b62be43f493af7, duration=9 ms] 2024-09-11 18:09:21.397 INFO 2981023 --- [ocp-async-7,70df5614415e0dcd,7a57a4db09dadf46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:21.901 INFO 2981023 --- [host-check-clockdiff4,3560ea2c5b6f5566,f68afcff3ea60035] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049361 0 0 , error= 2024-09-11 18:09:22.535 INFO 2981023 --- [ocp-schedules-29,c6fda00ec40cd2e2,4c251931697c3448] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:09:23.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,32274dc84328de5e,d563bcd0827982c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=32274dc84328de5e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:23.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,32274dc84328de5e,d563bcd0827982c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=32274dc84328de5e, duration=9 ms] 2024-09-11 18:09:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,58f1e032a0ce19bb,1a69d6e7353828d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,58f1e032a0ce19bb,1a69d6e7353828d6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:23.397 INFO 2981023 --- [ocp-async-9,f7d22c49cd2578c6,34f71de51b1886c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:23.966 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:23.966 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:24.412 INFO 2981023 --- [ocp-schedules-17,9126725835d7bef9,fd203a21cfd90080] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:09:25.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ae124ada237bdc4e,e1ce853fc4abc957] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ae124ada237bdc4e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:25.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ae124ada237bdc4e,e1ce853fc4abc957] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ae124ada237bdc4e, duration=10 ms] 2024-09-11 18:09:25.388 INFO 2981023 --- [ocp-schedules-9,3f7fb4149b6e85d7,3e6e0400ec201a6c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:09:25.400 INFO 2981023 --- [ocp-async-1,c6643fa8d0ca7836,89e248d051920a1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:25.440 WARN 2981023 --- [scanner-13616164-1,3f7fb4149b6e85d7,a39598705ab555c8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:09:25.563 INFO 2981023 --- [ocp-schedules-12,ac02c8df0115a28a,be9c3b43c679d8c7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:09:25.563 INFO 2981023 --- [ocp-schedules-12,ac02c8df0115a28a,be9c3b43c679d8c7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:09:25.563 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:09:25.563 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:09:25.563 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:09:25.601 INFO 2981023 --- [ocp-schedules-12,ac02c8df0115a28a,be9c3b43c679d8c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:09:25.602 INFO 2981023 --- [ocp-schedules-12,ac02c8df0115a28a,be9c3b43c679d8c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:09:25.607 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:09:25.608 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:09:25.608 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:09:25.609 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:09:25.610 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:09:25.610 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:09:25.669 INFO 2981023 --- [ocp-schedules-12,ac02c8df0115a28a,be9c3b43c679d8c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:09:25.672 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:09:25.672 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:09:25.681 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:09:25.683 INFO 2981023 --- [ocp-schedules-12,ac02c8df0115a28a,be9c3b43c679d8c7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:09:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,49536679f1bd57cf,07c2d63145cd969d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,49536679f1bd57cf,07c2d63145cd969d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:09:26.351922+08:00, lastProcessTime=2024-09-11T18:09:16.340086+08:00 2024-09-11 18:09:26.477 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:09:26.351922+08:00 2024-09-11 18:09:26.966 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:26.967 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:27.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,320fda3cab6d20cc,1c879eb92210e792] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=320fda3cab6d20cc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:27.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,320fda3cab6d20cc,1c879eb92210e792] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=320fda3cab6d20cc, duration=9 ms] 2024-09-11 18:09:27.400 INFO 2981023 --- [ocp-async-10,cb8dc90d6d994c3a,865e7360e5b8aad9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:27.782 INFO 2981023 --- [ocp-async-3,0f5b41ac1fb6dd08,9fadb8a8463bc7ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:09:29.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,340909c63aa7222d,9f7f8289e6a079f4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=340909c63aa7222d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:29.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,340909c63aa7222d,9f7f8289e6a079f4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=340909c63aa7222d, duration=9 ms] 2024-09-11 18:09:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,b41a64e24c4090b8,8fb3d0e869d6e9fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,b41a64e24c4090b8,8fb3d0e869d6e9fc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:29.400 INFO 2981023 --- [ocp-async-6,90a3e62e53b4b8d0,bfd36ba08034237e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:29.967 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:29.968 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:31.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,fd8a0a8901f4c26a,4532737efff6cc6c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fd8a0a8901f4c26a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:31.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,fd8a0a8901f4c26a,4532737efff6cc6c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fd8a0a8901f4c26a, duration=12 ms] 2024-09-11 18:09:31.400 INFO 2981023 --- [ocp-async-3,5968f0c726e895c2,55bb3c74f5293b4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,2fbdeff7c1f786e5,99331d9ab6da5924] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,2fbdeff7c1f786e5,99331d9ab6da5924] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:32.538 INFO 2981023 --- [ocp-schedules-21,e7e66c7f602a6711,44a06129d3893d6d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:09:32.968 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:32.969 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:33.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7b41ad07915a3ab3,1819e32bc65fefd3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7b41ad07915a3ab3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:33.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7b41ad07915a3ab3,1819e32bc65fefd3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7b41ad07915a3ab3, duration=9 ms] 2024-09-11 18:09:33.400 INFO 2981023 --- [ocp-async-8,e05056092cb32ad2,3701062713fa78d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:35.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c54007dc964d2eeb,89523eb7354e16b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c54007dc964d2eeb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:35.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c54007dc964d2eeb,89523eb7354e16b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c54007dc964d2eeb, duration=9 ms] 2024-09-11 18:09:35.211 INFO 2981023 --- [ocp-schedules-23,fc976d5294b87000,8b41791c9263e566] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:09:35.221 INFO 2981023 --- [ocp-schedules-23,fc976d5294b87000,8b41791c9263e566] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:09:35.221 INFO 2981023 --- [ocp-schedules-23,fc976d5294b87000,8b41791c9263e566] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:09:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,8b707e8a9b9e34cd,95f9ce0485ab47af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,8b707e8a9b9e34cd,95f9ce0485ab47af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:35.400 INFO 2981023 --- [ocp-async-9,a852f9d2e7d66495,b22a802126459bc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:35.657 INFO 2981023 --- [ocp-schedules-25,2f5e82b8b88832ab,ddad44556c6d6b57] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:09:35.701 WARN 2981023 --- [scanner-13616364-1,2f5e82b8b88832ab,c5f0ea7f99343bb5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:09:35.720 INFO 2981023 --- [alarm-detect-11,2f5e82b8b88832ab,549bbc0753e16c5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645575, data is empty. 2024-09-11 18:09:35.724 INFO 2981023 --- [alarm-detect-11,2f5e82b8b88832ab,549bbc0753e16c5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645575, data is empty. 2024-09-11 18:09:35.728 INFO 2981023 --- [alarm-detect-11,2f5e82b8b88832ab,549bbc0753e16c5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645575, data is empty. 2024-09-11 18:09:35.743 INFO 2981023 --- [alarm-detect-13,2f5e82b8b88832ab,9dc5fecfd72b52c0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645575, data is empty. 2024-09-11 18:09:35.749 INFO 2981023 --- [alarm-detect-13,2f5e82b8b88832ab,9dc5fecfd72b52c0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645575, data is empty. 2024-09-11 18:09:35.750 INFO 2981023 --- [alarm-detect-12,2f5e82b8b88832ab,274942bdff842fe8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645575, data is empty. 2024-09-11 18:09:35.755 INFO 2981023 --- [alarm-detect-12,2f5e82b8b88832ab,274942bdff842fe8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645575, data is empty. 2024-09-11 18:09:35.755 INFO 2981023 --- [alarm-detect-13,2f5e82b8b88832ab,9dc5fecfd72b52c0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645575, data is empty. 2024-09-11 18:09:35.759 INFO 2981023 --- [alarm-detect-12,2f5e82b8b88832ab,274942bdff842fe8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645575, data is empty. 2024-09-11 18:09:35.873 INFO 2981023 --- [alarm-detect-17,2f5e82b8b88832ab,85304cbd50c38fd5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645575, data is empty. 2024-09-11 18:09:35.879 INFO 2981023 --- [alarm-detect-17,2f5e82b8b88832ab,85304cbd50c38fd5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645575, data is empty. 2024-09-11 18:09:35.884 INFO 2981023 --- [alarm-detect-17,2f5e82b8b88832ab,85304cbd50c38fd5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645575, data is empty. 2024-09-11 18:09:35.969 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:35.969 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:09:36.337070+08:00, lastProcessTime=2024-09-11T18:09:26.351922+08:00 2024-09-11 18:09:36.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:09:36.337070+08:00 2024-09-11 18:09:37.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,cf8631a399f5cca7,62e701fc84e4ce0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cf8631a399f5cca7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:37.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,cf8631a399f5cca7,62e701fc84e4ce0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cf8631a399f5cca7, duration=11 ms] 2024-09-11 18:09:37.314 INFO 2981023 --- [prometheus-cache-monitor1,f82c801dedb1336f,190cd435a6853405] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5149} 2024-09-11 18:09:37.315 INFO 2981023 --- [prometheus-cache-monitor1,f82c801dedb1336f,190cd435a6853405] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:09:37.400 INFO 2981023 --- [ocp-async-5,dde3b4303bc58304,cde7ff378f62fc41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:37.783 INFO 2981023 --- [ocp-async-2,99c3d7b3ca9a1ded,baa304a846048e1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:09:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,58ab500205c83b54,79b1c690b36afa03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,58ab500205c83b54,79b1c690b36afa03] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:38.969 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:38.970 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:39.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7f988bc03e1fd9cc,c8ad7c21566ec131] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=7f988bc03e1fd9cc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:39.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,7f988bc03e1fd9cc,c8ad7c21566ec131] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=7f988bc03e1fd9cc, duration=9 ms] 2024-09-11 18:09:39.400 INFO 2981023 --- [ocp-async-10,1354e9368e74896a,0dab8cd73e65a86a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:40.003 INFO 2981023 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:09:40.003 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:09:40.006 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:09:40.006 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:09:40.009 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:09:40.009 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:09:40.009 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:09:40.009 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:09:40.011 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:09:40.011 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:09:40.011 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:09:40.011 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:09:40.033 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:09:40.038 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:09:40.038 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:09:40.038 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:09:40.038 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:09:40.038 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:09:40.043 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:09:40.290 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:09:40.543 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.531394295S 2024-09-11 18:09:40.543 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:09:40.610 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:09:40.610 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:09:40.626 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:09:40.662 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:09:40.662 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:09:40.662 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:09:40.662 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:09:40.662 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:09:40.663 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:09:40.668 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:09:40.747 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] o.s.boot.SpringApplication : Started application in 0.738 seconds (JVM running for 85180.62) 2024-09-11 18:09:40.747 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.738715185S 2024-09-11 18:09:40.950 INFO 2981023 --- [ocp-schedules-24,86e4fb5ae6155900,e18d40dbebe83362] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:09:41.096 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,781310a701e0e699,3cce8041af47d22f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=781310a701e0e699, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:41.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,781310a701e0e699,3cce8041af47d22f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=781310a701e0e699, duration=11 ms] 2024-09-11 18:09:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,706cd3de23583873,6641504126ae8c46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,706cd3de23583873,6641504126ae8c46] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:41.400 INFO 2981023 --- [ocp-async-2,918392dc2e02ae0f,3d0c3d8a523ca55f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:41.970 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:41.971 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:42.541 INFO 2981023 --- [ocp-schedules-2,e8f6d8db7b6fbe72,ee7abdc94b40f8e8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:09:43.400 INFO 2981023 --- [ocp-async-4,49342666174f472a,6f273db1257d6726] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:44.035 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,74316885894fd114,02a2393dadf6bfb2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=74316885894fd114, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:44.042 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,74316885894fd114,02a2393dadf6bfb2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=74316885894fd114, duration=10 ms] 2024-09-11 18:09:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,6671126c92a1295f,e571fff214ef296f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,6671126c92a1295f,e571fff214ef296f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:44.590 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,894eb368b05b406f,bb4e7271a543bdd8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=894eb368b05b406f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:09:44.611 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,894eb368b05b406f,bb4e7271a543bdd8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=894eb368b05b406f, duration=23 ms] 2024-09-11 18:09:44.943 INFO 2981023 --- [ocp-schedules-13,2d1aa240c935c66c,8e15e0dee99deded] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:09:44.963 WARN 2981023 --- [ocp-schedules-13,2d1aa240c935c66c,8e15e0dee99deded] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:09:44.971 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:44.972 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:45.101 INFO 2981023 --- [prometheus-second-schedule-1,39d38ee937b37bbb,30bb8485237b31d9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:09:45.401 INFO 2981023 --- [ocp-async-1,27abe949a1f6adb6,873d34200bf45275] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:45.918 INFO 2981023 --- [ocp-schedules-5,c65143f6f0ae6170,14933e754304bb73] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:09:45.966 WARN 2981023 --- [scanner-13616706-1,c65143f6f0ae6170,019d4e98fc8ac646] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:09:46.029 INFO 2981023 --- [ocp-schedules-8,cdb2e5e851fde662,4423ef6e0e5e61b4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:09:46.030 INFO 2981023 --- [ocp-schedules-10,b590a4934dcd50e7,d9e0e8e15d888bab] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:09:46.034 INFO 2981023 --- [ocp-schedules-11,6c159d77961bdc4f,1916fb6514f81bf0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:09:46.053 INFO 2981023 --- [host-check-clockdiff1,6c159d77961bdc4f,b8598d88d9613c09] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:09:46.053 INFO 2981023 --- [host-check-clockdiff4,6c159d77961bdc4f,2f99e7d4c4d6f995] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:09:46.053 INFO 2981023 --- [host-check-clockdiff3,6c159d77961bdc4f,6a85090e4f2b3b24] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:09:46.055 INFO 2981023 --- [host-check-clockdiff2,6c159d77961bdc4f,e47532143563986c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:09:46.059 INFO 2981023 --- [host-check-agent-status1,b590a4934dcd50e7,9dc160559401172b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:09:46.065 INFO 2981023 --- [host-check-agent-status1,b590a4934dcd50e7,9dc160559401172b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:09:46.067 INFO 2981023 --- [host-check-agent-status2,b590a4934dcd50e7,c59c0b323f848d56] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:09:46.067 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:09:46.067 INFO 2981023 --- [host-check-agent-status1,b590a4934dcd50e7,9dc160559401172b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:09:46.067 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:09:46.068 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:09:46.069 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:09:46.074 INFO 2981023 --- [host-check-agent-status2,b590a4934dcd50e7,c59c0b323f848d56] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:09:46.075 INFO 2981023 --- [host-check-agent-status2,b590a4934dcd50e7,c59c0b323f848d56] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:09:46.076 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:09:46.076 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:09:46.078 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:09:46.079 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:09:46.080 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:09:46.086 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:09:46.086 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:09:46.087 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:09:46.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,deaa059be8793412,fd43429c2ea060b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=deaa059be8793412, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:46.111 INFO 2981023 --- [host-check-agent-status1,b590a4934dcd50e7,9dc160559401172b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:09:46.115 INFO 2981023 --- [host-check-agent-status1,b590a4934dcd50e7,9dc160559401172b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:09:46.115 INFO 2981023 --- [host-check-agent-status1,b590a4934dcd50e7,9dc160559401172b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:09:46.115 INFO 2981023 --- [host-check-agent-status1,b590a4934dcd50e7,9dc160559401172b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:09:46.115 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:09:46.116 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:09:46.121630329+08:00 2024-09-11 18:09:46.116 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:09:46.119 INFO 2981023 --- [host-check-agent-status2,b590a4934dcd50e7,c59c0b323f848d56] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:09:46.120 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:09:46.116+08:00 2024-09-11 18:09:46.121 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:09:46.118082888+08:00 2024-09-11 18:09:46.121 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:09:46.121+08:00 2024-09-11 18:09:46.122 INFO 2981023 --- [host-check-agent-status2,b590a4934dcd50e7,c59c0b323f848d56] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:09:46.122 INFO 2981023 --- [host-check-agent-status2,b590a4934dcd50e7,c59c0b323f848d56] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:09:46.122 INFO 2981023 --- [host-check-agent-status2,b590a4934dcd50e7,c59c0b323f848d56] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:09:46.126 INFO 2981023 --- [host-check-agent-status3,b590a4934dcd50e7,cc26ca7a4d560d09] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:09:46.126 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:09:46.121+08:00 and status AVAILABLE 2024-09-11 18:09:46.126 INFO 2981023 --- [host-check-agent-status4,b590a4934dcd50e7,d35c08de92503a0e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:09:46.127 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:09:46.128 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:09:46.116+08:00 and status AVAILABLE 2024-09-11 18:09:46.134 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,deaa059be8793412,fd43429c2ea060b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=deaa059be8793412, duration=32 ms] 2024-09-11 18:09:46.136 INFO 2981023 --- [host-check-agent-status3,b590a4934dcd50e7,cc26ca7a4d560d09] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:09:46.138 INFO 2981023 --- [host-check-agent-status3,b590a4934dcd50e7,cc26ca7a4d560d09] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:09:46.139 INFO 2981023 --- [host-check-agent-status4,b590a4934dcd50e7,d35c08de92503a0e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:09:46.139 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:09:46.140 INFO 2981023 --- [host-check-agent-status4,b590a4934dcd50e7,d35c08de92503a0e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:09:46.141 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:09:46.143 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:09:46.142442520+08:00 2024-09-11 18:09:46.148 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:09:46.149 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:09:46.143+08:00 2024-09-11 18:09:46.152 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:09:46.143+08:00 and status AVAILABLE 2024-09-11 18:09:46.159 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:09:46.164 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:09:46.166 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:09:46.171 INFO 2981023 --- [host-check-agent-status3,b590a4934dcd50e7,cc26ca7a4d560d09] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:09:46.173 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:09:46.174 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:09:46.175 INFO 2981023 --- [ocp-schedules-9,354a5d60a6ac15b8,d9c53954755a6a50] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:09:46.177 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:09:46.183 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:09:46.185 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:09:46.187 INFO 2981023 --- [ocp-schedules-25,7d219037dcf2c345,f1b59051c7afdf2f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:09:46.187 INFO 2981023 --- [ocp-schedules-12,0a99e28785c9dd8c,3b85ce12dcdd697d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:09:46.187 INFO 2981023 --- [ocp-schedules-24,2caf1c356422dc73,78e0f080ad4e2a4e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:09:46.188 INFO 2981023 --- [ocp-schedules-19,35a72c2f710ff609,6ebe9c840f2ab76c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:09:46.190 INFO 2981023 --- [ocp-schedules-9,354a5d60a6ac15b8,d9c53954755a6a50] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:09:46.191 INFO 2981023 --- [ocp-schedules-24,2caf1c356422dc73,78e0f080ad4e2a4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:09:46.193 INFO 2981023 --- [ocp-schedules-25,7d219037dcf2c345,f1b59051c7afdf2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:09:46.194 INFO 2981023 --- [host-check-agent-status3,b590a4934dcd50e7,cc26ca7a4d560d09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:09:46.194 INFO 2981023 --- [host-check-agent-status3,b590a4934dcd50e7,cc26ca7a4d560d09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:09:46.194 INFO 2981023 --- [host-check-agent-status3,b590a4934dcd50e7,cc26ca7a4d560d09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:09:46.200 INFO 2981023 --- [ocp-schedules-12,0a99e28785c9dd8c,3b85ce12dcdd697d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:09:46.202 INFO 2981023 --- [ocp-schedules-12,0a99e28785c9dd8c,3b85ce12dcdd697d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:09:46.207 INFO 2981023 --- [host-check-clockdiff1,6c159d77961bdc4f,b8598d88d9613c09] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049386 6 6 , error= 2024-09-11 18:09:46.208 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:09:46.212975322+08:00 2024-09-11 18:09:46.215 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:09:46.208+08:00 2024-09-11 18:09:46.218 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:09:46.208+08:00 and status AVAILABLE 2024-09-11 18:09:46.218 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:09:46.219 INFO 2981023 --- [host-check-agent-status4,b590a4934dcd50e7,d35c08de92503a0e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:09:46.228 INFO 2981023 --- [host-check-agent-status4,b590a4934dcd50e7,d35c08de92503a0e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:09:46.228 INFO 2981023 --- [host-check-agent-status4,b590a4934dcd50e7,d35c08de92503a0e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:09:46.228 INFO 2981023 --- [host-check-agent-status4,b590a4934dcd50e7,d35c08de92503a0e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:09:46.237 INFO 2981023 --- [ocp-schedules-13,a351603630c4c793,c389acfdb2d53260] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:09:46.249 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:09:46.278 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.304 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:09:46.319 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:09:46.320 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:09:46.335 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.336 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:09:46.337 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:09:46.343 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:09:46.345 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:09:46.347 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:09:46.345921+08:00, lastProcessTime=2024-09-11T18:09:36.337070+08:00 2024-09-11 18:09:46.365 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.365 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:09:46.366 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:46.372 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.372 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.373 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:46.373 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:46.386 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:09:46.395 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,835a75b65ae2927c,859079ff49ffadab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=835a75b65ae2927c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:09:46.417 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:09:46.418 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.419 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:09:46.421 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:09:46.421 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.421 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:09:46.424 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:09:46.436 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:09:46.442 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.446 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.446 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:09:46.450 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:09:46.476 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:09:46.479 INFO 2981023 --- [host-check-clockdiff3,6c159d77961bdc4f,6a85090e4f2b3b24] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049386 0 0 , error= 2024-09-11 18:09:46.481 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049282825000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:09:46.491 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.491 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:09:46.492 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:09:46.493 INFO 2981023 --- [ocp-schedules-25,7d219037dcf2c345,f1b59051c7afdf2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:09:46.504 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:09:46.506 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:09:46.506 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049282825000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:09:46.506 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,835a75b65ae2927c,859079ff49ffadab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=835a75b65ae2927c, duration=126 ms] 2024-09-11 18:09:46.510 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:09:46.511 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.513 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.514 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:46.514 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.515 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:09:46.515 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:09:46.516 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:09:46.516 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.519 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.522 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:09:46.522 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:09:46.523 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:09:46.523 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:09:46.523 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:09:46.531 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:09:46.541 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:09:46.544 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:09:46.544 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:09:46.551 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:09:46.553 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:09:46.554 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:09:46.559 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:09:46.566 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:09:46.567 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:09:46.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:09:46.575 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:09:46.580 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.581 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.582 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:09:46.582 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:09:46.586 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:09:46.588 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:09:46.589 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:09:46.594 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:09:46.596 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:09:46.597 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:09:46.601 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.606 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:09:46.606 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:09:46.610 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:09:46.615 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:09:46.617 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:09:46.620 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.621 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.621 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:09:46.622 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:09:46.625 INFO 2981023 --- [host-check-host-status4,cdb2e5e851fde662,960305f1b3a89d6b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:09:46.626 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.626 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:09:46.627 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:09:46.627 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.633 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:09:46.634 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:09:46.639 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:09:46.646 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:09:46.658 INFO 2981023 --- [host-check-host-status1,cdb2e5e851fde662,5f14767992eb36c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:09:46.659 INFO 2981023 --- [host-check-host-status3,cdb2e5e851fde662,0af99db3fc2ace70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:09:46.659 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:09:46.661 INFO 2981023 --- [host-check-host-status2,cdb2e5e851fde662,1553ddb9346ac21a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:09:46.666 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:09:46.692 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.692 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.693 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:09:46.693 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:09:46.694 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.699 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.699 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.699 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:09:46.700 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:09:46.700 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.702 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:09:46.705 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:09:46.706 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:09:46.709 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:09:46.711 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.716 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.716 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:09:46.716 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:09:46.716 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:09:46.723 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:09:46.723 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048789295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:09:46.726 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:09:46.732 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:09:46.739 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.740 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.740 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:09:46.741 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:09:46.742 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.743 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.743 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:09:46.744 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:09:46.744 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.744 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:09:46.746 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:09:46.747 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:09:46.765 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:09:46.776 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:09:46.345921+08:00 2024-09-11 18:09:46.790 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:09:46.793 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:09:46.798 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:46.798 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:46.799 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:09:46.804 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:09:46.804 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726048812659000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:09:46.804 INFO 2981023 --- [ocp-schedules-24,2caf1c356422dc73,78e0f080ad4e2a4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:09:46.808 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:09:46.813 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:09:46.816 INFO 2981023 --- [ocp-schedules-1,564af10e86caa964,fe21233385fc4364] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:09:46.844 INFO 2981023 --- [ocp-schedules-17,d45d6010b3208034,8a2bb2d87102bc71] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5149} 2024-09-11 18:09:46.844 INFO 2981023 --- [ocp-schedules-17,d45d6010b3208034,8a2bb2d87102bc71] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:09:47.004 INFO 2981023 --- [host-check-clockdiff2,6c159d77961bdc4f,e47532143563986c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049387 5 5 , error= 2024-09-11 18:09:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,21c5f1b4aa3f5792,29a7e411ef1b9889] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,21c5f1b4aa3f5792,29a7e411ef1b9889] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:47.401 INFO 2981023 --- [ocp-async-2,b1b8e1f21f2fda0e,36f64f155a3754da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:47.787 INFO 2981023 --- [ocp-async-7,2f503083e904c3e2,c3055626319e534a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:09:47.950 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,e831b3efba6235c7,5862bebb212bece7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=e831b3efba6235c7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:09:47.962 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,e831b3efba6235c7,5862bebb212bece7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=e831b3efba6235c7, duration=15 ms] 2024-09-11 18:09:47.972 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:47.972 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:49.295 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,39a67d60bffd7685,e3257656281fbd39] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&version=2&ObClusterId=1719994437, client=10.31.35.10, traceId=39a67d60bffd7685, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkprdocp,1719994437,2,{"ObRegion":"linkprdocp","ObCluster":"linkprdocp","ObRegionId":1719994437,"ObClusterId":1719994437,"Type":"PRIMARY","timestamp":1726049389294288,"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:09:49.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,39a67d60bffd7685,e3257656281fbd39] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkprdocp, obClusterId=1719994437, rootServerJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, groupName=linkprdocp, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:09:49.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,39a67d60bffd7685,e3257656281fbd39] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:09:49.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,39a67d60bffd7685,e3257656281fbd39] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkprdocp, clusterId=1719994437, affectRows=1 2024-09-11 18:09:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,39a67d60bffd7685,e3257656281fbd39] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.10, traceId=39a67d60bffd7685, duration=9 ms] 2024-09-11 18:09:49.401 INFO 2981023 --- [ocp-async-3,7a0e8169bd93387e,2430c299aa9ed82a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:49.591 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1831a692c0b09304,6ff4d6335657eafa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=1831a692c0b09304, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:09:49.598 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1831a692c0b09304,6ff4d6335657eafa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=1831a692c0b09304, duration=11 ms] 2024-09-11 18:09:49.617 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c7d4438844feca46,59fce1ed7520ffb0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c7d4438844feca46, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:49.635 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c7d4438844feca46,59fce1ed7520ffb0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c7d4438844feca46, duration=21 ms] 2024-09-11 18:09:49.839 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,157e55aac2bfc4a2,91ca744846090620] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=157e55aac2bfc4a2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:09:49.860 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,157e55aac2bfc4a2,91ca744846090620] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=157e55aac2bfc4a2, duration=24 ms] 2024-09-11 18:09:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:09:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:09:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,51fc96af2f1b982a,9097a9679017502d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,51fc96af2f1b982a,9097a9679017502d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:50.326 INFO 2981023 --- [ocp-schedules-1,44500473d3133fb7,89013abb240a8737] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:09:50.972 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:50.974 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:51.401 INFO 2981023 --- [ocp-async-7,fc8049df580cf392,93c72e893c478eb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:51.475 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,fb2cf9bebc52b50e,2081ddb4c76a3f51] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=fb2cf9bebc52b50e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:09:51.484 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,fb2cf9bebc52b50e,2081ddb4c76a3f51] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=fb2cf9bebc52b50e, duration=12 ms] 2024-09-11 18:09:51.607 INFO 2981023 --- [ocp-schedules-15,c59cf3d9ff40be2e,4626e491e6f7d1c4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:09:51.608 INFO 2981023 --- [ocp-schedules-15,c59cf3d9ff40be2e,4626e491e6f7d1c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:09:51.608 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:09:51.608 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:09:51.608 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:09:51.608 INFO 2981023 --- [ocp-schedules-15,c59cf3d9ff40be2e,4626e491e6f7d1c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:09:51.640 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,fe6b41c1d37c0af1,e60606acce7e8ce2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=fe6b41c1d37c0af1, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:09:51.643 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,fe6b41c1d37c0af1,e60606acce7e8ce2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=fe6b41c1d37c0af1, duration=5 ms] 2024-09-11 18:09:52.010 INFO 2981023 --- [host-check-clockdiff4,6c159d77961bdc4f,2f99e7d4c4d6f995] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049392 0 0 , error= 2024-09-11 18:09:52.083 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,5711351f00e4494b,c11fa2ebab116f38] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5711351f00e4494b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:52.089 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,5711351f00e4494b,c11fa2ebab116f38] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5711351f00e4494b, duration=9 ms] 2024-09-11 18:09:52.543 INFO 2981023 --- [ocp-schedules-16,43c9910178980b0e,9cd9d57f0e8b2296] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:09:52.961 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,4bf01eea9b93185b,ae680cee6a8a3ced] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=4bf01eea9b93185b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:09:52.971 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,4bf01eea9b93185b,ae680cee6a8a3ced] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=4bf01eea9b93185b, duration=13 ms] 2024-09-11 18:09:53.130 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b7134aa550d26d3e,6f7599af069a342a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=b7134aa550d26d3e, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:09:53.132 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b7134aa550d26d3e,6f7599af069a342a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=b7134aa550d26d3e, duration=5 ms] 2024-09-11 18:09:53.284 INFO 2981023 --- [ocp-schedules-29,8fca0ae27231b032,43000f8a87ff712e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:09:53.284 INFO 2981023 --- [ocp-schedules-29,8fca0ae27231b032,43000f8a87ff712e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:09:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,c17624baf3130ed6,2d4bf3c585ada9e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,c17624baf3130ed6,2d4bf3c585ada9e4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:53.401 INFO 2981023 --- [ocp-async-6,c0691e2cf1b55ba9,3e03c585d853fedb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:53.974 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:53.976 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:54.095 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,84ffe0bbedaf138e,69f47b40a9dcd37d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=84ffe0bbedaf138e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:54.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,84ffe0bbedaf138e,69f47b40a9dcd37d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=84ffe0bbedaf138e, duration=21 ms] 2024-09-11 18:09:54.415 INFO 2981023 --- [ocp-schedules-20,979d40cca459570d,aae07c7068b7b2f1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:09:54.546 INFO 2981023 --- [ocp-schedules-24,5255297987ef367a,3afe3e9a94f3da9d] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:09:54.565 WARN 2981023 --- [ocp-schedules-24,5255297987ef367a,3afe3e9a94f3da9d] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:09:55.018 INFO 2981023 --- [ocp-schedules-17,9c41d4491b84d927,8831ee43f7d65d7d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:09:55.018 INFO 2981023 --- [ocp-schedules-17,9c41d4491b84d927,8831ee43f7d65d7d] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:09:55.023 INFO 2981023 --- [ocp-schedules-17,9c41d4491b84d927,8831ee43f7d65d7d] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:09:55.023 INFO 2981023 --- [ocp-schedules-17,9c41d4491b84d927,8831ee43f7d65d7d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:09:55.023 INFO 2981023 --- [ocp-schedules-17,9c41d4491b84d927,8831ee43f7d65d7d] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:09:55.033 INFO 2981023 --- [ocp-schedules-17,9c41d4491b84d927,8831ee43f7d65d7d] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:09:55.033 INFO 2981023 --- [ocp-schedules-17,9c41d4491b84d927,8831ee43f7d65d7d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:09:55.033 INFO 2981023 --- [ocp-schedules-17,9c41d4491b84d927,8831ee43f7d65d7d] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:09:55.044 INFO 2981023 --- [ocp-schedules-17,9c41d4491b84d927,8831ee43f7d65d7d] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:09:55.224 INFO 2981023 --- [ocp-schedules-12,06b99b3998ead106,c3d8a08f762ef993] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:09:55.232 INFO 2981023 --- [ocp-schedules-12,06b99b3998ead106,c3d8a08f762ef993] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:09:55.232 INFO 2981023 --- [ocp-schedules-12,06b99b3998ead106,c3d8a08f762ef993] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:09:55.340 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,39b28c44180135ad,4ab6793bcb05a13f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=39b28c44180135ad, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:09:55.360 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,39b28c44180135ad,4ab6793bcb05a13f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=39b28c44180135ad, duration=23 ms] 2024-09-11 18:09:55.401 INFO 2981023 --- [ocp-async-4,d9f9144054f15af9,30fb3a0bd221e239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:55.854 INFO 2981023 --- [ocp-schedules-2,a7204be742b6efe2,afaba2c473e27546] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:09:55.854 INFO 2981023 --- [ocp-schedules-2,a7204be742b6efe2,afaba2c473e27546] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:09:55.856 INFO 2981023 --- [ocp-schedules-2,a7204be742b6efe2,afaba2c473e27546] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:09:55.856 INFO 2981023 --- [ocp-schedules-2,a7204be742b6efe2,afaba2c473e27546] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:09:55.858 INFO 2981023 --- [ocp-schedules-2,a7204be742b6efe2,afaba2c473e27546] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:09:55.859 INFO 2981023 --- [ocp-schedules-2,a7204be742b6efe2,afaba2c473e27546] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:09:56.025 INFO 2981023 --- [ocp-schedules-13,d21dd90fd1b7d3c4,890a70ff09c9ebcf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:09:56.031 INFO 2981023 --- [ocp-schedules-13,d21dd90fd1b7d3c4,890a70ff09c9ebcf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:09:56.031 INFO 2981023 --- [ocp-schedules-13,d21dd90fd1b7d3c4,890a70ff09c9ebcf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:09:56.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,777aedc8a8a60231,e79f5c53ca4fce5d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=777aedc8a8a60231, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:09:56.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,777aedc8a8a60231,e79f5c53ca4fce5d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=777aedc8a8a60231, duration=9 ms] 2024-09-11 18:09:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,e3d58e6e1e585979,f164b9807dcc5daa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,e3d58e6e1e585979,f164b9807dcc5daa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:09:56.351771+08:00, lastProcessTime=2024-09-11T18:09:46.345921+08:00 2024-09-11 18:09:56.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:09:56.351771+08:00 2024-09-11 18:09:56.510 INFO 2981023 --- [ocp-schedules-8,ef222d19edbfa056,34bcf09a1cb31ec9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:09:56.549 WARN 2981023 --- [scanner-13616891-1,ef222d19edbfa056,cd06d2ee58bf735b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:09:56.610 INFO 2981023 --- [alarm-detect-9,ef222d19edbfa056,75f664db8491836d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645575, data is empty. 2024-09-11 18:09:56.614 INFO 2981023 --- [alarm-detect-9,ef222d19edbfa056,75f664db8491836d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645575, data is empty. 2024-09-11 18:09:56.623 INFO 2981023 --- [alarm-detect-9,ef222d19edbfa056,75f664db8491836d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645575, data is empty. 2024-09-11 18:09:56.644 INFO 2981023 --- [alarm-detect-2,ef222d19edbfa056,856d612bfa0cf563] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645575, data is empty. 2024-09-11 18:09:56.645 INFO 2981023 --- [alarm-detect-20,ef222d19edbfa056,9ff8223027ed75bc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645575, data is empty. 2024-09-11 18:09:56.647 INFO 2981023 --- [alarm-detect-2,ef222d19edbfa056,856d612bfa0cf563] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645575, data is empty. 2024-09-11 18:09:56.650 INFO 2981023 --- [alarm-detect-20,ef222d19edbfa056,9ff8223027ed75bc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645575, data is empty. 2024-09-11 18:09:56.653 INFO 2981023 --- [alarm-detect-2,ef222d19edbfa056,856d612bfa0cf563] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645575, data is empty. 2024-09-11 18:09:56.655 INFO 2981023 --- [alarm-detect-20,ef222d19edbfa056,9ff8223027ed75bc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645575, data is empty. 2024-09-11 18:09:56.795 INFO 2981023 --- [alarm-detect-13,ef222d19edbfa056,9d5839415bbbbe9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645575, data is empty. 2024-09-11 18:09:56.799 INFO 2981023 --- [alarm-detect-13,ef222d19edbfa056,9d5839415bbbbe9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645575, data is empty. 2024-09-11 18:09:56.808 INFO 2981023 --- [alarm-detect-13,ef222d19edbfa056,9d5839415bbbbe9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645575, data is empty. 2024-09-11 18:09:56.977 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:56.977 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:09:57.370 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d3220c869fb55d91,80fe7bd26210a675] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=d3220c869fb55d91, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:09:57.380 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d3220c869fb55d91,80fe7bd26210a675] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=d3220c869fb55d91, duration=12 ms] 2024-09-11 18:09:57.401 INFO 2981023 --- [ocp-async-8,b2e3681a7cdc8611,21a4b4aee48c3347] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:57.735 INFO 2981023 --- [ocp-schedules-11,ea43ba7209f94249,185197c9a19d60a1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:09:57.735 INFO 2981023 --- [ocp-schedules-11,ea43ba7209f94249,185197c9a19d60a1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:09:57.766 INFO 2981023 --- [Collect all tenants major compactions2,ea43ba7209f94249,ad30a4c70f9e096b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:09:57.767 INFO 2981023 --- [Collect all tenants major compactions1,ea43ba7209f94249,7ab93b3a1d974241] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:09:57.767 INFO 2981023 --- [Collect all tenants major compactions4,ea43ba7209f94249,136452f0c0f8017d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:09:57.767 INFO 2981023 --- [Collect all tenants major compactions3,ea43ba7209f94249,d3256d7b36e118cb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:09:57.776 INFO 2981023 --- [Collect all tenants major compactions6,ea43ba7209f94249,a073cf887463acde] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:09:57.776 INFO 2981023 --- [Collect all tenants major compactions5,ea43ba7209f94249,c0132213bb7b347a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:09:57.783 INFO 2981023 --- [ocp-async-10,3575c72b8219d940,670c360f6384d7f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:09:57.795 INFO 2981023 --- [Collect all tenants major compactions3,ea43ba7209f94249,d3256d7b36e118cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:57.796 INFO 2981023 --- [Collect all tenants major compactions3,ea43ba7209f94249,d3256d7b36e118cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:09:57.798 INFO 2981023 --- [Collect all tenants major compactions1,ea43ba7209f94249,7ab93b3a1d974241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:57.799 INFO 2981023 --- [Collect all tenants major compactions4,ea43ba7209f94249,136452f0c0f8017d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:57.800 INFO 2981023 --- [Collect all tenants major compactions3,ea43ba7209f94249,d3256d7b36e118cb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:09:57.800 INFO 2981023 --- [Collect all tenants major compactions6,ea43ba7209f94249,a073cf887463acde] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:57.800 INFO 2981023 --- [Collect all tenants major compactions4,ea43ba7209f94249,136452f0c0f8017d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:09:57.801 INFO 2981023 --- [Collect all tenants major compactions2,ea43ba7209f94249,ad30a4c70f9e096b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:57.801 INFO 2981023 --- [Collect all tenants major compactions1,ea43ba7209f94249,7ab93b3a1d974241] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:09:57.802 INFO 2981023 --- [Collect all tenants major compactions5,ea43ba7209f94249,c0132213bb7b347a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:57.802 INFO 2981023 --- [Collect all tenants major compactions4,ea43ba7209f94249,136452f0c0f8017d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:09:57.802 INFO 2981023 --- [Collect all tenants major compactions2,ea43ba7209f94249,ad30a4c70f9e096b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:09:57.803 INFO 2981023 --- [Collect all tenants major compactions3,ea43ba7209f94249,d3256d7b36e118cb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:09:57.803 INFO 2981023 --- [Collect all tenants major compactions6,ea43ba7209f94249,a073cf887463acde] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:09:57.804 INFO 2981023 --- [Collect all tenants major compactions5,ea43ba7209f94249,c0132213bb7b347a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:09:57.805 INFO 2981023 --- [Collect all tenants major compactions4,ea43ba7209f94249,136452f0c0f8017d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:09:57.806 INFO 2981023 --- [Collect all tenants major compactions6,ea43ba7209f94249,a073cf887463acde] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:09:57.806 INFO 2981023 --- [Collect all tenants major compactions6,ea43ba7209f94249,a073cf887463acde] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:09:57.806 INFO 2981023 --- [Collect all tenants major compactions2,ea43ba7209f94249,ad30a4c70f9e096b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:09:57.806 INFO 2981023 --- [Collect all tenants major compactions2,ea43ba7209f94249,ad30a4c70f9e096b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:09:57.806 INFO 2981023 --- [Collect all tenants major compactions5,ea43ba7209f94249,c0132213bb7b347a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:09:57.806 INFO 2981023 --- [Collect all tenants major compactions5,ea43ba7209f94249,c0132213bb7b347a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:09:57.807 INFO 2981023 --- [Collect all tenants major compactions1,ea43ba7209f94249,7ab93b3a1d974241] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:09:57.807 INFO 2981023 --- [Collect all tenants major compactions1,ea43ba7209f94249,7ab93b3a1d974241] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:09:57.808 INFO 2981023 --- [ocp-schedules-11,ea43ba7209f94249,185197c9a19d60a1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:09:57.979 INFO 2981023 --- [ocp-schedules-28,46c8e4661d024a3c,9ddd1c9146eed739] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:09:57.979+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:09:57.982 INFO 2981023 --- [ocp-schedules-28,46c8e4661d024a3c,9ddd1c9146eed739] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:09:58.115 INFO 2981023 --- [ocp-schedules-28,46c8e4661d024a3c,9ddd1c9146eed739] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 41 to SqlAggregateData 2024-09-11 18:09:58.117 INFO 2981023 --- [ocp-schedules-28,46c8e4661d024a3c,9ddd1c9146eed739] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:09:58.924 INFO 2981023 --- [ocp-schedules-21,51378c60fa761da3,b3e6bae49073e94d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:09:58.925 INFO 2981023 --- [ocp-schedules-21,51378c60fa761da3,b3e6bae49073e94d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:09:58.956 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:09:58.957 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.957 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:09:58.957 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:09:58.958 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.958 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.960 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:09:58.961 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:09:58.961 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:09:58.966 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.966 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:58.968 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.969 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:58.969 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.969 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:58.971 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.972 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:58.979 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.979 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:09:58.980 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.981 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:58.983 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.983 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:58.986 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.987 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:09:58.992 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.993 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:09:58.995 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.995 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:58.995 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:09:58.996 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4EA1D6DB-0-0] 2024-09-11 18:09:58.996 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:09:58.996 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.996 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:09:58.997 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.997 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:58.998 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:09:58.998 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:59.008 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.009 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:09:59.010 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.010 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:09:59.011 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.012 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:09:59.012 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.012 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.012 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:09:59.013 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.013 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:09:59.013 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:09:59.013 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:09:59.014 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.014 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:09:59.015 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:09:59.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.017 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:09:59.017 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.018 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:09:59.018 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.019 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:09:59.019 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.020 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:09:59.020 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:09:59.021 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.021 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.022 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:09:59.022 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:09:59.022 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:09:59.022 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:09:59.023 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4EA1D6E5-0-0] 2024-09-11 18:09:59.023 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:09:59.024 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:09:59.029 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.029 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:09:59.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.031 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:09:59.031 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.032 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:09:59.033 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.033 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:09:59.034 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.034 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:09:59.035 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.036 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:09:59.036 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:09:59.039 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:09:59.042 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:59.042 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:59.045 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.045 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.045 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:09:59.045 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:59.052 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:09:59.056 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:09:59.056 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:09:59.056 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.057 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:09:59.058 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.058 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:09:59.058 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:09:59.060 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:09:59.060 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:09:59.060 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:09:59.063 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:09:59.065 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:09:59.065 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:59.070 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:09:59.072 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:09:59.072 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:09:59.072 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:09:59.072 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:09:59.073 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:09:59.073 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:09:59.073 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:09:59.084 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:09:59.086 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:09:59.086 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:09:59.086 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:09:59.086 INFO 2981023 --- [ocp-schedules-21,51378c60fa761da3,b3e6bae49073e94d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:09:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,f4fb48efccfb0b36,bb1896460547ac0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049400, interval=60 2024-09-11 18:09:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,f4fb48efccfb0b36,bb1896460547ac0f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:09:59.402 INFO 2981023 --- [ocp-async-6,f809b134aaa89c21,a361f5c79b84fd47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:09:59.977 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:09:59.978 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:00.550 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,301451dbeded3bfb,3627d46547405095] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=301451dbeded3bfb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:00.560 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,301451dbeded3bfb,3627d46547405095] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=301451dbeded3bfb, duration=13 ms] 2024-09-11 18:10:00.874 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:05:01.0, nextRun=2024-09-11 18:10:00.0, currentDate=Wed Sep 11 18:10:00 HKT 2024 2024-09-11 18:10:00.874 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:05:01.0, nextRunTime=2024-09-11 18:10:00.0} 2024-09-11 18:10:00.878 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 18:10:00.878 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:10:00.878 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:10:00.878+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:10:00.896 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:05:01.0, nextRun=2024-09-11 18:10:00.0, currentDate=Wed Sep 11 18:10:00 HKT 2024 2024-09-11 18:10:00.896 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:05:01.0, nextRunTime=2024-09-11 18:10:00.0} 2024-09-11 18:10:00.911 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 18:10:00.918 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 18:10:00.918 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 18:10:00.918 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:10:00.918 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,6ad07910c3d61d35] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T18:10:00.918+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:10:01.402 INFO 2981023 --- [ocp-async-10,ee1f57c656f49ed6,562d80c46eabd85e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:01.833 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9ec0e6c24db1956f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048242, operation:EXECUTE, state: RUNNING 2024-09-11 18:10:01.843 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,9ec0e6c24db1956f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048243, operation:EXECUTE, state: RUNNING 2024-09-11 18:10:01.852 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048242, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048212, task_operation=execute, latest_execution_start_time=2024-09-11T18:10:01.828+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048242}, listMap={}}, executor=10.31.35.10 2024-09-11 18:10:01.861 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048243, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048213, task_operation=execute, latest_execution_start_time=2024-09-11T18:10:01.842+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048243}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 18:10:01.865 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 18:10:01.869 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:10:01.885 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 18:10:01.888 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 18:10:01.900 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 18:10:01.901 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 18:10:01.901 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 18:10:01.915 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 18:10:01.920 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 18:10:01.923 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:10:01.928 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,fffa1cccf568a7f0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048243, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:10:01.992 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:10:02.007 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:10:02.040 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:10:02.054 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 18:10:02.054 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 18:10:02.059 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:10:02.155 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:10:02.212 INFO 2981023 --- [ocp-schedules-30,3a4e189a6219bd79,e87cafe94cd06d4f] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:10:02.213 INFO 2981023 --- [ocp-schedules-30,3a4e189a6219bd79,e87cafe94cd06d4f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:10:02.226 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:10:02.258 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:10:02.274 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:10:02.319 INFO 2981023 --- [prometheus-minute-schedule-1,365bd69f25dc70d0,bd303c0099445390] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:02.321 INFO 2981023 --- [prometheus-minute-schedule-1,365bd69f25dc70d0,bd303c0099445390] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:02.330 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:10:02.348 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:10:02.362 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:10:02.369 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 18:10:02.369 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3395.459579986], b:[-5.807309276841022E12] 2024-09-11 18:10:02.374 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:10:02.436 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:10:02.466 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:10:02.478 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:10:02.491 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:10:02.500 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:10:02.506 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:10:02.512 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:10:02.514 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 18:10:02.515 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:10:02.529 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:10:02.537 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:10:02.546 INFO 2981023 --- [ocp-schedules-9,88cc7ec991ccaba1,77a228702524c63a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:10:02.548 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,2a2be3350e51da16] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048242, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:10:02.597 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048212, status:SUCCESS 2024-09-11 18:10:02.600 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048212, status:SUCCESS 2024-09-11 18:10:02.608 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048213, status:SUCCESS 2024-09-11 18:10:02.609 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048213, status:SUCCESS 2024-09-11 18:10:02.834 INFO 2981023 --- [ocp-schedules-23,22bea80dafb152a0,4bdc4440a93e2fc3] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:10:02.835 INFO 2981023 --- [ocp-schedules-23,22bea80dafb152a0,4bdc4440a93e2fc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:10:02.857 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:02.857 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:10:02.862 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:02.864 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:10:02.864 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049402864000} 2024-09-11 18:10:02.870 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:10:02.874 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000229, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:10:02.874 INFO 2981023 --- [ocp-schedules-23,22bea80dafb152a0,4bdc4440a93e2fc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:10:02.978 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:02.978 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:03.264 INFO 2981023 --- [ocp-schedules-3,76e0f06de4e2bd99,8561fef0c422ae18] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:10:03.324 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c26fa47d56be27ac,80b7812ef16efaea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=c26fa47d56be27ac, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:03.336 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c26fa47d56be27ac,80b7812ef16efaea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=c26fa47d56be27ac, duration=15 ms] 2024-09-11 18:10:03.402 INFO 2981023 --- [ocp-async-7,833e3c4675f204cf,6d77b7d3e514389d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:03.830 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:10:03.830 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:10:03.830 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:10:03.836 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:10:03.837 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000229, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:10:04.128 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000229 2024-09-11 18:10:05.015 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,109d59b98ef61b50] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:00:05.0, nextRun=2024-09-11 18:10:04.0, currentDate=Wed Sep 11 18:10:05 HKT 2024 2024-09-11 18:10:05.015 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,109d59b98ef61b50] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-09-11 18:00:05.0, nextRunTime=2024-09-11 18:10:04.0} 2024-09-11 18:10:05.022 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,109d59b98ef61b50] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-09-11 18:10:05.022 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,109d59b98ef61b50] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:10:05.022 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,109d59b98ef61b50] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:10:05.022+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:10:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,41b59ce35d6896bb,2f37b9fd09d8d330] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,41b59ce35d6896bb,2f37b9fd09d8d330] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:05.402 INFO 2981023 --- [ocp-async-9,9dd9d8591dd99024,4d6f186acd887a75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:05.869 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,b00ce5f90e98d502] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048244, operation:EXECUTE, state: RUNNING 2024-09-11 18:10:05.898 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048244, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048214, task_operation=execute, latest_execution_start_time=2024-09-11T18:10:05.867+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=4048244}, listMap={}}, executor=10.31.35.10 2024-09-11 18:10:05.905 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 18:10:05.905 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 18:10:05.906 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.907 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:10:05.909 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.909 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.910 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.910 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 18:10:05.923 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.924 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.924 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.924 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 18:10:05.933 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.933 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.934 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.934 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 18:10:05.943 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.943 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.943 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.944 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 18:10:05.953 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.953 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.954 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.954 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 18:10:05.957 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.958 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.958 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.958 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 18:10:05.961 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.961 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.962 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.962 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 18:10:05.965 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.965 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.966 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.966 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 18:10:05.967 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:10:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:10:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:10:05.971 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.971 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.972 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.972 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 18:10:05.976 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.977 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.977 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.978 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 18:10:05.978 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:05.980 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:05.983 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.983 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.984 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.984 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 18:10:05.992 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.992 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:05.993 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:05.993 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 18:10:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:10:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:10:06.005 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.006 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.006 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.007 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 18:10:06.007 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:10:06.007 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:10:06.007 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:10:06.011 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.011 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.011 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.011 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:10:06.012 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.012 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 18:10:06.012 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.012 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.017 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.017 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.018 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.018 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 18:10:06.022 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.022 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.022 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.023 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 18:10:06.026 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.026 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.027 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.027 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 18:10:06.030 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.031 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.031 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.032 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 18:10:06.036 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.036 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.037 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.037 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 18:10:06.041 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.041 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.042 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.042 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 18:10:06.046 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.046 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.048 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.049 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 18:10:06.053 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.054 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.054 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.055 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 18:10:06.058 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.059 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.059 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.060 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 18:10:06.063 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.063 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.064 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.064 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 18:10:06.070 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.070 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.071 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.071 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 18:10:06.075 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.076 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.076 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.076 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 18:10:06.079 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.080 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.080 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.081 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 18:10:06.085 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.085 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.086 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.086 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 18:10:06.090 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.090 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.091 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.091 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 18:10:06.095 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.095 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.095 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.096 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 18:10:06.096 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:07:41, start=2024-09-11T18:05:11, end=2024-09-11T18:07:41, updates=294, granule=120000ms, delay=150000ms 2024-09-11 18:10:06.099 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.100 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.100 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.100 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 18:10:06.104 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.104 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.105 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.105 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 18:10:06.109 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.110 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.111 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.111 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 18:10:06.114 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:07:41, target=2024-09-11T18:07:41, granule=120000ms, delay=0ms) 2024-09-11 18:10:06.114 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:07:41, target=2024-09-11T18:00:51, granule=600000ms, delay=410000ms) 2024-09-11 18:10:06.114 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=144ms, period=120000ms 2024-09-11 18:10:06.115 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.115 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.115 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.116 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 18:10:06.120 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.121 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.121 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.121 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 18:10:06.125 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.126 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.126 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.126 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 18:10:06.129 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.130 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.130 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.130 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 18:10:06.133 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.134 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.134 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.134 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 18:10:06.137 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.137 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.137 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.138 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 18:10:06.140 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.141 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.141 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.141 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 18:10:06.143 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.144 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.144 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.144 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 18:10:06.147 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.147 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.147 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.148 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 18:10:06.154 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.154 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.155 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.155 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 18:10:06.170 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.170 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.171 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.171 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 18:10:06.174 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.175 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.175 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.176 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 18:10:06.179 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.180 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.180 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.180 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 18:10:06.184 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.184 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.185 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.185 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 18:10:06.187 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.187 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.188 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.188 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 18:10:06.191 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.191 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.191 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.192 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 18:10:06.194 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 18:10:06.194 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 18:10:06.199 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.199 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:10:06.201 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.201 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.202 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.202 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 18:10:06.205 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.206 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:10:06.206 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:06.207 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 18:10:06.209 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 18:10:06.218 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9d9783568308948a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048244, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:10:06.281 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7fa0d871cea410fd,d32d81fc3c86fb88] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=7fa0d871cea410fd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:06.302 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7fa0d871cea410fd,d32d81fc3c86fb88] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=7fa0d871cea410fd, duration=25 ms] 2024-09-11 18:10:06.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:10:06.350750+08:00, lastProcessTime=2024-09-11T18:09:56.351771+08:00 2024-09-11 18:10:06.513 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:10:06.350750+08:00 2024-09-11 18:10:06.625 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048214, status:SUCCESS 2024-09-11 18:10:06.626 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048214, status:SUCCESS 2024-09-11 18:10:06.849 INFO 2981023 --- [ocp-schedules-10,40445ce426dc511a,f0ed6215f1085a75] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:10:06.899 WARN 2981023 --- [scanner-13617411-1,40445ce426dc511a,194ddc61d8cdd1a0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:10:07.402 INFO 2981023 --- [ocp-async-6,09e6629bcb5b5686,e85dcf891933e65f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:07.783 INFO 2981023 --- [ocp-async-5,251626fae83cdccd,c3545e8251675d09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:10:07.878 INFO 2981023 --- [ocp-schedules-21,a187ee29a6b24e08,d1e0f687f99d6827] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:10:07.900 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8621292ceff35348,f1c1a99bbeb88059] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=8621292ceff35348, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:07.910 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8621292ceff35348,f1c1a99bbeb88059] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=8621292ceff35348, duration=12 ms] 2024-09-11 18:10:07.986 INFO 2981023 --- [ocp-schedules-20,ebc21527574a46fc,5b975b9bf664ec0d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:10:07.986+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:10:07.989 INFO 2981023 --- [ocp-schedules-20,ebc21527574a46fc,5b975b9bf664ec0d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:10:07.993 INFO 2981023 --- [ocp-schedules-20,ebc21527574a46fc,5b975b9bf664ec0d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:10:07.993 INFO 2981023 --- [ocp-schedules-20,ebc21527574a46fc,5b975b9bf664ec0d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:10:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,c89bb076323efb23,8f2cead7d3620114] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,c89bb076323efb23,8f2cead7d3620114] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:08.980 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:08.982 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:09.402 INFO 2981023 --- [ocp-async-1,64634c6936a54ce9,a6c50abb230cb324] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:09.626 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9956dd07dbffe71b,f0b8ea26dab576c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=9956dd07dbffe71b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:09.648 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9956dd07dbffe71b,f0b8ea26dab576c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=9956dd07dbffe71b, duration=27 ms] 2024-09-11 18:10:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:10:10.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:10:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:10:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:10:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,d855912bee80229e,3193cce30c7b5c8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,d855912bee80229e,3193cce30c7b5c8e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:11.328 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,0bb58a1fa3ba0c6d,8e998776b3578de4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=0bb58a1fa3ba0c6d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:11.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,0bb58a1fa3ba0c6d,8e998776b3578de4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=0bb58a1fa3ba0c6d, duration=13 ms] 2024-09-11 18:10:11.402 INFO 2981023 --- [ocp-async-5,da322abc87d387fa,f01c12e641d84c99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:11.943 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049340, nodeCount=1000, affectRows=1000 2024-09-11 18:10:11.943 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049340, nodeCount=1000, affectRows=1000 2024-09-11 18:10:11.945 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049340, nodeCount=1000, validCount=1000 2024-09-11 18:10:11.946 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049340, nodeCount=999, validCount=999 2024-09-11 18:10:11.952 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049340, nodeCount=48, affectRows=48 2024-09-11 18:10:11.952 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049340, nodeCount=46, validCount=46 2024-09-11 18:10:11.982 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:11.983 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:12.548 INFO 2981023 --- [ocp-schedules-15,4c3c65f1dc9f87de,3639302af7d754e1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:10:12.654 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c5e645cbbfeeace7,39d3e2ac472603c5] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&version=2&ObClusterId=1718689631, client=10.31.35.5, traceId=c5e645cbbfeeace7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=rtdob,1718689631,2,{"ObRegion":"rtdob","ObCluster":"rtdob","ObRegionId":1718689631,"ObClusterId":1718689631,"Type":"PRIMARY","timestamp":1726049412658827,"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:10:12.662 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c5e645cbbfeeace7,39d3e2ac472603c5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=rtdob, obClusterId=1718689631, rootServerJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, groupName=rtdob, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:10:12.662 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c5e645cbbfeeace7,39d3e2ac472603c5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:10:12.662 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c5e645cbbfeeace7,39d3e2ac472603c5] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=rtdob, clusterId=1718689631, affectRows=1 2024-09-11 18:10:12.662 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c5e645cbbfeeace7,39d3e2ac472603c5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.5, traceId=c5e645cbbfeeace7, duration=8 ms] 2024-09-11 18:10:13.402 INFO 2981023 --- [ocp-async-3,8a3d6bdb6c00601f,5048d7a4fa4ab630] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:13.756 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,c58c3e5acc0f428e,9fc4412ef64a79c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=c58c3e5acc0f428e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:13.766 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,c58c3e5acc0f428e,9fc4412ef64a79c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=c58c3e5acc0f428e, duration=13 ms] 2024-09-11 18:10:14.289 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049340, nodeCount=480, affectRows=480 2024-09-11 18:10:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,c723b432c349fb05,3ff10873f5ce7383] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,694a8b3cfb85520e,c842a5e2f3fbe6d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:10:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,c723b432c349fb05,3ff10873f5ce7383] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:14.908 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c50889b3fdc8e7b0,e93b6377aa3fe7ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=c50889b3fdc8e7b0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:14.916 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,c50889b3fdc8e7b0,e93b6377aa3fe7ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=c50889b3fdc8e7b0, duration=11 ms] 2024-09-11 18:10:14.983 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:14.984 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:15.029 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,7215a9d45973cb80,040365b995f99a5c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=7215a9d45973cb80, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:15.032 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,7215a9d45973cb80,040365b995f99a5c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=7215a9d45973cb80, duration=6 ms] 2024-09-11 18:10:15.137 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b29ba9b97e84b36d,2182fcce0a9a1e3a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=b29ba9b97e84b36d, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:15.140 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b29ba9b97e84b36d,2182fcce0a9a1e3a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=b29ba9b97e84b36d, duration=6 ms] 2024-09-11 18:10:15.233 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a716da9c97b750d4,e3d2e27ca46d0ace] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=a716da9c97b750d4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:15.237 INFO 2981023 --- [ocp-schedules-12,ed7d6711f226b8a6,65002fb361bb40f7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:10:15.247 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,33a65b1c87f151ec,d07b39df7ee8f01b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=33a65b1c87f151ec, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:15.247 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a716da9c97b750d4,e3d2e27ca46d0ace] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=a716da9c97b750d4, duration=17 ms] 2024-09-11 18:10:15.247 INFO 2981023 --- [ocp-schedules-12,ed7d6711f226b8a6,65002fb361bb40f7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:10:15.247 INFO 2981023 --- [ocp-schedules-12,ed7d6711f226b8a6,65002fb361bb40f7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:10:15.250 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,33a65b1c87f151ec,d07b39df7ee8f01b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=33a65b1c87f151ec, duration=9 ms] 2024-09-11 18:10:15.371 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f72c00f11be2c40a,51667f4c9f35be70] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=f72c00f11be2c40a, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:15.374 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f72c00f11be2c40a,51667f4c9f35be70] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=f72c00f11be2c40a, duration=5 ms] 2024-09-11 18:10:15.403 INFO 2981023 --- [ocp-async-10,14446348a02ed585,5824ef7015660df3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:15.473 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,5dfaa600f78016fd,8ea4be1363c7920b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=5dfaa600f78016fd, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:15.475 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,5dfaa600f78016fd,8ea4be1363c7920b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=5dfaa600f78016fd, duration=5 ms] 2024-09-11 18:10:15.600 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b0cd61528fa783ad,b3a9f8c63c0f0203] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=b0cd61528fa783ad, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:15.603 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,b0cd61528fa783ad,b3a9f8c63c0f0203] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=b0cd61528fa783ad, duration=6 ms] 2024-09-11 18:10:15.707 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,b3a0753d2ed78878,8844a2076d5a970d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=b3a0753d2ed78878, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:15.709 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,b3a0753d2ed78878,8844a2076d5a970d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=b3a0753d2ed78878, duration=5 ms] 2024-09-11 18:10:15.827 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,28ce7ed1796a481a,eb4df25c37772b45] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=28ce7ed1796a481a, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:15.830 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,28ce7ed1796a481a,eb4df25c37772b45] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=28ce7ed1796a481a, duration=6 ms] 2024-09-11 18:10:16.021 INFO 2981023 --- [ocp-schedules-2,ce247b67ba09c270,d6216bd1af9ceb57] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:10:16.021 INFO 2981023 --- [host-check-clockdiff1,ce247b67ba09c270,113affef38c984d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:10:16.021 INFO 2981023 --- [host-check-clockdiff2,ce247b67ba09c270,58d6a9722966426b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:10:16.021 INFO 2981023 --- [ocp-schedules-20,89eb3bebdbf91ba7,ac90f729c3fdd33e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:10:16.021 INFO 2981023 --- [host-check-clockdiff3,ce247b67ba09c270,5903060de6a4f398] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:10:16.022 INFO 2981023 --- [host-check-clockdiff4,ce247b67ba09c270,77346a759cebdf44] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:10:16.025 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049340, nodeCount=480, validCount=480 2024-09-11 18:10:16.029 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:10:16.030 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:10:16.030 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:10:16.030 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:10:16.031 INFO 2981023 --- [ocp-schedules-4,cbbbd6d9c6ffc028,15bafaa244cae843] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:10:16.036 INFO 2981023 --- [host-check-agent-status1,cbbbd6d9c6ffc028,b51fbb8d8894ff6a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:10:16.037 INFO 2981023 --- [host-check-agent-status2,cbbbd6d9c6ffc028,a949da10242a1080] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:10:16.037 INFO 2981023 --- [host-check-agent-status3,cbbbd6d9c6ffc028,f0919a5902345b5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:10:16.037 INFO 2981023 --- [host-check-agent-status4,cbbbd6d9c6ffc028,3e518b5d315dc113] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:10:16.042 INFO 2981023 --- [host-check-clockdiff3,ce247b67ba09c270,5903060de6a4f398] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049416 0 0 , error= 2024-09-11 18:10:16.067 INFO 2981023 --- [host-check-agent-status2,cbbbd6d9c6ffc028,a949da10242a1080] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:10:16.068 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:10:16.068 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:10:16.072 INFO 2981023 --- [host-check-agent-status2,cbbbd6d9c6ffc028,a949da10242a1080] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:10:16.073 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:10:16.079 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:10:16.079 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:10:16.081 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:10:16.087 INFO 2981023 --- [host-check-agent-status3,cbbbd6d9c6ffc028,f0919a5902345b5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:10:16.087 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:10:16.098 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:10:16.153 INFO 2981023 --- [host-check-agent-status3,cbbbd6d9c6ffc028,f0919a5902345b5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:10:16.191 INFO 2981023 --- [ocp-schedules-24,aedccad6821ea841,4f8aa08c535933b6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:10:16.198 INFO 2981023 --- [ocp-schedules-13,09eb2f123082a686,d7c104f5ed505b12] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:10:16.198 INFO 2981023 --- [ocp-schedules-14,1201d72dfd013ad9,9bb355fe2d508fcd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:10:16.198 INFO 2981023 --- [ocp-schedules-23,e1caa8f4e767044e,d9343dc59ad6d354] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:10:16.270 INFO 2981023 --- [host-check-agent-status1,cbbbd6d9c6ffc028,b51fbb8d8894ff6a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:10:16.273 INFO 2981023 --- [host-check-agent-status4,cbbbd6d9c6ffc028,3e518b5d315dc113] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:10:16.280 INFO 2981023 --- [host-check-agent-status1,cbbbd6d9c6ffc028,b51fbb8d8894ff6a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:10:16.280 INFO 2981023 --- [host-check-agent-status4,cbbbd6d9c6ffc028,3e518b5d315dc113] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:10:16.289 INFO 2981023 --- [ocp-schedules-14,1201d72dfd013ad9,9bb355fe2d508fcd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:10:16.290 INFO 2981023 --- [ocp-schedules-13,09eb2f123082a686,d7c104f5ed505b12] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:10:16.291 INFO 2981023 --- [ocp-schedules-24,aedccad6821ea841,4f8aa08c535933b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:10:16.309 INFO 2981023 --- [ocp-schedules-13,09eb2f123082a686,d7c104f5ed505b12] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:10:16.309 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049402864000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:10:16.457 INFO 2981023 --- [ocp-schedules-11,b7b025a451d5418c,fe35f22c5a50b7c5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:10:16.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:10:16.460133+08:00, lastProcessTime=2024-09-11T18:10:06.350750+08:00 2024-09-11 18:10:16.470 INFO 2981023 --- [host-check-agent-status2,cbbbd6d9c6ffc028,a949da10242a1080] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:10:16.474 INFO 2981023 --- [host-check-agent-status2,cbbbd6d9c6ffc028,a949da10242a1080] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:10:16.474 INFO 2981023 --- [host-check-agent-status2,cbbbd6d9c6ffc028,a949da10242a1080] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:10:16.474 INFO 2981023 --- [host-check-agent-status2,cbbbd6d9c6ffc028,a949da10242a1080] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:10:16.477 INFO 2981023 --- [host-check-agent-status4,cbbbd6d9c6ffc028,3e518b5d315dc113] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:10:16.480 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:10:16.482 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:10:16.487302403+08:00 2024-09-11 18:10:16.484 INFO 2981023 --- [host-check-agent-status4,cbbbd6d9c6ffc028,3e518b5d315dc113] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:10:16.484 INFO 2981023 --- [host-check-agent-status4,cbbbd6d9c6ffc028,3e518b5d315dc113] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:10:16.484 INFO 2981023 --- [host-check-agent-status4,cbbbd6d9c6ffc028,3e518b5d315dc113] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:10:16.485 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:10:16.485 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:10:16.486 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:10:16.491717834+08:00 2024-09-11 18:10:16.486 INFO 2981023 --- [host-check-agent-status1,cbbbd6d9c6ffc028,b51fbb8d8894ff6a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:10:16.486 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:10:16.488 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:10:16.486+08:00 2024-09-11 18:10:16.488 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:10:16.488521208+08:00 2024-09-11 18:10:16.489 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:10:16.488373375+08:00 2024-09-11 18:10:16.489 INFO 2981023 --- [host-check-agent-status1,cbbbd6d9c6ffc028,b51fbb8d8894ff6a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:10:16.489 INFO 2981023 --- [host-check-agent-status1,cbbbd6d9c6ffc028,b51fbb8d8894ff6a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:10:16.489 INFO 2981023 --- [host-check-agent-status1,cbbbd6d9c6ffc028,b51fbb8d8894ff6a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:10:16.489 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:10:16.488+08:00 2024-09-11 18:10:16.490 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:10:16.482+08:00 2024-09-11 18:10:16.490 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:10:16.489+08:00 2024-09-11 18:10:16.491 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:10:16.486+08:00 and status AVAILABLE 2024-09-11 18:10:16.492 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:10:16.489+08:00 and status AVAILABLE 2024-09-11 18:10:16.493 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:10:16.488+08:00 and status AVAILABLE 2024-09-11 18:10:16.495 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:10:16.482+08:00 and status AVAILABLE 2024-09-11 18:10:16.501 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.501 INFO 2981023 --- [host-check-agent-status3,cbbbd6d9c6ffc028,f0919a5902345b5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:10:16.502 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:10:16.502 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:10:16.503 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:10:16.504 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.504 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.504 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:10:16.504 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:16.505 INFO 2981023 --- [host-check-agent-status3,cbbbd6d9c6ffc028,f0919a5902345b5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:10:16.505 INFO 2981023 --- [host-check-agent-status3,cbbbd6d9c6ffc028,f0919a5902345b5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:10:16.505 INFO 2981023 --- [host-check-agent-status3,cbbbd6d9c6ffc028,f0919a5902345b5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:10:16.505 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:10:16.505 INFO 2981023 --- [ocp-schedules-24,aedccad6821ea841,4f8aa08c535933b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:10:16.506 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.506 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:16.507 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:10:16.515 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:10:16.516 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:10:16.573 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:16.579 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:10:16.580 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:10:16.582 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:10:16.584 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:10:16.584 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:10:16.585 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:10:16.585 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:10:16.586 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:10:16.618 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:10:16.639 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:10:16.645 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:10:16.647 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:10:16.647 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:16.648 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.648 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:10:16.657 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:10:16.671 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:10:16.673 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:10:16.673 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049402864000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:10:16.676 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:10:16.676 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:10:16.678 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:10:16.687 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:10:16.687 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:10:16.688 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:10:16.690 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:16.691 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.691 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:16.692 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:10:16.692 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.695 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:10:16.695 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:10:16.706 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:10:16.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:10:16.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:10:16.711 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:10:16.732 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:10:16.732 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:10:16.735 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:10:16.737 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:10:16.741 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:10:16.746 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:16.747 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.747 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:10:16.748 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:10:16.748 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:10:16.753 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:10:16.756 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:10:16.760 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,3bd714774d66f7a0,22b5ff9f422e5c09] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=3bd714774d66f7a0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:16.760 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:10:16.764 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:10:16.767 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:10:16.768 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:10:16.777 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:10:16.777 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:10:16.781 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:10:16.781 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,3bd714774d66f7a0,22b5ff9f422e5c09] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=3bd714774d66f7a0, duration=26 ms] 2024-09-11 18:10:16.786 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:10:16.809 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:10:16.814 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:10:16.815 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:10:16.821 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:10:16.825 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:16.826 INFO 2981023 --- [host-check-host-status3,89eb3bebdbf91ba7,8b17e25d9eab9908] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:10:16.826 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.827 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:16.831 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:10:16.832 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.834 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:10:16.834 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:10:16.835 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:10:16.837 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:10:16.846 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:10:16.846 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:10:16.847 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:10:16.850 INFO 2981023 --- [host-check-host-status1,89eb3bebdbf91ba7,293ac49579acc04e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:10:16.856 INFO 2981023 --- [host-check-host-status4,89eb3bebdbf91ba7,8ec7c8e8898e9b89] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:10:16.864 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:10:16.460133+08:00 2024-09-11 18:10:16.865 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:10:16.867 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:10:16.874 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:16.874 INFO 2981023 --- [host-check-host-status2,89eb3bebdbf91ba7,74706924d5ac7647] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:10:16.875 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.876 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:10:16.876 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:10:16.880 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:10:16.883 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:10:16.884 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:10:16.899 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:10:16.900 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:10:16.904 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2e7e84f7193d9170,0d5c398ac6a289ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=2e7e84f7193d9170, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:16.907 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2e7e84f7193d9170,0d5c398ac6a289ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=2e7e84f7193d9170, duration=7 ms] 2024-09-11 18:10:16.908 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:16.909 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.909 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:10:16.914 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:10:16.914 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:10:16.930 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:10:16.935 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:16.936 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:16.937 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:10:16.943 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:10:16.943 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:10:16.943 INFO 2981023 --- [ocp-schedules-14,1201d72dfd013ad9,9bb355fe2d508fcd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:10:16.969 INFO 2981023 --- [host-check-clockdiff1,ce247b67ba09c270,113affef38c984d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049416 6 6 , error= 2024-09-11 18:10:16.970 INFO 2981023 --- [host-check-clockdiff2,ce247b67ba09c270,58d6a9722966426b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049416 5 5 , error= 2024-09-11 18:10:17.040 INFO 2981023 --- [ocp-schedules-9,d861ed5a7084a2f7,0440239bf6c708aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:10:17.064 WARN 2981023 --- [scanner-13619081-1,d861ed5a7084a2f7,856bc6c3e81fc302] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:10:17.119 INFO 2981023 --- [alarm-detect-3,d861ed5a7084a2f7,598bebeacdec5a70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645575, data is empty. 2024-09-11 18:10:17.124 INFO 2981023 --- [alarm-detect-3,d861ed5a7084a2f7,598bebeacdec5a70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645616, data is empty. 2024-09-11 18:10:17.131 INFO 2981023 --- [alarm-detect-3,d861ed5a7084a2f7,598bebeacdec5a70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645616, data is empty. 2024-09-11 18:10:17.141 INFO 2981023 --- [alarm-detect-3,d861ed5a7084a2f7,5961db47c9e48b1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645575, data is empty. 2024-09-11 18:10:17.144 INFO 2981023 --- [alarm-detect-19,d861ed5a7084a2f7,923144260ded8ee3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645575, data is empty. 2024-09-11 18:10:17.147 INFO 2981023 --- [alarm-detect-3,d861ed5a7084a2f7,5961db47c9e48b1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645616, data is empty. 2024-09-11 18:10:17.152 INFO 2981023 --- [alarm-detect-19,d861ed5a7084a2f7,923144260ded8ee3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645616, data is empty. 2024-09-11 18:10:17.154 INFO 2981023 --- [alarm-detect-3,d861ed5a7084a2f7,5961db47c9e48b1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645616, data is empty. 2024-09-11 18:10:17.170 INFO 2981023 --- [alarm-detect-19,d861ed5a7084a2f7,923144260ded8ee3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645616, data is empty. 2024-09-11 18:10:17.221 INFO 2981023 --- [alarm-detect-1,d861ed5a7084a2f7,d378a46bed9f1799] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726049385 ~ 1726049407 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726049400, value=1.79)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.79, values={1726049400=1.79}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:10:17.221 INFO 2981023 --- [alarm-detect-1,d861ed5a7084a2f7,d378a46bed9f1799] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726049385 ~ 1726049407 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726049400, value=1.558)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.558, values={1726049400=1.558}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:10:17.222 INFO 2981023 --- [alarm-detect-1,d861ed5a7084a2f7,d378a46bed9f1799] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.10-3, currentState=Pending 2024-09-11 18:10:17.222 INFO 2981023 --- [alarm-detect-1,d861ed5a7084a2f7,d378a46bed9f1799] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Firing 2024-09-11 18:10:17.222 INFO 2981023 --- [alarm-detect-1,d861ed5a7084a2f7,d378a46bed9f1799] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-3, alarm_template_id=0:host=10.31.35.5-3] 2024-09-11 18:10:17.267 INFO 2981023 --- [alarm-detect-1,d861ed5a7084a2f7,d378a46bed9f1799] c.o.o.alarm.service.OcpAlarmServiceImpl : delete alarm event 6007108 2024-09-11 18:10:17.269 INFO 2981023 --- [alarm-detect-1,d861ed5a7084a2f7,d378a46bed9f1799] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007109 2024-09-11 18:10:17.295 INFO 2981023 --- [alarm-detect-12,d861ed5a7084a2f7,16266dee95641426] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645575, data is empty. 2024-09-11 18:10:17.301 INFO 2981023 --- [alarm-detect-12,d861ed5a7084a2f7,16266dee95641426] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645616, data is empty. 2024-09-11 18:10:17.305 INFO 2981023 --- [alarm-detect-12,d861ed5a7084a2f7,16266dee95641426] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645616, data is empty. 2024-09-11 18:10:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,26a02f7088fa4077,5a99f9cf0168d65d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:17.318 INFO 2981023 --- [prometheus-minute-schedule-1,26a02f7088fa4077,5a99f9cf0168d65d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:17.403 INFO 2981023 --- [ocp-async-1,9c6da5f9b8a85265,c3ce85409c36d1ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:17.460 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ea4b7585be985454,6a43d469f51cb8d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ea4b7585be985454, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:17.472 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ea4b7585be985454,6a43d469f51cb8d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ea4b7585be985454, duration=16 ms] 2024-09-11 18:10:17.783 INFO 2981023 --- [ocp-async-8,aabaef5349626ccd,d8b9b6474c4a1d05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:10:17.984 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:17.985 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:18.248 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c429908d755ab9e9,e9e5ef1e693bd85a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=c429908d755ab9e9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:18.258 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c429908d755ab9e9,e9e5ef1e693bd85a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=c429908d755ab9e9, duration=13 ms] 2024-09-11 18:10:18.407 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,710373775a569c27,593e040318531882] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments?taskInstanceId=4048203&subtaskInstanceId=4048209, client=10.31.33.41, traceId=710373775a569c27, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=4048203,4048209,] 2024-09-11 18:10:18.414 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,710373775a569c27,593e040318531882] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203/subtasks/4048209/logSegments client=10.31.33.41, traceId=710373775a569c27, duration=9 ms] 2024-09-11 18:10:19.086 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a6a618a800c695b1,e118f3ffe984f23b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a6a618a800c695b1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:19.092 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a6a618a800c695b1,e118f3ffe984f23b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a6a618a800c695b1, duration=9 ms] 2024-09-11 18:10:19.403 INFO 2981023 --- [ocp-async-4,32d5d6cb91bcf18c,c23ff929bd808160] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:19.750 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c9c94abce4c3885a,7b9f1ccfe1a5a39e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/4048203?taskInstanceId=4048203, client=10.31.33.41, traceId=c9c94abce4c3885a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=4048203,] 2024-09-11 18:10:19.760 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c9c94abce4c3885a,7b9f1ccfe1a5a39e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/4048203 client=10.31.33.41, traceId=c9c94abce4c3885a, duration=12 ms] 2024-09-11 18:10:20.163 INFO 2981023 --- [host-check-clockdiff4,ce247b67ba09c270,77346a759cebdf44] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049420 0 0 , error= 2024-09-11 18:10:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,c55a456765604327,fcbd01ffb2b680a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,c55a456765604327,fcbd01ffb2b680a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:20.868 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,b3f51109026bb272,df3235dc78220e0b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=b3f51109026bb272, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 18:10:20.881 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a2423b5eae91723d,0296d0db34a85bed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=a2423b5eae91723d, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 18:10:20.891 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a2423b5eae91723d,0296d0db34a85bed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=a2423b5eae91723d, duration=12 ms] 2024-09-11 18:10:20.900 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,b3f51109026bb272,df3235dc78220e0b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=b3f51109026bb272, duration=34 ms] 2024-09-11 18:10:20.985 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:20.985 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:20.991 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,ae9c6988bfcaaecf,387eea457c14abc9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=ae9c6988bfcaaecf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:10:20.991 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,5b55b847e032017d,961276f6ed273191] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=5b55b847e032017d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:10:20.991 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3dd8d6bc150c851f,5396acd3610f945e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=3dd8d6bc150c851f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:10:20.992 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,ae9c6988bfcaaecf,387eea457c14abc9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=ae9c6988bfcaaecf, duration=3 ms] 2024-09-11 18:10:20.994 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3dd8d6bc150c851f,5396acd3610f945e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=3dd8d6bc150c851f, duration=5 ms] 2024-09-11 18:10:21.002 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,5b55b847e032017d,961276f6ed273191] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=5b55b847e032017d, duration=13 ms] 2024-09-11 18:10:21.006 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,ee77276b514f5524,728bccbc476974dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_ODP&type=TOP&maxPoints=360, client=10.31.33.41, traceId=ee77276b514f5524, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_ODP,] 2024-09-11 18:10:21.013 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,a05e7be8cc688f1c,7f42b535759055d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_TENANT&type=TOP&maxPoints=360, client=10.31.33.41, traceId=a05e7be8cc688f1c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_TENANT,] 2024-09-11 18:10:21.016 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,ee77276b514f5524,728bccbc476974dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=ee77276b514f5524, duration=14 ms] 2024-09-11 18:10:21.019 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,a05e7be8cc688f1c,7f42b535759055d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=a05e7be8cc688f1c, duration=13 ms] 2024-09-11 18:10:21.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6ed133e1e29de9af,8cdbfccb1e88abb8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_HOST&type=TOP&maxPoints=360, client=10.31.33.41, traceId=6ed133e1e29de9af, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_HOST,] 2024-09-11 18:10:21.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,30d7ac2d0274fd6f,114d2dd346c8b785] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_memory_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=30d7ac2d0274fd6f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_memory_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 18:10:21.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,6ed133e1e29de9af,8cdbfccb1e88abb8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=6ed133e1e29de9af, duration=6 ms] 2024-09-11 18:10:21.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,30d7ac2d0274fd6f,114d2dd346c8b785] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=30d7ac2d0274fd6f, duration=6 ms] 2024-09-11 18:10:21.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a20e8c4218de26fc,51752b7cae0d096b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_cpu_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=a20e8c4218de26fc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_cpu_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 18:10:21.128 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,a20e8c4218de26fc,51752b7cae0d096b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=a20e8c4218de26fc, duration=6 ms] 2024-09-11 18:10:21.129 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,1290af28b6b1a7fe,49cdf13be4f540e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_disk_used_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=1290af28b6b1a7fe, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_disk_used_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 18:10:21.132 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,1290af28b6b1a7fe,49cdf13be4f540e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=1290af28b6b1a7fe, duration=5 ms] 2024-09-11 18:10:21.133 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,b2f754868d7b3f00,0f8cc7f63db907c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_memory_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=b2f754868d7b3f00, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_memory_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 18:10:21.137 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,b2f754868d7b3f00,0f8cc7f63db907c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=b2f754868d7b3f00, duration=7 ms] 2024-09-11 18:10:21.232 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d3a58f97fcec4a21,0ea6f1122b84baa6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_cpu_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=d3a58f97fcec4a21, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_cpu_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 18:10:21.237 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,d3a58f97fcec4a21,0ea6f1122b84baa6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=d3a58f97fcec4a21, duration=7 ms] 2024-09-11 18:10:21.242 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,542c74b16dd5c46e,ac16cffa10903071] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_tenant_disk_usage_v4&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=542c74b16dd5c46e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_tenant_disk_usage_v4,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 18:10:21.246 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,542c74b16dd5c46e,ac16cffa10903071] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=542c74b16dd5c46e, duration=6 ms] 2024-09-11 18:10:21.252 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,ef3e7fce8875aea1,61c43959ded50e14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?foldStandby=false, client=10.31.33.41, traceId=ef3e7fce8875aea1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 18:10:21.256 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,423cf19f02265cce,d6fb7cb609466418] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=423cf19f02265cce, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:21.260 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,ef3e7fce8875aea1,61c43959ded50e14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=ef3e7fce8875aea1, duration=11 ms] 2024-09-11 18:10:21.263 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,423cf19f02265cce,d6fb7cb609466418] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=423cf19f02265cce, duration=9 ms] 2024-09-11 18:10:21.403 INFO 2981023 --- [ocp-async-8,dcd11c73337670b7,8e1942229a98fba8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:22.551 INFO 2981023 --- [ocp-schedules-27,b93fe8d8a94fe398,8efe81ddf5581055] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:10:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,08f8c4c9c19d4e3c,c7e4ae2677276568] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,08f8c4c9c19d4e3c,c7e4ae2677276568] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:23.403 INFO 2981023 --- [ocp-async-2,e4f24bf651177030,fb920134a75c685e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:23.985 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:23.987 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:24.418 INFO 2981023 --- [ocp-schedules-28,3aa73fe690770790,a9bdcb0d0eb711f8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:10:24.586 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?status=RUNNING, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:10:24.586 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/cluster, client=10.31.33.41, traceId=] 2024-09-11 18:10:24.586 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:10:25.403 INFO 2981023 --- [ocp-async-5,336a9e58ed38d86d,4aa7cd3e53a51f4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:25.686 INFO 2981023 --- [ocp-schedules-17,b3967345b8c0369a,ee0c95f33e985541] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:10:25.687 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:10:25.687 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:10:25.687 INFO 2981023 --- [ocp-schedules-17,b3967345b8c0369a,ee0c95f33e985541] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:10:25.687 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:10:25.706 INFO 2981023 --- [ocp-schedules-17,b3967345b8c0369a,ee0c95f33e985541] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:10:25.710 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:10:25.710 INFO 2981023 --- [ocp-schedules-17,b3967345b8c0369a,ee0c95f33e985541] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:10:25.712 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:10:25.714 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:10:25.715 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:10:25.715 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:10:25.716 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:10:25.890 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:10:25.891 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:10:25.957 INFO 2981023 --- [ocp-schedules-17,b3967345b8c0369a,ee0c95f33e985541] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:10:25.962 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:10:25.964 INFO 2981023 --- [ocp-schedules-17,b3967345b8c0369a,ee0c95f33e985541] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:10:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,098b0cb8e279e8a5,b577a783f6348529] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,098b0cb8e279e8a5,b577a783f6348529] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:10:26.352019+08:00, lastProcessTime=2024-09-11T18:10:16.460133+08:00 2024-09-11 18:10:26.478 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:10:26.483 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:10:26.352019+08:00 2024-09-11 18:10:26.811 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,bfa2bb373f19a833,01fc96a95cc4b18d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bfa2bb373f19a833, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:26.822 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,bfa2bb373f19a833,01fc96a95cc4b18d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bfa2bb373f19a833, duration=14 ms] 2024-09-11 18:10:26.987 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:26.988 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:27.298 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,acac71eefbd63207,c818a0d0254e0567] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.31.33.41, traceId=acac71eefbd63207, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-09-11 18:10:27.298 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,acac71eefbd63207,c818a0d0254e0567] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.31.33.41, traceId=acac71eefbd63207, duration=3 ms] 2024-09-11 18:10:27.323 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:10:27.323 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, client=10.31.33.41, traceId=] 2024-09-11 18:10:27.323 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:10:27.328 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:10:27.328 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, client=10.31.33.41, traceId=] 2024-09-11 18:10:27.328 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:10:27.352 INFO 2981023 --- [ocp-schedules-16,87fdc2d87b7a716b,951ee53a89ca6544] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:10:27.388 WARN 2981023 --- [scanner-13619464-1,87fdc2d87b7a716b,0e675b542101ed8d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:10:27.403 INFO 2981023 --- [ocp-async-6,a9b4d57b95aff410,58f6b80b06753f90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:27.708 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ba99c5d101293a52,889daa9dcdca5bbd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.31.33.41, traceId=ba99c5d101293a52, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-09-11 18:10:27.708 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,ba99c5d101293a52,889daa9dcdca5bbd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.31.33.41, traceId=ba99c5d101293a52, duration=2 ms] 2024-09-11 18:10:27.714 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,1e87f74451de413e,16d3287b49db9e2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.31.33.41, traceId=1e87f74451de413e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-09-11 18:10:27.717 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,1e87f74451de413e,16d3287b49db9e2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.31.33.41, traceId=1e87f74451de413e, duration=5 ms] 2024-09-11 18:10:27.723 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,fce1265d05621029,b485d4525764cc6e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.31.33.41, traceId=fce1265d05621029, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-09-11 18:10:27.723 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,fce1265d05621029,b485d4525764cc6e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.31.33.41, traceId=fce1265d05621029, duration=4 ms] 2024-09-11 18:10:27.724 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b116eafec40967ed,ef9a69ab865204c5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.31.33.41, traceId=b116eafec40967ed, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-09-11 18:10:27.724 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,b116eafec40967ed,ef9a69ab865204c5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.31.33.41, traceId=b116eafec40967ed, duration=6 ms] 2024-09-11 18:10:27.729 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:10:27.729 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, client=10.31.33.41, traceId=] 2024-09-11 18:10:27.729 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:10:27.783 INFO 2981023 --- [ocp-async-3,39d1345eaa0fded1,855006ccb760f7cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:10:27.831 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:10:27.831 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, client=10.31.33.41, traceId=] 2024-09-11 18:10:27.831 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:10:27.842 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,aa96a7dc0c5593ef,df146774f2c4219e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.31.33.41, traceId=aa96a7dc0c5593ef, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-09-11 18:10:27.842 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,aa96a7dc0c5593ef,df146774f2c4219e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.31.33.41, traceId=aa96a7dc0c5593ef, duration=1 ms] 2024-09-11 18:10:27.895 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a9f99544946f313d,d7c95cae650f7af7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.31.33.41, traceId=a9f99544946f313d, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-09-11 18:10:27.900 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,a9f99544946f313d,d7c95cae650f7af7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.31.33.41, traceId=a9f99544946f313d, duration=7 ms] 2024-09-11 18:10:28.243 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,9e4da1843aa412cc,61fbe823711ce7ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.31.33.41, traceId=9e4da1843aa412cc, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2024-09-11 18:10:28.246 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,1c33df106fa82401,c0ec9bae875cf47a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP&maxPoints=360, client=10.31.33.41, traceId=1c33df106fa82401, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,] 2024-09-11 18:10:28.251 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,1c33df106fa82401,c0ec9bae875cf47a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=1c33df106fa82401, duration=7 ms] 2024-09-11 18:10:28.251 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,9e4da1843aa412cc,61fbe823711ce7ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.31.33.41, traceId=9e4da1843aa412cc, duration=11 ms] 2024-09-11 18:10:28.683 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,79ccb1bfa12107d0,211a3ac963c38a13] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.31.33.41, traceId=79ccb1bfa12107d0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 18:10:28.691 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,817b37a880ef7b0a,dce5bc3dc7c94c14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.31.33.41, traceId=817b37a880ef7b0a, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2024-09-11 18:10:28.692 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,817b37a880ef7b0a,dce5bc3dc7c94c14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=817b37a880ef7b0a, duration=3 ms] 2024-09-11 18:10:28.696 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,abf95c51dd042b9a,80dd912099df01fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=abf95c51dd042b9a, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 18:10:28.740 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,abf95c51dd042b9a,80dd912099df01fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=abf95c51dd042b9a, duration=46 ms] 2024-09-11 18:10:28.806 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7016d13c5ab1002c,cdfce271b443630d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc&status=RUNNING, client=10.31.33.41, traceId=7016d13c5ab1002c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=[RUNNING],NONE,Page request [number: 0, size 10, sort: name: ASC],] 2024-09-11 18:10:28.821 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7016d13c5ab1002c,cdfce271b443630d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.31.33.41, traceId=7016d13c5ab1002c, duration=19 ms] 2024-09-11 18:10:28.847 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:28.848 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:28.851 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 18:10:28.851 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 18:10:28.852 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:28.854 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 18:10:28.861 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,79ccb1bfa12107d0,211a3ac963c38a13] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.31.33.41, traceId=79ccb1bfa12107d0, duration=181 ms] 2024-09-11 18:10:29.202 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,cc62d28c858bf1d9,c76bb03a5abaa456] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cc62d28c858bf1d9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:29.220 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,cc62d28c858bf1d9,c76bb03a5abaa456] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cc62d28c858bf1d9, duration=23 ms] 2024-09-11 18:10:29.262 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,518b06bf1fa508d6,3575e576efb7fb1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2024-09-11T17%3A10%3A28%2B08%3A00&endTime=2024-09-11T18%3A10%3A28%2B08%3A00&maxPoints=360, client=10.31.33.41, traceId=518b06bf1fa508d6, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T17:10:28+08:00,2024-09-11T18:10:28+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2024-09-11 18:10:29.274 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,518b06bf1fa508d6,3575e576efb7fb1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=518b06bf1fa508d6, duration=15 ms] 2024-09-11 18:10:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,9894a4e28f02af48,1e87e8690be11d9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:29.318 INFO 2981023 --- [prometheus-minute-schedule-1,9894a4e28f02af48,1e87e8690be11d9e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:29.403 INFO 2981023 --- [ocp-async-1,c837b1b0f5e096e4,929bc9d0bc6295eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:29.988 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:29.989 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:31.100 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5f9afabbb211e400,0b510c09ed764775] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5f9afabbb211e400, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:31.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5f9afabbb211e400,0b510c09ed764775] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5f9afabbb211e400, duration=11 ms] 2024-09-11 18:10:31.252 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,61357a4421e16fa1,fe1b5b85e07ea300] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=61357a4421e16fa1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:10:31.252 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,61357a4421e16fa1,fe1b5b85e07ea300] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=61357a4421e16fa1, duration=3 ms] 2024-09-11 18:10:31.266 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,598437fe57052f32,9623184a0c0db99d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=598437fe57052f32, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:10:31.266 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ddfe3d3b4db3cc06,708c3979399278a3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=ddfe3d3b4db3cc06, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:10:31.271 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,598437fe57052f32,9623184a0c0db99d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=598437fe57052f32, duration=7 ms] 2024-09-11 18:10:31.282 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,ddfe3d3b4db3cc06,708c3979399278a3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=ddfe3d3b4db3cc06, duration=18 ms] 2024-09-11 18:10:31.403 INFO 2981023 --- [ocp-async-3,a2609d14ec0f8d58,8f409b9cfd3fc4f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,1f0a08f28688d40c,61960c7f67f46b1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,1f0a08f28688d40c,61960c7f67f46b1d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:32.554 INFO 2981023 --- [ocp-schedules-12,326297eef24c0691,b2db15d19965f44a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:10:32.989 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:32.990 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:33.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2b7ca680893e2e0a,159742292b849cc5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2b7ca680893e2e0a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:33.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2b7ca680893e2e0a,159742292b849cc5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2b7ca680893e2e0a, duration=9 ms] 2024-09-11 18:10:33.404 INFO 2981023 --- [ocp-async-10,70fdb5a1722659dd,84e6bef2db86c81a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:35.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,4cb14f046c871436,4aa644a3e231c274] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4cb14f046c871436, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:35.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,4cb14f046c871436,4aa644a3e231c274] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4cb14f046c871436, duration=9 ms] 2024-09-11 18:10:35.251 INFO 2981023 --- [ocp-schedules-20,7404ebbdc91878eb,091b373bd29532c2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:10:35.259 INFO 2981023 --- [ocp-schedules-20,7404ebbdc91878eb,091b373bd29532c2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:10:35.259 INFO 2981023 --- [ocp-schedules-20,7404ebbdc91878eb,091b373bd29532c2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:10:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,5eeaa77a849552dd,84231aa29a43244e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,5eeaa77a849552dd,84231aa29a43244e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:35.404 INFO 2981023 --- [ocp-async-7,22a7a25a3e99a86d,846a20d796fe2905] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:35.990 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:35.990 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:36.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:10:36.352094+08:00, lastProcessTime=2024-09-11T18:10:26.352019+08:00 2024-09-11 18:10:36.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:10:36.352094+08:00 2024-09-11 18:10:37.098 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,fe1b5878549762a3,0a0a1ec03ae375e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fe1b5878549762a3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:37.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,fe1b5878549762a3,0a0a1ec03ae375e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fe1b5878549762a3, duration=10 ms] 2024-09-11 18:10:37.314 INFO 2981023 --- [prometheus-cache-monitor1,61754eee5fcf1a41,18c552448a6003c6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2050,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5149} 2024-09-11 18:10:37.315 INFO 2981023 --- [prometheus-cache-monitor1,61754eee5fcf1a41,18c552448a6003c6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:10:37.404 INFO 2981023 --- [ocp-async-9,ea26fd597b740892,1aa55462b7ea5399] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:37.553 INFO 2981023 --- [ocp-schedules-7,c1f6afbd55ee2137,6e9e57dc935d3bea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:10:37.611 WARN 2981023 --- [scanner-13619680-1,c1f6afbd55ee2137,df28c8ae40de48ff] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:10:37.627 INFO 2981023 --- [alarm-detect-15,c1f6afbd55ee2137,d6b9c85d10f35eda] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645616, data is empty. 2024-09-11 18:10:37.635 INFO 2981023 --- [alarm-detect-15,c1f6afbd55ee2137,d6b9c85d10f35eda] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645616, data is empty. 2024-09-11 18:10:37.639 INFO 2981023 --- [alarm-detect-15,c1f6afbd55ee2137,d6b9c85d10f35eda] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645616, data is empty. 2024-09-11 18:10:37.657 INFO 2981023 --- [alarm-detect-2,c1f6afbd55ee2137,3240af6e9f826595] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645616, data is empty. 2024-09-11 18:10:37.672 INFO 2981023 --- [alarm-detect-2,c1f6afbd55ee2137,3240af6e9f826595] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645616, data is empty. 2024-09-11 18:10:37.675 INFO 2981023 --- [alarm-detect-2,c1f6afbd55ee2137,3240af6e9f826595] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645616, data is empty. 2024-09-11 18:10:37.679 INFO 2981023 --- [alarm-detect-15,c1f6afbd55ee2137,9e8ba7448ec4d00d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645616, data is empty. 2024-09-11 18:10:37.682 INFO 2981023 --- [alarm-detect-15,c1f6afbd55ee2137,9e8ba7448ec4d00d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645616, data is empty. 2024-09-11 18:10:37.685 INFO 2981023 --- [alarm-detect-15,c1f6afbd55ee2137,9e8ba7448ec4d00d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645616, data is empty. 2024-09-11 18:10:37.785 INFO 2981023 --- [ocp-async-2,298b1c1c8490b405,a4166472d26aaf8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:10:37.795 INFO 2981023 --- [alarm-detect-1,c1f6afbd55ee2137,0bbbd6b136a50d70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645616, data is empty. 2024-09-11 18:10:37.798 INFO 2981023 --- [alarm-detect-1,c1f6afbd55ee2137,0bbbd6b136a50d70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645616, data is empty. 2024-09-11 18:10:37.802 INFO 2981023 --- [alarm-detect-1,c1f6afbd55ee2137,0bbbd6b136a50d70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645616, data is empty. 2024-09-11 18:10:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,6ecd679a62f248c8,4b94682fb21c0b94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,6ecd679a62f248c8,4b94682fb21c0b94] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:38.990 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:38.991 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:39.096 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4515ff84f417c196,9739be00d7a9bc22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4515ff84f417c196, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:39.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4515ff84f417c196,9739be00d7a9bc22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4515ff84f417c196, duration=9 ms] 2024-09-11 18:10:39.404 INFO 2981023 --- [ocp-async-6,44e0bb67db59a236,ffa246ff4bd443cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:40.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:10:40.003 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:10:41.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,bd902db37118aabe,10cabf94f23c65f3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bd902db37118aabe, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:41.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,bd902db37118aabe,10cabf94f23c65f3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bd902db37118aabe, duration=10 ms] 2024-09-11 18:10:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,756e301885ccdb2a,c8502a310905b65b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,756e301885ccdb2a,c8502a310905b65b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:41.404 INFO 2981023 --- [ocp-async-2,802a74e32fa54e0c,7a64d7da18c1e544] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:41.991 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:41.991 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:42.557 INFO 2981023 --- [ocp-schedules-15,61a0eabf761312f8,1ddfa6cd9a66822d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:10:43.095 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f163bd5650c7823a,191aea50feb1d32f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f163bd5650c7823a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:43.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f163bd5650c7823a,191aea50feb1d32f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f163bd5650c7823a, duration=9 ms] 2024-09-11 18:10:43.404 INFO 2981023 --- [ocp-async-5,c98653fc90650ed5,aa1573733d25af7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,1c352ac5aa20be5c,1616ca8a61db5e7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,1c352ac5aa20be5c,1616ca8a61db5e7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:44.991 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:44.992 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:45.096 INFO 2981023 --- [ocp-schedules-18,1d22d1a8499545dc,df3e6368e0b566a7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:10:45.096 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,70f923bc7e48081c,235f11c5684b1647] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=70f923bc7e48081c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:45.101 INFO 2981023 --- [prometheus-second-schedule-1,a501918fd58f7ed2,2453876dad681a36] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:10:45.103 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,70f923bc7e48081c,235f11c5684b1647] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=70f923bc7e48081c, duration=10 ms] 2024-09-11 18:10:45.123 WARN 2981023 --- [ocp-schedules-18,1d22d1a8499545dc,df3e6368e0b566a7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:10:45.404 INFO 2981023 --- [ocp-async-4,b1653a0ac6f68d3c,87a860440b9e62dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:46.026 INFO 2981023 --- [ocp-schedules-5,907512c53b8102f7,ca2bfdbc84d14b59] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:10:46.026 INFO 2981023 --- [ocp-schedules-30,d93ad9bdbf6ff12f,19b60d4632a3dc84] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:10:46.026 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:10:46.026 INFO 2981023 --- [host-check-clockdiff1,d93ad9bdbf6ff12f,8276b5f270d9ee4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:10:46.026 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:10:46.027 INFO 2981023 --- [ocp-schedules-26,33397809e3e5095d,a5348f0804603c72] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:10:46.027 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:10:46.027 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:10:46.028 INFO 2981023 --- [host-check-clockdiff4,d93ad9bdbf6ff12f,887965532531cca9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:10:46.028 INFO 2981023 --- [host-check-agent-status2,33397809e3e5095d,0b5e7a9c9780a43a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:10:46.029 INFO 2981023 --- [host-check-agent-status1,33397809e3e5095d,0c4906345ac6fe29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:10:46.030 INFO 2981023 --- [host-check-clockdiff2,d93ad9bdbf6ff12f,88ecd1044da99fc6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:10:46.040 INFO 2981023 --- [host-check-clockdiff3,d93ad9bdbf6ff12f,e051ab634ca6bb51] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:10:46.041 INFO 2981023 --- [host-check-agent-status3,33397809e3e5095d,38eaca9092358659] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:10:46.041 INFO 2981023 --- [host-check-agent-status4,33397809e3e5095d,85599da77b3272ca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:10:46.050 INFO 2981023 --- [host-check-clockdiff2,d93ad9bdbf6ff12f,88ecd1044da99fc6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049446 6 6 , error= 2024-09-11 18:10:46.066 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:10:46.068 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:10:46.068 INFO 2981023 --- [host-check-agent-status1,33397809e3e5095d,0c4906345ac6fe29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:10:46.070 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:10:46.071 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:10:46.080 INFO 2981023 --- [host-check-agent-status2,33397809e3e5095d,0b5e7a9c9780a43a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:10:46.087 INFO 2981023 --- [host-check-agent-status1,33397809e3e5095d,0c4906345ac6fe29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:10:46.091 INFO 2981023 --- [host-check-clockdiff3,d93ad9bdbf6ff12f,e051ab634ca6bb51] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049446 0 0 , error= 2024-09-11 18:10:46.110 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:10:46.111 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:10:46.113 INFO 2981023 --- [host-check-agent-status2,33397809e3e5095d,0b5e7a9c9780a43a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:10:46.119 INFO 2981023 --- [host-check-agent-status3,33397809e3e5095d,38eaca9092358659] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:10:46.155 INFO 2981023 --- [host-check-agent-status3,33397809e3e5095d,38eaca9092358659] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:10:46.163 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:10:46.187 INFO 2981023 --- [ocp-schedules-27,fc9946b623f8fe02,43c0a8eb6a6bc7a7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:10:46.187 INFO 2981023 --- [ocp-schedules-10,5f165d2ce5ff0f4a,70bc9d7a7cffe9cf] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:10:46.188 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:10:46.189 INFO 2981023 --- [ocp-schedules-23,059e25a45f95e816,6904a2c9d090e799] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:10:46.194 INFO 2981023 --- [ocp-schedules-18,05740c2103dda693,5e1f8a49fb25e53a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:10:46.199 INFO 2981023 --- [ocp-schedules-23,059e25a45f95e816,6904a2c9d090e799] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:10:46.266 INFO 2981023 --- [ocp-schedules-17,daeef924162c808d,0521e79c877af7fd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:10:46.266 INFO 2981023 --- [ocp-schedules-27,fc9946b623f8fe02,43c0a8eb6a6bc7a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:10:46.267 INFO 2981023 --- [ocp-schedules-10,5f165d2ce5ff0f4a,70bc9d7a7cffe9cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:10:46.267 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:10:46.269 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049402864000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:10:46.269 INFO 2981023 --- [ocp-schedules-10,5f165d2ce5ff0f4a,70bc9d7a7cffe9cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:10:46.273 INFO 2981023 --- [host-check-agent-status4,33397809e3e5095d,85599da77b3272ca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:10:46.284 INFO 2981023 --- [host-check-agent-status4,33397809e3e5095d,85599da77b3272ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:10:46.322 INFO 2981023 --- [ocp-schedules-17,daeef924162c808d,0521e79c877af7fd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:10:46.357 INFO 2981023 --- [ocp-schedules-11,a4aced63c259cfa5,77d6bd1aa83e9173] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:10:46.358 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:10:46.376 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:10:46.379 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:10:46.378268295+08:00 2024-09-11 18:10:46.383 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:10:46.379+08:00 2024-09-11 18:10:46.385 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:10:46.379+08:00 and status AVAILABLE 2024-09-11 18:10:46.386 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:10:46.388 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:10:46.388137144+08:00 2024-09-11 18:10:46.389 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:10:46.388+08:00 2024-09-11 18:10:46.391 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:10:46.391 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:10:46.388+08:00 and status AVAILABLE 2024-09-11 18:10:46.393 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:10:46.390794+08:00, lastProcessTime=2024-09-11T18:10:36.352094+08:00 2024-09-11 18:10:46.393 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:10:46.398647894+08:00 2024-09-11 18:10:46.394 INFO 2981023 --- [host-check-agent-status1,33397809e3e5095d,0c4906345ac6fe29] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:10:46.394 INFO 2981023 --- [host-check-agent-status3,33397809e3e5095d,38eaca9092358659] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:10:46.394 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:10:46.393+08:00 2024-09-11 18:10:46.395 INFO 2981023 --- [host-check-agent-status4,33397809e3e5095d,85599da77b3272ca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:10:46.395 INFO 2981023 --- [host-check-agent-status2,33397809e3e5095d,0b5e7a9c9780a43a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:10:46.396 INFO 2981023 --- [host-check-agent-status1,33397809e3e5095d,0c4906345ac6fe29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:10:46.396 INFO 2981023 --- [host-check-agent-status1,33397809e3e5095d,0c4906345ac6fe29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:10:46.396 INFO 2981023 --- [host-check-agent-status1,33397809e3e5095d,0c4906345ac6fe29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:10:46.397 INFO 2981023 --- [host-check-agent-status3,33397809e3e5095d,38eaca9092358659] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:10:46.397 INFO 2981023 --- [host-check-agent-status3,33397809e3e5095d,38eaca9092358659] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:10:46.397 INFO 2981023 --- [host-check-agent-status3,33397809e3e5095d,38eaca9092358659] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:10:46.398 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:10:46.393+08:00 and status AVAILABLE 2024-09-11 18:10:46.399 INFO 2981023 --- [host-check-agent-status4,33397809e3e5095d,85599da77b3272ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:10:46.399 INFO 2981023 --- [host-check-agent-status4,33397809e3e5095d,85599da77b3272ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:10:46.399 INFO 2981023 --- [host-check-agent-status4,33397809e3e5095d,85599da77b3272ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:10:46.399 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:10:46.399 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:10:46.400 INFO 2981023 --- [host-check-agent-status2,33397809e3e5095d,0b5e7a9c9780a43a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:10:46.400 INFO 2981023 --- [host-check-agent-status2,33397809e3e5095d,0b5e7a9c9780a43a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:10:46.400 INFO 2981023 --- [host-check-agent-status2,33397809e3e5095d,0b5e7a9c9780a43a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:10:46.400 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:10:46.405659999+08:00 2024-09-11 18:10:46.401 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:10:46.400+08:00 2024-09-11 18:10:46.402 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:10:46.400+08:00 and status AVAILABLE 2024-09-11 18:10:46.405 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:10:46.412 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:10:46.458 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:10:46.461 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.462 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:10:46.463 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:10:46.463 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:10:46.465 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:10:46.467 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.467 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:10:46.468 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:10:46.468 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:10:46.469 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:10:46.469 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:10:46.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.470 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:46.470 INFO 2981023 --- [ocp-schedules-27,fc9946b623f8fe02,43c0a8eb6a6bc7a7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:10:46.470 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:10:46.472 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:10:46.474 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:10:46.476 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.477 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:46.478 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.478 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.479 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:10:46.479 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:46.479 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:10:46.480 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.488 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:10:46.488 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:10:46.495 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:10:46.509 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:10:46.513 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:10:46.524 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:10:46.525 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:10:46.527 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.528 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.528 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:10:46.535 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:10:46.540 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:10:46.554 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:10:46.567 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:10:46.574 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.575 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.575 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:10:46.576 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:10:46.577 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.577 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:10:46.578 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:10:46.604 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:10:46.604 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049402864000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:10:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:46.606 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:10:46.606 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.609 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:10:46.610 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:10:46.612 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:10:46.612 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:10:46.615 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:10:46.616 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:10:46.620 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:10:46.623 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:10:46.626 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:10:46.626 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:10:46.634 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:10:46.641 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:10:46.647 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:10:46.656 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.659 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:10:46.659 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:10:46.662 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:10:46.664 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:10:46.667 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.668 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.669 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:10:46.669 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:10:46.670 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.670 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:10:46.671 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:10:46.672 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.678 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:10:46.680 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:10:46.682 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:10:46.682 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.682 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.682 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:10:46.683 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:10:46.686 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:10:46.687 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:10:46.690 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.691 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.691 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:10:46.692 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:10:46.694 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:10:46.694 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:10:46.701 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:10:46.702 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:10:46.705 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:10:46.711 INFO 2981023 --- [host-check-host-status4,907512c53b8102f7,7895dc4ab006c973] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:10:46.718 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:10:46.719 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:10:46.722 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.723 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.723 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:10:46.724 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:10:46.724 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.727 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:10:46.728 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:10:46.729 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:10:46.731 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:10:46.734 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.735 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.736 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:10:46.737 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:10:46.737 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:10:46.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:10:46.739 INFO 2981023 --- [host-check-host-status3,907512c53b8102f7,523ca97125778047] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:10:46.742 INFO 2981023 --- [host-check-host-status1,907512c53b8102f7,8604084316a63e61] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:10:46.749 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:10:46.754 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:10:46.755 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:10:46.759 INFO 2981023 --- [host-check-host-status2,907512c53b8102f7,e9d4da236fe68bac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:10:46.774 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:10:46.783 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.784 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.785 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:10:46.785 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:10:46.789 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:10:46.793 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:10:46.795 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.795 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:10:46.796 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:10:46.798 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:10:46.798 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.822 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:10:46.836 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:10:46.390794+08:00 2024-09-11 18:10:46.839 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:10:46.847 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.848 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.848 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:10:46.850 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:10:46.856 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:10:46.857 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:10:46.858 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:10:46.858 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:46.866 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:10:46.871 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:10:46.871 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:10:46.872 INFO 2981023 --- [ocp-schedules-23,059e25a45f95e816,6904a2c9d090e799] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:10:46.874 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:10:46.880 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:10:46.882 INFO 2981023 --- [ocp-schedules-4,15a5b1096857e038,afd88350d4b3482d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:10:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,44cac8323d067f77,ad14a4148e85868a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,44cac8323d067f77,ad14a4148e85868a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:47.404 INFO 2981023 --- [ocp-async-2,2fc6ad5e5f089f16,8e98065437ca5e14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:47.784 INFO 2981023 --- [ocp-async-8,624befed03774341,e83b226f2915cfb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:10:47.823 INFO 2981023 --- [host-check-clockdiff1,d93ad9bdbf6ff12f,8276b5f270d9ee4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049447 6 6 , error= 2024-09-11 18:10:47.850 INFO 2981023 --- [ocp-schedules-4,140336a6439b841e,e92280a189e06584] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:10:47.894 WARN 2981023 --- [scanner-13619960-1,140336a6439b841e,c79ea4722453ba88] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:10:47.938 INFO 2981023 --- [alarm-detect-19,140336a6439b841e,327d251cb0176a37] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726049415 ~ 1726049437 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726049430, value=1.697)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.697, values={1726049430=1.697}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:10:47.992 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:47.994 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:49.259 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,9f1aee64b0ae7752,26e3a38fc3fc0b98] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9f1aee64b0ae7752, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:49.277 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,9f1aee64b0ae7752,26e3a38fc3fc0b98] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9f1aee64b0ae7752, duration=23 ms] 2024-09-11 18:10:49.404 INFO 2981023 --- [ocp-async-3,869decb86e9c98ae,51fa8e7ea9432def] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:10:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:10:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,9a1bc63bc9ba9530,0585826adf626712] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,9a1bc63bc9ba9530,0585826adf626712] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:50.329 INFO 2981023 --- [ocp-schedules-23,c15a0e26174d250f,18a9113865e4fff2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:10:50.994 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:50.996 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:51.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,d86677f135f4361f,d30b59f4ad35ea14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d86677f135f4361f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:51.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,d86677f135f4361f,d30b59f4ad35ea14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d86677f135f4361f, duration=9 ms] 2024-09-11 18:10:51.405 INFO 2981023 --- [ocp-async-4,8733684811d2ed49,b8a2d1f67d6f7d09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:51.609 INFO 2981023 --- [ocp-schedules-24,82bb60f4d2272582,6bfc8bb85c3d5398] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:10:51.610 INFO 2981023 --- [ocp-schedules-24,82bb60f4d2272582,6bfc8bb85c3d5398] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:10:51.610 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:10:51.610 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:10:51.610 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:10:51.610 INFO 2981023 --- [ocp-schedules-24,82bb60f4d2272582,6bfc8bb85c3d5398] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:10:52.003 INFO 2981023 --- [host-check-clockdiff4,d93ad9bdbf6ff12f,887965532531cca9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049451 0 0 , error= 2024-09-11 18:10:52.559 INFO 2981023 --- [ocp-schedules-8,5c1d154ba62554a7,85a6c14ed50669de] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:10:53.098 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,9feac4710294fd95,6dcbef0588e104b8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9feac4710294fd95, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:53.104 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,9feac4710294fd95,6dcbef0588e104b8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9feac4710294fd95, duration=9 ms] 2024-09-11 18:10:53.290 INFO 2981023 --- [ocp-schedules-17,d1268e2de89e92de,6d9b09f431e48fa8] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:10:53.290 INFO 2981023 --- [ocp-schedules-17,d1268e2de89e92de,6d9b09f431e48fa8] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:10:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,1f8a53c3d17556ff,560c0b2be85b7733] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,1f8a53c3d17556ff,560c0b2be85b7733] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:53.405 INFO 2981023 --- [ocp-async-7,fb5a76a70d255700,12565606bb7ab020] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:53.996 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:53.997 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:54.421 INFO 2981023 --- [ocp-schedules-2,9a9e9c85022dbf94,9fc4751ae442e183] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:10:54.602 INFO 2981023 --- [ocp-schedules-14,23ccda531c290588,e1baf2dfea139fb9] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:10:54.620 WARN 2981023 --- [ocp-schedules-14,23ccda531c290588,e1baf2dfea139fb9] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:10:55.097 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,fef6e9782f5e409c,aa6bce7da41b9425] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=fef6e9782f5e409c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:55.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,fef6e9782f5e409c,aa6bce7da41b9425] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=fef6e9782f5e409c, duration=15 ms] 2024-09-11 18:10:55.262 INFO 2981023 --- [ocp-schedules-11,47d1b5d24c4ad961,e3f36911ecd8e436] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:10:55.271 INFO 2981023 --- [ocp-schedules-11,47d1b5d24c4ad961,e3f36911ecd8e436] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:10:55.271 INFO 2981023 --- [ocp-schedules-11,47d1b5d24c4ad961,e3f36911ecd8e436] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:10:55.405 INFO 2981023 --- [ocp-async-9,73084166de768152,afac3fd7284491b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:55.861 INFO 2981023 --- [ocp-schedules-27,03ae4bb3c24eb617,c569eba939ea92ac] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:10:55.861 INFO 2981023 --- [ocp-schedules-27,03ae4bb3c24eb617,c569eba939ea92ac] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:10:55.862 INFO 2981023 --- [ocp-schedules-27,03ae4bb3c24eb617,c569eba939ea92ac] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:10:55.862 INFO 2981023 --- [ocp-schedules-27,03ae4bb3c24eb617,c569eba939ea92ac] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:10:55.862 INFO 2981023 --- [ocp-schedules-27,03ae4bb3c24eb617,c569eba939ea92ac] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:10:55.862 INFO 2981023 --- [ocp-schedules-27,03ae4bb3c24eb617,c569eba939ea92ac] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:10:56.031 INFO 2981023 --- [ocp-schedules-2,eb70ca74da8b55a1,659431394fbb4d71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:10:56.037 INFO 2981023 --- [ocp-schedules-2,eb70ca74da8b55a1,659431394fbb4d71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:10:56.037 INFO 2981023 --- [ocp-schedules-2,eb70ca74da8b55a1,659431394fbb4d71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:10:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,c56fb448ff337cdd,58786f550edc3731] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,c56fb448ff337cdd,58786f550edc3731] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:10:56.351938+08:00, lastProcessTime=2024-09-11T18:10:46.390794+08:00 2024-09-11 18:10:56.496 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:10:56.351938+08:00 2024-09-11 18:10:56.997 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:56.998 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:10:57.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,41da3316c7c128d0,95d46bec520f1f73] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=41da3316c7c128d0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:57.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,41da3316c7c128d0,95d46bec520f1f73] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=41da3316c7c128d0, duration=9 ms] 2024-09-11 18:10:57.405 INFO 2981023 --- [ocp-async-8,79f523761cee96d5,e89d8d853bd75fa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:57.784 INFO 2981023 --- [ocp-async-5,2ced438c0026c6e1,1061b22ccdf8bcf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:10:57.808 INFO 2981023 --- [ocp-schedules-6,ff08f7cfb18adb1a,cbb7a147c9b6ff62] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:10:57.808 INFO 2981023 --- [ocp-schedules-6,ff08f7cfb18adb1a,cbb7a147c9b6ff62] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:10:57.839 INFO 2981023 --- [Collect all tenants major compactions1,ff08f7cfb18adb1a,2e4d1ad0e64145bb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:10:57.840 INFO 2981023 --- [Collect all tenants major compactions3,ff08f7cfb18adb1a,496fa35f563d1db5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:10:57.840 INFO 2981023 --- [Collect all tenants major compactions2,ff08f7cfb18adb1a,a43f0e1d38e12e2f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:10:57.846 INFO 2981023 --- [Collect all tenants major compactions4,ff08f7cfb18adb1a,ff82343700a60947] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:10:57.848 INFO 2981023 --- [Collect all tenants major compactions6,ff08f7cfb18adb1a,bf4fc6356cd7835a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:10:57.849 INFO 2981023 --- [Collect all tenants major compactions5,ff08f7cfb18adb1a,5c7002301586bb3b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:10:57.868 INFO 2981023 --- [Collect all tenants major compactions3,ff08f7cfb18adb1a,496fa35f563d1db5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:57.868 INFO 2981023 --- [Collect all tenants major compactions2,ff08f7cfb18adb1a,a43f0e1d38e12e2f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:57.869 INFO 2981023 --- [Collect all tenants major compactions2,ff08f7cfb18adb1a,a43f0e1d38e12e2f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:10:57.869 INFO 2981023 --- [Collect all tenants major compactions3,ff08f7cfb18adb1a,496fa35f563d1db5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:10:57.870 INFO 2981023 --- [Collect all tenants major compactions1,ff08f7cfb18adb1a,2e4d1ad0e64145bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:57.870 INFO 2981023 --- [Collect all tenants major compactions5,ff08f7cfb18adb1a,5c7002301586bb3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:57.871 INFO 2981023 --- [Collect all tenants major compactions1,ff08f7cfb18adb1a,2e4d1ad0e64145bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:10:57.872 INFO 2981023 --- [Collect all tenants major compactions5,ff08f7cfb18adb1a,5c7002301586bb3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:10:57.872 INFO 2981023 --- [Collect all tenants major compactions4,ff08f7cfb18adb1a,ff82343700a60947] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:57.872 INFO 2981023 --- [Collect all tenants major compactions2,ff08f7cfb18adb1a,a43f0e1d38e12e2f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:10:57.872 INFO 2981023 --- [Collect all tenants major compactions2,ff08f7cfb18adb1a,a43f0e1d38e12e2f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:10:57.872 INFO 2981023 --- [Collect all tenants major compactions3,ff08f7cfb18adb1a,496fa35f563d1db5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:10:57.872 INFO 2981023 --- [Collect all tenants major compactions3,ff08f7cfb18adb1a,496fa35f563d1db5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:10:57.873 INFO 2981023 --- [Collect all tenants major compactions5,ff08f7cfb18adb1a,5c7002301586bb3b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:10:57.873 INFO 2981023 --- [Collect all tenants major compactions5,ff08f7cfb18adb1a,5c7002301586bb3b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:10:57.873 INFO 2981023 --- [Collect all tenants major compactions4,ff08f7cfb18adb1a,ff82343700a60947] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:10:57.874 INFO 2981023 --- [Collect all tenants major compactions1,ff08f7cfb18adb1a,2e4d1ad0e64145bb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:10:57.874 INFO 2981023 --- [Collect all tenants major compactions1,ff08f7cfb18adb1a,2e4d1ad0e64145bb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:10:57.875 INFO 2981023 --- [Collect all tenants major compactions4,ff08f7cfb18adb1a,ff82343700a60947] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:10:57.875 INFO 2981023 --- [Collect all tenants major compactions4,ff08f7cfb18adb1a,ff82343700a60947] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:10:57.875 INFO 2981023 --- [Collect all tenants major compactions6,ff08f7cfb18adb1a,bf4fc6356cd7835a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:57.876 INFO 2981023 --- [Collect all tenants major compactions6,ff08f7cfb18adb1a,bf4fc6356cd7835a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:10:57.878 INFO 2981023 --- [Collect all tenants major compactions6,ff08f7cfb18adb1a,bf4fc6356cd7835a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:10:57.878 INFO 2981023 --- [Collect all tenants major compactions6,ff08f7cfb18adb1a,bf4fc6356cd7835a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:10:57.878 INFO 2981023 --- [ocp-schedules-6,ff08f7cfb18adb1a,cbb7a147c9b6ff62] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:10:58.033 INFO 2981023 --- [ocp-schedules-8,a258d38c3d7a41f6,f7fde1d89f5cd22c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:10:58.079 WARN 2981023 --- [scanner-13620214-1,a258d38c3d7a41f6,eb756bcf084acb01] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:10:58.104 INFO 2981023 --- [alarm-detect-14,a258d38c3d7a41f6,60ef2574ac39e984] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645616, data is empty. 2024-09-11 18:10:58.108 INFO 2981023 --- [alarm-detect-14,a258d38c3d7a41f6,60ef2574ac39e984] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645616, data is empty. 2024-09-11 18:10:58.113 INFO 2981023 --- [alarm-detect-14,a258d38c3d7a41f6,60ef2574ac39e984] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645616, data is empty. 2024-09-11 18:10:58.123 INFO 2981023 --- [alarm-detect-14,a258d38c3d7a41f6,981fd8d4011f0155] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645616, data is empty. 2024-09-11 18:10:58.127 INFO 2981023 --- [alarm-detect-14,a258d38c3d7a41f6,981fd8d4011f0155] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645616, data is empty. 2024-09-11 18:10:58.128 INFO 2981023 --- [alarm-detect-8,a258d38c3d7a41f6,0c180dd601dd2010] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645616, data is empty. 2024-09-11 18:10:58.135 INFO 2981023 --- [alarm-detect-14,a258d38c3d7a41f6,981fd8d4011f0155] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645616, data is empty. 2024-09-11 18:10:58.136 INFO 2981023 --- [alarm-detect-8,a258d38c3d7a41f6,0c180dd601dd2010] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645616, data is empty. 2024-09-11 18:10:58.139 INFO 2981023 --- [alarm-detect-8,a258d38c3d7a41f6,0c180dd601dd2010] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645616, data is empty. 2024-09-11 18:10:58.170 INFO 2981023 --- [ocp-schedules-19,774a1e74c3b74dc9,f0541bb8274121c0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:10:58.170+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:10:58.180 INFO 2981023 --- [ocp-schedules-19,774a1e74c3b74dc9,f0541bb8274121c0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:10:58.253 INFO 2981023 --- [alarm-detect-10,a258d38c3d7a41f6,2b1a8ca65376d272] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645616, data is empty. 2024-09-11 18:10:58.257 INFO 2981023 --- [alarm-detect-10,a258d38c3d7a41f6,2b1a8ca65376d272] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645616, data is empty. 2024-09-11 18:10:58.273 INFO 2981023 --- [alarm-detect-10,a258d38c3d7a41f6,2b1a8ca65376d272] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645616, data is empty. 2024-09-11 18:10:58.352 INFO 2981023 --- [ocp-schedules-19,774a1e74c3b74dc9,f0541bb8274121c0] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 39 to SqlAggregateData 2024-09-11 18:10:58.353 INFO 2981023 --- [ocp-schedules-19,774a1e74c3b74dc9,f0541bb8274121c0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:10:59.086 INFO 2981023 --- [ocp-schedules-14,4b246b3ad1695d00,878023e27f246807] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:10:59.087 INFO 2981023 --- [ocp-schedules-14,4b246b3ad1695d00,878023e27f246807] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:10:59.116 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:10:59.116 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:10:59.116 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.116 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.119 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:10:59.119 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.120 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:10:59.120 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:10:59.122 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:10:59.124 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.124 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.124 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.125 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.126 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.126 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.129 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.130 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.135 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.135 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:10:59.136 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.137 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.140 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.140 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.142 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.142 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:10:59.147 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.148 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.170 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:10:59.170 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4F51D782-0-0] 2024-09-11 18:10:59.170 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:10:59.170 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.171 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:10:59.174 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.174 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.174 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:10:59.174 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.182 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.183 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:10:59.184 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.185 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:10:59.185 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.186 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:10:59.187 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.188 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:10:59.188 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.189 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:10:59.189 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:10:59.194 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:10:59.198 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.198 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:10:59.198 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.199 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:10:59.199 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:10:59.199 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE4F51D78C-0-0] 2024-09-11 18:10:59.199 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:10:59.200 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.200 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:10:59.201 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:10:59.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.203 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:10:59.203 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.204 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:10:59.204 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.205 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:10:59.206 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.206 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.206 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:10:59.207 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.207 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:10:59.208 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:10:59.208 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.208 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.209 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:10:59.209 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:10:59.209 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:10:59.209 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.210 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:10:59.211 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.211 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:59.211 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:10:59.211 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:10:59.212 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.213 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:10:59.213 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.213 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.213 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.214 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:10:59.214 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:10:59.216 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:10:59.224 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:59.226 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.226 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:10:59.229 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:10:59.229 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:10:59.230 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.231 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:10:59.232 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:10:59.232 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:10:59.232 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:10:59.233 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:10:59.235 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:10:59.235 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.236 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:10:59.242 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:10:59.242 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:10:59.243 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:10:59.244 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.244 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:10:59.244 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.244 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:10:59.245 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:10:59.249 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:10:59.251 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:10:59.251 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:10:59.251 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:10:59.256 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:10:59.258 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:10:59.258 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:10:59.258 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:10:59.258 INFO 2981023 --- [ocp-schedules-14,4b246b3ad1695d00,878023e27f246807] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:10:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,d513bb6c023cdc92,7689f943e5e50708] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049460, interval=60 2024-09-11 18:10:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,d513bb6c023cdc92,7689f943e5e50708] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:10:59.405 INFO 2981023 --- [ocp-async-1,0b7b0d55ecccb111,7099a9d0782e4e61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:10:59.722 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,22cb49d27b029bd0,2461306a27696fdf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=22cb49d27b029bd0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:10:59.728 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,22cb49d27b029bd0,2461306a27696fdf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=22cb49d27b029bd0, duration=9 ms] 2024-09-11 18:10:59.998 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:10:59.999 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:01.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,cc90e29ba0e7bf59,598f63993ca9dc43] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cc90e29ba0e7bf59, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:01.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,cc90e29ba0e7bf59,598f63993ca9dc43] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cc90e29ba0e7bf59, duration=14 ms] 2024-09-11 18:11:01.405 INFO 2981023 --- [ocp-async-5,2d0b04422aadc40c,b2e9f91a1023e52b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,6092ecc9fb3c2b01,6549003024580c33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,6092ecc9fb3c2b01,6549003024580c33] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:02.562 INFO 2981023 --- [ocp-schedules-18,baf595cb8b6cdea6,f27649c6322b3856] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:11:02.999 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:02.999 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:03.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1467d127ee9b6b28,fc2ebca09644faaa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1467d127ee9b6b28, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:03.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1467d127ee9b6b28,fc2ebca09644faaa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1467d127ee9b6b28, duration=9 ms] 2024-09-11 18:11:03.405 INFO 2981023 --- [ocp-async-4,dff76d54a8e65b6e,49744cdad101a201] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:03.729 INFO 2981023 --- [ocp-schedules-14,2e743b97819fb8ab,41ed60f9424e3a44] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:11:05.105 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c6292aa961e6cbe0,3be93c70527f633e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=c6292aa961e6cbe0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:05.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c6292aa961e6cbe0,3be93c70527f633e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=c6292aa961e6cbe0, duration=9 ms] 2024-09-11 18:11:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f1cd1bc881ea0dad,e44f5a5065da1684] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,f1cd1bc881ea0dad,e44f5a5065da1684] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:05.405 INFO 2981023 --- [ocp-async-2,671c1fcd16684591,c602d9600614f24a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:05.999 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:06.000 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:11:06.337531+08:00, lastProcessTime=2024-09-11T18:10:56.351938+08:00 2024-09-11 18:11:06.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:11:06.337531+08:00 2024-09-11 18:11:07.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2f4c54c5dd833d31,3f01b7c945a4e310] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2f4c54c5dd833d31, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:07.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,2f4c54c5dd833d31,3f01b7c945a4e310] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2f4c54c5dd833d31, duration=9 ms] 2024-09-11 18:11:07.406 INFO 2981023 --- [ocp-async-1,18bb068267a9682f,b806d7160d6d2562] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:07.784 INFO 2981023 --- [ocp-async-7,f89517c9a0845b6d,922450a713c6d860] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:11:07.893 INFO 2981023 --- [ocp-schedules-2,39362ee033817b51,b157fc6ea6a984ab] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:11:08.007 INFO 2981023 --- [ocp-schedules-10,e23c28891befbeff,3e28c12049845ee2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:11:08.007+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:11:08.008 INFO 2981023 --- [ocp-schedules-10,e23c28891befbeff,3e28c12049845ee2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:11:08.009 INFO 2981023 --- [ocp-schedules-10,e23c28891befbeff,3e28c12049845ee2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:11:08.010 INFO 2981023 --- [ocp-schedules-10,e23c28891befbeff,3e28c12049845ee2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:11:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,98471c29c4dc4dbc,e49a9cbbbf7554b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,98471c29c4dc4dbc,e49a9cbbbf7554b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:08.321 INFO 2981023 --- [ocp-schedules-20,432d37c0ef49b9ac,ee629ff3c48a92b1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:11:08.367 WARN 2981023 --- [scanner-13620498-1,432d37c0ef49b9ac,f64d9f2f5959b457] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:11:09.000 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:09.001 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:09.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,cb13abda693d1f78,cb384e48af8c49c8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=cb13abda693d1f78, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:09.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,cb13abda693d1f78,cb384e48af8c49c8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=cb13abda693d1f78, duration=9 ms] 2024-09-11 18:11:09.406 INFO 2981023 --- [ocp-async-10,ac6c96156540c189,5bb75861302c785a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:09.450 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:07:09.039, end=2024-09-11T18:09:09.446} 2024-09-11 18:11:09.454 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdo 2024-09-11 18:11:09.454 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:11:09.457 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:09.039, end=2024-09-11T18:09:09.446},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:11:09.457 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:11:09.457 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:11:09.463 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:11:09.463 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:09.039, end=2024-09-11T18:09:09.446},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:11:09.463 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:11:09.463 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:11:09.464 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:11:09.466 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:09.039, end=2024-09-11T18:09:09.446},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:11:09.466 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:11:09.466 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:11:09.487 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:11:09.487 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:11:09.489 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:09.039, end=2024-09-11T18:09:09.446},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:11:09.489 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:11:09.489 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:11:09.540 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069540000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.540Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.542 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069542000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.542Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.543 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069543000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.543Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.548 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069548000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.548Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.550 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069550000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.550Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.551 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069551000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.551Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.551 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069551000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.551Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.556 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069556000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.556Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.562 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.562 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:11:09.562 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.562 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.568 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 67 2024-09-11 18:11:09.568 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.572 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:11:09.572 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.574 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.574 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.575 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.575 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.579 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.579 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.581 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.581 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.584 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.584 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.586 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.586 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.586 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.586 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.587 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.587 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.589 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.589 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.590 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:11:09.590 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.591 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.591 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.592 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.592 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.593 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.593 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.593 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.593 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.597 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.597 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.600 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.600 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.600 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.600 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.602 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.602 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.602 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.602 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.604 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.605 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.605 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.607 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.607 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.608 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.608 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.608 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.608 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.609 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.610 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.610 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.611 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.611 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.612 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.612 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.614 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.614 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.614 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.54, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.615 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.616 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.542, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.616 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.617 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.548, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.618 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.618 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.620 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.55, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.622 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.543, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.622 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.623 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.623 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.627 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.551, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.628 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.631 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.551, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.636 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.556, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.653 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069653000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.653Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.657 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069657000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.657Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.659 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069659000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.659Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.661 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069661000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.661Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.668 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:11:09.668 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.668 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069668000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.668Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.669 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069669000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.669Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.672 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.672 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.673 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.673 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.674 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069674000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.674Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.679 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.679 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.685 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:11:09.685 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.685 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.685 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.686 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.686 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.687 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069687000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1ED31157F7096F47941EA2233AA4101C, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.687Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.690 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.690 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.692 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.692 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.694 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.694 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.695 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.695 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.698 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.698 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.698 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.698 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.699 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.699 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.699 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.699 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.702 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.702 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.703 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.703 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.704 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.706 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.706 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.709 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.709 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.710 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.710 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.711 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.711 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.711 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.711 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.713 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.713 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.713 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.713 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.713 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.715 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.715 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.716 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.653, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.716 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.716 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.717 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.717 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.718 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.659, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.719 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.721 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.721 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.724 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.724 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.661, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.726 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.726 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.726 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.726 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.727 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.727 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.727 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.728 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.730 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.730 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.657, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.730 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.730 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.732 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.734 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.668, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.734 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.687, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=1ED31157F7096F47941EA2233AA4101C, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.735 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.669, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.736 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.674, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.740 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069740000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.740Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.746 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:11:09.746 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.751 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.751 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.752 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.752 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.757 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.757 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.758 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:11:09.761 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.74, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.761 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:11:09.762 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:09.039, end=2024-09-11T18:09:09.446},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:11:09.762 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:11:09.762 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:11:09.763 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:11:09.764 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:09.039, end=2024-09-11T18:09:09.446},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:11:09.764 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:11:09.764 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:11:09.773 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:11:09.788 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963069788000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:11:09.788Z, endTime=2024-09-11T10:09:09.446Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:11:09.789 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:11:09.789 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:11:09.795 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.795 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.796 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:11:09.796 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:11:09.801 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:11:09.801 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049349446000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:11:09.802 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:11:09.802 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:11:09.788, end=2024-09-11T18:09:09.446}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:11:09.802 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:11:10.004 INFO 2981023 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:11:10.004 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:11:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:11:10.922 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:11:11.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,520817c8037f48db,22178ea8fbecf92b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=520817c8037f48db, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:11.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,520817c8037f48db,22178ea8fbecf92b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=520817c8037f48db, duration=9 ms] 2024-09-11 18:11:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,86032fc52ba7a614,375787f35f34f982] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,86032fc52ba7a614,375787f35f34f982] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:11.406 INFO 2981023 --- [ocp-async-7,5a73e13aa901034c,c2b89ee51fa2290f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:12.001 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:12.002 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:12.112 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049400, nodeCount=1000, affectRows=1000 2024-09-11 18:11:12.114 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049400, nodeCount=1000, validCount=1000 2024-09-11 18:11:12.121 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049400, nodeCount=1000, affectRows=1000 2024-09-11 18:11:12.123 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049400, nodeCount=999, validCount=999 2024-09-11 18:11:12.126 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049400, nodeCount=50, affectRows=50 2024-09-11 18:11:12.126 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049400, nodeCount=48, validCount=48 2024-09-11 18:11:12.565 INFO 2981023 --- [ocp-schedules-1,344e74bb72f30e5e,3d94dba35327ca04] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:11:13.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8698dfc214cfc96c,df6ae4717d76d384] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=8698dfc214cfc96c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:13.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8698dfc214cfc96c,df6ae4717d76d384] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=8698dfc214cfc96c, duration=9 ms] 2024-09-11 18:11:13.406 INFO 2981023 --- [ocp-async-3,f47fa8d25d2263af,4a14a27e36efdacc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,c5a58cfcf421811e,f8e1336abbc0fdb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,b70701c4a1b03800,0c2b29dd150848ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:11:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,c5a58cfcf421811e,f8e1336abbc0fdb4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:15.002 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:15.003 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:15.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f6fe5b2c61fa3bb9,e9f6d9c7e9cbb774] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f6fe5b2c61fa3bb9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:15.131 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f6fe5b2c61fa3bb9,e9f6d9c7e9cbb774] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f6fe5b2c61fa3bb9, duration=11 ms] 2024-09-11 18:11:15.273 INFO 2981023 --- [ocp-schedules-28,b3b4d6f52f552c01,00a4b29276b305c0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:11:15.278 INFO 2981023 --- [ocp-schedules-28,b3b4d6f52f552c01,00a4b29276b305c0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:11:15.278 INFO 2981023 --- [ocp-schedules-28,b3b4d6f52f552c01,00a4b29276b305c0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:11:15.352 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049400, nodeCount=480, affectRows=480 2024-09-11 18:11:15.406 INFO 2981023 --- [ocp-async-5,58f4d587b488921a,9e7c595b4f8928b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:16.022 INFO 2981023 --- [ocp-schedules-22,696a0a00dafb974f,7301842a5f9a2563] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:11:16.023 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:11:16.023 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:11:16.024 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:11:16.024 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:11:16.038 INFO 2981023 --- [ocp-schedules-27,ffa509f9f706ab2a,6157918d3009fd45] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:11:16.041 INFO 2981023 --- [ocp-schedules-23,36da6e2d605808ae,35dd259e61ea5185] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:11:16.041 INFO 2981023 --- [host-check-clockdiff1,ffa509f9f706ab2a,8a8f3a516e6ba10f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:11:16.041 INFO 2981023 --- [host-check-clockdiff2,ffa509f9f706ab2a,e18b7705e8b9c92c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:11:16.045 INFO 2981023 --- [host-check-clockdiff3,ffa509f9f706ab2a,e8fa7a4bcea446c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:11:16.045 INFO 2981023 --- [host-check-clockdiff4,ffa509f9f706ab2a,59ede1e716a54a5e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:11:16.046 INFO 2981023 --- [host-check-agent-status1,36da6e2d605808ae,c235891c4a70e7e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:11:16.047 INFO 2981023 --- [host-check-agent-status2,36da6e2d605808ae,d83de7491f4d4c6f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:11:16.048 INFO 2981023 --- [host-check-agent-status3,36da6e2d605808ae,c8c8f713e7b03059] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:11:16.048 INFO 2981023 --- [host-check-agent-status4,36da6e2d605808ae,5751800490153a80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:11:16.055 INFO 2981023 --- [host-check-clockdiff1,ffa509f9f706ab2a,8a8f3a516e6ba10f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049476 5 5 , error= 2024-09-11 18:11:16.065 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:11:16.066 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:11:16.066 INFO 2981023 --- [host-check-clockdiff3,ffa509f9f706ab2a,e8fa7a4bcea446c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049476 0 0 , error= 2024-09-11 18:11:16.067 INFO 2981023 --- [host-check-clockdiff2,ffa509f9f706ab2a,e18b7705e8b9c92c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049476 5 5 , error= 2024-09-11 18:11:16.072 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:11:16.072 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:11:16.075 INFO 2981023 --- [host-check-agent-status1,36da6e2d605808ae,c235891c4a70e7e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:11:16.075 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:11:16.076 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:11:16.077 INFO 2981023 --- [host-check-agent-status1,36da6e2d605808ae,c235891c4a70e7e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:11:16.077 INFO 2981023 --- [host-check-agent-status4,36da6e2d605808ae,5751800490153a80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:11:16.079 INFO 2981023 --- [host-check-agent-status4,36da6e2d605808ae,5751800490153a80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:11:16.079 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:11:16.080 INFO 2981023 --- [host-check-agent-status3,36da6e2d605808ae,c8c8f713e7b03059] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:11:16.080 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:11:16.094 INFO 2981023 --- [host-check-agent-status3,36da6e2d605808ae,c8c8f713e7b03059] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:11:16.189 INFO 2981023 --- [ocp-schedules-11,68eed55064a30074,2c09ae9d98b4b7fc] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:11:16.201 INFO 2981023 --- [ocp-schedules-5,f051afa61f43bf1b,cc0092590e76824e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:11:16.204 INFO 2981023 --- [ocp-schedules-21,66d632560d894539,621e7b9d6e76fdcf] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:11:16.206 INFO 2981023 --- [ocp-schedules-20,49b30d17f48fccf5,e81e83f48d120537] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:11:16.213 INFO 2981023 --- [ocp-schedules-21,66d632560d894539,621e7b9d6e76fdcf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:11:16.275 INFO 2981023 --- [ocp-schedules-11,68eed55064a30074,2c09ae9d98b4b7fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:11:16.276 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049402864000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:11:16.281 INFO 2981023 --- [ocp-schedules-5,f051afa61f43bf1b,cc0092590e76824e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:11:16.282 INFO 2981023 --- [host-check-agent-status2,36da6e2d605808ae,d83de7491f4d4c6f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:11:16.288 INFO 2981023 --- [host-check-agent-status2,36da6e2d605808ae,d83de7491f4d4c6f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:11:16.305 INFO 2981023 --- [ocp-schedules-5,f051afa61f43bf1b,cc0092590e76824e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:11:16.356 INFO 2981023 --- [ocp-schedules-10,45bf8dc3dced0f85,07437a340b99f171] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:11:16.360 INFO 2981023 --- [host-check-agent-status1,36da6e2d605808ae,c235891c4a70e7e2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:11:16.361 INFO 2981023 --- [host-check-agent-status3,36da6e2d605808ae,c8c8f713e7b03059] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:11:16.366 INFO 2981023 --- [host-check-agent-status3,36da6e2d605808ae,c8c8f713e7b03059] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:11:16.366 INFO 2981023 --- [host-check-agent-status3,36da6e2d605808ae,c8c8f713e7b03059] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:11:16.366 INFO 2981023 --- [host-check-agent-status3,36da6e2d605808ae,c8c8f713e7b03059] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:11:16.367 INFO 2981023 --- [host-check-agent-status1,36da6e2d605808ae,c235891c4a70e7e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:11:16.367 INFO 2981023 --- [host-check-agent-status1,36da6e2d605808ae,c235891c4a70e7e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:11:16.367 INFO 2981023 --- [host-check-agent-status1,36da6e2d605808ae,c235891c4a70e7e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:11:16.368 INFO 2981023 --- [host-check-agent-status4,36da6e2d605808ae,5751800490153a80] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:11:16.370 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:11:16.370 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:11:16.375889421+08:00 2024-09-11 18:11:16.370 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:11:16.380 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:11:16.384455782+08:00 2024-09-11 18:11:16.381 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:11:16.381 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:11:16.380+08:00 2024-09-11 18:11:16.382 INFO 2981023 --- [host-check-agent-status4,36da6e2d605808ae,5751800490153a80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:11:16.382 INFO 2981023 --- [host-check-agent-status4,36da6e2d605808ae,5751800490153a80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:11:16.382 INFO 2981023 --- [host-check-agent-status4,36da6e2d605808ae,5751800490153a80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:11:16.382 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:11:16.370+08:00 2024-09-11 18:11:16.385 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:11:16.384316850+08:00 2024-09-11 18:11:16.386 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:11:16.370+08:00 and status AVAILABLE 2024-09-11 18:11:16.387 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:11:16.380+08:00 and status AVAILABLE 2024-09-11 18:11:16.387 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:11:16.385+08:00 2024-09-11 18:11:16.391 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:11:16.385+08:00 and status AVAILABLE 2024-09-11 18:11:16.394 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:11:16.393416+08:00, lastProcessTime=2024-09-11T18:11:06.337531+08:00 2024-09-11 18:11:16.396 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.396 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:11:16.397 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:11:16.398 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:11:16.399 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:11:16.399 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:11:16.399247477+08:00 2024-09-11 18:11:16.401 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:11:16.399+08:00 2024-09-11 18:11:16.402 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:11:16.403 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:11:16.399+08:00 and status AVAILABLE 2024-09-11 18:11:16.405 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:11:16.409 INFO 2981023 --- [host-check-agent-status2,36da6e2d605808ae,d83de7491f4d4c6f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:11:16.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.410 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:11:16.411 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:11:16.411 INFO 2981023 --- [host-check-agent-status2,36da6e2d605808ae,d83de7491f4d4c6f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:11:16.412 INFO 2981023 --- [host-check-agent-status2,36da6e2d605808ae,d83de7491f4d4c6f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:11:16.412 INFO 2981023 --- [host-check-agent-status2,36da6e2d605808ae,d83de7491f4d4c6f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:11:16.419 INFO 2981023 --- [ocp-schedules-11,68eed55064a30074,2c09ae9d98b4b7fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:11:16.424 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:11:16.454 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.455 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:16.456 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.457 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:16.464 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:11:16.465 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:11:16.465 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:11:16.466 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:11:16.468 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.469 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:16.470 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:11:16.472 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:11:16.474 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:11:16.474 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:11:16.496 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:11:16.507 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:11:16.508 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:11:16.508 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:11:16.511 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:16.512 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.512 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:11:16.525 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:11:16.530 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:11:16.534 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:11:16.536 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:11:16.536 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049402864000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:11:16.537 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:11:16.555 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:11:16.555 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:11:16.559 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:11:16.562 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:11:16.563 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.565 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:16.565 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:11:16.566 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.569 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:11:16.570 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:11:16.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:11:16.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:11:16.593 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:11:16.601 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:11:16.606 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:11:16.607 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:11:16.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:11:16.620 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:11:16.630 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:11:16.634 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:11:16.642 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:11:16.642 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:11:16.675 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:11:16.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:16.678 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.679 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:11:16.679 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:11:16.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:11:16.680 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:11:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:16.683 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.683 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:11:16.688 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:11:16.693 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:11:16.694 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:11:16.699 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:11:16.701 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:11:16.701 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:11:16.704 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:11:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:16.710 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.710 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:16.711 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:11:16.711 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.714 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:11:16.714 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:11:16.715 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:11:16.722 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:11:16.728 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:11:16.729 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:11:16.743 INFO 2981023 --- [host-check-host-status4,696a0a00dafb974f,54f9f504ecaf4626] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:11:16.743 INFO 2981023 --- [host-check-host-status2,696a0a00dafb974f,f37c7cc34262d934] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:11:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:11:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:11:16.747 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:11:16.754 INFO 2981023 --- [host-check-host-status1,696a0a00dafb974f,8a1585a882467f36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:11:16.766 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:11:16.766 INFO 2981023 --- [host-check-host-status3,696a0a00dafb974f,55c8e7609ba4288e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:11:16.773 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:11:16.393416+08:00 2024-09-11 18:11:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:11:16.780 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:11:16.784 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:11:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:11:16.789 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:11:16.799 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:11:16.804 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:11:16.805 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:16.806 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.806 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:11:16.811 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:11:16.811 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:11:16.843 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:11:16.849 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:16.849 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:16.850 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:11:16.855 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:11:16.855 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:11:16.855 INFO 2981023 --- [ocp-schedules-21,66d632560d894539,621e7b9d6e76fdcf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:11:17.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,96a5ed1f5dd4adf7,3718cc00e67cd8dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=96a5ed1f5dd4adf7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:17.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,96a5ed1f5dd4adf7,3718cc00e67cd8dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=96a5ed1f5dd4adf7, duration=12 ms] 2024-09-11 18:11:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,176096ded2d6882b,41a8c2cc79ec0b13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,176096ded2d6882b,41a8c2cc79ec0b13] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:17.406 INFO 2981023 --- [ocp-async-4,3855c9867be9e26d,1de3f969c8454d35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:17.546 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049400, nodeCount=480, validCount=480 2024-09-11 18:11:17.784 INFO 2981023 --- [ocp-async-8,347885d41873a07c,e608cb698ff5c4c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:11:18.003 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:18.003 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:18.511 INFO 2981023 --- [ocp-schedules-12,6aee33f777f0a619,376fab7a7093731b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:11:18.575 WARN 2981023 --- [scanner-13622239-1,6aee33f777f0a619,d0bf2aa6997b41e9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:11:18.619 INFO 2981023 --- [alarm-detect-15,6aee33f777f0a619,514333939221d059] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645616, data is empty. 2024-09-11 18:11:18.621 INFO 2981023 --- [alarm-detect-15,6aee33f777f0a619,514333939221d059] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645655, data is empty. 2024-09-11 18:11:18.627 INFO 2981023 --- [alarm-detect-15,6aee33f777f0a619,514333939221d059] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645655, data is empty. 2024-09-11 18:11:18.631 INFO 2981023 --- [alarm-detect-12,6aee33f777f0a619,8b06075657be5d88] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645616, data is empty. 2024-09-11 18:11:18.635 INFO 2981023 --- [alarm-detect-12,6aee33f777f0a619,8b06075657be5d88] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645655, data is empty. 2024-09-11 18:11:18.638 INFO 2981023 --- [alarm-detect-7,6aee33f777f0a619,43beda4864040116] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645616, data is empty. 2024-09-11 18:11:18.638 INFO 2981023 --- [alarm-detect-12,6aee33f777f0a619,8b06075657be5d88] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645655, data is empty. 2024-09-11 18:11:18.644 INFO 2981023 --- [alarm-detect-7,6aee33f777f0a619,43beda4864040116] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645655, data is empty. 2024-09-11 18:11:18.649 INFO 2981023 --- [alarm-detect-7,6aee33f777f0a619,43beda4864040116] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645655, data is empty. 2024-09-11 18:11:18.744 INFO 2981023 --- [alarm-detect-12,6aee33f777f0a619,68e8c11005466b40] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645616, data is empty. 2024-09-11 18:11:18.750 INFO 2981023 --- [alarm-detect-12,6aee33f777f0a619,68e8c11005466b40] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645655, data is empty. 2024-09-11 18:11:18.755 INFO 2981023 --- [alarm-detect-12,6aee33f777f0a619,68e8c11005466b40] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645655, data is empty. 2024-09-11 18:11:19.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,905e9c31a6a00f1f,51e30a856b1470b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=905e9c31a6a00f1f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:19.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,905e9c31a6a00f1f,51e30a856b1470b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=905e9c31a6a00f1f, duration=9 ms] 2024-09-11 18:11:19.406 INFO 2981023 --- [ocp-async-9,c5860f2acb51335a,6b096fbfcb01035a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,3dd15a04657cafb0,bf3765d9aa1d7f52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,3dd15a04657cafb0,bf3765d9aa1d7f52] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:21.004 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:21.004 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:21.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,696d528a38c3dbd4,661c2dad73915fa5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=696d528a38c3dbd4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:21.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,696d528a38c3dbd4,661c2dad73915fa5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=696d528a38c3dbd4, duration=9 ms] 2024-09-11 18:11:21.406 INFO 2981023 --- [ocp-async-8,84b52264b473284d,a257be4dba0a7553] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:22.042 INFO 2981023 --- [host-check-clockdiff4,ffa509f9f706ab2a,59ede1e716a54a5e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049482 0 0 , error= 2024-09-11 18:11:22.568 INFO 2981023 --- [ocp-schedules-22,aa906cf1db6f0638,75159908190a2172] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:11:23.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,5adbef7d3e31480c,64e62183db9882a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5adbef7d3e31480c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:23.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,5adbef7d3e31480c,64e62183db9882a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5adbef7d3e31480c, duration=9 ms] 2024-09-11 18:11:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,29d43de4cdf10ff1,108c2e97b868110f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,29d43de4cdf10ff1,108c2e97b868110f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:23.407 INFO 2981023 --- [ocp-async-2,af57bc7ba8ff0fb7,6fb5c69a9ddbd822] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:24.004 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:24.005 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:24.422 INFO 2981023 --- [ocp-schedules-21,7f9a9f6744e58034,8b5feb377f820bcd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:11:24.466 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2b914d8899b489ab,af0300f248a655af] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&version=2&ObClusterId=1719994443, client=10.31.35.6, traceId=2b914d8899b489ab, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkrtd,1719994443,2,{"ObRegion":"linkrtd","ObCluster":"linkrtd","ObRegionId":1719994443,"ObClusterId":1719994443,"Type":"PRIMARY","timestamp":1726049484469805,"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:11:24.473 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2b914d8899b489ab,af0300f248a655af] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkrtd, obClusterId=1719994443, rootServerJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049484466000}, groupName=linkrtd, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:11:24.473 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2b914d8899b489ab,af0300f248a655af] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:11:24.473 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2b914d8899b489ab,af0300f248a655af] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkrtd, clusterId=1719994443, affectRows=1 2024-09-11 18:11:24.474 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2b914d8899b489ab,af0300f248a655af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.6, traceId=2b914d8899b489ab, duration=9 ms] 2024-09-11 18:11:25.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,e674bbcc97d98c89,03438c4a1ecc3aaa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e674bbcc97d98c89, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:25.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,e674bbcc97d98c89,03438c4a1ecc3aaa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e674bbcc97d98c89, duration=9 ms] 2024-09-11 18:11:25.407 INFO 2981023 --- [ocp-async-6,1fcd5cb332252d1b,f6849c07c862513c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:25.967 INFO 2981023 --- [ocp-schedules-2,18b5914e5749db8b,25b9835da4de8bc8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:11:25.968 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:11:25.968 INFO 2981023 --- [ocp-schedules-2,18b5914e5749db8b,25b9835da4de8bc8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:11:25.968 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:11:25.968 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:11:25.972 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:11:25.972 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:11:25.973 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:11:25.973 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:11:25.973 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:11:25.974 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:11:25.974 INFO 2981023 --- [ocp-schedules-2,18b5914e5749db8b,25b9835da4de8bc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:11:25.975 INFO 2981023 --- [ocp-schedules-2,18b5914e5749db8b,25b9835da4de8bc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:11:25.992 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:11:25.992 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:11:25.993 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:11:25.996 INFO 2981023 --- [ocp-schedules-2,18b5914e5749db8b,25b9835da4de8bc8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:11:25.996 INFO 2981023 --- [ocp-schedules-2,18b5914e5749db8b,25b9835da4de8bc8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:11:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,703c5da2bad1d047,9af3511cccff1913] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,703c5da2bad1d047,9af3511cccff1913] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:11:26.337240+08:00, lastProcessTime=2024-09-11T18:11:16.393416+08:00 2024-09-11 18:11:26.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:11:26.337240+08:00 2024-09-11 18:11:27.005 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:27.006 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:27.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,1b1252d978bcb6cf,75afcc1a62f890a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1b1252d978bcb6cf, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:27.119 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,1b1252d978bcb6cf,75afcc1a62f890a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1b1252d978bcb6cf, duration=10 ms] 2024-09-11 18:11:27.407 INFO 2981023 --- [ocp-async-3,f0c22472f58e8190,0eb7579fb4e0cfd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:27.785 INFO 2981023 --- [ocp-async-1,f225b494c93b60ff,bb31bdc69c5586da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:11:28.792 INFO 2981023 --- [ocp-schedules-10,33484d09f7f5caee,5b43290f768eb933] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:11:28.804 WARN 2981023 --- [scanner-13622532-1,33484d09f7f5caee,731f29e2ce8994e5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:11:29.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,5bd2141d6fe91760,25ce9909dfe3e658] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=5bd2141d6fe91760, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:29.128 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,5bd2141d6fe91760,25ce9909dfe3e658] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=5bd2141d6fe91760, duration=9 ms] 2024-09-11 18:11:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,0315d4b16ec44762,e87808b21446b956] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,0315d4b16ec44762,e87808b21446b956] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:29.407 INFO 2981023 --- [ocp-async-4,f507a78c11a77e55,8ea45ab35da10132] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:30.006 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:30.006 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:31.127 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a38d3285feb4ef7a,eb9244d96ba30dd7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=a38d3285feb4ef7a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:31.135 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a38d3285feb4ef7a,eb9244d96ba30dd7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=a38d3285feb4ef7a, duration=18 ms] 2024-09-11 18:11:31.407 INFO 2981023 --- [ocp-async-1,5bf14ca87e6fa057,e8c8492838c0437a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,f9882639177cb259,aefea36a766bda16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,f9882639177cb259,aefea36a766bda16] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:32.570 INFO 2981023 --- [ocp-schedules-16,3c57e8942dc32862,104718a23c558fd2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:11:33.006 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:33.007 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:33.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e98931b193427d09,682e1bc123768d30] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e98931b193427d09, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:33.121 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e98931b193427d09,682e1bc123768d30] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e98931b193427d09, duration=9 ms] 2024-09-11 18:11:33.407 INFO 2981023 --- [ocp-async-5,c902e4d71816bcde,5eb58cc5dceb86c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:35.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,814d9b30b172cde4,1e618c274cddf48b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=814d9b30b172cde4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:35.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,814d9b30b172cde4,1e618c274cddf48b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=814d9b30b172cde4, duration=9 ms] 2024-09-11 18:11:35.279 INFO 2981023 --- [ocp-schedules-10,97eb15bf8e796b40,f03695e30edb9246] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:11:35.284 INFO 2981023 --- [ocp-schedules-10,97eb15bf8e796b40,f03695e30edb9246] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:11:35.284 INFO 2981023 --- [ocp-schedules-10,97eb15bf8e796b40,f03695e30edb9246] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:11:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,1e3876c8d4f77bd9,c0fdcefff1d1018f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,1e3876c8d4f77bd9,c0fdcefff1d1018f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:35.407 INFO 2981023 --- [ocp-async-10,39b116b2544e6630,4ddf87dc2dc90add] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:36.007 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:36.008 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:11:36.336818+08:00, lastProcessTime=2024-09-11T18:11:26.337240+08:00 2024-09-11 18:11:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:11:36.336818+08:00 2024-09-11 18:11:37.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,49e32fb5d9d505ea,7f43879a5a45c1a1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=49e32fb5d9d505ea, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:37.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,49e32fb5d9d505ea,7f43879a5a45c1a1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=49e32fb5d9d505ea, duration=11 ms] 2024-09-11 18:11:37.314 INFO 2981023 --- [prometheus-cache-monitor1,0b2aa2c04164c345,07349951ca3ec4e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8197,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:11:37.315 INFO 2981023 --- [prometheus-cache-monitor1,0b2aa2c04164c345,07349951ca3ec4e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:11:37.407 INFO 2981023 --- [ocp-async-7,ec1589ed4f315222,23c9deba5055de98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:37.785 INFO 2981023 --- [ocp-async-2,a07be5ff89593dd7,49771704a42e393e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:11:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,c0fd81a595375eef,b3aa30bd37ca9d49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,c0fd81a595375eef,b3aa30bd37ca9d49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:39.008 INFO 2981023 --- [ocp-schedules-23,19167b45b05f79b7,3eb0c6b1933bfb2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:11:39.008 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:39.011 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:39.048 WARN 2981023 --- [scanner-13622787-1,19167b45b05f79b7,cfc11a9d120dac1a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:11:39.068 INFO 2981023 --- [alarm-detect-8,19167b45b05f79b7,d5d3405c82036387] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645655, data is empty. 2024-09-11 18:11:39.071 INFO 2981023 --- [alarm-detect-8,19167b45b05f79b7,d5d3405c82036387] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645655, data is empty. 2024-09-11 18:11:39.074 INFO 2981023 --- [alarm-detect-8,19167b45b05f79b7,d5d3405c82036387] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645655, data is empty. 2024-09-11 18:11:39.098 INFO 2981023 --- [alarm-detect-19,19167b45b05f79b7,144de6a954612560] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645655, data is empty. 2024-09-11 18:11:39.100 INFO 2981023 --- [alarm-detect-2,19167b45b05f79b7,d27b9a4fee506538] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645655, data is empty. 2024-09-11 18:11:39.102 INFO 2981023 --- [alarm-detect-19,19167b45b05f79b7,144de6a954612560] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645655, data is empty. 2024-09-11 18:11:39.106 INFO 2981023 --- [alarm-detect-2,19167b45b05f79b7,d27b9a4fee506538] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645655, data is empty. 2024-09-11 18:11:39.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,431742828a58d1f2,4f08cf63f7f95da4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=431742828a58d1f2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:39.111 INFO 2981023 --- [alarm-detect-19,19167b45b05f79b7,144de6a954612560] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645655, data is empty. 2024-09-11 18:11:39.112 INFO 2981023 --- [alarm-detect-2,19167b45b05f79b7,d27b9a4fee506538] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645655, data is empty. 2024-09-11 18:11:39.126 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,431742828a58d1f2,4f08cf63f7f95da4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=431742828a58d1f2, duration=26 ms] 2024-09-11 18:11:39.220 INFO 2981023 --- [alarm-detect-2,19167b45b05f79b7,6bb17f492b81d845] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645655, data is empty. 2024-09-11 18:11:39.224 INFO 2981023 --- [alarm-detect-2,19167b45b05f79b7,6bb17f492b81d845] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645655, data is empty. 2024-09-11 18:11:39.227 INFO 2981023 --- [alarm-detect-2,19167b45b05f79b7,6bb17f492b81d845] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645655, data is empty. 2024-09-11 18:11:39.408 INFO 2981023 --- [ocp-async-3,1d2aa1cf14652d16,02a857766d401c92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:40.002 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:11:40.002 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:11:41.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,429a808b29eafc07,b517aa58a355738e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=429a808b29eafc07, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:41.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,429a808b29eafc07,b517aa58a355738e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=429a808b29eafc07, duration=8 ms] 2024-09-11 18:11:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,d71476a9dd5bf6d7,9d4285092730d4d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,d71476a9dd5bf6d7,9d4285092730d4d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:41.408 INFO 2981023 --- [ocp-async-2,c2cc898b28eab40e,ec786825720073d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:42.012 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:42.012 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:42.573 INFO 2981023 --- [ocp-schedules-9,8f70e12f798cc4f8,9c53ac6add37eb49] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:11:43.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,ad965ac715718bb5,389436607c9dfe43] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ad965ac715718bb5, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:43.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,ad965ac715718bb5,389436607c9dfe43] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ad965ac715718bb5, duration=9 ms] 2024-09-11 18:11:43.408 INFO 2981023 --- [ocp-async-6,de396f6b8bdfbe54,5031749d293f135d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,0052a8a8e0dee0bd,b404e86d639e59ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,0052a8a8e0dee0bd,b404e86d639e59ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:45.012 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:45.013 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:45.101 INFO 2981023 --- [prometheus-second-schedule-1,0065a97f7e496a11,906a8092d2ae9a5a] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:11:45.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,39465766730c0034,7a278e8a58726104] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=39465766730c0034, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:45.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,39465766730c0034,7a278e8a58726104] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=39465766730c0034, duration=12 ms] 2024-09-11 18:11:45.255 INFO 2981023 --- [ocp-schedules-2,35a787be98be1466,d0e80b0b9a117457] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:11:45.273 WARN 2981023 --- [ocp-schedules-2,35a787be98be1466,d0e80b0b9a117457] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:11:45.408 INFO 2981023 --- [ocp-async-9,24aee24127833f14,ed14643ac2f654b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:45.998 INFO 2981023 --- [ocp-schedules-24,02cbf02a3c95cc59,3b0551cc56a55f88] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:11:45.998 INFO 2981023 --- [host-check-clockdiff1,02cbf02a3c95cc59,461f4346254f2730] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:11:45.998 INFO 2981023 --- [host-check-clockdiff3,02cbf02a3c95cc59,c2142afc632884fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:11:45.999 INFO 2981023 --- [host-check-clockdiff4,02cbf02a3c95cc59,b2b601d7a2f52243] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:11:45.999 INFO 2981023 --- [host-check-clockdiff2,02cbf02a3c95cc59,3b062199647514c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:11:46.005 INFO 2981023 --- [ocp-schedules-11,9e75ca9f4435de0e,d01af0a7e68ee0f4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:11:46.014 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:11:46.014 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:11:46.014 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:11:46.014 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:11:46.017 INFO 2981023 --- [ocp-schedules-8,4168a4cac395f8c7,010bd7abd52e4d77] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:11:46.017 INFO 2981023 --- [host-check-agent-status1,4168a4cac395f8c7,aea8708baf29c615] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:11:46.018 INFO 2981023 --- [host-check-agent-status2,4168a4cac395f8c7,7cd0e74f58dfeb58] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:11:46.019 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:11:46.020 INFO 2981023 --- [host-check-agent-status3,4168a4cac395f8c7,f3f1aac7ec10a15a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:11:46.020 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:11:46.027 INFO 2981023 --- [host-check-clockdiff3,02cbf02a3c95cc59,c2142afc632884fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049506 0 0 , error= 2024-09-11 18:11:46.027 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:11:46.028 INFO 2981023 --- [host-check-agent-status4,4168a4cac395f8c7,0b771dae71c478f3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:11:46.029 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:11:46.029 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:11:46.030 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:11:46.030 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:11:46.032 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:11:46.032 INFO 2981023 --- [host-check-agent-status2,4168a4cac395f8c7,7cd0e74f58dfeb58] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:11:46.032 INFO 2981023 --- [host-check-agent-status1,4168a4cac395f8c7,aea8708baf29c615] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:11:46.034 INFO 2981023 --- [host-check-agent-status2,4168a4cac395f8c7,7cd0e74f58dfeb58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:11:46.035 INFO 2981023 --- [host-check-agent-status1,4168a4cac395f8c7,aea8708baf29c615] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:11:46.036 INFO 2981023 --- [host-check-agent-status3,4168a4cac395f8c7,f3f1aac7ec10a15a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:11:46.038 INFO 2981023 --- [host-check-agent-status3,4168a4cac395f8c7,f3f1aac7ec10a15a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:11:46.038 INFO 2981023 --- [host-check-agent-status4,4168a4cac395f8c7,0b771dae71c478f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:11:46.040 INFO 2981023 --- [host-check-agent-status4,4168a4cac395f8c7,0b771dae71c478f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:11:46.055 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:11:46.057 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:11:46.057 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:11:46.063109008+08:00 2024-09-11 18:11:46.059 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:11:46.058+08:00 2024-09-11 18:11:46.059 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:11:46.059478247+08:00 2024-09-11 18:11:46.060 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:11:46.059+08:00 2024-09-11 18:11:46.061 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:11:46.058+08:00 and status AVAILABLE 2024-09-11 18:11:46.062 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:11:46.059+08:00 and status AVAILABLE 2024-09-11 18:11:46.063 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:11:46.063 INFO 2981023 --- [host-check-agent-status3,4168a4cac395f8c7,f3f1aac7ec10a15a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:11:46.063 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:11:46.065 INFO 2981023 --- [host-check-agent-status2,4168a4cac395f8c7,7cd0e74f58dfeb58] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:11:46.065 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:11:46.070683373+08:00 2024-09-11 18:11:46.066 INFO 2981023 --- [host-check-agent-status1,4168a4cac395f8c7,aea8708baf29c615] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:11:46.066 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:11:46.065+08:00 2024-09-11 18:11:46.066 INFO 2981023 --- [host-check-agent-status3,4168a4cac395f8c7,f3f1aac7ec10a15a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:11:46.066 INFO 2981023 --- [host-check-agent-status3,4168a4cac395f8c7,f3f1aac7ec10a15a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:11:46.066 INFO 2981023 --- [host-check-agent-status3,4168a4cac395f8c7,f3f1aac7ec10a15a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:11:46.067 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:11:46.066616968+08:00 2024-09-11 18:11:46.067 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:11:46.065+08:00 and status AVAILABLE 2024-09-11 18:11:46.068 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:11:46.067+08:00 2024-09-11 18:11:46.068 INFO 2981023 --- [host-check-agent-status4,4168a4cac395f8c7,0b771dae71c478f3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:11:46.068 INFO 2981023 --- [host-check-agent-status1,4168a4cac395f8c7,aea8708baf29c615] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:11:46.068 INFO 2981023 --- [host-check-agent-status1,4168a4cac395f8c7,aea8708baf29c615] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:11:46.068 INFO 2981023 --- [host-check-agent-status1,4168a4cac395f8c7,aea8708baf29c615] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:11:46.069 INFO 2981023 --- [host-check-agent-status2,4168a4cac395f8c7,7cd0e74f58dfeb58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:11:46.069 INFO 2981023 --- [host-check-agent-status2,4168a4cac395f8c7,7cd0e74f58dfeb58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:11:46.069 INFO 2981023 --- [host-check-agent-status2,4168a4cac395f8c7,7cd0e74f58dfeb58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:11:46.069 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:11:46.069 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:11:46.067+08:00 and status AVAILABLE 2024-09-11 18:11:46.072 INFO 2981023 --- [host-check-agent-status4,4168a4cac395f8c7,0b771dae71c478f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:11:46.072 INFO 2981023 --- [host-check-agent-status4,4168a4cac395f8c7,0b771dae71c478f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:11:46.072 INFO 2981023 --- [host-check-agent-status4,4168a4cac395f8c7,0b771dae71c478f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:11:46.073 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:11:46.075 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:11:46.076 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:11:46.079 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:11:46.081 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:11:46.081 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:11:46.087 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:11:46.095 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:11:46.098 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:11:46.099 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:11:46.107 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:11:46.120 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:11:46.124 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:11:46.126 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:11:46.133 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:11:46.140 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:11:46.159 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:11:46.165 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:11:46.168 INFO 2981023 --- [ocp-schedules-28,1502e9daf517f3d0,b7249a5545193d99] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:11:46.180 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:11:46.184 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:11:46.184 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:11:46.187 INFO 2981023 --- [ocp-schedules-13,a88cb949ea8b097b,de2d3f78aac1363c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:11:46.189 INFO 2981023 --- [ocp-schedules-13,a88cb949ea8b097b,de2d3f78aac1363c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:11:46.189 INFO 2981023 --- [ocp-schedules-30,3825fe70d8404cc9,715d2bad4d33ac0b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:11:46.189 INFO 2981023 --- [ocp-schedules-28,1502e9daf517f3d0,b7249a5545193d99] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:11:46.189 INFO 2981023 --- [ocp-schedules-3,cbc92d9c04ff6fcc,a874b4ee10031e70] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:11:46.189 INFO 2981023 --- [ocp-schedules-16,05af1065993ff8e6,6ff6c51054509d26] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:11:46.192 INFO 2981023 --- [ocp-schedules-30,3825fe70d8404cc9,715d2bad4d33ac0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:11:46.192 INFO 2981023 --- [ocp-schedules-16,05af1065993ff8e6,6ff6c51054509d26] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:11:46.206 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:11:46.209 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:11:46.210 INFO 2981023 --- [ocp-schedules-27,1f3b38a1651d6b50,e944c337f5927b4c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 18:11:46.210 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:11:46.251 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049484466000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:11:46.261 INFO 2981023 --- [ocp-schedules-27,1f3b38a1651d6b50,e944c337f5927b4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:11:46.267 INFO 2981023 --- [ocp-schedules-3,2026f2002b444db0,857a29852e8b6408] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 18:11:46.267 INFO 2981023 --- [ocp-schedules-3,2026f2002b444db0,857a29852e8b6408] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 18:11:46.274 INFO 2981023 --- [ocp-schedules-28,a2f770046f0f3961,a306705e012e0d98] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:11:46.280 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.280 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.281 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:11:46.281 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:11:46.281 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:11:46.282 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:11:46.285 INFO 2981023 --- [ocp-schedules-30,3825fe70d8404cc9,715d2bad4d33ac0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:11:46.286 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:11:46.288 INFO 2981023 --- [ocp-schedules-13,a88cb949ea8b097b,de2d3f78aac1363c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:11:46.295 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:11:46.300 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.300 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:11:46.300 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:11:46.301 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.309 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:11:46.312 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:11:46.315 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:11:46.319 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:11:46.319 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:11:46.321 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.321 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:11:46.321 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.323 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:46.328 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,d12a2e898cd79b5e,27489b3dcb6e61da] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=d12a2e898cd79b5e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 18:11:46.328 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,d12a2e898cd79b5e,27489b3dcb6e61da] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=d12a2e898cd79b5e, duration=1 ms] 2024-09-11 18:11:46.329 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.330 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:46.333 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:11:46.336 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.336 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:11:46.341 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.342 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 18:11:46.348 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.348 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:11:46.358 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71912120e27438e1,99b53fb562bce8b2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=71912120e27438e1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 18:11:46.358 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,71912120e27438e1,99b53fb562bce8b2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=71912120e27438e1, duration=0 ms] 2024-09-11 18:11:46.361 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.362 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:11:46.364 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:11:46.369 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:11:46.369287+08:00, lastProcessTime=2024-09-11T18:11:36.336818+08:00 2024-09-11 18:11:46.371 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.375 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 18:11:46.376 INFO 2981023 --- [ocp-schedules-27,1f3b38a1651d6b50,e944c337f5927b4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 18:11:46.376 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:11:46.377 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:11:46.382 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.383 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:46.388 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:11:46.392 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:11:46.398 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.398 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.399 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:11:46.405 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:11:46.406 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:11:46.413 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.414 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:11:46.415 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.415 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:11:46.416 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:11:46.416 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.416 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:11:46.416 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049484466000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:11:46.421 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:11:46.423 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:11:46.425 INFO 2981023 --- [host-check-host-status1,9e75ca9f4435de0e,9165a1c8ea6013cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:11:46.427 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:11:46.428 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.433 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.433 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.438 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:11:46.439 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.440 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:46.440 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:11:46.441 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.442 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:11:46.444 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:11:46.444 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:11:46.445 INFO 2981023 --- [host-check-host-status4,9e75ca9f4435de0e,5200e18c3bba2605] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:11:46.450 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:11:46.452 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:11:46.453 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:11:46.458 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:11:46.458 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:11:46.464 INFO 2981023 --- [host-check-host-status3,9e75ca9f4435de0e,7ba3635f3c04f9cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:11:46.479 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.479 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.480 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:11:46.482 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:11:46.482 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.485 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:11:46.486 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:11:46.493 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:11:46.493 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.496 INFO 2981023 --- [pool-10-thread-1,67f55688f5be79db,ad96f01856499d4a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 18:11:46.498 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:11:46.505 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:11:46.505 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:11:46.509 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.510 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.510 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:11:46.511 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:11:46.514 INFO 2981023 --- [host-check-host-status2,9e75ca9f4435de0e,b240d58fa64625ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:11:46.515 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.516 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.516 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:11:46.517 INFO 2981023 --- [pool-10-thread-2,67f55688f5be79db,921a79273274bd5d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 18:11:46.517 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:11:46.517 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.517 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:11:46.518 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:11:46.518 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:11:46.518 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.519 INFO 2981023 --- [pool-10-thread-3,67f55688f5be79db,65161c84e7f7215c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 18:11:46.519 INFO 2981023 --- [pool-10-thread-3,67f55688f5be79db,65161c84e7f7215c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 18:11:46.523 INFO 2981023 --- [pool-10-thread-1,67f55688f5be79db,ad96f01856499d4a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 18:11:46.523 INFO 2981023 --- [pool-10-thread-1,67f55688f5be79db,ad96f01856499d4a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 18:11:46.524 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:11:46.525 INFO 2981023 --- [pool-10-thread-4,67f55688f5be79db,207ee614d462ea03] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 18:11:46.528 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:11:46.530 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:11:46.537 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:11:46.537 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 18:11:46.537 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 18:11:46.538 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 18:11:46.538 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 18:11:46.550 INFO 2981023 --- [ForkJoinPool-6-worker-14,67f55688f5be79db,d79b9e4e42b8675f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 18:11:46.550 INFO 2981023 --- [ForkJoinPool-6-worker-14,67f55688f5be79db,d79b9e4e42b8675f] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 18:11:46.559 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.560 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.561 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:11:46.573 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:11:46.586 WARN 2981023 --- [ForkJoinPool-6-worker-14,67f55688f5be79db,d79b9e4e42b8675f] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 18:11:46.590 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:11:46.591 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:11:46.602 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:11:46.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:11:46.607 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.608 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:11:46.609 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.609 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:11:46.610 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 18:11:46.613 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.613 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:11:46.617 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.617 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:11:46.619 INFO 2981023 --- [ForkJoinPool-6-worker-12,67f55688f5be79db,d7083219720c5b7f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.622 INFO 2981023 --- [ForkJoinPool-6-worker-12,67f55688f5be79db,d7083219720c5b7f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:11:46.631 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:11:46.635 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.639 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:11:46.652 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:11:46.659 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.660 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:11:46, 2024-09-11 18:11:46] 2024-09-11 18:11:46.664 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 18:11:46.664 INFO 2981023 --- [ForkJoinPool-6-worker-7,67f55688f5be79db,1ae915b903883f41] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 18:11:46.675 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.676 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.677 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:11:46.678 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:11:46.678 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.679 INFO 2981023 --- [ForkJoinPool-6-worker-12,67f55688f5be79db,d7083219720c5b7f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:11:46.679 INFO 2981023 --- [ForkJoinPool-6-worker-12,67f55688f5be79db,d7083219720c5b7f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.681 INFO 2981023 --- [ForkJoinPool-6-worker-12,67f55688f5be79db,d7083219720c5b7f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:11:46.682 INFO 2981023 --- [ForkJoinPool-6-worker-12,67f55688f5be79db,d7083219720c5b7f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:11:46.682 INFO 2981023 --- [ForkJoinPool-6-worker-12,67f55688f5be79db,d7083219720c5b7f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:11:46.694 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:11:46.696 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.700 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:11:46.702 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 18:11:46.702 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.706 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:11:46.708 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.708 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.708 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:11:46.709 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:11:46.709 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.710 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:11:46.711 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 18:11:46.712 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:11:46.731 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:11:46.732 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.734 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:11:46, 2024-09-11 18:11:46] 2024-09-11 18:11:46.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.740 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.740 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:11:46.744 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:11:46.752 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:11:46.753 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.755 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:11:46, 2024-09-11 18:11:46] 2024-09-11 18:11:46.755 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.756 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 18:11:46.756 INFO 2981023 --- [ForkJoinPool-6-worker-5,67f55688f5be79db,49844a272e7ba3cc] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 18:11:46.759 INFO 2981023 --- [pool-10-thread-4,67f55688f5be79db,207ee614d462ea03] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 18:11:46.761 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:11:46.762 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:11:46.762 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:11:46.765 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:11:46.766 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.775 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.779 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 18:11:46.780 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:11:46.781 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.782 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:11:46.785 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.786 INFO 2981023 --- [ForkJoinPool-6-worker-10,67f55688f5be79db,e71e6346ac426a5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:11:46.787 INFO 2981023 --- [pool-10-thread-2,67f55688f5be79db,921a79273274bd5d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 18:11:46.791 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.792 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.792 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:11:46.796 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:46.800 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.800 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:11:46.801 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:11:46.801 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:11:46.801 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:11:46.801 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:46.809 INFO 2981023 --- [ocp-schedules-16,05af1065993ff8e6,6ff6c51054509d26] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:11:46.816 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:11:46.369287+08:00 2024-09-11 18:11:46.845 INFO 2981023 --- [ocp-schedules-14,3b3d5fb03461ad3b,34c5cb6f061417eb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8197,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:11:46.845 INFO 2981023 --- [ocp-schedules-14,3b3d5fb03461ad3b,34c5cb6f061417eb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:11:46.847 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:11:46.858 INFO 2981023 --- [pool-10-thread-3,67f55688f5be79db,65161c84e7f7215c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 18:11:46.862 INFO 2981023 --- [pool-10-thread-3,67f55688f5be79db,65161c84e7f7215c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 18:11:46.864 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:11:46.869 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:11:46.872 INFO 2981023 --- [ocp-schedules-5,cc9e536086b9da20,79bcfadd24180b66] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:11:46.944 INFO 2981023 --- [host-check-clockdiff2,02cbf02a3c95cc59,3b062199647514c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049506 5 5 , error= 2024-09-11 18:11:47.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2cf9d77b4939587f,a36732564dbc57fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2cf9d77b4939587f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:47.121 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,2cf9d77b4939587f,a36732564dbc57fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2cf9d77b4939587f, duration=10 ms] 2024-09-11 18:11:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,4d4929088d9ea68e,d63b62b12dc46130] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,4d4929088d9ea68e,d63b62b12dc46130] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:47.408 INFO 2981023 --- [ocp-async-2,5859597f1ee5f20d,f37af80a3beecdbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:47.785 INFO 2981023 --- [ocp-async-5,e8bef2ee4cc4646e,de78a70dbcbc5ad9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:11:48.013 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:48.014 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:49.107 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6fceb702962a8768,21e258297861ac70] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6fceb702962a8768, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:49.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6fceb702962a8768,21e258297861ac70] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6fceb702962a8768, duration=15 ms] 2024-09-11 18:11:49.269 INFO 2981023 --- [ocp-schedules-14,1c2fdfdd5309fb7f,33add6b89d280943] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:11:49.314 WARN 2981023 --- [scanner-13623085-1,1c2fdfdd5309fb7f,19608f24d9fe48fb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:11:49.409 INFO 2981023 --- [ocp-async-1,f76cce2b51a6dd59,2d29fe346d319e1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:11:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:11:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,e59203dd923e20ca,04124418aedf33bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:50.316 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:11:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,e59203dd923e20ca,04124418aedf33bc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.330 INFO 2981023 --- [ocp-schedules-10,48af2f6a13c3d32c,ec30d18dbe99cc38] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:11:50.333 INFO 2981023 --- [ocp-schedules-12,3b6aa41fd3738f4b,9755dce291f9fd4b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:11:51.014 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:51.016 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:51.111 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,288cecef5984b5b6,ba09c5f3ce856513] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=288cecef5984b5b6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:51.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,288cecef5984b5b6,ba09c5f3ce856513] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=288cecef5984b5b6, duration=9 ms] 2024-09-11 18:11:51.408 INFO 2981023 --- [ocp-async-5,636de3e120a59ae9,f0a96022d051ecbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:51.542 INFO 2981023 --- [host-check-clockdiff1,02cbf02a3c95cc59,461f4346254f2730] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049511 5 5 , error= 2024-09-11 18:11:51.610 INFO 2981023 --- [ocp-schedules-15,1ed9f71c34d55505,e2f3984f2d1e28d8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:11:51.611 INFO 2981023 --- [ocp-schedules-15,1ed9f71c34d55505,e2f3984f2d1e28d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:11:51.612 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:11:51.612 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:11:51.612 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:11:51.612 INFO 2981023 --- [ocp-schedules-15,1ed9f71c34d55505,e2f3984f2d1e28d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:11:51.981 INFO 2981023 --- [host-check-clockdiff4,02cbf02a3c95cc59,b2b601d7a2f52243] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049511 0 0 , error= 2024-09-11 18:11:52.576 INFO 2981023 --- [ocp-schedules-23,bde4421851708b8e,0772f7eebdaa5cab] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:11:53.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,0fe0ebd3407667bd,f4e89c2dca35fb24] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=0fe0ebd3407667bd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:53.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,0fe0ebd3407667bd,f4e89c2dca35fb24] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=0fe0ebd3407667bd, duration=9 ms] 2024-09-11 18:11:53.297 INFO 2981023 --- [ocp-schedules-15,14d5614a043ff38f,6134e3f413a0ab04] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:11:53.297 INFO 2981023 --- [ocp-schedules-15,14d5614a043ff38f,6134e3f413a0ab04] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:11:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,e1dc0d88a33eaca2,f72550d825f45277] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,e1dc0d88a33eaca2,f72550d825f45277] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:53.408 INFO 2981023 --- [ocp-async-4,4e1881dd8351f272,c2bbaf0b04c36efc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:54.016 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:54.016 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:54.424 INFO 2981023 --- [ocp-schedules-4,3dc448aa15baf52e,d5d09b77ab2dacfb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:11:54.650 INFO 2981023 --- [ocp-schedules-16,19d6f6001bc52938,1a44b5f2dda6d5ab] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:11:54.670 WARN 2981023 --- [ocp-schedules-16,19d6f6001bc52938,1a44b5f2dda6d5ab] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:11:55.117 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,6d505200af545d59,4ec23e05469a69f5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6d505200af545d59, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:55.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,6d505200af545d59,4ec23e05469a69f5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6d505200af545d59, duration=9 ms] 2024-09-11 18:11:55.287 INFO 2981023 --- [ocp-schedules-11,c439b3a049492428,8bfef874c7fee338] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:11:55.297 INFO 2981023 --- [ocp-schedules-11,c439b3a049492428,8bfef874c7fee338] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:11:55.297 INFO 2981023 --- [ocp-schedules-11,c439b3a049492428,8bfef874c7fee338] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:11:55.408 INFO 2981023 --- [ocp-async-6,24f83ed22e0a6c88,652019a77c04d87f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:55.862 INFO 2981023 --- [ocp-schedules-10,45fb2d57e362bacf,e12b7f4e719ad869] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:11:55.862 INFO 2981023 --- [ocp-schedules-10,45fb2d57e362bacf,e12b7f4e719ad869] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:11:55.863 INFO 2981023 --- [ocp-schedules-10,45fb2d57e362bacf,e12b7f4e719ad869] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:11:55.863 INFO 2981023 --- [ocp-schedules-10,45fb2d57e362bacf,e12b7f4e719ad869] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:11:55.863 INFO 2981023 --- [ocp-schedules-10,45fb2d57e362bacf,e12b7f4e719ad869] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:11:55.863 INFO 2981023 --- [ocp-schedules-10,45fb2d57e362bacf,e12b7f4e719ad869] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:11:56.038 INFO 2981023 --- [ocp-schedules-30,23344400316101f2,0d9c573c57db50dc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:11:56.044 INFO 2981023 --- [ocp-schedules-30,23344400316101f2,0d9c573c57db50dc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:11:56.044 INFO 2981023 --- [ocp-schedules-30,23344400316101f2,0d9c573c57db50dc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:11:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,c5acddaab8da7df4,a8cbb4c14a179104] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,c5acddaab8da7df4,a8cbb4c14a179104] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:11:56.351510+08:00, lastProcessTime=2024-09-11T18:11:46.369287+08:00 2024-09-11 18:11:56.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:11:56.351510+08:00 2024-09-11 18:11:57.016 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:11:57.017 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:11:57.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,dea6a1877eda4676,ad8e52c9417f757f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=dea6a1877eda4676, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:57.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,dea6a1877eda4676,ad8e52c9417f757f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=dea6a1877eda4676, duration=9 ms] 2024-09-11 18:11:57.409 INFO 2981023 --- [ocp-async-9,e1725d832da57e16,e4294d5f7dfe60c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:57.785 INFO 2981023 --- [ocp-async-3,65268b9aae8e97b0,9b0514406a4945cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:11:57.879 INFO 2981023 --- [ocp-schedules-11,1d78175792755325,ef79432015c8ff89] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:11:57.879 INFO 2981023 --- [ocp-schedules-11,1d78175792755325,ef79432015c8ff89] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:11:57.901 INFO 2981023 --- [Collect all tenants major compactions1,1d78175792755325,be5b909059f7c4d2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:11:57.901 INFO 2981023 --- [Collect all tenants major compactions3,1d78175792755325,ab00599dab98a7eb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:11:57.901 INFO 2981023 --- [Collect all tenants major compactions2,1d78175792755325,f0c6bf793cf9d71b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:11:57.901 INFO 2981023 --- [Collect all tenants major compactions4,1d78175792755325,eca7fe898ca33071] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:11:57.903 INFO 2981023 --- [Collect all tenants major compactions6,1d78175792755325,d1b97e45ef543fb6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:11:57.903 INFO 2981023 --- [Collect all tenants major compactions5,1d78175792755325,93fa43f42c12c54f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:11:57.918 INFO 2981023 --- [Collect all tenants major compactions3,1d78175792755325,ab00599dab98a7eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:57.918 INFO 2981023 --- [Collect all tenants major compactions1,1d78175792755325,be5b909059f7c4d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:57.918 INFO 2981023 --- [Collect all tenants major compactions2,1d78175792755325,f0c6bf793cf9d71b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:57.919 INFO 2981023 --- [Collect all tenants major compactions1,1d78175792755325,be5b909059f7c4d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:11:57.920 INFO 2981023 --- [Collect all tenants major compactions3,1d78175792755325,ab00599dab98a7eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:11:57.920 INFO 2981023 --- [Collect all tenants major compactions2,1d78175792755325,f0c6bf793cf9d71b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:11:57.921 INFO 2981023 --- [Collect all tenants major compactions4,1d78175792755325,eca7fe898ca33071] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:57.921 INFO 2981023 --- [Collect all tenants major compactions5,1d78175792755325,93fa43f42c12c54f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:57.922 INFO 2981023 --- [Collect all tenants major compactions6,1d78175792755325,d1b97e45ef543fb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:57.922 INFO 2981023 --- [Collect all tenants major compactions4,1d78175792755325,eca7fe898ca33071] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:11:57.922 INFO 2981023 --- [Collect all tenants major compactions5,1d78175792755325,93fa43f42c12c54f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:11:57.923 INFO 2981023 --- [Collect all tenants major compactions6,1d78175792755325,d1b97e45ef543fb6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:11:57.923 INFO 2981023 --- [Collect all tenants major compactions2,1d78175792755325,f0c6bf793cf9d71b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:11:57.923 INFO 2981023 --- [Collect all tenants major compactions1,1d78175792755325,be5b909059f7c4d2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:11:57.923 INFO 2981023 --- [Collect all tenants major compactions3,1d78175792755325,ab00599dab98a7eb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:11:57.923 INFO 2981023 --- [Collect all tenants major compactions1,1d78175792755325,be5b909059f7c4d2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:11:57.923 INFO 2981023 --- [Collect all tenants major compactions3,1d78175792755325,ab00599dab98a7eb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:11:57.923 INFO 2981023 --- [Collect all tenants major compactions2,1d78175792755325,f0c6bf793cf9d71b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:11:57.924 INFO 2981023 --- [Collect all tenants major compactions4,1d78175792755325,eca7fe898ca33071] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:11:57.924 INFO 2981023 --- [Collect all tenants major compactions4,1d78175792755325,eca7fe898ca33071] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:11:57.925 INFO 2981023 --- [Collect all tenants major compactions6,1d78175792755325,d1b97e45ef543fb6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:11:57.925 INFO 2981023 --- [Collect all tenants major compactions6,1d78175792755325,d1b97e45ef543fb6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:11:57.925 INFO 2981023 --- [Collect all tenants major compactions5,1d78175792755325,93fa43f42c12c54f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:11:57.925 INFO 2981023 --- [Collect all tenants major compactions5,1d78175792755325,93fa43f42c12c54f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:11:57.926 INFO 2981023 --- [ocp-schedules-11,1d78175792755325,ef79432015c8ff89] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:11:58.368 INFO 2981023 --- [ocp-schedules-21,1128fa840329b622,ce2a39f93b8fdd22] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:11:58.368+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:11:58.372 INFO 2981023 --- [ocp-schedules-21,1128fa840329b622,ce2a39f93b8fdd22] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:11:58.516 INFO 2981023 --- [ocp-schedules-21,1128fa840329b622,ce2a39f93b8fdd22] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 17 to SqlAggregateData 2024-09-11 18:11:58.518 INFO 2981023 --- [ocp-schedules-21,1128fa840329b622,ce2a39f93b8fdd22] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:11:59.106 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d2bf0238064e0530,519715431f06a787] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=d2bf0238064e0530, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:11:59.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d2bf0238064e0530,519715431f06a787] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=d2bf0238064e0530, duration=11 ms] 2024-09-11 18:11:59.258 INFO 2981023 --- [ocp-schedules-4,9393de5e640efbb2,a8fccb413b4c9dc6] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:11:59.259 INFO 2981023 --- [ocp-schedules-4,9393de5e640efbb2,a8fccb413b4c9dc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:11:59.287 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:11:59.287 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.291 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:11:59.296 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:11:59.296 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.299 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:11:59.300 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:11:59.301 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.304 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.305 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:11:59.305 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.308 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.308 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.309 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.310 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.310 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.313 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,0d5316edf758a79f,201895735c22eac1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049520, interval=60 2024-09-11 18:11:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,0d5316edf758a79f,201895735c22eac1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:11:59.328 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.328 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.329 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.329 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.329 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.329 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:11:59.337 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.339 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:11:59.339 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.340 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.340 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:11:59.340 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5041D6D7-0-0] 2024-09-11 18:11:59.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:11:59.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:11:59.343 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.343 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.346 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.346 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:11:59.353 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.353 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:11:59.356 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.356 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:11:59.358 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.359 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:11:59.359 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.360 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:11:59.361 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.361 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:11:59.362 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:11:59.366 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:11:59.366 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:11:59.367 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:11:59.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5041D6E1-0-0] 2024-09-11 18:11:59.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:11:59.369 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.369 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:11:59.370 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.370 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:11:59.371 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.372 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:11:59.373 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.373 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:11:59.374 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.374 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.375 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:11:59.375 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:11:59.375 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.375 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.376 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:11:59.376 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:11:59.376 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.376 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.377 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:11:59.377 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:11:59.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.378 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:11:59.378 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:11:59.379 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.379 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:11:59.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:11:59.380 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:11:59.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.381 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:11:59.381 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:11:59.382 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:11:59.384 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:11:59.386 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:59.388 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.388 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.396 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:59.396 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:11:59.397 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.397 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.397 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:11:59.398 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:11:59.401 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:11:59.403 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:11:59.405 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:11:59.405 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:11:59.405 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:11:59.405 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:11:59.405 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.406 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:11:59.408 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:11:59.409 INFO 2981023 --- [ocp-async-2,88bcb0baefdbd91e,d2ca2a17540388ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:11:59.412 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:11:59.414 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:11:59.414 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:11:59.414 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.414 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:11:59.414 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.414 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:11:59.420 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:11:59.422 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:11:59.422 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:11:59.422 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:11:59.425 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:11:59.427 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:11:59.427 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:11:59.427 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:11:59.428 INFO 2981023 --- [ocp-schedules-4,9393de5e640efbb2,a8fccb413b4c9dc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:11:59.458 INFO 2981023 --- [ocp-schedules-13,83a44cb7db8f04f8,d5d04cecba459d24] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:11:59.549 INFO 2981023 --- [alarm-detect-9,83a44cb7db8f04f8,bff71598da0e87dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645655, data is empty. 2024-09-11 18:11:59.553 INFO 2981023 --- [alarm-detect-9,83a44cb7db8f04f8,bff71598da0e87dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645672, data is empty. 2024-09-11 18:11:59.557 INFO 2981023 --- [alarm-detect-9,83a44cb7db8f04f8,bff71598da0e87dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645672, data is empty. 2024-09-11 18:11:59.576 INFO 2981023 --- [alarm-detect-1,83a44cb7db8f04f8,3ba9b735781ada9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645655, data is empty. 2024-09-11 18:11:59.577 WARN 2981023 --- [scanner-13623343-1,83a44cb7db8f04f8,01172420e80b902e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:11:59.581 INFO 2981023 --- [alarm-detect-1,83a44cb7db8f04f8,3ba9b735781ada9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645672, data is empty. 2024-09-11 18:11:59.584 INFO 2981023 --- [alarm-detect-14,83a44cb7db8f04f8,fd82a18c93a1b3ee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645655, data is empty. 2024-09-11 18:11:59.587 INFO 2981023 --- [alarm-detect-1,83a44cb7db8f04f8,3ba9b735781ada9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645672, data is empty. 2024-09-11 18:11:59.591 INFO 2981023 --- [alarm-detect-14,83a44cb7db8f04f8,fd82a18c93a1b3ee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645672, data is empty. 2024-09-11 18:11:59.596 INFO 2981023 --- [alarm-detect-14,83a44cb7db8f04f8,fd82a18c93a1b3ee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645672, data is empty. 2024-09-11 18:11:59.695 INFO 2981023 --- [alarm-detect-7,83a44cb7db8f04f8,e0bb8434100d39fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645655, data is empty. 2024-09-11 18:11:59.700 INFO 2981023 --- [alarm-detect-7,83a44cb7db8f04f8,e0bb8434100d39fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645672, data is empty. 2024-09-11 18:11:59.706 INFO 2981023 --- [alarm-detect-7,83a44cb7db8f04f8,e0bb8434100d39fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645672, data is empty. 2024-09-11 18:12:00.017 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:00.019 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:01.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2fd88090783e2d80,65456c97d8d6a5c7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=2fd88090783e2d80, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:01.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,2fd88090783e2d80,65456c97d8d6a5c7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=2fd88090783e2d80, duration=11 ms] 2024-09-11 18:12:01.409 INFO 2981023 --- [ocp-async-3,9d6cde7b52a4ba43,be9777314de1aa34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,0e13beb4231fc4fe,11274774cc37a072] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,0e13beb4231fc4fe,11274774cc37a072] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:02.578 INFO 2981023 --- [ocp-schedules-13,10277f559eaa3676,05892bf91a2f64ab] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:12:02.874 INFO 2981023 --- [ocp-schedules-9,67f5c2445486314a,cfe2674a676814d8] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:12:02.875 INFO 2981023 --- [ocp-schedules-9,67f5c2445486314a,cfe2674a676814d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:02.896 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:02.896 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:12:02.901 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:02.903 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:12:02.904 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049522904000} 2024-09-11 18:12:02.909 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:12:02.913 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000230, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:12:02.913 INFO 2981023 --- [ocp-schedules-9,67f5c2445486314a,cfe2674a676814d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:12:02.984 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:12:02.984 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:12:02.984 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:12:02.988 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000230, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:12:02.991 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:12:03.019 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:03.019 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:03.121 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,b61c456ed5184480,b236a80dd2d21ba7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b61c456ed5184480, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:03.127 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,b61c456ed5184480,b236a80dd2d21ba7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b61c456ed5184480, duration=9 ms] 2024-09-11 18:12:03.153 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000230 2024-09-11 18:12:03.409 INFO 2981023 --- [ocp-async-8,5b2e92ffef88ebfd,30d3848428d0fc72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:04.178 INFO 2981023 --- [ocp-schedules-13,aad252915846d6f8,3fae8a9ee6b8c8d5] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:12:05.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ce4ca327516a0f25,9cf8129f74b8e3d5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=ce4ca327516a0f25, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:05.125 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ce4ca327516a0f25,9cf8129f74b8e3d5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=ce4ca327516a0f25, duration=13 ms] 2024-09-11 18:12:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,35afd1d22014e8bd,961ef21ee161fae7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,35afd1d22014e8bd,961ef21ee161fae7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:05.409 INFO 2981023 --- [ocp-async-10,e86b2dd5af76d6ed,4be8f0b4f1ec1050] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:05.968 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:12:05.988 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:12:05.993 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:12:05.999 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:12:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:12:05.999 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:12:06.001 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T18:09:41, target=2024-09-11T18:07:41, granule=120000ms, delay=120000ms) 2024-09-11 18:12:06.001 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:07:41, target=2024-09-11T18:00:51, granule=600000ms, delay=410000ms) 2024-09-11 18:12:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=33ms, period=120000ms 2024-09-11 18:12:06.019 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:06.020 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:12:06.353017+08:00, lastProcessTime=2024-09-11T18:11:56.351510+08:00 2024-09-11 18:12:06.484 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:12:06.353017+08:00 2024-09-11 18:12:07.146 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,9680fcc7a0c646b3,ea56e50d37edf49b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=9680fcc7a0c646b3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:07.152 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,9680fcc7a0c646b3,ea56e50d37edf49b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=9680fcc7a0c646b3, duration=9 ms] 2024-09-11 18:12:07.409 INFO 2981023 --- [ocp-async-2,d533e56a1a0c8579,8f2f1da49b0f396f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:07.785 INFO 2981023 --- [ocp-async-4,3b439cc7d0a2c9c3,bb9a779ba5487c06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:12:07.907 INFO 2981023 --- [ocp-schedules-30,58f47e194b42616a,a11ab85db224526e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:12:08.023 INFO 2981023 --- [ocp-schedules-3,34802bf6204a4df6,0845fbe19fa40adf] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:12:08.023+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:12:08.024 INFO 2981023 --- [ocp-schedules-3,34802bf6204a4df6,0845fbe19fa40adf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:12:08.026 INFO 2981023 --- [ocp-schedules-3,34802bf6204a4df6,0845fbe19fa40adf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:12:08.027 INFO 2981023 --- [ocp-schedules-3,34802bf6204a4df6,0845fbe19fa40adf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:12:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,e325c2ae04d08dd8,b5b9dca7f4906590] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,e325c2ae04d08dd8,b5b9dca7f4906590] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:09.020 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:09.021 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:09.109 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3d5b52f817aef932,090e599b8dd66efa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=3d5b52f817aef932, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:09.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3d5b52f817aef932,090e599b8dd66efa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=3d5b52f817aef932, duration=9 ms] 2024-09-11 18:12:09.410 INFO 2981023 --- [ocp-async-9,7b32f907bd5c1951,e248dc37e7286aff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:09.743 INFO 2981023 --- [ocp-schedules-10,f47d3fc73d8c3df4,268b43e18feed4f0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:12:09.815 WARN 2981023 --- [scanner-13623632-1,f47d3fc73d8c3df4,10227514d7547ed2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:12:10.002 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:12:10.003 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:12:10.105 INFO 2981023 --- [ocp-schedules-18,935eb983d3c8d09f,12daef4b34e630d5] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-09-11 18:12:10.105 INFO 2981023 --- [ocp-schedules-18,935eb983d3c8d09f,12daef4b34e630d5] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-09-11 18:12:10.105 INFO 2981023 --- [ocp-schedules-18,935eb983d3c8d09f,12daef4b34e630d5] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=11798 2024-09-11 18:12:10.327 INFO 2981023 --- [ocp-schedules-18,935eb983d3c8d09f,12daef4b34e630d5] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=11798 2024-09-11 18:12:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:12:10.920 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:12:11.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,059be52cd76a5a0e,1ecaeaec21164cab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=059be52cd76a5a0e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:11.121 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,059be52cd76a5a0e,1ecaeaec21164cab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=059be52cd76a5a0e, duration=9 ms] 2024-09-11 18:12:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,a85f11102ee10f22,e64a5784f5d1f2e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,a85f11102ee10f22,e64a5784f5d1f2e2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:11.410 INFO 2981023 --- [ocp-async-4,6557f05507b55965,51ea909663aca59e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:12.021 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:12.024 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:12.289 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049460, nodeCount=1000, affectRows=1000 2024-09-11 18:12:12.292 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049460, nodeCount=999, validCount=999 2024-09-11 18:12:12.307 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049460, nodeCount=1000, affectRows=1000 2024-09-11 18:12:12.308 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049460, nodeCount=1000, validCount=1000 2024-09-11 18:12:12.309 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049460, nodeCount=49, affectRows=49 2024-09-11 18:12:12.309 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049460, nodeCount=47, validCount=47 2024-09-11 18:12:12.580 INFO 2981023 --- [ocp-schedules-7,dda758b59f93812f,ae312ea65e8ae1bd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:12:12.662 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049460, nodeCount=480, affectRows=480 2024-09-11 18:12:13.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,b267c09bca3d838d,2455f6e579963e1d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b267c09bca3d838d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:13.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,b267c09bca3d838d,2455f6e579963e1d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b267c09bca3d838d, duration=11 ms] 2024-09-11 18:12:13.301 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 18:12:13.311 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:13.314 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:13.369 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:12:13.370 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:12:13.382 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:13.382 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:13.393 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:12:13.394 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:12:13.405 INFO 2981023 --- [ocp-schedules-1,9d77ef8dc7b001bd,f531a5a9522880ff] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2024-09-11 18:12:13.410 INFO 2981023 --- [ocp-async-1,530457ba3106bada,31d4eca0961c8fbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,38ee84bbbd85dfdd,5fae4e610d97b515] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,25de938f18390aaf,ae423b2cfe66772e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:12:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,38ee84bbbd85dfdd,5fae4e610d97b515] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:14.437 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049460, nodeCount=480, validCount=480 2024-09-11 18:12:15.024 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:15.024 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:15.199 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,1e92fc6feb0e0386,ce3f84f10d119661] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=1e92fc6feb0e0386, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:15.205 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,1e92fc6feb0e0386,ce3f84f10d119661] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=1e92fc6feb0e0386, duration=9 ms] 2024-09-11 18:12:15.301 INFO 2981023 --- [ocp-schedules-12,78261f6fd757c3f5,04343aca15cea914] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:12:15.311 INFO 2981023 --- [ocp-schedules-12,78261f6fd757c3f5,04343aca15cea914] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:12:15.311 INFO 2981023 --- [ocp-schedules-12,78261f6fd757c3f5,04343aca15cea914] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:12:15.410 INFO 2981023 --- [ocp-async-3,9d269ac58c036ec9,6fe0684714c4a58b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:16.024 INFO 2981023 --- [ocp-schedules-23,7e5a5a4fe922c502,60ccdf1fe62e65c8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:12:16.025 INFO 2981023 --- [ocp-schedules-18,a5a70a0044eddce2,718050f9866cb7f0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:12:16.025 INFO 2981023 --- [host-check-clockdiff1,7e5a5a4fe922c502,0d98257105fdfbf0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:12:16.026 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:12:16.026 INFO 2981023 --- [host-check-clockdiff2,7e5a5a4fe922c502,bae5790a38d1bf58] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:12:16.026 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:12:16.028 INFO 2981023 --- [ocp-schedules-10,3eae2a0c0b29120d,4240af3082aaf3ac] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:12:16.028 INFO 2981023 --- [host-check-clockdiff3,7e5a5a4fe922c502,0ff350ad04dde2b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:12:16.028 INFO 2981023 --- [host-check-clockdiff4,7e5a5a4fe922c502,bdfc175db5256199] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:12:16.032 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:12:16.035 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:12:16.046 INFO 2981023 --- [host-check-agent-status1,3eae2a0c0b29120d,cda1bb018029d3f3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:12:16.046 INFO 2981023 --- [host-check-agent-status2,3eae2a0c0b29120d,aed19515407b172c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:12:16.047 INFO 2981023 --- [host-check-agent-status3,3eae2a0c0b29120d,bf269e39f89b111d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:12:16.047 INFO 2981023 --- [host-check-agent-status4,3eae2a0c0b29120d,eb856207759fcc55] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:12:16.052 INFO 2981023 --- [host-check-clockdiff2,7e5a5a4fe922c502,bae5790a38d1bf58] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049536 5 5 , error= 2024-09-11 18:12:16.059 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:12:16.059 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:12:16.060 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:12:16.060 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:12:16.066 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:16.071 INFO 2981023 --- [host-check-clockdiff3,7e5a5a4fe922c502,0ff350ad04dde2b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049536 0 0 , error= 2024-09-11 18:12:16.074 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:16.075 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:16.088 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:16.091 INFO 2981023 --- [host-check-agent-status3,3eae2a0c0b29120d,bf269e39f89b111d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:16.092 INFO 2981023 --- [host-check-agent-status3,3eae2a0c0b29120d,bf269e39f89b111d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:16.094 INFO 2981023 --- [host-check-agent-status2,3eae2a0c0b29120d,aed19515407b172c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:12:16.094 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:12:16.098 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:12:16.096607696+08:00 2024-09-11 18:12:16.098 INFO 2981023 --- [host-check-agent-status2,3eae2a0c0b29120d,aed19515407b172c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:12:16.099 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:12:16.098+08:00 2024-09-11 18:12:16.103 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:12:16.098+08:00 and status AVAILABLE 2024-09-11 18:12:16.103 INFO 2981023 --- [host-check-agent-status4,3eae2a0c0b29120d,eb856207759fcc55] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:12:16.107 INFO 2981023 --- [host-check-agent-status1,3eae2a0c0b29120d,cda1bb018029d3f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:16.108 INFO 2981023 --- [host-check-agent-status1,3eae2a0c0b29120d,cda1bb018029d3f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:16.109 INFO 2981023 --- [host-check-agent-status4,3eae2a0c0b29120d,eb856207759fcc55] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:12:16.115 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:12:16.116 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:12:16.117 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:12:16.117286229+08:00 2024-09-11 18:12:16.117 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:12:16.117+08:00 2024-09-11 18:12:16.118 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:12:16.123675648+08:00 2024-09-11 18:12:16.119 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:12:16.117+08:00 and status AVAILABLE 2024-09-11 18:12:16.119 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:12:16.118+08:00 2024-09-11 18:12:16.121 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:12:16.118+08:00 and status AVAILABLE 2024-09-11 18:12:16.131 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:12:16.141 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:12:16.141 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:12:16.143 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:12:16.148935001+08:00 2024-09-11 18:12:16.144 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:12:16.147 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:12:16.143+08:00 2024-09-11 18:12:16.159 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:12:16.143+08:00 and status AVAILABLE 2024-09-11 18:12:16.174 INFO 2981023 --- [host-check-agent-status1,3eae2a0c0b29120d,cda1bb018029d3f3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:12:16.177 INFO 2981023 --- [host-check-agent-status1,3eae2a0c0b29120d,cda1bb018029d3f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:12:16.177 INFO 2981023 --- [host-check-agent-status1,3eae2a0c0b29120d,cda1bb018029d3f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:12:16.177 INFO 2981023 --- [host-check-agent-status1,3eae2a0c0b29120d,cda1bb018029d3f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:12:16.181 INFO 2981023 --- [host-check-agent-status3,3eae2a0c0b29120d,bf269e39f89b111d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:12:16.184 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:12:16.184 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:12:16.189 INFO 2981023 --- [ocp-schedules-3,ac179f9db84bc800,826bf3bdf7e64530] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:12:16.193 INFO 2981023 --- [ocp-schedules-15,e161610275f5e919,d9797016a6d3fca9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:12:16.193 INFO 2981023 --- [ocp-schedules-4,6d211a10e19a0f9f,fc2ad4aade0726e9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:12:16.193 INFO 2981023 --- [ocp-schedules-26,2f8e20bd55e1e4e8,5869301586560605] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:12:16.194 INFO 2981023 --- [host-check-agent-status3,3eae2a0c0b29120d,bf269e39f89b111d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:12:16.194 INFO 2981023 --- [host-check-agent-status3,3eae2a0c0b29120d,bf269e39f89b111d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:12:16.194 INFO 2981023 --- [host-check-agent-status3,3eae2a0c0b29120d,bf269e39f89b111d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:12:16.198 INFO 2981023 --- [ocp-schedules-3,ac179f9db84bc800,826bf3bdf7e64530] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:16.200 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:12:16.201 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:12:16.204 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:12:16.206 INFO 2981023 --- [ocp-schedules-4,6d211a10e19a0f9f,fc2ad4aade0726e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:16.209 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:16.209 INFO 2981023 --- [ocp-schedules-15,e161610275f5e919,d9797016a6d3fca9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:16.225 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049522904000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:12:16.233 INFO 2981023 --- [host-check-agent-status2,3eae2a0c0b29120d,aed19515407b172c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:12:16.239 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:16.240 INFO 2981023 --- [host-check-agent-status2,3eae2a0c0b29120d,aed19515407b172c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:12:16.240 INFO 2981023 --- [host-check-agent-status2,3eae2a0c0b29120d,aed19515407b172c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:12:16.240 INFO 2981023 --- [host-check-agent-status2,3eae2a0c0b29120d,aed19515407b172c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:12:16.241 INFO 2981023 --- [host-check-agent-status4,3eae2a0c0b29120d,eb856207759fcc55] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:12:16.243 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:16.244 INFO 2981023 --- [ocp-schedules-17,6feb1ac8ab448e8e,72020a7ba9be2849] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:12:16.245 INFO 2981023 --- [host-check-agent-status4,3eae2a0c0b29120d,eb856207759fcc55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:12:16.245 INFO 2981023 --- [host-check-agent-status4,3eae2a0c0b29120d,eb856207759fcc55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:12:16.245 INFO 2981023 --- [host-check-agent-status4,3eae2a0c0b29120d,eb856207759fcc55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:12:16.249 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:16.271 INFO 2981023 --- [ocp-schedules-15,e161610275f5e919,d9797016a6d3fca9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:12:16.274 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:12:16.282 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.283 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:12:16.283 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:12:16.285 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:12:16.287 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:12:16.291 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:12:16.295 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:12:16.297 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.300 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.301 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:12:16.301 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:16.301 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:12:16.302 INFO 2981023 --- [ocp-schedules-3,ac179f9db84bc800,826bf3bdf7e64530] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:12:16.311 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:12:16.315 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:12:16.318 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:12:16.320 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.320 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:12:16.320 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:16.336 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:12:16.345 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.346 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:16.348 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:12:16.349 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:16.350 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.351 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:12:16.359 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:12:16.360 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:12:16.361 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:12:16.361286+08:00, lastProcessTime=2024-09-11T18:12:06.353017+08:00 2024-09-11 18:12:16.362 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:12:16.363 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:12:16.383 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:12:16.383 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:12:16.383 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:16.387 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:12:16.388 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.388 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:16.388 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:12:16.389 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.391 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:12:16.391 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:12:16.393 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:12:16.398 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:12:16.398 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:12:16.406 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:12:16.408 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:12:16.408 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:12:16.431 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:12:16.431 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049522904000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:12:16.432 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:12:16.436 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:12:16.437 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:12:16.438 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:12:16.439 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:12:16.443 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:16.443 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.444 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:12:16.444 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:12:16.444 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:12:16.448 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:12:16.452 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:16.453 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:16.460 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:12:16.461 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:12:16.465 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:12:16.473 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:12:16.474 INFO 2981023 --- [host-check-host-status4,a5a70a0044eddce2,33c6495216efa493] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:12:16.477 INFO 2981023 --- [host-check-host-status1,a5a70a0044eddce2,c505d8149b59524e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:12:16.484 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:16.485 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:12:16.485 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.486 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:12:16.488 INFO 2981023 --- [host-check-host-status2,a5a70a0044eddce2,8861f421b92cb1af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:12:16.510 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:12:16.517 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:16.517 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.517 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:16.518 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:12:16.518 INFO 2981023 --- [host-check-host-status3,a5a70a0044eddce2,de316b68a0d0b26c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:12:16.518 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.521 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:12:16.521 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:12:16.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:12:16.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:12:16.548 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:12:16.554 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:16.554 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.555 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:12:16.556 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:12:16.558 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:12:16.561 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:16.562 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:16.563 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:12:16.575 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:12:16.577 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:12:16.584 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:12:16.584 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:12:16.591 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:12:16.361286+08:00 2024-09-11 18:12:16.609 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:12:16.613 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:16.614 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:16.614 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:12:16.619 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:12:16.619 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:12:16.619 INFO 2981023 --- [ocp-schedules-4,6d211a10e19a0f9f,fc2ad4aade0726e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:12:17.113 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f0557dbaf5e6462f,fcb05ec2cf004709] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=f0557dbaf5e6462f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:17.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,f0557dbaf5e6462f,fcb05ec2cf004709] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=f0557dbaf5e6462f, duration=9 ms] 2024-09-11 18:12:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,6aeccdd495acd183,893fd5195b5d35be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,6aeccdd495acd183,893fd5195b5d35be] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:17.410 INFO 2981023 --- [ocp-async-8,88198c9c8d26d06a,3b483f8c692a0cc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:17.785 INFO 2981023 --- [ocp-async-5,0ba873111530b204,f104971687791c09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:12:18.024 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:18.025 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:19.101 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,56c2e638a10b9519,cb6ec2a5ccb1f76a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=56c2e638a10b9519, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:19.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,56c2e638a10b9519,cb6ec2a5ccb1f76a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=56c2e638a10b9519, duration=10 ms] 2024-09-11 18:12:19.410 INFO 2981023 --- [ocp-async-2,cf092284a8b10f48,df689b8dbc59c631] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:19.739 INFO 2981023 --- [host-check-clockdiff1,7e5a5a4fe922c502,0d98257105fdfbf0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049539 6 6 , error= 2024-09-11 18:12:19.975 INFO 2981023 --- [ocp-schedules-12,74e571752136cadf,53bdac2937dcefea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:12:20.018 WARN 2981023 --- [scanner-13625349-1,74e571752136cadf,59eeafc08b981a59] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:12:20.048 INFO 2981023 --- [alarm-detect-18,74e571752136cadf,99ef20859b3ca971] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645672, data is empty. 2024-09-11 18:12:20.053 INFO 2981023 --- [alarm-detect-18,74e571752136cadf,99ef20859b3ca971] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645672, data is empty. 2024-09-11 18:12:20.059 INFO 2981023 --- [alarm-detect-18,74e571752136cadf,99ef20859b3ca971] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645672, data is empty. 2024-09-11 18:12:20.071 INFO 2981023 --- [alarm-detect-17,74e571752136cadf,2cb6fc34dd372b38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645672, data is empty. 2024-09-11 18:12:20.075 INFO 2981023 --- [alarm-detect-17,74e571752136cadf,2cb6fc34dd372b38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645672, data is empty. 2024-09-11 18:12:20.080 INFO 2981023 --- [alarm-detect-9,74e571752136cadf,e52ca8c672aef550] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645672, data is empty. 2024-09-11 18:12:20.083 INFO 2981023 --- [alarm-detect-17,74e571752136cadf,2cb6fc34dd372b38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645672, data is empty. 2024-09-11 18:12:20.087 INFO 2981023 --- [alarm-detect-9,74e571752136cadf,e52ca8c672aef550] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645672, data is empty. 2024-09-11 18:12:20.091 INFO 2981023 --- [alarm-detect-9,74e571752136cadf,e52ca8c672aef550] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645672, data is empty. 2024-09-11 18:12:20.207 INFO 2981023 --- [alarm-detect-9,74e571752136cadf,e2d5fb6ebbf27d5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645672, data is empty. 2024-09-11 18:12:20.212 INFO 2981023 --- [alarm-detect-9,74e571752136cadf,e2d5fb6ebbf27d5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645672, data is empty. 2024-09-11 18:12:20.215 INFO 2981023 --- [alarm-detect-9,74e571752136cadf,e2d5fb6ebbf27d5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645672, data is empty. 2024-09-11 18:12:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,a10a35fa0cea02f1,420915776ab0e757] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,a10a35fa0cea02f1,420915776ab0e757] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:21.025 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:21.026 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:21.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,80adfb27ffa26056,de882053c7791491] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=80adfb27ffa26056, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:21.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,80adfb27ffa26056,de882053c7791491] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=80adfb27ffa26056, duration=9 ms] 2024-09-11 18:12:21.410 INFO 2981023 --- [ocp-async-5,85e80f00487ccd77,1d2072ea55ad41bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:21.922 INFO 2981023 --- [host-check-clockdiff4,7e5a5a4fe922c502,bdfc175db5256199] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049541 0 0 , error= 2024-09-11 18:12:22.583 INFO 2981023 --- [ocp-schedules-30,24bd03327d61bcf1,f9690fdc0cbbf92d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:12:23.115 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,e81996e62d0f5287,eb0336ffeb92d049] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=e81996e62d0f5287, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:23.121 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,e81996e62d0f5287,eb0336ffeb92d049] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=e81996e62d0f5287, duration=10 ms] 2024-09-11 18:12:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,4c7411d3dd8bccf6,bd4638f46bccbb82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,4c7411d3dd8bccf6,bd4638f46bccbb82] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:23.411 INFO 2981023 --- [ocp-async-7,7a2977b0605d6822,36860541ef59516c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:24.026 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:24.026 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:24.427 INFO 2981023 --- [ocp-schedules-12,badc42e0133062cf,baa87cdc9c6723d6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:12:25.110 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,605170af6e4a1e14,b4ba5a21f6d694cf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=605170af6e4a1e14, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:25.116 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,605170af6e4a1e14,b4ba5a21f6d694cf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=605170af6e4a1e14, duration=9 ms] 2024-09-11 18:12:25.411 INFO 2981023 --- [ocp-async-10,23fd383241137989,fda2d632c669945a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:26.000 INFO 2981023 --- [ocp-schedules-21,d5a9e6554bbb6500,2d4395df85213d16] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:12:26.000 INFO 2981023 --- [ocp-schedules-21,d5a9e6554bbb6500,2d4395df85213d16] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:12:26.000 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:12:26.000 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:12:26.000 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:12:26.021 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:26.022 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:26.025 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:12:26.025 INFO 2981023 --- [ocp-schedules-21,d5a9e6554bbb6500,2d4395df85213d16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:26.026 INFO 2981023 --- [ocp-schedules-21,d5a9e6554bbb6500,2d4395df85213d16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:26.026 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:12:26.026 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:12:26.027 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:12:26.039 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:12:26.042 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:12:26.044 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:12:26.045 INFO 2981023 --- [ocp-schedules-21,d5a9e6554bbb6500,2d4395df85213d16] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:12:26.047 INFO 2981023 --- [ocp-schedules-21,d5a9e6554bbb6500,2d4395df85213d16] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:12:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,2eb496da2596f6e5,05ded4eed0f00b94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,2eb496da2596f6e5,05ded4eed0f00b94] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:12:26.351565+08:00, lastProcessTime=2024-09-11T18:12:16.361286+08:00 2024-09-11 18:12:26.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:12:26.351565+08:00 2024-09-11 18:12:27.027 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:27.027 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:27.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,391cf75f6a54d92a,ae4889ab3b9eea1d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=391cf75f6a54d92a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:27.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,391cf75f6a54d92a,ae4889ab3b9eea1d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=391cf75f6a54d92a, duration=9 ms] 2024-09-11 18:12:27.411 INFO 2981023 --- [ocp-async-6,7576f134b6ecfd12,104b785513df2571] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:27.785 INFO 2981023 --- [ocp-async-1,34be63cb742ccb6c,1d58dd49beb2e92d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:12:29.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4814a83ba28eef1f,177259caf9e93291] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=4814a83ba28eef1f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:29.118 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,4814a83ba28eef1f,177259caf9e93291] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=4814a83ba28eef1f, duration=10 ms] 2024-09-11 18:12:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,4540d575ca229cf9,e015a4e4bb08d780] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,4540d575ca229cf9,e015a4e4bb08d780] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:29.411 INFO 2981023 --- [ocp-async-8,8f77b5d2c36a1e56,c501d1e102004f38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:30.027 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:30.028 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:30.252 INFO 2981023 --- [ocp-schedules-8,cf0e853d1e59c07f,4e2399a1e5cf8b84] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:12:30.302 WARN 2981023 --- [scanner-13625647-1,cf0e853d1e59c07f,1216276965f69978] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:12:31.112 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,bb93fa7886c5499b,a5d937678d588a00] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bb93fa7886c5499b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:31.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,bb93fa7886c5499b,a5d937678d588a00] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bb93fa7886c5499b, duration=12 ms] 2024-09-11 18:12:31.411 INFO 2981023 --- [ocp-async-1,28315d509c64bc5c,659658b455688769] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,f6eacb9accb3ae5e,e10cf512f98d02f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,f6eacb9accb3ae5e,e10cf512f98d02f4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:32.585 INFO 2981023 --- [ocp-schedules-19,8b164bb233fd106a,5e0a74197e4b8338] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:12:33.028 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:33.029 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:33.114 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6515812a88fdc381,06426b9cd62629fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=6515812a88fdc381, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:33.120 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6515812a88fdc381,06426b9cd62629fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=6515812a88fdc381, duration=9 ms] 2024-09-11 18:12:33.411 INFO 2981023 --- [ocp-async-3,f6fb0cf92b2bdbd4,85d55075545dbcf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:34.373 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,db75787771b66770,cfc55e105ec64723] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=db75787771b66770, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:12:34.391 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,db75787771b66770,cfc55e105ec64723] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=db75787771b66770, duration=34 ms] 2024-09-11 18:12:34.458 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2e14703b11b7828f,8042af9851d9bdc2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=2e14703b11b7828f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:12:34.473 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8c5e4ff8b561cef4,5a74e25483d60bc3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=8c5e4ff8b561cef4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:12:34.483 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,2e14703b11b7828f,8042af9851d9bdc2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=2e14703b11b7828f, duration=31 ms] 2024-09-11 18:12:34.484 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,8c5e4ff8b561cef4,5a74e25483d60bc3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=8c5e4ff8b561cef4, duration=32 ms] 2024-09-11 18:12:34.586 WARN 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] update failed, sql:[alter system add server ? zone ?], error message:[PreparedStatementCallback; SQL [alter system add server ? zone ?]; (conn=3221488696) Timeout; nested exception is java.sql.SQLTransientConnectionException: (conn=3221488696) Timeout] 2024-09-11 18:12:34.592 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2306-000621D2D95387AF-0-0] 2024-09-11 18:12:34.596 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=600094, sql=alter system add server ? zone ? 2024-09-11 18:12:34.601 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:34.605 ERROR 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Timeout java.sql.SQLException: Timeout at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2347) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2210) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2098) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:247) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:170) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeUpdate(JDBC4PreparedStatement.java:204) at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeUpdate(DruidPooledPreparedStatement.java:255) at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:967) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:650) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:962) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1017) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1027) at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.updateInner(ConnectTemplate.java:293) at com.oceanbase.ocp.obsdk.connector.ConnectTemplate.update(ConnectTemplate.java:282) at com.oceanbase.ocp.obsdk.operator.cluster.MysqlClusterOperator.addServer(MysqlClusterOperator.java:613) at com.oceanbase.ocp.service.task.business.server.ObServerTaskHandler.addObServer(ObServerTaskHandler.java:196) at com.oceanbase.ocp.service.task.business.server.AddObServerTask.run(AddObServerTask.java:53) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 18:12:34.621 INFO 2981023 --- [manual-subtask-executor16,81ea8781302714e9,6cbdc495294ae9c3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048209, operation:EXECUTE, state: FAILED 2024-09-11 18:12:34.925 INFO 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=4048203, name='Add OB zone', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1000006, tenantId=null, hostId=null, creator='admin', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={cluster_name=linkrtd, startup_option_string=large_query_threshold=1s,log_disk_percentage=85,system_cpu_quota=1,memory_limit_percentage=85,system_memory=4G,datafile_disk_percentage=85,cpu_count=6, mix_rpm_names={"zone2":"oceanbase-ce-4.2.4.0-100000082024070810.el7.x86_64.rpm"}, prohibit_rollback=false, service_name=linkrtd:1719994443, ob_log_disk_path=/oceanbase/redo, ob_run_path=/data/rtdob/oceanbase, service_version=4.2.4.0, ob_cluster_id=1719994443, ob_install_path=/data/rtdob/oceanbase, ob_sql_port=2881, host.2000010.zone=zone2, cluster_id=1000006, service_type=OB_CLUSTER, ob_disk_path_style=OBD, ob_svr_port=2882, dep_rpm_names={"zone2":["oceanbase-ce-libs-4.2.4.0-100000082024070810.el7.x86_64.rpm","oceanbase-ce-utils-4.2.4.0-100000082024070810.el7.x86_64.rpm"]}, ob_run_user=admin, ob_data_disk_path=/data/oceanbase, zone2.idc=default_idc, zone2.region=sys_region, mix_obs_rpm={}}, listMap={root_server_ips=[10.31.35.6], tenant_names=[sys], server_ids=[1000029], exists_server_addrs=[10.31.35.6:2882], host_ids=[2000010], exists_running_server_addrs=[10.31.35.6:2882], zone_names=[zone2]}}, createTime=2024-09-11T17:59:47+08:00, startTime=2024-09-11T17:59:47.664+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=4048215, name=Start observer process with param, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.StartObServerProcessWithParamTask, seriesId=19, startTime=2024-09-11T18:01:31.957+08:00, endTime=2024-09-11T18:02:32.237+08:00}, SubtaskInstanceEntity{id=4048201, name=Update zone status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.zone.UpdateObZoneStatusTask, seriesId=23, startTime=null, endTime=null}, SubtaskInstanceEntity{id=4048213, name=Wait node, state=SUCCESSFUL, o 2024-09-11 18:12:34.939 WARN 2981023 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:188) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:184) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:146) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:144) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:104) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:77) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:119) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:106) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 18:12:34.939 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048203, status:FAILED 2024-09-11 18:12:35.102 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b696572f66537095,1b73e25ba5358199] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=b696572f66537095, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:12:35.108 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b696572f66537095,1b73e25ba5358199] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=b696572f66537095, duration=9 ms] 2024-09-11 18:12:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,b4cac71bcd057b7d,2e8cdecb3d5620b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:35.314 INFO 2981023 --- [ocp-schedules-9,196471160d141971,33fc780462fb93bb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:12:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,b4cac71bcd057b7d,2e8cdecb3d5620b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:35.331 INFO 2981023 --- [ocp-schedules-9,196471160d141971,33fc780462fb93bb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:12:35.331 INFO 2981023 --- [ocp-schedules-9,196471160d141971,33fc780462fb93bb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:12:35.411 INFO 2981023 --- [ocp-async-9,fbebac6fa5ff3ab2,81fbf1b1b67eb93a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:36.029 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:36.029 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:12:36.337713+08:00, lastProcessTime=2024-09-11T18:12:26.351565+08:00 2024-09-11 18:12:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:12:36.337713+08:00 2024-09-11 18:12:37.314 INFO 2981023 --- [prometheus-cache-monitor1,ba66ce8845ae1e83,3cb7c0a76786b6fc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2050,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:12:37.315 INFO 2981023 --- [prometheus-cache-monitor1,ba66ce8845ae1e83,3cb7c0a76786b6fc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:12:37.411 INFO 2981023 --- [ocp-async-4,652405bacc21a98a,e19f0a502cfbe149] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:37.785 INFO 2981023 --- [ocp-async-7,77ffdc6d94f96684,4b1bce8abc989899] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:12:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,41b467e5100b28f9,173aec9de195d319] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,41b467e5100b28f9,173aec9de195d319] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:39.029 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:39.030 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:39.412 INFO 2981023 --- [ocp-async-6,cbaa2cf25ba2e672,b2a2878e83e935fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:40.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:12:40.003 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:12:40.434 INFO 2981023 --- [ocp-schedules-20,64f741bda0ec9ed7,6952b9ede6b25dd2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:12:40.483 WARN 2981023 --- [scanner-13625901-1,64f741bda0ec9ed7,d8bc38db3ca659f0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:12:40.500 INFO 2981023 --- [alarm-detect-5,64f741bda0ec9ed7,3c6849cfdeccbbc7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645672, data is empty. 2024-09-11 18:12:40.506 INFO 2981023 --- [alarm-detect-5,64f741bda0ec9ed7,3c6849cfdeccbbc7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645672, data is empty. 2024-09-11 18:12:40.511 INFO 2981023 --- [alarm-detect-5,64f741bda0ec9ed7,3c6849cfdeccbbc7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645672, data is empty. 2024-09-11 18:12:40.525 INFO 2981023 --- [alarm-detect-10,64f741bda0ec9ed7,f554b29dd19ea365] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645672, data is empty. 2024-09-11 18:12:40.528 INFO 2981023 --- [alarm-detect-1,64f741bda0ec9ed7,a4f89e67f604b29c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645672, data is empty. 2024-09-11 18:12:40.530 INFO 2981023 --- [alarm-detect-10,64f741bda0ec9ed7,f554b29dd19ea365] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645672, data is empty. 2024-09-11 18:12:40.532 INFO 2981023 --- [alarm-detect-1,64f741bda0ec9ed7,a4f89e67f604b29c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645672, data is empty. 2024-09-11 18:12:40.534 INFO 2981023 --- [alarm-detect-10,64f741bda0ec9ed7,f554b29dd19ea365] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645672, data is empty. 2024-09-11 18:12:40.536 INFO 2981023 --- [alarm-detect-1,64f741bda0ec9ed7,a4f89e67f604b29c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645672, data is empty. 2024-09-11 18:12:40.642 INFO 2981023 --- [alarm-detect-20,64f741bda0ec9ed7,c5bcf99ca0219c94] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645672, data is empty. 2024-09-11 18:12:40.645 INFO 2981023 --- [alarm-detect-20,64f741bda0ec9ed7,c5bcf99ca0219c94] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645672, data is empty. 2024-09-11 18:12:40.649 INFO 2981023 --- [alarm-detect-20,64f741bda0ec9ed7,c5bcf99ca0219c94] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645672, data is empty. 2024-09-11 18:12:40.950 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:12:40.950 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:12:40.953 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:12:40.953 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:12:40.953 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:12:40.953 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:12:40.955 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:12:40.955 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:12:40.955 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:12:40.955 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:12:40.977 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:12:40.982 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:12:40.982 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:12:40.982 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:12:40.982 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:12:40.982 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:12:40.986 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:12:41.223 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:12:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,4dbf172f97adb794,98841eac181a9099] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,4dbf172f97adb794,98841eac181a9099] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:41.412 INFO 2981023 --- [ocp-async-7,9b9af0aa3098deb9,64425017c78d2a5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:41.500 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.545353353S 2024-09-11 18:12:41.501 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:12:41.554 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:12:41.554 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:12:41.573 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:12:41.605 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:12:41.605 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:12:41.605 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:12:41.605 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:12:41.605 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:12:41.605 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:12:41.611 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:12:41.719 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] o.s.boot.SpringApplication : Started application in 0.766 seconds (JVM running for 85361.592) 2024-09-11 18:12:41.719 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.76651661S 2024-09-11 18:12:41.936 INFO 2981023 --- [ocp-schedules-14,04fa702b2bcb2716,1766a1064cf2a0ad] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:12:42.030 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:42.031 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:42.588 INFO 2981023 --- [ocp-schedules-20,008b9e4aaa8460b3,382960eeab71dab9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:12:43.412 INFO 2981023 --- [ocp-async-10,b7edc0aa28ab4397,f6c14bc5f3dc1142] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,3ab70c6e4eefa6f4,a368d1a8e57e7609] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:44.317 INFO 2981023 --- [prometheus-minute-schedule-1,3ab70c6e4eefa6f4,a368d1a8e57e7609] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:45.031 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:45.033 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:45.101 INFO 2981023 --- [prometheus-second-schedule-1,b26e8721eb8a59a8,6f809da6212700f6] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:12:45.412 INFO 2981023 --- [ocp-async-2,e76ad1732dbe1e14,3b10090414d06312] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:45.422 INFO 2981023 --- [ocp-schedules-17,013cd2652dee8716,1fff7b1ed450b284] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:12:45.442 WARN 2981023 --- [ocp-schedules-17,013cd2652dee8716,1fff7b1ed450b284] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:12:45.683 INFO 2981023 --- [ocp-schedules-26,54d56c028fa2f96d,c1173c5014076c49] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-09-11 18:12:45.685 INFO 2981023 --- [ocp-schedules-26,54d56c028fa2f96d,c1173c5014076c49] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:45.742 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:45.743 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:45.744 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:45.745 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 18:00:06.461339, 2024-09-11 18:12:45.691, 1000] 2024-09-11 18:12:45.746 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000006, startTime 2024-09-11 18:00:06.461339, endTime 2024-09-11 18:12:45.691 2024-09-11 18:12:45.747 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 10:40:18.658505, 2024-09-11 18:12:45.722, 1000] 2024-09-11 18:12:45.748 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 02:03:06.281634, 2024-09-11 18:12:45.724, 1000] 2024-09-11 18:12:45.749 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-09-11 02:03:06.281634, endTime 2024-09-11 18:12:45.724 2024-09-11 18:12:45.749 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2024-09-11 10:40:18.658505, endTime 2024-09-11 18:12:45.722 2024-09-11 18:12:45.759 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:45.762 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:45.783 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 18:02:45.492504, 2024-09-11 18:12:45.751, 1000] 2024-09-11 18:12:45.789 INFO 2981023 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:45.789 INFO 2981023 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:45.790 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:45.798 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:45.814 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 08:00:10.321716, 2024-09-11 18:12:45.783, 1000] 2024-09-11 18:12:45.815 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-09-11 08:00:10.321716, endTime 2024-09-11 18:12:45.783 2024-09-11 18:12:45.823 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 18:01:30.332457, 2024-09-11 18:12:45.782, 1000] 2024-09-11 18:12:45.826 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2024-09-11 18:01:30.332457, endTime 2024-09-11 18:12:45.782 2024-09-11 18:12:45.873 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1000 server events for cluster 1000006, startTime 2024-09-11 18:02:45.492504, endTime 2024-09-11 18:12:45.751, limit 1000, offset 0 2024-09-11 18:12:46.025 INFO 2981023 --- [ocp-schedules-2,ca6936392abd34b4,e5d75a126351680d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:12:46.025 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:12:46.027 INFO 2981023 --- [ocp-schedules-9,c58ad849ff1555f3,49ff55dd1d6712f9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:12:46.032 INFO 2981023 --- [ocp-schedules-7,97cfcee09b14e87d,5b71a0c9a2b5140c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:12:46.032 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:12:46.032 INFO 2981023 --- [host-check-agent-status1,97cfcee09b14e87d,2645c9df4718cc6e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:12:46.033 INFO 2981023 --- [host-check-clockdiff1,c58ad849ff1555f3,fcfef37b5db693c7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:12:46.037 INFO 2981023 --- [host-check-clockdiff2,c58ad849ff1555f3,c30e22b9019b65be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:12:46.044 INFO 2981023 --- [host-check-clockdiff3,c58ad849ff1555f3,4eb0dc4c40135428] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:12:46.044 INFO 2981023 --- [host-check-clockdiff4,c58ad849ff1555f3,96d3ffb95cdf1a23] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:12:46.055 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:46.059 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:46.063 INFO 2981023 --- [host-check-agent-status2,97cfcee09b14e87d,7acc6b65becedad9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:12:46.063 INFO 2981023 --- [host-check-agent-status3,97cfcee09b14e87d,3a57b600ce6e4f2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:12:46.065 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:12:46.066 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:12:46.074 INFO 2981023 --- [host-check-clockdiff1,c58ad849ff1555f3,fcfef37b5db693c7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049566 5 5 , error= 2024-09-11 18:12:46.074 INFO 2981023 --- [host-check-agent-status4,97cfcee09b14e87d,21f6a657b269aa68] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:12:46.088 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:46.089 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:46.093 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:12:46.093 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:12:46.097 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:12:46.097 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:12:46.132 INFO 2981023 --- [host-check-agent-status1,97cfcee09b14e87d,2645c9df4718cc6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:46.146 INFO 2981023 --- [host-check-clockdiff3,c58ad849ff1555f3,4eb0dc4c40135428] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049566 0 0 , error= 2024-09-11 18:12:46.147 INFO 2981023 --- [host-check-clockdiff2,c58ad849ff1555f3,c30e22b9019b65be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049566 5 5 , error= 2024-09-11 18:12:46.148 INFO 2981023 --- [host-check-agent-status1,97cfcee09b14e87d,2645c9df4718cc6e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:46.169 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:12:46.175 INFO 2981023 --- [host-check-agent-status3,97cfcee09b14e87d,3a57b600ce6e4f2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:46.186 INFO 2981023 --- [ocp-schedules-30,a32b01057444563c,dd51ab328fe4765d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:12:46.198 INFO 2981023 --- [ocp-schedules-28,1fb0ac4b34ba1f7a,9b3bb38770e073cd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:12:46.198 INFO 2981023 --- [ocp-schedules-23,1a911fcfe16f4a27,3d23bd34888be2db] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:12:46.200 INFO 2981023 --- [ocp-schedules-24,c7510476da6af12c,acab5ecc099f05c0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:12:46.200 INFO 2981023 --- [ocp-schedules-22,43b13df4d20a3f3e,bfe6d49cad63beaf] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:12:46.203 INFO 2981023 --- [host-check-agent-status2,97cfcee09b14e87d,7acc6b65becedad9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:12:46.203 INFO 2981023 --- [ocp-schedules-23,1a911fcfe16f4a27,3d23bd34888be2db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:46.205 INFO 2981023 --- [ocp-schedules-28,1fb0ac4b34ba1f7a,9b3bb38770e073cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:46.205 INFO 2981023 --- [host-check-agent-status3,97cfcee09b14e87d,3a57b600ce6e4f2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:46.234 INFO 2981023 --- [host-check-agent-status2,97cfcee09b14e87d,7acc6b65becedad9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:12:46.246 INFO 2981023 --- [ocp-schedules-23,1a911fcfe16f4a27,3d23bd34888be2db] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:12:46.246 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049522904000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:12:46.266 INFO 2981023 --- [ocp-schedules-24,c7510476da6af12c,acab5ecc099f05c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:46.366 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:12:46.372 INFO 2981023 --- [host-check-agent-status4,97cfcee09b14e87d,21f6a657b269aa68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:12:46.374 INFO 2981023 --- [host-check-agent-status4,97cfcee09b14e87d,21f6a657b269aa68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:12:46.378 INFO 2981023 --- [ocp-schedules-30,a32b01057444563c,dd51ab328fe4765d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:12:46.404 INFO 2981023 --- [ocp-schedules-27,b3caff99bb64c531,0fa834a67eed9de2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:12:46.409 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:12:46.411 INFO 2981023 --- [host-check-agent-status3,97cfcee09b14e87d,3a57b600ce6e4f2b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:12:46.413 INFO 2981023 --- [host-check-agent-status1,97cfcee09b14e87d,2645c9df4718cc6e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:12:46.418 INFO 2981023 --- [host-check-agent-status1,97cfcee09b14e87d,2645c9df4718cc6e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:12:46.418 INFO 2981023 --- [host-check-agent-status1,97cfcee09b14e87d,2645c9df4718cc6e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:12:46.418 INFO 2981023 --- [host-check-agent-status1,97cfcee09b14e87d,2645c9df4718cc6e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:12:46.418 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:12:46.411414259+08:00 2024-09-11 18:12:46.419 INFO 2981023 --- [host-check-agent-status3,97cfcee09b14e87d,3a57b600ce6e4f2b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:12:46.419 INFO 2981023 --- [host-check-agent-status3,97cfcee09b14e87d,3a57b600ce6e4f2b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:12:46.419 INFO 2981023 --- [host-check-agent-status3,97cfcee09b14e87d,3a57b600ce6e4f2b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:12:46.455 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:12:46.418+08:00 2024-09-11 18:12:46.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:12:46.456107+08:00, lastProcessTime=2024-09-11T18:12:36.337713+08:00 2024-09-11 18:12:46.459 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.459 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:12:46.460 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:12:46.464 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:12:46.418+08:00 and status AVAILABLE 2024-09-11 18:12:46.470 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.471 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:12:46.472 INFO 2981023 --- [host-check-agent-status2,97cfcee09b14e87d,7acc6b65becedad9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:12:46.476 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:12:46.476 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.476 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:46.476 INFO 2981023 --- [ocp-schedules-28,1fb0ac4b34ba1f7a,9b3bb38770e073cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:12:46.476 INFO 2981023 --- [host-check-agent-status2,97cfcee09b14e87d,7acc6b65becedad9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:12:46.476 INFO 2981023 --- [host-check-agent-status2,97cfcee09b14e87d,7acc6b65becedad9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:12:46.476 INFO 2981023 --- [host-check-agent-status2,97cfcee09b14e87d,7acc6b65becedad9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:12:46.478 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:12:46.479 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:12:46.484687704+08:00 2024-09-11 18:12:46.480 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:12:46.480 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:12:46.480 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:12:46.480433402+08:00 2024-09-11 18:12:46.480 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:12:46.485951189+08:00 2024-09-11 18:12:46.481 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:12:46.481 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:12:46.480+08:00 2024-09-11 18:12:46.481 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:12:46.480+08:00 2024-09-11 18:12:46.484 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:12:46.479+08:00 2024-09-11 18:12:46.486 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:12:46.479+08:00 and status AVAILABLE 2024-09-11 18:12:46.487 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:12:46.480+08:00 and status AVAILABLE 2024-09-11 18:12:46.489 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:12:46.480+08:00 and status AVAILABLE 2024-09-11 18:12:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:46.496 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:12:46.503 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:12:46.508 INFO 2981023 --- [host-check-agent-status4,97cfcee09b14e87d,21f6a657b269aa68] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:12:46.510 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:12:46.512 INFO 2981023 --- [host-check-agent-status4,97cfcee09b14e87d,21f6a657b269aa68] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:12:46.512 INFO 2981023 --- [host-check-agent-status4,97cfcee09b14e87d,21f6a657b269aa68] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:12:46.512 INFO 2981023 --- [host-check-agent-status4,97cfcee09b14e87d,21f6a657b269aa68] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:12:46.517 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:12:46.520 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:12:46.557 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:46.558 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.558 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:12:46.559 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:12:46.559 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:46.559 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:12:46.560 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.561 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.561 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:46.563 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:46.563 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:46.567 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:12:46.574 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:12:46.583 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:12:46.592 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:12:46.592 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:12:46.603 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.606 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.607 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:12:46.609 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:12:46.619 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:12:46.628 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:12:46.639 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:12:46.640 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.643 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.643 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:12:46.644 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:12:46.644 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.654 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:12:46.660 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:12:46.665 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:12:46.665 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049522904000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:12:46.667 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:12:46.669 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.669 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:46.670 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:12:46.671 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:12:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.678 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:12:46.678 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:12:46.679 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:12:46.686 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:12:46.696 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:12:46.696 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:12:46.703 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:12:46.704 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:12:46.705 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:12:46.717 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:12:46.718 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:12:46.725 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:12:46.728 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.728 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.728 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:12:46.730 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.731 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:12:46.732 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:12:46.742 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:12:46.744 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.745 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.745 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:12:46.745 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:12:46.745 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:12:46.746 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:12:46.750 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.750 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:12:46.751 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:12:46.751 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.751 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.752 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:12:46.752 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.752 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:12:46.753 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:12:46.755 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:12:46.757 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:12:46.758 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:12:46.772 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:12:46.774 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:12:46.775 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:12:46.782 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:12:46.782 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:12:46.785 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.785 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.786 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:12:46.787 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:12:46.787 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.789 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1000 server events to database for cluster 1000006 2024-09-11 18:12:46.790 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:12:46.792 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:12:46.796 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:12:46.800 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:12:46.800 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:12:46.801 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.802 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.803 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.804 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:12:46.804 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.804 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:12:46.804 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:12:46.806 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:12:46.806 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.807 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:12:46.814 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:12:46.821 INFO 2981023 --- [host-check-host-status1,ca6936392abd34b4,8423ea69e4ddf983] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:12:46.826 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:12:46.832 INFO 2981023 --- [host-check-host-status4,ca6936392abd34b4,e9d1d9fa7abd03d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:12:46.837 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 18:02:45.492504, 2024-09-11 18:12:45.751, 1000, 1000] 2024-09-11 18:12:46.847 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:12:46.858 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.858 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:12:46.858 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:12:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:12:46.859 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:12:46.865 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:12:46.871 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:12:46.875 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:12:46.877 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:12:46.888 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.891 INFO 2981023 --- [host-check-host-status3,ca6936392abd34b4,facbbc48c9524824] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:12:46.893 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:12:46.905 INFO 2981023 --- [host-check-host-status2,ca6936392abd34b4,a5c33b1fda4e93b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:12:46.905 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:12:46.906 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.906 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:12:46.907 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:12:46.914 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.936 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:12:46.941 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.950 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:12:46.951 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:12:46.456107+08:00 2024-09-11 18:12:46.954 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.963 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:46.963 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:12:46.963 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:46.964 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:12:46.972 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:12:46.972 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:12:46.972 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1000 server events for cluster 1000006, startTime 2024-09-11 18:02:45.492504, endTime 2024-09-11 18:12:45.751, limit 1000, offset 1000 2024-09-11 18:12:46.972 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:12:46.972 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:12:46.974 INFO 2981023 --- [ocp-schedules-24,c7510476da6af12c,acab5ecc099f05c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:12:46.983 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:12:47.001 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:12:47.008 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:12:47.011 INFO 2981023 --- [ocp-schedules-16,b87842892ba20d1a,b8f55a47809276d3] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:12:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,8d37674041abdbc6,9e7474e08c49ed23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,8d37674041abdbc6,9e7474e08c49ed23] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:47.412 INFO 2981023 --- [ocp-async-9,1da4e674abefa376,03bc3a3c32c73412] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:47.450 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1000 server events to database for cluster 1000006 2024-09-11 18:12:47.457 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:47.458 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:47.476 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 18:02:45.492504, 2024-09-11 18:12:45.751, 1000, 2000] 2024-09-11 18:12:47.480 INFO 2981023 --- [ocp-schedules-30,284c11d77e00f6eb,3203fc216725160a] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 2 2024-09-11 18:12:47.481 INFO 2981023 --- [ocp-schedules-30,284c11d77e00f6eb,3203fc216725160a] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-09-11 18:12:47.481 INFO 2981023 --- [ocp-schedules-30,284c11d77e00f6eb,3203fc216725160a] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 2 2024-09-11 18:12:47.595 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1000 server events for cluster 1000006, startTime 2024-09-11 18:02:45.492504, endTime 2024-09-11 18:12:45.751, limit 1000, offset 2000 2024-09-11 18:12:47.786 INFO 2981023 --- [ocp-async-3,cf3d8b99a69d34eb,474c9317d4b88c40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:12:48.033 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:48.034 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:48.069 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1000 server events to database for cluster 1000006 2024-09-11 18:12:48.077 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:48.078 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:48.106 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 18:02:45.492504, 2024-09-11 18:12:45.751, 1000, 3000] 2024-09-11 18:12:48.198 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1000 server events for cluster 1000006, startTime 2024-09-11 18:02:45.492504, endTime 2024-09-11 18:12:45.751, limit 1000, offset 3000 2024-09-11 18:12:48.617 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1000 server events to database for cluster 1000006 2024-09-11 18:12:48.624 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:48.625 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:48.643 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 18:02:45.492504, 2024-09-11 18:12:45.751, 1000, 4000] 2024-09-11 18:12:48.738 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1000 server events for cluster 1000006, startTime 2024-09-11 18:02:45.492504, endTime 2024-09-11 18:12:45.751, limit 1000, offset 4000 2024-09-11 18:12:49.156 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1000 server events to database for cluster 1000006 2024-09-11 18:12:49.162 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:49.163 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:49.182 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 18:02:45.492504, 2024-09-11 18:12:45.751, 1000, 5000] 2024-09-11 18:12:49.273 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 783 server events for cluster 1000006, startTime 2024-09-11 18:02:45.492504, endTime 2024-09-11 18:12:45.751, limit 1000, offset 5000 2024-09-11 18:12:49.412 INFO 2981023 --- [ocp-async-10,975ae553e5193451,03939e0a875edbb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:49.592 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 783 server events to database for cluster 1000006 2024-09-11 18:12:49.599 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:49.600 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:49.619 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-09-11 18:02:45.492504, 2024-09-11 18:12:45.751, 1000, 6000] 2024-09-11 18:12:49.627 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000006, startTime 2024-09-11 18:02:45.492504, endTime 2024-09-11 18:12:45.751 2024-09-11 18:12:49.628 INFO 2981023 --- [ocp-schedules-26,54d56c028fa2f96d,c1173c5014076c49] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, tryCount=3, successCount=3 2024-09-11 18:12:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:12:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:12:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,6629244c7a6f225b,4caa33b660648ef8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,6629244c7a6f225b,4caa33b660648ef8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:50.336 INFO 2981023 --- [ocp-schedules-1,eebee4100b6da8f6,4a002c9c5213bdd5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:12:50.698 INFO 2981023 --- [ocp-schedules-10,a1c41512a0dcbebb,40c2015307d2ff4f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:12:50.751 WARN 2981023 --- [scanner-13626219-1,a1c41512a0dcbebb,f8165729c8a6b533] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:12:51.034 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:51.034 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:51.412 INFO 2981023 --- [ocp-async-3,b0babae94091948f,cd53e87240d18259] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:51.612 INFO 2981023 --- [ocp-schedules-30,b71209d25f943a47,5020d0de04647177] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:12:51.613 INFO 2981023 --- [ocp-schedules-30,b71209d25f943a47,5020d0de04647177] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:51.614 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:12:51.614 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:12:51.614 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:12:51.614 INFO 2981023 --- [ocp-schedules-30,b71209d25f943a47,5020d0de04647177] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:12:52.031 INFO 2981023 --- [host-check-clockdiff4,c58ad849ff1555f3,96d3ffb95cdf1a23] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049572 0 0 , error= 2024-09-11 18:12:52.591 INFO 2981023 --- [ocp-schedules-3,63c1bbf4dca7d268,d18ba3f4da1e9a0f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:12:53.304 INFO 2981023 --- [ocp-schedules-8,22b7bdd37c06e485,7ad18fd84416e59d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:12:53.304 INFO 2981023 --- [ocp-schedules-8,22b7bdd37c06e485,7ad18fd84416e59d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:12:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,41ed67e311c9d84c,737d73fc736d31df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:53.315 INFO 2981023 --- [prometheus-minute-schedule-1,41ed67e311c9d84c,737d73fc736d31df] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:53.412 INFO 2981023 --- [ocp-async-7,e2438060748491f8,3ca91f7defe8ed90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:54.035 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:54.035 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:54.429 INFO 2981023 --- [ocp-schedules-2,0f75184ea3ae5d15,b3ba562e11cb0d6c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:12:54.698 INFO 2981023 --- [ocp-schedules-22,7a5800cf74edca60,99d38a043d929901] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:12:54.718 WARN 2981023 --- [ocp-schedules-22,7a5800cf74edca60,99d38a043d929901] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:12:55.044 INFO 2981023 --- [ocp-schedules-17,f1fe58d002f47f9f,eab4efff9d2b00ff] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:12:55.044 INFO 2981023 --- [ocp-schedules-17,f1fe58d002f47f9f,eab4efff9d2b00ff] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:12:55.047 INFO 2981023 --- [ocp-schedules-17,f1fe58d002f47f9f,eab4efff9d2b00ff] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:12:55.047 INFO 2981023 --- [ocp-schedules-17,f1fe58d002f47f9f,eab4efff9d2b00ff] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:12:55.047 INFO 2981023 --- [ocp-schedules-17,f1fe58d002f47f9f,eab4efff9d2b00ff] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:12:55.050 INFO 2981023 --- [ocp-schedules-17,f1fe58d002f47f9f,eab4efff9d2b00ff] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:12:55.050 INFO 2981023 --- [ocp-schedules-17,f1fe58d002f47f9f,eab4efff9d2b00ff] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:12:55.050 INFO 2981023 --- [ocp-schedules-17,f1fe58d002f47f9f,eab4efff9d2b00ff] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:12:55.059 INFO 2981023 --- [ocp-schedules-17,f1fe58d002f47f9f,eab4efff9d2b00ff] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:12:55.334 INFO 2981023 --- [ocp-schedules-9,c4f4c3f5c1057e87,494eee60fbb1eb38] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:12:55.342 INFO 2981023 --- [ocp-schedules-9,c4f4c3f5c1057e87,494eee60fbb1eb38] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:12:55.342 INFO 2981023 --- [ocp-schedules-9,c4f4c3f5c1057e87,494eee60fbb1eb38] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:12:55.412 INFO 2981023 --- [ocp-async-4,3dcbe423cf2e4911,50e6ac1afe7ef546] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:55.864 INFO 2981023 --- [ocp-schedules-4,34a0a32792f65d68,67df043563e95680] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:12:55.864 INFO 2981023 --- [ocp-schedules-4,34a0a32792f65d68,67df043563e95680] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:12:55.866 INFO 2981023 --- [ocp-schedules-4,34a0a32792f65d68,67df043563e95680] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:12:55.866 INFO 2981023 --- [ocp-schedules-4,34a0a32792f65d68,67df043563e95680] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:12:55.868 INFO 2981023 --- [ocp-schedules-4,34a0a32792f65d68,67df043563e95680] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:12:55.868 INFO 2981023 --- [ocp-schedules-4,34a0a32792f65d68,67df043563e95680] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:12:56.044 INFO 2981023 --- [ocp-schedules-18,892f5a2f1c2afbd7,bf5b4204e9bba857] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:12:56.050 INFO 2981023 --- [ocp-schedules-18,892f5a2f1c2afbd7,bf5b4204e9bba857] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:12:56.050 INFO 2981023 --- [ocp-schedules-18,892f5a2f1c2afbd7,bf5b4204e9bba857] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:12:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,419c7b2b3ecd1151,4cd338a51bfc69de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,419c7b2b3ecd1151,4cd338a51bfc69de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:56.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:12:56.337555+08:00, lastProcessTime=2024-09-11T18:12:46.456107+08:00 2024-09-11 18:12:56.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:12:56.337555+08:00 2024-09-11 18:12:57.035 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:12:57.036 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:12:57.413 INFO 2981023 --- [ocp-async-5,4f863d86850ba525,c14c8b9cc8888fa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:57.786 INFO 2981023 --- [ocp-async-1,07254686a29bde30,bf8f64d78f8845a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:12:57.926 INFO 2981023 --- [ocp-schedules-12,260077b1ce5ca664,bf90b5dd58a6e292] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:12:57.926 INFO 2981023 --- [ocp-schedules-12,260077b1ce5ca664,bf90b5dd58a6e292] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:12:57.941 INFO 2981023 --- [Collect all tenants major compactions1,260077b1ce5ca664,fb4b7247dd64071a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:12:57.941 INFO 2981023 --- [Collect all tenants major compactions2,260077b1ce5ca664,d0d8ea71d74f18d4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:12:57.943 INFO 2981023 --- [Collect all tenants major compactions4,260077b1ce5ca664,6bf40211820e90f2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:12:57.943 INFO 2981023 --- [Collect all tenants major compactions5,260077b1ce5ca664,63b2e47406d135e8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:12:57.944 INFO 2981023 --- [Collect all tenants major compactions3,260077b1ce5ca664,c9ebc3fb4266149c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:12:57.945 INFO 2981023 --- [Collect all tenants major compactions6,260077b1ce5ca664,18eb4398523636ae] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:12:57.957 INFO 2981023 --- [Collect all tenants major compactions2,260077b1ce5ca664,d0d8ea71d74f18d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:57.958 INFO 2981023 --- [Collect all tenants major compactions4,260077b1ce5ca664,6bf40211820e90f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:57.958 INFO 2981023 --- [Collect all tenants major compactions2,260077b1ce5ca664,d0d8ea71d74f18d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:12:57.959 INFO 2981023 --- [Collect all tenants major compactions3,260077b1ce5ca664,c9ebc3fb4266149c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:57.959 INFO 2981023 --- [Collect all tenants major compactions1,260077b1ce5ca664,fb4b7247dd64071a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:57.960 INFO 2981023 --- [Collect all tenants major compactions6,260077b1ce5ca664,18eb4398523636ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:57.960 INFO 2981023 --- [Collect all tenants major compactions4,260077b1ce5ca664,6bf40211820e90f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:12:57.961 INFO 2981023 --- [Collect all tenants major compactions1,260077b1ce5ca664,fb4b7247dd64071a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:12:57.961 INFO 2981023 --- [Collect all tenants major compactions3,260077b1ce5ca664,c9ebc3fb4266149c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:12:57.961 INFO 2981023 --- [Collect all tenants major compactions6,260077b1ce5ca664,18eb4398523636ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:12:57.961 INFO 2981023 --- [Collect all tenants major compactions5,260077b1ce5ca664,63b2e47406d135e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:57.963 INFO 2981023 --- [Collect all tenants major compactions5,260077b1ce5ca664,63b2e47406d135e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:12:57.963 INFO 2981023 --- [Collect all tenants major compactions2,260077b1ce5ca664,d0d8ea71d74f18d4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:12:57.963 INFO 2981023 --- [Collect all tenants major compactions3,260077b1ce5ca664,c9ebc3fb4266149c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:12:57.963 INFO 2981023 --- [Collect all tenants major compactions1,260077b1ce5ca664,fb4b7247dd64071a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:12:57.964 INFO 2981023 --- [Collect all tenants major compactions5,260077b1ce5ca664,63b2e47406d135e8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:12:57.965 INFO 2981023 --- [Collect all tenants major compactions4,260077b1ce5ca664,6bf40211820e90f2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:12:57.965 INFO 2981023 --- [Collect all tenants major compactions6,260077b1ce5ca664,18eb4398523636ae] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:12:57.968 INFO 2981023 --- [Collect all tenants major compactions4,260077b1ce5ca664,6bf40211820e90f2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:12:57.968 INFO 2981023 --- [Collect all tenants major compactions2,260077b1ce5ca664,d0d8ea71d74f18d4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:12:57.971 INFO 2981023 --- [Collect all tenants major compactions1,260077b1ce5ca664,fb4b7247dd64071a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:12:57.972 INFO 2981023 --- [Collect all tenants major compactions3,260077b1ce5ca664,c9ebc3fb4266149c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:12:57.973 INFO 2981023 --- [Collect all tenants major compactions6,260077b1ce5ca664,18eb4398523636ae] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:12:57.973 INFO 2981023 --- [Collect all tenants major compactions5,260077b1ce5ca664,63b2e47406d135e8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:12:57.974 INFO 2981023 --- [ocp-schedules-12,260077b1ce5ca664,bf90b5dd58a6e292] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:12:58.532 INFO 2981023 --- [ocp-schedules-19,c860882106508413,f23fb133ede36929] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:12:58.532+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:12:58.535 INFO 2981023 --- [ocp-schedules-19,c860882106508413,f23fb133ede36929] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:12:58.645 INFO 2981023 --- [ocp-schedules-19,c860882106508413,f23fb133ede36929] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 20 to SqlAggregateData 2024-09-11 18:12:58.648 INFO 2981023 --- [ocp-schedules-19,c860882106508413,f23fb133ede36929] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster linkrtd, exclude result is empty. 2024-09-11 18:12:58.979 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:58.984 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkp 2024-09-11 18:12:58.994 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.012 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.016 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.024 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.024 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.025 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.026 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.027 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.031 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.035 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.038 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.042 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.043 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.044 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.044 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.045 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.049 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.052 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.055 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.060 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.060 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.061 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.062 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.062 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:06:58.872, end=2024-09-11T18:11:58.976} 2024-09-11 18:12:59.062 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 18:12:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,18c9cfe1fc4bbdce,aff63751fe4a33c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049580, interval=60 2024-09-11 18:12:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,18c9cfe1fc4bbdce,aff63751fe4a33c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:12:59.413 INFO 2981023 --- [ocp-async-9,d3dd7ede0d6fdd68,44f008e3602d5191] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:12:59.428 INFO 2981023 --- [ocp-schedules-1,092588d4f396ab9d,bb22d4ae2eae2e2d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:12:59.429 INFO 2981023 --- [ocp-schedules-1,092588d4f396ab9d,bb22d4ae2eae2e2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:12:59.441 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,5b0af5f9e933f9db,d8743067060d2ef3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=5b0af5f9e933f9db, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:12:59.446 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2f1d3b94b25fe07e,ad8205b7e7940cc9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=2f1d3b94b25fe07e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:12:59.451 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,3512047d5d1755fd,05172a9f6ab33c30] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=3512047d5d1755fd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:12:59.454 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,5b0af5f9e933f9db,d8743067060d2ef3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=5b0af5f9e933f9db, duration=18 ms] 2024-09-11 18:12:59.458 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,3512047d5d1755fd,05172a9f6ab33c30] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=3512047d5d1755fd, duration=10 ms] 2024-09-11 18:12:59.461 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2f1d3b94b25fe07e,ad8205b7e7940cc9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=2f1d3b94b25fe07e, duration=27 ms] 2024-09-11 18:12:59.471 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:12:59.472 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.475 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:12:59.476 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:12:59.476 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.478 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:12:59.478 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.480 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:12:59.481 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:12:59.483 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.483 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.484 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.485 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.485 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.486 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.491 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.491 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.496 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.496 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.496 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.496 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:12:59.499 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.500 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.503 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.503 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:12:59.508 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.508 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:12:59.512 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.512 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.513 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:12:59.513 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5141D6DC-0-0] 2024-09-11 18:12:59.513 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:12:59.513 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:12:59.514 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:12:59.515 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.515 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.525 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.526 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:12:59.526 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.527 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:12:59.528 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.528 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:12:59.529 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:12:59.529 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.529 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.530 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:12:59.530 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:12:59.530 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:12:59.531 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.531 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:12:59.532 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.533 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:12:59.533 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.534 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:12:59.534 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.535 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:12:59.535 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.536 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:12:59.537 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.537 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:12:59.537 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:12:59.538 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.538 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.538 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:12:59.539 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:12:59.539 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5141D6E6-0-0] 2024-09-11 18:12:59.539 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:12:59.539 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:12:59.540 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:12:59.541 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:12:59.546 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.547 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:12:59.548 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.548 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:12:59.549 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.550 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:12:59.550 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.551 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:12:59.552 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.552 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:12:59.553 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.554 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:12:59.555 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:12:59.556 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:59.556 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:59.557 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:12:59.558 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.558 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.558 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:12:59.558 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.566 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:12:59.570 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:12:59.570 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:12:59.570 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.571 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:12:59.572 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:12:59.572 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:12:59.574 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:12:59.574 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:12:59.574 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:12:59.576 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:12:59.580 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:12:59.580 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.580 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.581 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:12:59.583 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:12:59.583 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:12:59.584 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:12:59.586 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:12:59.588 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:12:59.588 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:12:59.588 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:12:59.595 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:12:59.597 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:12:59.597 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:12:59.597 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:12:59.597 INFO 2981023 --- [ocp-schedules-1,092588d4f396ab9d,bb22d4ae2eae2e2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:13:00.036 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:00.037 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:00.885 INFO 2981023 --- [ocp-schedules-5,e3254ab6993a870b,f8e38c93b45d7918] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:13:00.954 WARN 2981023 --- [scanner-13626427-1,e3254ab6993a870b,410e7b05f0a04003] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:13:00.986 INFO 2981023 --- [alarm-detect-3,e3254ab6993a870b,35216a68114c6ecd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645672, data is empty. 2024-09-11 18:13:00.998 INFO 2981023 --- [alarm-detect-3,e3254ab6993a870b,35216a68114c6ecd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645692, data is empty. 2024-09-11 18:13:01.000 INFO 2981023 --- [alarm-detect-20,e3254ab6993a870b,14b0149c2c055e0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645672, data is empty. 2024-09-11 18:13:01.003 INFO 2981023 --- [alarm-detect-17,e3254ab6993a870b,cadd7279bedc5066] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645672, data is empty. 2024-09-11 18:13:01.004 INFO 2981023 --- [alarm-detect-3,e3254ab6993a870b,35216a68114c6ecd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645692, data is empty. 2024-09-11 18:13:01.005 INFO 2981023 --- [alarm-detect-20,e3254ab6993a870b,14b0149c2c055e0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645692, data is empty. 2024-09-11 18:13:01.011 INFO 2981023 --- [alarm-detect-17,e3254ab6993a870b,cadd7279bedc5066] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645692, data is empty. 2024-09-11 18:13:01.012 INFO 2981023 --- [alarm-detect-20,e3254ab6993a870b,14b0149c2c055e0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645692, data is empty. 2024-09-11 18:13:01.015 INFO 2981023 --- [alarm-detect-17,e3254ab6993a870b,cadd7279bedc5066] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645692, data is empty. 2024-09-11 18:13:01.413 INFO 2981023 --- [ocp-async-1,8f69d4d9b1c14515,d661b6943d2e99d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:01.494 INFO 2981023 --- [alarm-detect-18,e3254ab6993a870b,de74f4e6729ce14f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645672, data is empty. 2024-09-11 18:13:01.502 INFO 2981023 --- [alarm-detect-18,e3254ab6993a870b,de74f4e6729ce14f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645692, data is empty. 2024-09-11 18:13:01.507 INFO 2981023 --- [alarm-detect-18,e3254ab6993a870b,de74f4e6729ce14f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645692, data is empty. 2024-09-11 18:13:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,35d96c55cdda8107,eafbaaffff86af5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,35d96c55cdda8107,eafbaaffff86af5a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:02.594 INFO 2981023 --- [ocp-schedules-2,78629f9dfecfaac5,b8938091f3adc16e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:13:03.037 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:03.037 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:03.413 INFO 2981023 --- [ocp-async-2,74ed62d0ac9cf0d1,24eb93d6f23937e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:04.784 INFO 2981023 --- [ocp-schedules-10,45f6390695d85f10,eb953390cc07c44a] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:13:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,d3464db72706b30f,d6beb87763dbd28f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,d3464db72706b30f,d6beb87763dbd28f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:05.413 INFO 2981023 --- [ocp-async-8,effc5a69e6c949d1,e1913b6c28553e79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:06.038 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:06.038 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:06.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:13:06.353334+08:00, lastProcessTime=2024-09-11T18:12:56.337555+08:00 2024-09-11 18:13:06.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:13:06.353334+08:00 2024-09-11 18:13:07.413 INFO 2981023 --- [ocp-async-9,bf619cdb5f0897fb,690f8926ccaf8218] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:07.786 INFO 2981023 --- [ocp-async-7,7d4156843eb8d9d3,1eb11824bac43d69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:13:07.930 INFO 2981023 --- [ocp-schedules-2,f28e4a198dd2f9d6,352bfdb416082a30] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:13:08.040 INFO 2981023 --- [ocp-schedules-21,9a1881aad70b073c,a22a2902d5a73c6d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:13:08.040+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:13:08.041 INFO 2981023 --- [ocp-schedules-21,9a1881aad70b073c,a22a2902d5a73c6d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:13:08.042 INFO 2981023 --- [ocp-schedules-21,9a1881aad70b073c,a22a2902d5a73c6d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:13:08.042 INFO 2981023 --- [ocp-schedules-21,9a1881aad70b073c,a22a2902d5a73c6d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:13:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,326cd66924fbcf8c,7b9edd8284088115] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,326cd66924fbcf8c,7b9edd8284088115] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:09.038 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:09.040 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:09.413 INFO 2981023 --- [ocp-async-3,83c218d3dd611fc9,37bb9080ec62a21c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:09.817 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:09:09.446, end=2024-09-11T18:11:09.814} 2024-09-11 18:13:09.820 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp 2024-09-11 18:13:09.820 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:13:09.821 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:09:09.446, end=2024-09-11T18:11:09.814},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:13:09.821 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:13:09.821 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:13:09.822 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:13:09.824 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:09:09.446, end=2024-09-11T18:11:09.814},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:13:09.824 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:13:09.824 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:13:09.889 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963189888000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:09.888Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:09.889 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963189889000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:09.889Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:09.894 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963189894000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:09.894Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:09.898 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963189898000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:09.898Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:09.898 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963189898000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:09.898Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:09.899 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963189899000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:09.899Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:09.905 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963189905000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:09.905Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:09.905 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963189905000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:09.905Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:09.916 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:09.916 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:09.919 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:09.919 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:09.919 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:09.919 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:09.929 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.929 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.929 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:09.929 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:09.940 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.940 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.940 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:13:09.940 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:09.941 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:13:09.941 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:09.942 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.942 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.942 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.942 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.945 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 67 2024-09-11 18:13:09.945 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:09.945 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:13:09.945 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:09.947 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:09.947 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:09.949 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:09.949 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:09.951 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:09.951 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:09.951 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:09.951 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:09.955 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.955 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.955 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.955 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.958 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:09.958 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:09.960 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:09.960 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:09.963 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.963 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.964 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.964 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.965 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.965 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.967 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.967 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.967 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:09.967 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:09.968 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.968 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.969 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:09.969 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:09.970 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:09.970 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:09.971 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.971 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.972 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.972 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.972 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.972 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.972 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:09.975 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:09.976 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:09.978 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:09.979 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:09.894, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:09.981 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:09.898, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:09.982 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:09.888, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:09.983 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.983 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.983 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:09.983 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:09.985 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:09.898, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:09.985 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:09.889, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:09.986 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:09.986 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:09.899, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:09.989 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:09.997 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:09.905, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.005 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:09.905, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.013 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:13:10.015 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:13:10.025 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963190025000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:10.025Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:10.025 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963190025000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:10.025Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:10.028 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963190028000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:10.028Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:10.036 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963190036000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1ED31157F7096F47941EA2233AA4101C, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:10.036Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:10.038 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963190038000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:10.038Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:10.040 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963190040000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:10.040Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:10.044 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:13:10.044 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:10.056 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:10.056 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:10.056 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:13:10.056 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:10.057 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963190057000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:10.057Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:10.058 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963190058000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:10.058Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:10.058 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:10.058 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:10.059 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.059 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.063 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:10.064 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:10.066 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:10.066 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:10.069 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:10.069 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:10.075 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.075 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.076 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.076 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.076 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.076 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.076 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:10.076 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:10.079 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:10.079 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:10.080 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.080 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.080 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:13:10.080 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:10.081 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:10.081 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:10.082 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:10.082 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:10.085 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:10.085 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:10.088 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.088 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.088 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.088 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.090 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:10.091 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.091 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:10.091 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:10.091 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.098 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:10.025, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.098 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.098 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.098 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.098 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.099 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:10.099 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:10.100 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.100 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.101 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.101 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.101 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:10.102 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.102 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.102 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:10.103 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:10.103 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:10.104 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:10.105 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.105 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.106 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:10.107 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:10.111 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:10.028, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.111 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:10.025, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.112 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:10.038, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.112 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:10.057, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.113 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:10.04, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.114 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.114 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.115 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.115 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.116 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:10.116 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:13:10.119 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:10.036, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=1ED31157F7096F47941EA2233AA4101C, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.120 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:10.058, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.120 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:13:10.121 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:09:09.446, end=2024-09-11T18:11:09.814},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:13:10.121 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:13:10.121 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:13:10.122 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:13:10.122 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:09:09.446, end=2024-09-11T18:11:09.814},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:13:10.122 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:13:10.122 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:13:10.123 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:13:10.124 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:09:09.446, end=2024-09-11T18:11:09.814},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:13:10.124 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:13:10.124 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:13:10.134 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:13:10.147 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963190147000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:13:10.147Z, endTime=2024-09-11T10:11:09.814Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:13:10.149 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:13:10.149 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:13:10.155 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.155 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.156 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:13:10.156 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:13:10.161 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:13:10.161 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049469814000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:13:10.162 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:13:10.162 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:13:10.147, end=2024-09-11T18:11:09.814}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:13:10.162 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:13:10.163 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:09:09.446, end=2024-09-11T18:11:09.814},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:13:10.163 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:13:10.163 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:13:10.174 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:13:10.174 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:13:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:13:10.923 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:13:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,1d86d6274adc2694,bcd95e390b91cdc4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:11.319 INFO 2981023 --- [prometheus-minute-schedule-1,1d86d6274adc2694,bcd95e390b91cdc4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:11.414 INFO 2981023 --- [ocp-async-7,80e97afeb2842cbd,5107c16af33384d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:11.541 INFO 2981023 --- [ocp-schedules-13,6b7d606c48cf8a1f,77629a0f26877bff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:13:11.589 WARN 2981023 --- [scanner-13628187-1,6b7d606c48cf8a1f,4ae07bd0771abe2e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:13:12.040 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:12.041 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:12.437 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049520, nodeCount=1000, affectRows=1000 2024-09-11 18:13:12.440 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049520, nodeCount=999, validCount=999 2024-09-11 18:13:12.466 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049520, nodeCount=1000, affectRows=1000 2024-09-11 18:13:12.469 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049520, nodeCount=1000, validCount=1000 2024-09-11 18:13:12.471 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049520, nodeCount=51, affectRows=51 2024-09-11 18:13:12.471 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049520, nodeCount=49, validCount=49 2024-09-11 18:13:12.596 INFO 2981023 --- [ocp-schedules-4,c1926c1f3e840d5c,a21cbe12ae4837fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:13:13.414 INFO 2981023 --- [ocp-async-4,c81774ba980d2820,32f8b669e4618120] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,3ab8e76b14a45ea8,a5ccb70722c24110] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,f8e2668dec444aa5,24c01fafcccc1df8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:13:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,3ab8e76b14a45ea8,a5ccb70722c24110] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:14.542 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049520, nodeCount=480, affectRows=480 2024-09-11 18:13:15.041 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:15.042 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:15.345 INFO 2981023 --- [ocp-schedules-24,30a32241ed4042ff,47d2fffcf4910385] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:13:15.353 INFO 2981023 --- [ocp-schedules-24,30a32241ed4042ff,47d2fffcf4910385] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:13:15.353 INFO 2981023 --- [ocp-schedules-24,30a32241ed4042ff,47d2fffcf4910385] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:13:15.414 INFO 2981023 --- [ocp-async-10,bdc587c07ab08ca6,3705ec929d145f17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:16.026 INFO 2981023 --- [ocp-schedules-29,e9b2478f47dc28e2,2987c43163161e37] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:13:16.028 INFO 2981023 --- [ocp-schedules-30,5e63a2bf28effa0d,33259604ef63f28a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:13:16.028 INFO 2981023 --- [ocp-schedules-17,e9bd4e7b92c7a615,c812f7477a789be0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:13:16.029 INFO 2981023 --- [host-check-agent-status1,e9b2478f47dc28e2,4f3c97942c3a070a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:13:16.029 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:13:16.030 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:13:16.030 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:13:16.031 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:13:16.031 INFO 2981023 --- [host-check-clockdiff1,5e63a2bf28effa0d,37c1f0eea0cacf39] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:13:16.031 INFO 2981023 --- [host-check-clockdiff2,5e63a2bf28effa0d,491e4c122d5f73a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:13:16.040 INFO 2981023 --- [host-check-clockdiff3,5e63a2bf28effa0d,04ac616e15d81258] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:13:16.046 INFO 2981023 --- [host-check-clockdiff4,5e63a2bf28effa0d,8ab668f2b8ba582c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:13:16.051 INFO 2981023 --- [host-check-agent-status2,e9b2478f47dc28e2,d1069be8df196500] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:13:16.052 INFO 2981023 --- [host-check-agent-status3,e9b2478f47dc28e2,cfc85b829b7f8f7a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:13:16.052 INFO 2981023 --- [host-check-agent-status4,e9b2478f47dc28e2,286d5afe638a308b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:13:16.067 INFO 2981023 --- [host-check-agent-status4,e9b2478f47dc28e2,286d5afe638a308b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:13:16.095 INFO 2981023 --- [host-check-agent-status4,e9b2478f47dc28e2,286d5afe638a308b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:13:16.102 INFO 2981023 --- [host-check-clockdiff2,5e63a2bf28effa0d,491e4c122d5f73a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049596 5 5 , error= 2024-09-11 18:13:16.102 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:13:16.102 INFO 2981023 --- [host-check-clockdiff3,5e63a2bf28effa0d,04ac616e15d81258] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049596 0 0 , error= 2024-09-11 18:13:16.103 INFO 2981023 --- [host-check-agent-status3,e9b2478f47dc28e2,cfc85b829b7f8f7a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:13:16.104 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:13:16.104 INFO 2981023 --- [host-check-agent-status3,e9b2478f47dc28e2,cfc85b829b7f8f7a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:13:16.107 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:13:16.107 INFO 2981023 --- [host-check-agent-status1,e9b2478f47dc28e2,4f3c97942c3a070a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:13:16.107 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:13:16.108 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:13:16.109 INFO 2981023 --- [host-check-agent-status1,e9b2478f47dc28e2,4f3c97942c3a070a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:13:16.109 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:13:16.131 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:13:16.137 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:13:16.187 INFO 2981023 --- [host-check-agent-status2,e9b2478f47dc28e2,d1069be8df196500] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:13:16.188 INFO 2981023 --- [ocp-schedules-27,c94aa790b024f7cf,1b0b9485ebbd4160] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:13:16.188 INFO 2981023 --- [ocp-schedules-20,31e7eaa643ce7720,a090bac0cbe59dd4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:13:16.188 INFO 2981023 --- [ocp-schedules-1,fd0ddac95cf5cb2a,cf5eafe7c6faa136] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:13:16.188 INFO 2981023 --- [ocp-schedules-3,ef6c91d7a918bb3a,fa7f78c1108492b7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:13:16.199 INFO 2981023 --- [ocp-schedules-1,fd0ddac95cf5cb2a,cf5eafe7c6faa136] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:13:16.200 INFO 2981023 --- [host-check-agent-status2,e9b2478f47dc28e2,d1069be8df196500] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:13:16.200 INFO 2981023 --- [ocp-schedules-27,c94aa790b024f7cf,1b0b9485ebbd4160] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:13:16.200 INFO 2981023 --- [ocp-schedules-20,31e7eaa643ce7720,a090bac0cbe59dd4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:13:16.214 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049522904000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:13:16.215 INFO 2981023 --- [ocp-schedules-20,31e7eaa643ce7720,a090bac0cbe59dd4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:13:16.292 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:13:16.294 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:13:16.293844657+08:00 2024-09-11 18:13:16.294 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:13:16.294+08:00 2024-09-11 18:13:16.297 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:13:16.294+08:00 and status AVAILABLE 2024-09-11 18:13:16.358 INFO 2981023 --- [ocp-schedules-14,563db76420261b02,b11016b678309d68] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:13:16.361 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:13:16.374 INFO 2981023 --- [host-check-agent-status1,e9b2478f47dc28e2,4f3c97942c3a070a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:13:16.374 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:13:16.374 INFO 2981023 --- [host-check-agent-status4,e9b2478f47dc28e2,286d5afe638a308b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:13:16.376 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:13:16.381638951+08:00 2024-09-11 18:13:16.376 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:13:16.376+08:00 2024-09-11 18:13:16.377 INFO 2981023 --- [host-check-agent-status1,e9b2478f47dc28e2,4f3c97942c3a070a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:13:16.377 INFO 2981023 --- [host-check-agent-status1,e9b2478f47dc28e2,4f3c97942c3a070a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:13:16.377 INFO 2981023 --- [host-check-agent-status1,e9b2478f47dc28e2,4f3c97942c3a070a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:13:16.378 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:13:16.376+08:00 and status AVAILABLE 2024-09-11 18:13:16.384 INFO 2981023 --- [host-check-agent-status4,e9b2478f47dc28e2,286d5afe638a308b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:13:16.384 INFO 2981023 --- [host-check-agent-status4,e9b2478f47dc28e2,286d5afe638a308b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:13:16.384 INFO 2981023 --- [host-check-agent-status4,e9b2478f47dc28e2,286d5afe638a308b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:13:16.394 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:13:16.395 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:13:16.392401+08:00, lastProcessTime=2024-09-11T18:13:06.353334+08:00 2024-09-11 18:13:16.395 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:13:16.395043009+08:00 2024-09-11 18:13:16.396 INFO 2981023 --- [host-check-agent-status2,e9b2478f47dc28e2,d1069be8df196500] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:13:16.396 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:13:16.395+08:00 2024-09-11 18:13:16.398 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:13:16.395+08:00 and status AVAILABLE 2024-09-11 18:13:16.399 INFO 2981023 --- [host-check-agent-status2,e9b2478f47dc28e2,d1069be8df196500] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:13:16.399 INFO 2981023 --- [host-check-agent-status2,e9b2478f47dc28e2,d1069be8df196500] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:13:16.399 INFO 2981023 --- [host-check-agent-status2,e9b2478f47dc28e2,d1069be8df196500] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:13:16.399 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.399 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:13:16.400 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:13:16.400 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:13:16.403 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.404 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:16.405 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:13:16.406 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:13:16.406 INFO 2981023 --- [host-check-agent-status3,e9b2478f47dc28e2,cfc85b829b7f8f7a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:13:16.406 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:13:16.408 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:13:16.411 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:13:16.416528803+08:00 2024-09-11 18:13:16.411 INFO 2981023 --- [host-check-agent-status3,e9b2478f47dc28e2,cfc85b829b7f8f7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:13:16.411 INFO 2981023 --- [host-check-agent-status3,e9b2478f47dc28e2,cfc85b829b7f8f7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:13:16.411 INFO 2981023 --- [host-check-agent-status3,e9b2478f47dc28e2,cfc85b829b7f8f7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:13:16.408 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.413 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:13:16.411+08:00 2024-09-11 18:13:16.414 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:13:16.414 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:13:16.414 INFO 2981023 --- [ocp-schedules-27,c94aa790b024f7cf,1b0b9485ebbd4160] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:13:16.455 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:13:16.411+08:00 and status AVAILABLE 2024-09-11 18:13:16.457 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:13:16.459 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:13:16.460 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:13:16.461 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:13:16.471 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:13:16.476 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.476 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:16.479 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:13:16.491 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:13:16.493 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:13:16.493 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.494 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:13:16.495 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:13:16.498 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:16.502 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:16.502 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.503 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:13:16.511 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:13:16.514 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:13:16.523 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:13:16.526 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:13:16.526 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:13:16.528 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:13:16.550 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:13:16.554 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:13:16.561 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:16.562 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.562 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:16.563 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:13:16.563 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.566 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:13:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:13:16.570 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:13:16.573 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:13:16.574 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:13:16.574 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049522904000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:13:16.581 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:13:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:13:16.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:13:16.602 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:13:16.602 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:13:16.606 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:13:16.612 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:13:16.613 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:13:16.619 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:13:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:13:16.629 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:16.629 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.629 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:13:16.630 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:13:16.637 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:13:16.637 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:13:16.638 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:13:16.639 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:13:16.645 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:13:16.647 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:13:16.649 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:13:16.650 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:13:16.660 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:16.661 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.661 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:13:16.662 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:13:16.668 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:13:16.670 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:13:16.674 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:13:16.678 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:13:16.680 INFO 2981023 --- [host-check-host-status4,e9bd4e7b92c7a615,4d002f00bdb645e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:13:16.688 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:13:16.692 INFO 2981023 --- [host-check-host-status3,e9bd4e7b92c7a615,2a007d8a9daf9b36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:13:16.699 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:13:16.701 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:16.702 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.702 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:16.703 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:13:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.706 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:13:16.706 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:13:16.709 INFO 2981023 --- [host-check-host-status1,e9bd4e7b92c7a615,1d1dc2c1e0e9c88d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:13:16.715 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:13:16.715 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:13:16.721 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:13:16.730 INFO 2981023 --- [host-check-host-status2,e9bd4e7b92c7a615,869ca2d608c82e62] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:13:16.744 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:13:16.745 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:13:16.752 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:16.752 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:16.753 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:13:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.754 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:13:16.754 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:13:16.759 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:13:16.761 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:13:16.761 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:13:16.762 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049520, nodeCount=480, validCount=480 2024-09-11 18:13:16.763 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:13:16.764 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:13:16.771 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:13:16.392401+08:00 2024-09-11 18:13:16.775 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:13:16.804 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:13:16.810 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:16.810 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:13:16.816 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:13:16.816 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:13:16.816 INFO 2981023 --- [ocp-schedules-1,fd0ddac95cf5cb2a,cf5eafe7c6faa136] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:13:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,bd8bda9726dcf8b1,c3a5946047e40483] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,bd8bda9726dcf8b1,c3a5946047e40483] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:17.414 INFO 2981023 --- [ocp-async-5,8a10467ebe5cdc4e,14fede47942aeb1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:17.786 INFO 2981023 --- [ocp-async-1,3880d666e19cf0ae,b818d33cbc390e34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:13:18.042 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:18.042 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:19.414 INFO 2981023 --- [ocp-async-9,6e3e6333cc4e93fe,d90849c3959edecb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:19.735 INFO 2981023 --- [host-check-clockdiff1,5e63a2bf28effa0d,37c1f0eea0cacf39] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049599 6 6 , error= 2024-09-11 18:13:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,461318b9760ef0f0,faa814dbeda87aaf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,461318b9760ef0f0,faa814dbeda87aaf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:21.042 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:21.043 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:21.414 INFO 2981023 --- [ocp-async-2,1cee3076c049846d,f3586f09202582ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:21.752 INFO 2981023 --- [ocp-schedules-19,a11d705033d747cb,c5a394fe3dd0dcf8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:13:21.793 WARN 2981023 --- [scanner-13628450-1,a11d705033d747cb,a367bdad11eee70b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:13:21.816 INFO 2981023 --- [alarm-detect-17,a11d705033d747cb,fd7bb57becd7aa65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645692, data is empty. 2024-09-11 18:13:21.820 INFO 2981023 --- [alarm-detect-17,a11d705033d747cb,fd7bb57becd7aa65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645692, data is empty. 2024-09-11 18:13:21.823 INFO 2981023 --- [alarm-detect-17,a11d705033d747cb,fd7bb57becd7aa65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645692, data is empty. 2024-09-11 18:13:21.845 INFO 2981023 --- [alarm-detect-4,a11d705033d747cb,3a28147d5b38da51] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645692, data is empty. 2024-09-11 18:13:21.845 INFO 2981023 --- [alarm-detect-10,a11d705033d747cb,cd0fd33e39309f86] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645692, data is empty. 2024-09-11 18:13:21.849 INFO 2981023 --- [alarm-detect-4,a11d705033d747cb,3a28147d5b38da51] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645692, data is empty. 2024-09-11 18:13:21.852 INFO 2981023 --- [alarm-detect-10,a11d705033d747cb,cd0fd33e39309f86] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645692, data is empty. 2024-09-11 18:13:21.854 INFO 2981023 --- [alarm-detect-4,a11d705033d747cb,3a28147d5b38da51] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645692, data is empty. 2024-09-11 18:13:21.855 INFO 2981023 --- [alarm-detect-10,a11d705033d747cb,cd0fd33e39309f86] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645692, data is empty. 2024-09-11 18:13:21.961 INFO 2981023 --- [alarm-detect-4,a11d705033d747cb,35fbbd0b3455466d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645692, data is empty. 2024-09-11 18:13:21.964 INFO 2981023 --- [alarm-detect-4,a11d705033d747cb,35fbbd0b3455466d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645692, data is empty. 2024-09-11 18:13:21.968 INFO 2981023 --- [alarm-detect-4,a11d705033d747cb,35fbbd0b3455466d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645692, data is empty. 2024-09-11 18:13:22.067 INFO 2981023 --- [host-check-clockdiff4,5e63a2bf28effa0d,8ab668f2b8ba582c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049602 0 0 , error= 2024-09-11 18:13:22.599 INFO 2981023 --- [ocp-schedules-26,acc5950f02e8d923,3dfc51623950aa56] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:13:22.817 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9465ddf4de64291a,c760d511288c7236] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=9465ddf4de64291a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:13:22.817 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,5f0520a682a89230,d68d2c685b9807f0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=5f0520a682a89230, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:13:22.818 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,5f0520a682a89230,d68d2c685b9807f0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=5f0520a682a89230, duration=4 ms] 2024-09-11 18:13:22.821 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9465ddf4de64291a,c760d511288c7236] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=9465ddf4de64291a, duration=9 ms] 2024-09-11 18:13:22.827 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,a066bd279ea0088f,c16a1d6560354b22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=a066bd279ea0088f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:13:22.828 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,a066bd279ea0088f,c16a1d6560354b22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=a066bd279ea0088f, duration=2 ms] 2024-09-11 18:13:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,348d944d523a8278,fb147bfd98c18d24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,348d944d523a8278,fb147bfd98c18d24] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:23.414 INFO 2981023 --- [ocp-async-8,4cb851de019c83c2,8b01dd43d7e3f785] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:23.419 INFO 2981023 --- [ocp-schedules-30,8571efc80df08155,62a8dadaf23087eb] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 18:13:23.419 INFO 2981023 --- [ocp-schedules-30,8571efc80df08155,62a8dadaf23087eb] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 18:13:23.435 INFO 2981023 --- [pool-580-thread-2,8571efc80df08155,b22a189515e24a45] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:13:23.435 INFO 2981023 --- [pool-580-thread-1,8571efc80df08155,7618a49cdfb81ff0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:13:23.436 INFO 2981023 --- [pool-580-thread-3,8571efc80df08155,91432824b7b62f96] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:13:23.454 INFO 2981023 --- [pool-580-thread-1,8571efc80df08155,7618a49cdfb81ff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:23.456 INFO 2981023 --- [pool-580-thread-3,8571efc80df08155,91432824b7b62f96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:23.456 INFO 2981023 --- [pool-580-thread-1,8571efc80df08155,7618a49cdfb81ff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049303000000, 1726049603000000] 2024-09-11 18:13:23.457 INFO 2981023 --- [pool-580-thread-2,8571efc80df08155,b22a189515e24a45] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:23.458 INFO 2981023 --- [pool-580-thread-3,8571efc80df08155,91432824b7b62f96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049303000000, 1726049603000000] 2024-09-11 18:13:23.460 INFO 2981023 --- [pool-580-thread-2,8571efc80df08155,b22a189515e24a45] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049303000000, 1726049603000000] 2024-09-11 18:13:23.462 INFO 2981023 --- [pool-580-thread-3,8571efc80df08155,91432824b7b62f96] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:13:23.547 INFO 2981023 --- [pool-580-thread-1,8571efc80df08155,7618a49cdfb81ff0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:13:23.602 INFO 2981023 --- [pool-580-thread-2,8571efc80df08155,b22a189515e24a45] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:13:23.603 INFO 2981023 --- [ocp-schedules-30,8571efc80df08155,62a8dadaf23087eb] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 18:13:24.043 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:24.043 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:24.432 INFO 2981023 --- [ocp-schedules-19,e073d51afc60f39a,a4012d41bfd9fa6f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:13:25.414 INFO 2981023 --- [ocp-async-6,e1b2b131805d9ae8,44b7a5223c18b7ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:26.050 INFO 2981023 --- [ocp-schedules-1,c9156148e2461ead,b5f1069ecd563c30] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:13:26.050 INFO 2981023 --- [ocp-schedules-1,c9156148e2461ead,b5f1069ecd563c30] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:13:26.050 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:13:26.050 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:13:26.051 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:13:26.055 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:13:26.055 INFO 2981023 --- [ocp-schedules-1,c9156148e2461ead,b5f1069ecd563c30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:13:26.056 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:13:26.056 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:13:26.056 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:13:26.056 INFO 2981023 --- [ocp-schedules-1,c9156148e2461ead,b5f1069ecd563c30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:13:26.057 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:13:26.057 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:13:26.073 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:13:26.075 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:13:26.077 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:13:26.079 INFO 2981023 --- [ocp-schedules-1,c9156148e2461ead,b5f1069ecd563c30] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:13:26.080 INFO 2981023 --- [ocp-schedules-1,c9156148e2461ead,b5f1069ecd563c30] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:13:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,f827d9e3df55164f,d4110afafc633042] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,f827d9e3df55164f,d4110afafc633042] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:13:26.336526+08:00, lastProcessTime=2024-09-11T18:13:16.392401+08:00 2024-09-11 18:13:26.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:13:26.336526+08:00 2024-09-11 18:13:27.043 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:27.044 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:27.414 INFO 2981023 --- [ocp-async-9,4199f376beee794d,ab28d88d4380a757] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:27.786 INFO 2981023 --- [ocp-async-7,42ed138b9bf9661e,ed2403559f348174] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:13:28.318 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.320 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, 2024-09-11 18:13:28.331 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.340 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 31 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.398 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 9 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.420 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 157 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.457 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.459 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.459 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:13:28.460 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.461 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.462 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.462 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.463 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.464 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.464 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.465 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.466 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.467 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.467 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.468 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.469 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.473 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.473 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:13:28.476 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.481 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.486 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.491 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.491 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:13:28.493 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.497 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.501 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:07:28.076, end=2024-09-11T18:12:28.311} 2024-09-11 18:13:28.501 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 2 dubious sqls 2024-09-11 18:13:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,3aadc9a0f2de7afa,be3e6ee71cc80f04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,3aadc9a0f2de7afa,be3e6ee71cc80f04] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:29.415 INFO 2981023 --- [ocp-async-3,94e3b5086d02bba6,d39e887cfe98aa16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:30.044 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:30.045 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:31.415 INFO 2981023 --- [ocp-async-7,afafb7a7044112c8,c2d211a815c8aa17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:32.011 INFO 2981023 --- [ocp-schedules-18,064a701c96549fc3,4525c3eaee7adfef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:13:32.061 WARN 2981023 --- [scanner-13628780-1,064a701c96549fc3,33bec77317272727] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:13:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,ebe6520e2a69084b,0738fe48ac23717c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,ebe6520e2a69084b,0738fe48ac23717c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:32.602 INFO 2981023 --- [ocp-schedules-21,982479ea3cbdc457,ce58749bcfc873f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:13:33.045 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:33.045 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:33.415 INFO 2981023 --- [ocp-async-1,47361186acb85a72,77dd27b5afc4651a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,0397778a70168cac,77deb80da6f09283] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,0397778a70168cac,77deb80da6f09283] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:35.355 INFO 2981023 --- [ocp-schedules-14,431f48d0dbae26c8,f18dd60b0af57d76] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:13:35.360 INFO 2981023 --- [ocp-schedules-14,431f48d0dbae26c8,f18dd60b0af57d76] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:13:35.360 INFO 2981023 --- [ocp-schedules-14,431f48d0dbae26c8,f18dd60b0af57d76] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:13:35.415 INFO 2981023 --- [ocp-async-2,1022df0419c13b5e,01aeaf00df5cc1d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:36.045 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:36.046 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:13:36.336743+08:00, lastProcessTime=2024-09-11T18:13:26.336526+08:00 2024-09-11 18:13:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:13:36.336743+08:00 2024-09-11 18:13:37.314 INFO 2981023 --- [prometheus-cache-monitor1,860425509b8763e8,b7ec9f49a002770e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8198,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:13:37.315 INFO 2981023 --- [prometheus-cache-monitor1,860425509b8763e8,b7ec9f49a002770e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:13:37.415 INFO 2981023 --- [ocp-async-3,736664f8f70a979f,9714155d3909af1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:37.786 INFO 2981023 --- [ocp-async-8,9cb0ed87ff94042a,536dfa685a48b962] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:13:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,52bb55c905a83f82,3935388b22569169] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,52bb55c905a83f82,3935388b22569169] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:39.046 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:39.048 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:39.415 INFO 2981023 --- [ocp-async-9,c11972f477309a7f,d027f74631597b04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:40.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:13:40.003 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:13:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,4717bf488f81fcac,a004b2f91e460bb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,4717bf488f81fcac,a004b2f91e460bb5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:41.415 INFO 2981023 --- [ocp-async-8,e4332c51ebd65c54,a57e5a2027cf6657] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:42.048 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:42.049 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:42.198 INFO 2981023 --- [ocp-schedules-3,3f7e11141e17f1b7,246b13173cdb6f44] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:13:42.239 WARN 2981023 --- [scanner-13628965-1,3f7e11141e17f1b7,81a584d87ae260ad] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:13:42.254 INFO 2981023 --- [alarm-detect-6,3f7e11141e17f1b7,f777201768f4a2fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645692, data is empty. 2024-09-11 18:13:42.258 INFO 2981023 --- [alarm-detect-6,3f7e11141e17f1b7,f777201768f4a2fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645692, data is empty. 2024-09-11 18:13:42.260 INFO 2981023 --- [alarm-detect-6,3f7e11141e17f1b7,f777201768f4a2fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645692, data is empty. 2024-09-11 18:13:42.294 INFO 2981023 --- [alarm-detect-5,3f7e11141e17f1b7,1b9ec63196fc9bdc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645692, data is empty. 2024-09-11 18:13:42.297 INFO 2981023 --- [alarm-detect-5,3f7e11141e17f1b7,1b9ec63196fc9bdc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645692, data is empty. 2024-09-11 18:13:42.299 INFO 2981023 --- [alarm-detect-13,3f7e11141e17f1b7,eaac7924761c6589] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645692, data is empty. 2024-09-11 18:13:42.301 INFO 2981023 --- [alarm-detect-5,3f7e11141e17f1b7,1b9ec63196fc9bdc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645692, data is empty. 2024-09-11 18:13:42.304 INFO 2981023 --- [alarm-detect-13,3f7e11141e17f1b7,eaac7924761c6589] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645692, data is empty. 2024-09-11 18:13:42.339 INFO 2981023 --- [alarm-detect-13,3f7e11141e17f1b7,eaac7924761c6589] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645692, data is empty. 2024-09-11 18:13:42.443 INFO 2981023 --- [alarm-detect-10,3f7e11141e17f1b7,b1a27c8c8622cd56] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645692, data is empty. 2024-09-11 18:13:42.446 INFO 2981023 --- [alarm-detect-10,3f7e11141e17f1b7,b1a27c8c8622cd56] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645692, data is empty. 2024-09-11 18:13:42.452 INFO 2981023 --- [alarm-detect-10,3f7e11141e17f1b7,b1a27c8c8622cd56] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645692, data is empty. 2024-09-11 18:13:42.604 INFO 2981023 --- [ocp-schedules-30,974c92a706645ac7,6a62353d83dcac53] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:13:43.415 INFO 2981023 --- [ocp-async-6,314e6c1784105651,8c94ec9cc7288567] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,57bc2144cc63db69,4c7ff65bf92b02b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,57bc2144cc63db69,4c7ff65bf92b02b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:45.049 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:45.050 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:45.101 INFO 2981023 --- [prometheus-second-schedule-1,bef92cf5dc9d3bdf,469e5fcb0a6d4cac] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:13:45.416 INFO 2981023 --- [ocp-async-7,b89add6e37f80dac,47715ae59983be5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:45.576 INFO 2981023 --- [ocp-schedules-5,b978412f29301834,ba7d5bec35728e7e] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:13:45.595 WARN 2981023 --- [ocp-schedules-5,b978412f29301834,ba7d5bec35728e7e] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:13:46.015 INFO 2981023 --- [ocp-schedules-7,31b0682f5cf88746,b5e839f2d8a11cff] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:13:46.016 INFO 2981023 --- [host-check-agent-status2,31b0682f5cf88746,2664c9401241a232] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:13:46.016 INFO 2981023 --- [host-check-agent-status1,31b0682f5cf88746,145efb29eee85417] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:13:46.016 INFO 2981023 --- [host-check-agent-status3,31b0682f5cf88746,1e1a3a71799bd160] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:13:46.017 INFO 2981023 --- [host-check-agent-status4,31b0682f5cf88746,4289681c934f268c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:13:46.028 INFO 2981023 --- [ocp-schedules-19,7cfead81531638ab,155c79dc3495e2d2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:13:46.036 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:13:46.037 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:13:46.037 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:13:46.038 INFO 2981023 --- [ocp-schedules-29,a24c6edd2eaee644,842884996de7bd2a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:13:46.038 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:13:46.042 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:13:46.043 INFO 2981023 --- [host-check-clockdiff1,a24c6edd2eaee644,30d07d8b6150bf74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:13:46.043 INFO 2981023 --- [host-check-clockdiff2,a24c6edd2eaee644,ca47a1d2e73c85b9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:13:46.044 INFO 2981023 --- [host-check-clockdiff3,a24c6edd2eaee644,c43a88fa7147f0b1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:13:46.045 INFO 2981023 --- [host-check-clockdiff4,a24c6edd2eaee644,a23693b5af6c4098] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:13:46.052 INFO 2981023 --- [host-check-clockdiff2,a24c6edd2eaee644,ca47a1d2e73c85b9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049626 6 6 , error= 2024-09-11 18:13:46.056 INFO 2981023 --- [host-check-agent-status1,31b0682f5cf88746,145efb29eee85417] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:13:46.057 INFO 2981023 --- [host-check-agent-status1,31b0682f5cf88746,145efb29eee85417] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:13:46.058 INFO 2981023 --- [host-check-agent-status4,31b0682f5cf88746,4289681c934f268c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:13:46.058 INFO 2981023 --- [host-check-agent-status4,31b0682f5cf88746,4289681c934f268c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:13:46.059 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:13:46.060 INFO 2981023 --- [host-check-clockdiff3,a24c6edd2eaee644,c43a88fa7147f0b1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049626 0 0 , error= 2024-09-11 18:13:46.061 INFO 2981023 --- [host-check-agent-status2,31b0682f5cf88746,2664c9401241a232] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:13:46.064 INFO 2981023 --- [host-check-agent-status2,31b0682f5cf88746,2664c9401241a232] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:13:46.064 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:13:46.065 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:13:46.066 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:13:46.066 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:13:46.068 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:13:46.080 INFO 2981023 --- [host-check-agent-status3,31b0682f5cf88746,1e1a3a71799bd160] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:13:46.099 INFO 2981023 --- [host-check-agent-status3,31b0682f5cf88746,1e1a3a71799bd160] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:13:46.100 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:13:46.171 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:13:46.189 INFO 2981023 --- [ocp-schedules-30,c241654fa59a46ca,8429ebc2124d2c71] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:13:46.189 INFO 2981023 --- [ocp-schedules-21,cc5614d4c0bb954e,950a0bc37e0b164c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:13:46.189 INFO 2981023 --- [ocp-schedules-12,ca2667d35cf07468,781ead230f20ea15] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:13:46.189 INFO 2981023 --- [ocp-schedules-23,9913a03c06e36115,ccbf73e904e6dcd5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:13:46.257 INFO 2981023 --- [ocp-schedules-30,c241654fa59a46ca,8429ebc2124d2c71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:13:46.261 INFO 2981023 --- [ocp-schedules-14,17cf5976fb21a758,1a09d8f006d942e9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:13:46.271 INFO 2981023 --- [ocp-schedules-21,cc5614d4c0bb954e,950a0bc37e0b164c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:13:46.278 INFO 2981023 --- [ocp-schedules-14,17cf5976fb21a758,1a09d8f006d942e9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:13:46.300 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:13:46.302 INFO 2981023 --- [ocp-schedules-12,ca2667d35cf07468,781ead230f20ea15] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:13:46.305 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:13:46.308225172+08:00 2024-09-11 18:13:46.313 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:13:46.317 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:13:46.305+08:00 2024-09-11 18:13:46.323 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:13:46.305+08:00 and status AVAILABLE 2024-09-11 18:13:46.335 INFO 2981023 --- [ocp-schedules-24,99e8469dc18018e1,80686ba9e7c78480] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:13:46.335 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:13:46.338 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049522904000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:13:46.339 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:13:46.343529016+08:00 2024-09-11 18:13:46.341 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:13:46.339+08:00 2024-09-11 18:13:46.345 INFO 2981023 --- [ocp-schedules-30,c241654fa59a46ca,8429ebc2124d2c71] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:13:46.351 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:13:46.339+08:00 and status AVAILABLE 2024-09-11 18:13:46.355 INFO 2981023 --- [host-check-agent-status4,31b0682f5cf88746,4289681c934f268c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:13:46.357 INFO 2981023 --- [host-check-agent-status3,31b0682f5cf88746,1e1a3a71799bd160] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:13:46.360 INFO 2981023 --- [host-check-agent-status4,31b0682f5cf88746,4289681c934f268c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:13:46.360 INFO 2981023 --- [host-check-agent-status4,31b0682f5cf88746,4289681c934f268c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:13:46.360 INFO 2981023 --- [host-check-agent-status4,31b0682f5cf88746,4289681c934f268c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:13:46.365 INFO 2981023 --- [host-check-agent-status3,31b0682f5cf88746,1e1a3a71799bd160] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:13:46.365 INFO 2981023 --- [host-check-agent-status3,31b0682f5cf88746,1e1a3a71799bd160] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:13:46.365 INFO 2981023 --- [host-check-agent-status3,31b0682f5cf88746,1e1a3a71799bd160] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:13:46.374 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:13:46.383 INFO 2981023 --- [host-check-agent-status1,31b0682f5cf88746,145efb29eee85417] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:13:46.388 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:13:46.388 INFO 2981023 --- [host-check-agent-status1,31b0682f5cf88746,145efb29eee85417] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:13:46.388 INFO 2981023 --- [host-check-agent-status1,31b0682f5cf88746,145efb29eee85417] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:13:46.388 INFO 2981023 --- [host-check-agent-status1,31b0682f5cf88746,145efb29eee85417] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:13:46.389 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:13:46.389135717+08:00 2024-09-11 18:13:46.390 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:13:46.390 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:13:46.389+08:00 2024-09-11 18:13:46.393 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:13:46.390511+08:00, lastProcessTime=2024-09-11T18:13:36.336743+08:00 2024-09-11 18:13:46.397 INFO 2981023 --- [host-check-agent-status2,31b0682f5cf88746,2664c9401241a232] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:13:46.400 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:13:46.389+08:00 and status AVAILABLE 2024-09-11 18:13:46.400 INFO 2981023 --- [host-check-agent-status2,31b0682f5cf88746,2664c9401241a232] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:13:46.400 INFO 2981023 --- [host-check-agent-status2,31b0682f5cf88746,2664c9401241a232] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:13:46.400 INFO 2981023 --- [host-check-agent-status2,31b0682f5cf88746,2664c9401241a232] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:13:46.408 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:13:46.408 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:13:46.408648454+08:00 2024-09-11 18:13:46.409 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:13:46.409+08:00 2024-09-11 18:13:46.454 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:13:46.455 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:13:46.409+08:00 and status AVAILABLE 2024-09-11 18:13:46.460 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:13:46.463 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:13:46.464 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:13:46.469 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:13:46.471 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:13:46.472 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:13:46.473 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:13:46.475 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:13:46.475 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:13:46.485 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.485 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.486 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:13:46.486 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:13:46.486 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:13:46.487 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:13:46.488 INFO 2981023 --- [ocp-schedules-21,cc5614d4c0bb954e,950a0bc37e0b164c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:13:46.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:46.501 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:13:46.503 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.503 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.503 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:46.503 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:13:46.504 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:13:46.505 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.506 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:13:46.520 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:13:46.521 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.521 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:46.523 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:13:46.525 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:13:46.531 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:13:46.533 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:13:46.547 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:13:46.549 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:13:46.564 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:13:46.566 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:13:46.568 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:13:46.575 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:13:46.585 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.586 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.587 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:13:46.587 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:13:46.587 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.596 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:13:46.596 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049522904000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:13:46.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.607 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.608 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:13:46.614 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:13:46.614 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:13:46.618 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:13:46.618 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:13:46.622 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:13:46.632 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:13:46.635 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:13:46.642 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:13:46.642 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:13:46.652 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:13:46.656 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:46.658 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:13:46.658 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.661 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:13:46.662 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:13:46.663 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:13:46.665 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:13:46.668 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:13:46.669 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:13:46.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:13:46.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:13:46.684 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:13:46.685 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:13:46.702 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.703 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.703 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:13:46.704 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:13:46.704 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.705 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:13:46.706 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:13:46.706 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.709 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:13:46.712 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:13:46.712 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.713 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:13:46.713 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.714 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:13:46.714 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:13:46.717 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:13:46.719 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:13:46.722 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:13:46.729 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:13:46.730 INFO 2981023 --- [host-check-host-status3,7cfead81531638ab,7f2a885d8ed50793] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:13:46.730 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:13:46.737 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:13:46.739 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:13:46.740 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:13:46.743 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:13:46.762 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.762 INFO 2981023 --- [host-check-host-status1,7cfead81531638ab,71b16b18776332f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:13:46.762 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.762 INFO 2981023 --- [host-check-host-status4,7cfead81531638ab,9a1c9b2bcb09d914] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:13:46.763 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:13:46.766 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:13:46.767 INFO 2981023 --- [host-check-host-status2,7cfead81531638ab,0b7d25ca9b0c689f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:13:46.773 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.775 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.775 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:13:46.776 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:13:46.776 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.792 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:13:46.811 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:13:46.820 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.821 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.821 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:13:46.822 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:13:46.822 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.825 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:13:46.825 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:13:46.831 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.833 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.833 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:13:46.834 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:13:46.834 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:13:46.834 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:13:46.834 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.838 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:13:46.390511+08:00 2024-09-11 18:13:46.845 INFO 2981023 --- [ocp-schedules-10,03789f8994d005ea,aa98a32716dea4bb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8198,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:13:46.845 INFO 2981023 --- [ocp-schedules-10,03789f8994d005ea,aa98a32716dea4bb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:13:46.853 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:13:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.861 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:13:46.861 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:13:46.864 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:13:46.866 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:13:46.867 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:13:46.879 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:13:46.885 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:13:46.889 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:13:46.895 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.895 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.896 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:13:46.901 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:13:46.901 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:13:46.902 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:13:46.908 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:13:46.908 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:13:46.911 INFO 2981023 --- [ocp-schedules-28,f28ec44bc9728743,939df6a08e8f6b60] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:13:46.914 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:46.914 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:46.915 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:13:46.920 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:13:46.920 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:13:46.920 INFO 2981023 --- [ocp-schedules-12,ca2667d35cf07468,781ead230f20ea15] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:13:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,f0993da47734f0cd,d4fb5af241da8af5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:47.315 INFO 2981023 --- [prometheus-minute-schedule-1,f0993da47734f0cd,d4fb5af241da8af5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:47.416 INFO 2981023 --- [ocp-async-8,23dc7535c30a9255,d86a3128a488011d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:47.786 INFO 2981023 --- [ocp-async-7,5565b2183c16d9f4,63ed74e87e5e57ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:13:48.050 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:48.051 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:49.416 INFO 2981023 --- [ocp-async-3,30102ecb6e2bbd03,74af02fcbe7fec99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:13:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:13:50.179 INFO 2981023 --- [host-check-clockdiff1,a24c6edd2eaee644,30d07d8b6150bf74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049630 6 6 , error= 2024-09-11 18:13:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,a249d138fa4f242e,027f7c41d626b1ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,a249d138fa4f242e,027f7c41d626b1ef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:50.339 INFO 2981023 --- [ocp-schedules-26,b256b44f0f73bc17,653e0211b846091f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:13:51.051 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:51.053 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:51.416 INFO 2981023 --- [ocp-async-7,7ccbbe14a984db5d,e6c47fb2dd27dfb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:51.614 INFO 2981023 --- [ocp-schedules-7,d792af0484d8a159,ebe98fd2a5f724de] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:13:51.616 INFO 2981023 --- [ocp-schedules-7,d792af0484d8a159,ebe98fd2a5f724de] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:13:51.616 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:13:51.616 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:13:51.616 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:13:51.617 INFO 2981023 --- [ocp-schedules-7,d792af0484d8a159,ebe98fd2a5f724de] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:13:51.998 INFO 2981023 --- [host-check-clockdiff4,a24c6edd2eaee644,a23693b5af6c4098] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049631 0 0 , error= 2024-09-11 18:13:52.488 INFO 2981023 --- [ocp-schedules-6,935088e059eb89aa,d9b111af15b10b98] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:13:52.545 WARN 2981023 --- [scanner-13629332-1,935088e059eb89aa,b2d5295e0ce8bfc5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:13:52.608 INFO 2981023 --- [ocp-schedules-1,d6bea4e4a5b5909b,efb918537728476a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:13:53.307 INFO 2981023 --- [ocp-schedules-19,02c2c7f2f3e0a06d,47120e1574ae2568] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:13:53.307 INFO 2981023 --- [ocp-schedules-19,02c2c7f2f3e0a06d,47120e1574ae2568] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:13:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,a3c38960be7a8a36,f86d3d7b744dacc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:53.315 INFO 2981023 --- [prometheus-minute-schedule-1,a3c38960be7a8a36,f86d3d7b744dacc5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:53.416 INFO 2981023 --- [ocp-async-2,1071ea5d16d86033,2222f09175c57bc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:54.053 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:54.054 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:54.399 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,936308db61f00a40,b083572396a1ca36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=936308db61f00a40, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:13:54.402 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,936308db61f00a40,b083572396a1ca36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=936308db61f00a40, duration=7 ms] 2024-09-11 18:13:54.406 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3ef67b7cf36812a4,a9e3ba17a3e1015f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=3ef67b7cf36812a4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:13:54.407 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,3ef67b7cf36812a4,a9e3ba17a3e1015f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=3ef67b7cf36812a4, duration=2 ms] 2024-09-11 18:13:54.410 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,dd054a17948ad7f1,f339bcfbf7bb20b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=dd054a17948ad7f1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:13:54.411 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,dd054a17948ad7f1,f339bcfbf7bb20b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=dd054a17948ad7f1, duration=3 ms] 2024-09-11 18:13:54.433 INFO 2981023 --- [ocp-schedules-26,a9117616e3f56ed8,a573cb040ce15bae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:13:54.746 INFO 2981023 --- [ocp-schedules-5,8d279cdaab92b14e,940b7e1009b4bf17] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:13:54.764 WARN 2981023 --- [ocp-schedules-5,8d279cdaab92b14e,940b7e1009b4bf17] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:13:54.897 INFO 2981023 --- [ocp-schedules-7,2abd4cd82cb13a1d,d6c512ecd42b6c07] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T18:13:54.897+08:00, tenantSize=6 2024-09-11 18:13:54.905 INFO 2981023 --- [ocp-schedules-7,2abd4cd82cb13a1d,d6c512ecd42b6c07] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726049634897000 2024-09-11 18:13:54.905 INFO 2981023 --- [ocp-schedules-7,2abd4cd82cb13a1d,d6c512ecd42b6c07] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726049334834000 2024-09-11 18:13:54.928 INFO 2981023 --- [ocp-schedules-7,2abd4cd82cb13a1d,d6c512ecd42b6c07] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 18:13:55.363 INFO 2981023 --- [ocp-schedules-29,bc7a91830d44f8d9,a06df53bbb4f4a4c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:13:55.372 INFO 2981023 --- [ocp-schedules-29,bc7a91830d44f8d9,a06df53bbb4f4a4c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:13:55.372 INFO 2981023 --- [ocp-schedules-29,bc7a91830d44f8d9,a06df53bbb4f4a4c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:13:55.416 INFO 2981023 --- [ocp-async-6,8bfdf6b26b2687d6,2ec50beb4b2330e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:55.870 INFO 2981023 --- [ocp-schedules-3,8662eb2070116fc7,188e4c65004638e0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:13:55.870 INFO 2981023 --- [ocp-schedules-3,8662eb2070116fc7,188e4c65004638e0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:13:55.871 INFO 2981023 --- [ocp-schedules-3,8662eb2070116fc7,188e4c65004638e0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:13:55.871 INFO 2981023 --- [ocp-schedules-3,8662eb2070116fc7,188e4c65004638e0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:13:55.871 INFO 2981023 --- [ocp-schedules-3,8662eb2070116fc7,188e4c65004638e0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:13:55.871 INFO 2981023 --- [ocp-schedules-3,8662eb2070116fc7,188e4c65004638e0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:13:56.051 INFO 2981023 --- [ocp-schedules-26,631ea0896e92cdd8,9dbfcff896597cd4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:13:56.058 INFO 2981023 --- [ocp-schedules-26,631ea0896e92cdd8,9dbfcff896597cd4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:13:56.058 INFO 2981023 --- [ocp-schedules-26,631ea0896e92cdd8,9dbfcff896597cd4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:13:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,223d4afb1b8852b1,022cc652a4a23648] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,223d4afb1b8852b1,022cc652a4a23648] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:56.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:13:56.353354+08:00, lastProcessTime=2024-09-11T18:13:46.390511+08:00 2024-09-11 18:13:56.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:13:56.353354+08:00 2024-09-11 18:13:57.054 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:13:57.055 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:13:57.416 INFO 2981023 --- [ocp-async-10,a55802a58c360fbb,213c60dc5891422c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:57.786 INFO 2981023 --- [ocp-async-1,e7a4ebc975b7699c,f035bd6e02391706] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:13:57.974 INFO 2981023 --- [ocp-schedules-9,2bba1c0f39e8c44a,bff27db6da165076] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:13:57.974 INFO 2981023 --- [ocp-schedules-9,2bba1c0f39e8c44a,bff27db6da165076] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:13:57.992 INFO 2981023 --- [Collect all tenants major compactions1,2bba1c0f39e8c44a,f4ae9b94512224b9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:13:57.993 INFO 2981023 --- [Collect all tenants major compactions2,2bba1c0f39e8c44a,7fd9c223a032d93e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:13:57.994 INFO 2981023 --- [Collect all tenants major compactions3,2bba1c0f39e8c44a,b92774bf7faa0344] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:13:57.997 INFO 2981023 --- [Collect all tenants major compactions4,2bba1c0f39e8c44a,ad97d4e8ba03e1f4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:13:58.010 INFO 2981023 --- [Collect all tenants major compactions2,2bba1c0f39e8c44a,7fd9c223a032d93e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:58.010 INFO 2981023 --- [Collect all tenants major compactions1,2bba1c0f39e8c44a,f4ae9b94512224b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:58.011 INFO 2981023 --- [Collect all tenants major compactions6,2bba1c0f39e8c44a,aef58a0d4cbeac77] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:13:58.011 INFO 2981023 --- [Collect all tenants major compactions5,2bba1c0f39e8c44a,6f21d6f43440250e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:13:58.012 INFO 2981023 --- [Collect all tenants major compactions3,2bba1c0f39e8c44a,b92774bf7faa0344] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:58.012 INFO 2981023 --- [Collect all tenants major compactions2,2bba1c0f39e8c44a,7fd9c223a032d93e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:13:58.012 INFO 2981023 --- [Collect all tenants major compactions1,2bba1c0f39e8c44a,f4ae9b94512224b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:13:58.014 INFO 2981023 --- [Collect all tenants major compactions3,2bba1c0f39e8c44a,b92774bf7faa0344] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:13:58.014 INFO 2981023 --- [Collect all tenants major compactions4,2bba1c0f39e8c44a,ad97d4e8ba03e1f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:58.015 INFO 2981023 --- [Collect all tenants major compactions4,2bba1c0f39e8c44a,ad97d4e8ba03e1f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:13:58.016 INFO 2981023 --- [Collect all tenants major compactions2,2bba1c0f39e8c44a,7fd9c223a032d93e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:13:58.016 INFO 2981023 --- [Collect all tenants major compactions2,2bba1c0f39e8c44a,7fd9c223a032d93e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:13:58.017 INFO 2981023 --- [Collect all tenants major compactions3,2bba1c0f39e8c44a,b92774bf7faa0344] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:13:58.017 INFO 2981023 --- [Collect all tenants major compactions3,2bba1c0f39e8c44a,b92774bf7faa0344] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:13:58.017 INFO 2981023 --- [Collect all tenants major compactions1,2bba1c0f39e8c44a,f4ae9b94512224b9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:13:58.017 INFO 2981023 --- [Collect all tenants major compactions1,2bba1c0f39e8c44a,f4ae9b94512224b9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:13:58.017 INFO 2981023 --- [Collect all tenants major compactions4,2bba1c0f39e8c44a,ad97d4e8ba03e1f4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:13:58.017 INFO 2981023 --- [Collect all tenants major compactions4,2bba1c0f39e8c44a,ad97d4e8ba03e1f4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:13:58.021 INFO 2981023 --- [Collect all tenants major compactions5,2bba1c0f39e8c44a,6f21d6f43440250e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:58.022 INFO 2981023 --- [Collect all tenants major compactions5,2bba1c0f39e8c44a,6f21d6f43440250e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:13:58.023 INFO 2981023 --- [Collect all tenants major compactions6,2bba1c0f39e8c44a,aef58a0d4cbeac77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:58.024 INFO 2981023 --- [Collect all tenants major compactions5,2bba1c0f39e8c44a,6f21d6f43440250e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:13:58.024 INFO 2981023 --- [Collect all tenants major compactions5,2bba1c0f39e8c44a,6f21d6f43440250e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:13:58.024 INFO 2981023 --- [Collect all tenants major compactions6,2bba1c0f39e8c44a,aef58a0d4cbeac77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:13:58.026 INFO 2981023 --- [Collect all tenants major compactions6,2bba1c0f39e8c44a,aef58a0d4cbeac77] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:13:58.026 INFO 2981023 --- [Collect all tenants major compactions6,2bba1c0f39e8c44a,aef58a0d4cbeac77] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:13:58.027 INFO 2981023 --- [ocp-schedules-9,2bba1c0f39e8c44a,bff27db6da165076] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:13:58.661 INFO 2981023 --- [ocp-schedules-1,232d5ba1054e0c4c,7eee5518517fff9d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:13:58.661+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:13:58.665 INFO 2981023 --- [ocp-schedules-1,232d5ba1054e0c4c,7eee5518517fff9d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:13:58.818 INFO 2981023 --- [ocp-schedules-1,232d5ba1054e0c4c,7eee5518517fff9d] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 41 to SqlAggregateData 2024-09-11 18:13:58.819 INFO 2981023 --- [ocp-schedules-1,232d5ba1054e0c4c,7eee5518517fff9d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:13:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,eb5569ba561f6805,a6965893a99f199f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049640, interval=60 2024-09-11 18:13:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,eb5569ba561f6805,a6965893a99f199f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:13:59.416 INFO 2981023 --- [ocp-async-2,525a3020e9eaa677,4aad6d4cb6a85c26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:13:59.597 INFO 2981023 --- [ocp-schedules-9,6b63db9b8f8bcf24,04ca89bcb46cc3c7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:13:59.598 INFO 2981023 --- [ocp-schedules-9,6b63db9b8f8bcf24,04ca89bcb46cc3c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:13:59.628 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:13:59.628 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.629 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:13:59.629 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.633 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:13:59.633 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:13:59.637 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.637 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:13:59.637 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.637 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.637 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.638 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.640 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:13:59.644 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.644 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.644 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.644 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.649 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.649 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.653 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.653 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:13:59.657 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.657 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.659 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.659 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.660 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:13:59.660 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.660 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:13:59.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5271D6DB-0-0] 2024-09-11 18:13:59.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:13:59.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:13:59.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.666 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:13:59.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.671 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.671 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:13:59.678 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.678 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:13:59.678 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.679 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:13:59.679 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.680 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:13:59.680 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:13:59.682 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.683 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:13:59.683 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:13:59.683 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5271D6E5-0-0] 2024-09-11 18:13:59.683 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:13:59.685 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:13:59.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:13:59.690 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.691 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:13:59.691 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.692 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:13:59.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:13:59.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.695 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:13:59.695 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.696 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:13:59.697 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.698 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:13:59.698 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:13:59.700 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:13:59.701 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.702 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:13:59.702 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:59.702 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.703 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:13:59.704 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.704 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.705 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:13:59.705 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.705 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.706 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:13:59.707 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.707 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:13:59.708 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.709 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:13:59.709 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.710 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:13:59.710 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.711 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:13:59.711 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.712 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:13:59.712 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:59.712 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:13:59.714 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.714 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:13:59.714 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:13:59.719 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:13:59.722 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:13:59.722 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:13:59.722 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:13:59.727 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:13:59.727 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:13:59.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:13:59.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.729 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:13:59.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.735 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:13:59.737 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:13:59.737 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.738 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:13:59.741 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:13:59.742 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:13:59.742 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:13:59.742 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:13:59.744 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:13:59.745 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:13:59.745 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:13:59.746 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:13:59.751 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:13:59.753 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:13:59.753 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:13:59.753 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:13:59.753 INFO 2981023 --- [ocp-schedules-9,6b63db9b8f8bcf24,04ca89bcb46cc3c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:14:00.055 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:00.055 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:01.417 INFO 2981023 --- [ocp-async-1,5630d9ac701e6d07,6754ae4776fc6e7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:02.236 INFO 2981023 --- [ocp-schedules-22,1c475c4925c6612d,3360dee7f916196e] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:14:02.236 INFO 2981023 --- [ocp-schedules-22,1c475c4925c6612d,3360dee7f916196e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:14:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,a9b230d7aea543b5,27f2fbfb425d7eb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,a9b230d7aea543b5,27f2fbfb425d7eb3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:02.615 INFO 2981023 --- [ocp-schedules-12,d563572b088b7ea4,ed8aaf3807622c5d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:14:02.687 INFO 2981023 --- [ocp-schedules-29,b504d6121af32013,1ac9c2afc53bb5b8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:14:02.733 WARN 2981023 --- [scanner-13629533-1,b504d6121af32013,1bba7d8ed9a51d7d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:14:02.760 INFO 2981023 --- [alarm-detect-19,b504d6121af32013,179182d0193792d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645692, data is empty. 2024-09-11 18:14:02.764 INFO 2981023 --- [alarm-detect-19,b504d6121af32013,179182d0193792d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645733, data is empty. 2024-09-11 18:14:02.768 INFO 2981023 --- [alarm-detect-19,b504d6121af32013,179182d0193792d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645733, data is empty. 2024-09-11 18:14:02.789 INFO 2981023 --- [alarm-detect-5,b504d6121af32013,5ef402fed22a6617] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645692, data is empty. 2024-09-11 18:14:02.790 INFO 2981023 --- [alarm-detect-14,b504d6121af32013,f35aec603fe814f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645692, data is empty. 2024-09-11 18:14:02.794 INFO 2981023 --- [alarm-detect-14,b504d6121af32013,f35aec603fe814f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645733, data is empty. 2024-09-11 18:14:02.794 INFO 2981023 --- [alarm-detect-5,b504d6121af32013,5ef402fed22a6617] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645733, data is empty. 2024-09-11 18:14:02.798 INFO 2981023 --- [alarm-detect-5,b504d6121af32013,5ef402fed22a6617] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645733, data is empty. 2024-09-11 18:14:02.800 INFO 2981023 --- [alarm-detect-14,b504d6121af32013,f35aec603fe814f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645733, data is empty. 2024-09-11 18:14:02.908 INFO 2981023 --- [alarm-detect-2,b504d6121af32013,7d88193f049a5875] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645692, data is empty. 2024-09-11 18:14:02.913 INFO 2981023 --- [ocp-schedules-28,7663d50bb6470542,312cef3db0277a35] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:14:02.915 INFO 2981023 --- [alarm-detect-2,b504d6121af32013,7d88193f049a5875] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645733, data is empty. 2024-09-11 18:14:02.917 INFO 2981023 --- [ocp-schedules-28,7663d50bb6470542,312cef3db0277a35] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:14:02.920 INFO 2981023 --- [alarm-detect-2,b504d6121af32013,7d88193f049a5875] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645733, data is empty. 2024-09-11 18:14:02.963 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:02.964 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:14:02.969 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:02.971 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:14:02.972 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049642972000} 2024-09-11 18:14:02.977 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:14:02.981 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000231, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:14:02.981 INFO 2981023 --- [ocp-schedules-28,7663d50bb6470542,312cef3db0277a35] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:14:03.055 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:03.056 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:03.207 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:14:03.207 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:14:03.208 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:14:03.212 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000231, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:14:03.214 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:14:03.417 INFO 2981023 --- [ocp-async-7,fe8ca011b8edc5d0,739a259f06f5957b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:04.199 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000231 2024-09-11 18:14:05.228 INFO 2981023 --- [ocp-schedules-29,0c55fdc58cf510a4,8e15cd1cfd403fb0] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:14:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,ef916ba5a5b34966,04299e72f48c9145] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:05.320 INFO 2981023 --- [prometheus-minute-schedule-1,ef916ba5a5b34966,04299e72f48c9145] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:05.417 INFO 2981023 --- [ocp-async-8,a9f08fd6f8f291df,8b10e634ad08425c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:14:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:14:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:14:05.993 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:14:05.997 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:14:06.002 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:14:06.002 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:14:06.002 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:14:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:14:06.006 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:06.006 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:06.052 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:11:41, start=2024-09-11T18:07:41, end=2024-09-11T18:10:11, updates=292, granule=120000ms, delay=240000ms 2024-09-11 18:14:06.056 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:06.057 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:11:41, target=2024-09-11T18:10:11, granule=120000ms, delay=90000ms) 2024-09-11 18:14:06.057 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:10:11, target=2024-09-11T18:00:51, granule=600000ms, delay=560000ms) 2024-09-11 18:14:06.057 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=87ms, period=120000ms 2024-09-11 18:14:06.058 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:14:06.352613+08:00, lastProcessTime=2024-09-11T18:13:56.353354+08:00 2024-09-11 18:14:06.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:14:06.352613+08:00 2024-09-11 18:14:07.417 INFO 2981023 --- [ocp-async-6,7c142e094960fbcf,decba15a6f2596fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:07.786 INFO 2981023 --- [ocp-async-5,9de712ebfce65d3f,8371ea38c1fbb9f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:14:07.944 INFO 2981023 --- [ocp-schedules-13,b94ee82b3aa736b3,440f3a7f21c1645c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:14:08.059 INFO 2981023 --- [ocp-schedules-7,45c61ba60929266e,1a29d7eeb8a90121] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:14:08.059+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:14:08.060 INFO 2981023 --- [ocp-schedules-7,45c61ba60929266e,1a29d7eeb8a90121] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:14:08.062 INFO 2981023 --- [ocp-schedules-7,45c61ba60929266e,1a29d7eeb8a90121] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:14:08.062 INFO 2981023 --- [ocp-schedules-7,45c61ba60929266e,1a29d7eeb8a90121] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:14:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,97e570ce95bd67d5,084e2c64aae58080] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,97e570ce95bd67d5,084e2c64aae58080] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:09.058 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:09.060 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:09.417 INFO 2981023 --- [ocp-async-10,be835b662b934bef,21f332c9affc8637] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:10.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:14:10.003 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:14:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:14:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:14:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,f698b951f532eb6c,b11e37c45f857fa3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,f698b951f532eb6c,b11e37c45f857fa3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:11.417 INFO 2981023 --- [ocp-async-5,40436cd856d19f72,48b982814c909932] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:11.796 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049580, nodeCount=418, affectRows=418 2024-09-11 18:14:12.060 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:12.060 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:12.564 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049580, nodeCount=1000, affectRows=1000 2024-09-11 18:14:12.566 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049580, nodeCount=999, validCount=999 2024-09-11 18:14:12.570 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049580, nodeCount=1000, affectRows=1000 2024-09-11 18:14:12.573 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049580, nodeCount=1000, validCount=1000 2024-09-11 18:14:12.578 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049580, nodeCount=48, affectRows=48 2024-09-11 18:14:12.578 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049580, nodeCount=46, validCount=46 2024-09-11 18:14:12.618 INFO 2981023 --- [ocp-schedules-12,63977564412e472e,bc81c9fe633fc89b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:14:12.964 INFO 2981023 --- [ocp-schedules-22,985371e420dbe1ac,77d7efacbee8cf29] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:14:13.009 WARN 2981023 --- [scanner-13631341-1,985371e420dbe1ac,fd193f1699ca388d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:14:13.417 INFO 2981023 --- [ocp-async-2,3293af6170c8f372,a3a5cc7f5126c0e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:13.618 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049580, nodeCount=418, validCount=418 2024-09-11 18:14:13.630 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049580, nodeCount=62, affectRows=62 2024-09-11 18:14:13.841 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049580, nodeCount=62, validCount=62 2024-09-11 18:14:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,37cd13fd1b357905,ed581114881aa6fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,697dc64c606c4451,137d84ca3b182264] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:14:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,37cd13fd1b357905,ed581114881aa6fe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:15.060 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:15.061 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:15.374 INFO 2981023 --- [ocp-schedules-30,e4a33631d8b665a8,5c4dbdae49c9e975] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:14:15.382 INFO 2981023 --- [ocp-schedules-30,e4a33631d8b665a8,5c4dbdae49c9e975] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:14:15.382 INFO 2981023 --- [ocp-schedules-30,e4a33631d8b665a8,5c4dbdae49c9e975] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:14:15.417 INFO 2981023 --- [ocp-async-1,4e8a2022296d08c2,e780d22d05ca301e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:16.016 INFO 2981023 --- [ocp-schedules-29,234de43eb83d6757,767e7e0f34142cf8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:14:16.016 INFO 2981023 --- [host-check-agent-status1,234de43eb83d6757,2d9dfd5503ced5c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:14:16.017 INFO 2981023 --- [host-check-agent-status2,234de43eb83d6757,22a14e1a585daa21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:14:16.017 INFO 2981023 --- [host-check-agent-status3,234de43eb83d6757,1b7823c9afe501c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:14:16.017 INFO 2981023 --- [ocp-schedules-5,53c40cd11edfc8ec,9da4306745d913b4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:14:16.018 INFO 2981023 --- [host-check-clockdiff1,53c40cd11edfc8ec,c07dfd8db00e1572] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:14:16.018 INFO 2981023 --- [host-check-clockdiff2,53c40cd11edfc8ec,e0953ab571396f95] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:14:16.019 INFO 2981023 --- [host-check-clockdiff3,53c40cd11edfc8ec,20396eba3d58eb87] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:14:16.020 INFO 2981023 --- [host-check-agent-status4,234de43eb83d6757,0297c41672987a47] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:14:16.021 INFO 2981023 --- [ocp-schedules-25,286622ff6b018d39,1850045ccdc08fa0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:14:16.039 INFO 2981023 --- [host-check-clockdiff4,53c40cd11edfc8ec,9b9a3512e28cd56b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:14:16.043 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:14:16.044 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:14:16.044 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:14:16.045 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:14:16.047 INFO 2981023 --- [host-check-clockdiff3,53c40cd11edfc8ec,20396eba3d58eb87] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049656 0 0 , error= 2024-09-11 18:14:16.057 INFO 2981023 --- [host-check-agent-status3,234de43eb83d6757,1b7823c9afe501c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:16.061 INFO 2981023 --- [host-check-agent-status4,234de43eb83d6757,0297c41672987a47] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:14:16.062 INFO 2981023 --- [host-check-agent-status3,234de43eb83d6757,1b7823c9afe501c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:16.062 INFO 2981023 --- [host-check-agent-status1,234de43eb83d6757,2d9dfd5503ced5c4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:16.064 INFO 2981023 --- [host-check-agent-status1,234de43eb83d6757,2d9dfd5503ced5c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:16.064 INFO 2981023 --- [host-check-agent-status2,234de43eb83d6757,22a14e1a585daa21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:14:16.065 INFO 2981023 --- [host-check-agent-status4,234de43eb83d6757,0297c41672987a47] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:14:16.066 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:14:16.066 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:16.067 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:14:16.067 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:16.068 INFO 2981023 --- [host-check-agent-status2,234de43eb83d6757,22a14e1a585daa21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:14:16.187 INFO 2981023 --- [ocp-schedules-4,75c2c509b4854467,be6ec8151a9a9ffc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:14:16.187 INFO 2981023 --- [ocp-schedules-26,f075b126fb8b4563,4600ddba3bb20aa3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:14:16.191 INFO 2981023 --- [ocp-schedules-20,977f8a4a226f0a29,683ceb357b79f7b6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:14:16.191 INFO 2981023 --- [ocp-schedules-23,20a5fa17e1889fc5,34604fff9a363d32] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:14:16.257 INFO 2981023 --- [ocp-schedules-26,f075b126fb8b4563,4600ddba3bb20aa3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:14:16.259 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049642972000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:14:16.266 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:14:16.273 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:16.280 INFO 2981023 --- [ocp-schedules-20,977f8a4a226f0a29,683ceb357b79f7b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:14:16.280 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:14:16.282 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:16.287 INFO 2981023 --- [ocp-schedules-4,75c2c509b4854467,be6ec8151a9a9ffc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:14:16.290 INFO 2981023 --- [ocp-schedules-4,75c2c509b4854467,be6ec8151a9a9ffc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:14:16.386 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:14:16.390 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:14:16.388412308+08:00 2024-09-11 18:14:16.390 INFO 2981023 --- [host-check-agent-status4,234de43eb83d6757,0297c41672987a47] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:14:16.391 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:14:16.390+08:00 2024-09-11 18:14:16.393 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:14:16.390+08:00 and status AVAILABLE 2024-09-11 18:14:16.394 INFO 2981023 --- [host-check-agent-status4,234de43eb83d6757,0297c41672987a47] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:14:16.394 INFO 2981023 --- [host-check-agent-status4,234de43eb83d6757,0297c41672987a47] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:14:16.394 INFO 2981023 --- [host-check-agent-status4,234de43eb83d6757,0297c41672987a47] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:14:16.456 INFO 2981023 --- [host-check-agent-status2,234de43eb83d6757,22a14e1a585daa21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:14:16.457 INFO 2981023 --- [ocp-schedules-14,0151c5b85f1aa134,a127f05667514bc5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:14:16.462 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.462 INFO 2981023 --- [host-check-agent-status2,234de43eb83d6757,22a14e1a585daa21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:14:16.462 INFO 2981023 --- [host-check-agent-status2,234de43eb83d6757,22a14e1a585daa21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:14:16.462 INFO 2981023 --- [host-check-agent-status2,234de43eb83d6757,22a14e1a585daa21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:14:16.462 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:14:16.463 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.463 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:14:16.464 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:14:16.464 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:14:16.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:14:16.463751+08:00, lastProcessTime=2024-09-11T18:14:06.352613+08:00 2024-09-11 18:14:16.465 INFO 2981023 --- [ocp-schedules-26,f075b126fb8b4563,4600ddba3bb20aa3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:14:16.465 INFO 2981023 --- [host-check-agent-status3,234de43eb83d6757,1b7823c9afe501c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:14:16.468 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:14:16.473 INFO 2981023 --- [host-check-agent-status3,234de43eb83d6757,1b7823c9afe501c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:14:16.473 INFO 2981023 --- [host-check-agent-status3,234de43eb83d6757,1b7823c9afe501c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:14:16.473 INFO 2981023 --- [host-check-agent-status3,234de43eb83d6757,1b7823c9afe501c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:14:16.475 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:14:16.475 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:14:16.475386657+08:00 2024-09-11 18:14:16.475 INFO 2981023 --- [host-check-agent-status1,234de43eb83d6757,2d9dfd5503ced5c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:14:16.478 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:14:16.475+08:00 2024-09-11 18:14:16.480 INFO 2981023 --- [host-check-agent-status1,234de43eb83d6757,2d9dfd5503ced5c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:14:16.480 INFO 2981023 --- [host-check-agent-status1,234de43eb83d6757,2d9dfd5503ced5c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:14:16.480 INFO 2981023 --- [host-check-agent-status1,234de43eb83d6757,2d9dfd5503ced5c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:14:16.481 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:14:16.475+08:00 and status AVAILABLE 2024-09-11 18:14:16.483 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:14:16.483 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:14:16.484 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:14:16.489281484+08:00 2024-09-11 18:14:16.484 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:14:16.489445235+08:00 2024-09-11 18:14:16.484 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:14:16.484+08:00 2024-09-11 18:14:16.485 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:14:16.484+08:00 2024-09-11 18:14:16.489 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:14:16.484+08:00 and status AVAILABLE 2024-09-11 18:14:16.491 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.493 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:14:16.484+08:00 and status AVAILABLE 2024-09-11 18:14:16.494 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:14:16.494 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:16.495 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:14:16.497 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:14:16.497 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:14:16.499 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.500 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:16.508 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:14:16.514 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:16.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.516 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:16.516 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:14:16.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:16.518 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:14:16.521 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:14:16.523 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:16.524 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:16.543 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:14:16.568 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:14:16.569 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:14:16.581 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:14:16.587 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:14:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:14:16.605 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:14:16.605 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049642972000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:14:16.611 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:14:16.616 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:14:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:16.619 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.619 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:14:16.619 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:16.620 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:14:16.621 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.623 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:14:16.625 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:14:16.627 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:14:16.628 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:14:16.629 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:14:16.632 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:14:16.643 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:14:16.654 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:14:16.655 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:14:16.655 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:14:16.664 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:14:16.665 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:14:16.674 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:14:16.679 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:14:16.680 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:14:16.682 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.687 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:14:16.692 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:14:16.693 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:16.693 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:14:16.693 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.694 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:14:16.694 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:14:16.701 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:14:16.702 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:14:16.704 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:14:16.706 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:14:16.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:16.709 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:16.711 INFO 2981023 --- [host-check-host-status4,286622ff6b018d39,52e5885e1e8cd4dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:14:16.718 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:14:16.730 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:14:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:16.731 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:14:16.732 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.733 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:16.734 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:14:16.734 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.737 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:14:16.737 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:14:16.740 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:14:16.743 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:14:16.747 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:14:16.747 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:14:16.761 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:14:16.769 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:14:16.769 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:14:16.769 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:14:16.774 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:16.777 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.777 INFO 2981023 --- [host-check-host-status1,286622ff6b018d39,3b2df433e07c06c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:14:16.777 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:14:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:14:16.788 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:14:16.789 INFO 2981023 --- [host-check-host-status2,286622ff6b018d39,aa474712bf85b649] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:14:16.798 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:16.799 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:16.809 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:14:16.810 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:14:16.812 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:14:16.463751+08:00 2024-09-11 18:14:16.819 INFO 2981023 --- [host-check-host-status3,286622ff6b018d39,1419e71562b8a33d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:14:16.820 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:16.821 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.821 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:14:16.822 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:14:16.827 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:14:16.827 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:14:16.854 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:14:16.859 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:16.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:16.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:14:16.865 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:14:16.865 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:14:16.866 INFO 2981023 --- [ocp-schedules-20,977f8a4a226f0a29,683ceb357b79f7b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:14:16.983 INFO 2981023 --- [host-check-clockdiff2,53c40cd11edfc8ec,e0953ab571396f95] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049656 6 6 , error= 2024-09-11 18:14:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,281bb2cfbc6ab119,117f49bc2b59e53a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,281bb2cfbc6ab119,117f49bc2b59e53a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:17.417 INFO 2981023 --- [ocp-async-10,bc3b026f5b61b5e7,986724d850b28600] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:17.786 INFO 2981023 --- [ocp-async-7,67de82ed1c3ad734,3d987863798bbca0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:14:18.061 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:18.062 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:19.417 INFO 2981023 --- [ocp-async-3,be653855bc24fc31,077b22f871b07a4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,ed8ee35aec8bcf11,9cd29066163fbf4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,ed8ee35aec8bcf11,9cd29066163fbf4e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:21.062 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:21.062 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:21.214 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:14:21.215 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000007 2024-09-11 18:14:21.215 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000009 2024-09-11 18:14:21.215 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2024-09-11 18:14:21.215 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000010 2024-09-11 18:14:21.250 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:14:21.254 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:14:21.255 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:14:21.267 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:14:21.278 WARN 2981023 --- [ForkJoinPool.commonPool-worker-1,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:14:21.285 WARN 2981023 --- [ForkJoinPool.commonPool-worker-0,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:14:21.286 WARN 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:14:21.294 WARN 2981023 --- [ForkJoinPool.commonPool-worker-2,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:14:21.303 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:14:21.303 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:14:21.303 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:14:21.303 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000010 agent sql collect property 2024-09-11 18:14:21.303 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:14:21.303 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:14:21.304 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:14:21.304 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000009 agent sql collect property 2024-09-11 18:14:21.315 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:14:21.315 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2000009 to set sql collect property inactive 2024-09-11 18:14:21.315 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:14:21.315 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:14:21.315 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:14:21.316 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000007 agent sql collect property 2024-09-11 18:14:21.317 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:21.317 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:14:21.318 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:21.332 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:14:21.332 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:14:21.332 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:14:21.332 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2024-09-11 18:14:21.339 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:14:21.339 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (monagent.ob.err.code.prefix=errcode=-), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.cgroup.status=active), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.run.user=admin), (ob.logcleaner.enabled=true), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.pipeline.awr.collect.disabled=true), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.ob.rpc.port=2882), (monagent.monitordb.user=root@ocp_monitor), (monagent.host.ip=10.31.35.7), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"includeKeywordsRelation":"Or"," 2024-09-11 18:14:21.341 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994437), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.10), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkprdocp), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"include 2024-09-11 18:14:21.342 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:14:21.342 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:21.345 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:21.357 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000010, do nothing 2024-09-11 18:14:21.367 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (monagent.ob.err.code.prefix=errcode=-), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.cgroup.status=active), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.run.user=admin), (ob.logcleaner.enabled=true), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.pipeline.awr.collect.disabled=true), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.ob.rpc.port=2882), (monagent.monitordb.user=root@ocp_monitor), (monagent.host.ip=10.31.35.6), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"includeKeywordsRelation":"Or"," 2024-09-11 18:14:21.369 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000009, do nothing 2024-09-11 18:14:21.369 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1718689631), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.5), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=rtdob), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"in 2024-09-11 18:14:21.382 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2024-09-11 18:14:21.382 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000007, do nothing 2024-09-11 18:14:21.384 INFO 2981023 --- [ocp-schedules-24,a7e3a133b5c570cd,9297996fc33cda67] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-09-11 18:14:21.418 INFO 2981023 --- [ocp-async-7,a41832e2a3dbd406,a99670cd06a96469] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:21.606 INFO 2981023 --- [host-check-clockdiff1,53c40cd11edfc8ec,c07dfd8db00e1572] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049661 5 5 , error= 2024-09-11 18:14:21.999 INFO 2981023 --- [host-check-clockdiff4,53c40cd11edfc8ec,9b9a3512e28cd56b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049661 0 0 , error= 2024-09-11 18:14:22.620 INFO 2981023 --- [ocp-schedules-11,be4900e45c4e7b4a,8773128d8d1c5d89] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:14:23.155 INFO 2981023 --- [ocp-schedules-26,601c3e449806bb67,f4362d9d19c52c44] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:14:23.207 WARN 2981023 --- [scanner-13631554-1,601c3e449806bb67,9c88808f5db819d1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:14:23.222 INFO 2981023 --- [alarm-detect-16,601c3e449806bb67,52140de62da3b578] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645733, data is empty. 2024-09-11 18:14:23.226 INFO 2981023 --- [alarm-detect-16,601c3e449806bb67,52140de62da3b578] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645733, data is empty. 2024-09-11 18:14:23.231 INFO 2981023 --- [alarm-detect-16,601c3e449806bb67,52140de62da3b578] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645733, data is empty. 2024-09-11 18:14:23.254 INFO 2981023 --- [alarm-detect-1,601c3e449806bb67,0698214754d7ae4d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645733, data is empty. 2024-09-11 18:14:23.258 INFO 2981023 --- [alarm-detect-10,601c3e449806bb67,7962f0a50b09ace1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645733, data is empty. 2024-09-11 18:14:23.261 INFO 2981023 --- [alarm-detect-1,601c3e449806bb67,0698214754d7ae4d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645733, data is empty. 2024-09-11 18:14:23.265 INFO 2981023 --- [alarm-detect-10,601c3e449806bb67,7962f0a50b09ace1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645733, data is empty. 2024-09-11 18:14:23.266 INFO 2981023 --- [alarm-detect-1,601c3e449806bb67,0698214754d7ae4d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645733, data is empty. 2024-09-11 18:14:23.268 INFO 2981023 --- [alarm-detect-10,601c3e449806bb67,7962f0a50b09ace1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645733, data is empty. 2024-09-11 18:14:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,fc91169d01dfc32b,ccf350397df5a83e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:23.323 INFO 2981023 --- [prometheus-minute-schedule-1,fc91169d01dfc32b,ccf350397df5a83e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:23.385 INFO 2981023 --- [alarm-detect-16,601c3e449806bb67,8d03878f4b7c0518] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645733, data is empty. 2024-09-11 18:14:23.389 INFO 2981023 --- [alarm-detect-16,601c3e449806bb67,8d03878f4b7c0518] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645733, data is empty. 2024-09-11 18:14:23.393 INFO 2981023 --- [alarm-detect-16,601c3e449806bb67,8d03878f4b7c0518] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645733, data is empty. 2024-09-11 18:14:23.418 INFO 2981023 --- [ocp-async-4,10002d190c4ea178,c5f37051c2db445c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:24.062 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:24.064 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:24.436 INFO 2981023 --- [ocp-schedules-27,c0e596614907552f,7f8d83c2cdccb030] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:14:25.418 INFO 2981023 --- [ocp-async-5,27f4f1c439402e4e,3c213adb4398e944] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:26.083 INFO 2981023 --- [ocp-schedules-28,519e253416aa8d9b,fe1f7b501a9b12b0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:14:26.083 INFO 2981023 --- [ocp-schedules-28,519e253416aa8d9b,fe1f7b501a9b12b0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:14:26.083 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:14:26.083 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:14:26.084 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:14:26.087 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:14:26.088 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:14:26.088 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:26.091 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:14:26.091 INFO 2981023 --- [ocp-schedules-28,519e253416aa8d9b,fe1f7b501a9b12b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:26.091 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:14:26.092 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:26.092 INFO 2981023 --- [ocp-schedules-28,519e253416aa8d9b,fe1f7b501a9b12b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:26.118 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:14:26.120 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:14:26.121 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:14:26.122 INFO 2981023 --- [ocp-schedules-28,519e253416aa8d9b,fe1f7b501a9b12b0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:14:26.124 INFO 2981023 --- [ocp-schedules-28,519e253416aa8d9b,fe1f7b501a9b12b0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:14:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,fbcdaee2059e740d,045d5d587073de2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,fbcdaee2059e740d,045d5d587073de2c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:14:26.352824+08:00, lastProcessTime=2024-09-11T18:14:16.463751+08:00 2024-09-11 18:14:26.498 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:14:26.352824+08:00 2024-09-11 18:14:27.064 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:27.065 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:27.418 INFO 2981023 --- [ocp-async-2,1c345e8f9976adac,8ce89fd4ed43b8ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:27.787 INFO 2981023 --- [ocp-async-6,9eb870d85b4e84c1,1b134449096a0c5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:14:27.954 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 18:14:27.954 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 18:14:27.955 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.955 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:14:27.956 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.956 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:27.956 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.957 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 18:14:27.967 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.967 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:27.968 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.968 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 18:14:27.976 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.977 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:27.977 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.977 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 18:14:27.986 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.986 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:27.987 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.987 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 18:14:27.996 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.997 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:27.997 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:27.997 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 18:14:28.001 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.001 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.002 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.002 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 18:14:28.005 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.005 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.006 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.006 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 18:14:28.009 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.009 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.010 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.010 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 18:14:28.014 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.014 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.015 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.015 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 18:14:28.018 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.019 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.019 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.019 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 18:14:28.023 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.023 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.023 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.024 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 18:14:28.030 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.030 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.031 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.031 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 18:14:28.041 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.041 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.042 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.042 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 18:14:28.046 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.046 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.047 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.047 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 18:14:28.051 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.051 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.052 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.052 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 18:14:28.056 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.057 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.058 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.058 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 18:14:28.062 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.063 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.063 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.064 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 18:14:28.067 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.068 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.068 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.069 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 18:14:28.072 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.072 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.073 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.073 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 18:14:28.077 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.078 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.078 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.079 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 18:14:28.082 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.082 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.083 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.083 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 18:14:28.086 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.087 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.087 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.087 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 18:14:28.090 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.090 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.091 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.091 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 18:14:28.094 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.094 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.095 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.095 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 18:14:28.098 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.098 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.098 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.099 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 18:14:28.101 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.102 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.102 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.102 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 18:14:28.105 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.106 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.106 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.106 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 18:14:28.109 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.110 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.110 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.110 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 18:14:28.113 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.113 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.114 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.114 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 18:14:28.117 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.117 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.117 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.118 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 18:14:28.120 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.120 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.122 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.122 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 18:14:28.125 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.125 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.125 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.126 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 18:14:28.129 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.129 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.129 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.130 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 18:14:28.133 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.134 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.134 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.135 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 18:14:28.141 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.142 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.142 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.143 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 18:14:28.148 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.149 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.149 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.150 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 18:14:28.155 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.155 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.156 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.156 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 18:14:28.161 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.161 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.162 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.162 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 18:14:28.165 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.166 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.166 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.166 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 18:14:28.169 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.170 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.170 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.171 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 18:14:28.174 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.174 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.175 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.175 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 18:14:28.179 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.179 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.179 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.180 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 18:14:28.191 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.191 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.192 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.192 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 18:14:28.211 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.211 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.212 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.212 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 18:14:28.216 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.216 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.216 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.217 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 18:14:28.220 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.221 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.221 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.222 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 18:14:28.226 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.226 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.226 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.227 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 18:14:28.229 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.229 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.229 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.230 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 18:14:28.232 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.233 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.233 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.233 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 18:14:28.236 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 18:14:28.236 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 18:14:28.236 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.237 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:14:28.238 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.238 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.239 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.240 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 18:14:28.243 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.243 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:14:28.244 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:28.244 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 18:14:28.246 INFO 2981023 --- [ocp-schedules-2,b60b84710696beea,d921a0b1cbfbbbc5] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 18:14:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,fbe9ec773c42d58e,0231717a449b12dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,fbe9ec773c42d58e,0231717a449b12dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:29.418 INFO 2981023 --- [ocp-async-10,dc3a1d9543c6acea,4ae51fe7dc68170e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:30.065 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:30.066 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:31.418 INFO 2981023 --- [ocp-async-6,207cdba46e34ddd8,207837184d2eeff3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,b7e305a2f9b3a7a1,b9856aa38411a894] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,b7e305a2f9b3a7a1,b9856aa38411a894] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:32.623 INFO 2981023 --- [ocp-schedules-13,6633930955e7f3a2,c9e7951f7619d7a7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:14:33.066 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:33.067 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:33.418 INFO 2981023 --- [ocp-async-9,0c573adacd0c6303,0d60a0c5f2ff04fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:33.436 INFO 2981023 --- [ocp-schedules-3,e8e95b1a203d9740,b222411941649e05] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:14:33.484 WARN 2981023 --- [scanner-13631896-1,e8e95b1a203d9740,e62fb44fc81f08cb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:14:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,893dbe41bb69d7a5,807e3d85b6a5c133] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:35.315 INFO 2981023 --- [prometheus-minute-schedule-1,893dbe41bb69d7a5,807e3d85b6a5c133] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:35.385 INFO 2981023 --- [ocp-schedules-6,b6801ab5c6c52aec,6dd15949246a1608] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:14:35.398 INFO 2981023 --- [ocp-schedules-6,b6801ab5c6c52aec,6dd15949246a1608] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:14:35.398 INFO 2981023 --- [ocp-schedules-6,b6801ab5c6c52aec,6dd15949246a1608] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:14:35.418 INFO 2981023 --- [ocp-async-1,7c6ad346608c64e0,fac2172a27162634] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:36.067 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:36.067 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:14:36.337225+08:00, lastProcessTime=2024-09-11T18:14:26.352824+08:00 2024-09-11 18:14:36.478 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:14:36.337225+08:00 2024-09-11 18:14:37.314 INFO 2981023 --- [prometheus-cache-monitor1,2966eedd13458eef,faba69149bfec87b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8198,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:14:37.315 INFO 2981023 --- [prometheus-cache-monitor1,2966eedd13458eef,faba69149bfec87b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:14:37.419 INFO 2981023 --- [ocp-async-8,69ddee6adfdc025c,d985b7de7c79ead0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:37.787 INFO 2981023 --- [ocp-async-4,4ceb1c866ceafdef,e7b1dbf4a4f9ade0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:14:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,7912a0816538871e,b12f700d83c25584] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,7912a0816538871e,b12f700d83c25584] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:39.067 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:39.068 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:39.419 INFO 2981023 --- [ocp-async-2,1bc7f6aa74f901dc,89deef72b6227bb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:40.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:14:40.003 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:14:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,e0fc22ad941093bb,9980c85624232810] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,e0fc22ad941093bb,9980c85624232810] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:41.419 INFO 2981023 --- [ocp-async-4,ea76a57151e939bf,d8e93a033f2fd9f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:42.068 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:42.069 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:42.625 INFO 2981023 --- [ocp-schedules-4,0442aaac1599bd2e,e0772d7b3a3e7bf5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:14:43.419 INFO 2981023 --- [ocp-async-5,9167b7abed23af91,e258193e9dfd315b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:43.626 INFO 2981023 --- [ocp-schedules-20,22f73560d2153cf6,31e081239d517b5c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:14:43.671 WARN 2981023 --- [scanner-13632074-1,22f73560d2153cf6,07f7df99ce4df6a5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:14:43.689 INFO 2981023 --- [alarm-detect-9,22f73560d2153cf6,d1f2654879d1a6b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645733, data is empty. 2024-09-11 18:14:43.691 INFO 2981023 --- [alarm-detect-9,22f73560d2153cf6,d1f2654879d1a6b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645733, data is empty. 2024-09-11 18:14:43.694 INFO 2981023 --- [alarm-detect-9,22f73560d2153cf6,d1f2654879d1a6b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645733, data is empty. 2024-09-11 18:14:43.717 INFO 2981023 --- [alarm-detect-4,22f73560d2153cf6,0d5be11b48f41faa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645733, data is empty. 2024-09-11 18:14:43.720 INFO 2981023 --- [alarm-detect-4,22f73560d2153cf6,0d5be11b48f41faa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645733, data is empty. 2024-09-11 18:14:43.723 INFO 2981023 --- [alarm-detect-4,22f73560d2153cf6,0d5be11b48f41faa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645733, data is empty. 2024-09-11 18:14:43.724 INFO 2981023 --- [alarm-detect-2,22f73560d2153cf6,512f4fadeef0edca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645733, data is empty. 2024-09-11 18:14:43.729 INFO 2981023 --- [alarm-detect-2,22f73560d2153cf6,512f4fadeef0edca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645733, data is empty. 2024-09-11 18:14:43.734 INFO 2981023 --- [alarm-detect-2,22f73560d2153cf6,512f4fadeef0edca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645733, data is empty. 2024-09-11 18:14:43.833 INFO 2981023 --- [alarm-detect-11,22f73560d2153cf6,86de8e25965c41b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645733, data is empty. 2024-09-11 18:14:43.837 INFO 2981023 --- [alarm-detect-11,22f73560d2153cf6,86de8e25965c41b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645733, data is empty. 2024-09-11 18:14:43.841 INFO 2981023 --- [alarm-detect-11,22f73560d2153cf6,86de8e25965c41b3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645733, data is empty. 2024-09-11 18:14:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,ea33cb72ccc783a2,35236d7ea2b31fc4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,ea33cb72ccc783a2,35236d7ea2b31fc4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:45.069 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:45.069 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:45.101 INFO 2981023 --- [prometheus-second-schedule-1,368e505619dbdde2,a6c59a9a578078e7] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:14:45.419 INFO 2981023 --- [ocp-async-3,a2d8f6b47f9de09d,9e91d8ae19e510c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:45.727 INFO 2981023 --- [ocp-schedules-10,b623331f8c0bd4ff,dda14d22a84b2979] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:14:45.744 WARN 2981023 --- [ocp-schedules-10,b623331f8c0bd4ff,dda14d22a84b2979] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:14:46.019 INFO 2981023 --- [ocp-schedules-8,5236c8c9b92b7efc,d18da1bb9b49ef66] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:14:46.019 INFO 2981023 --- [ocp-schedules-19,9d916a59269dd7b8,45bf5bea9d9c7980] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:14:46.019 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:14:46.020 INFO 2981023 --- [host-check-clockdiff1,9d916a59269dd7b8,560c89e2da0fabc5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:14:46.020 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:14:46.020 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:14:46.021 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:14:46.022 INFO 2981023 --- [host-check-clockdiff2,9d916a59269dd7b8,64a8b9d581d2cb57] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:14:46.023 INFO 2981023 --- [host-check-clockdiff3,9d916a59269dd7b8,9a9bc1b2255344e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:14:46.024 INFO 2981023 --- [host-check-clockdiff4,9d916a59269dd7b8,36360f3a0c3ffdb8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:14:46.025 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:46.028 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:46.032 INFO 2981023 --- [ocp-schedules-2,c294b31261a53078,6bcaeebf8f91d552] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:14:46.033 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:14:46.033 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:14:46.034 INFO 2981023 --- [host-check-agent-status3,c294b31261a53078,fbb52e9f17111b28] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:14:46.034 INFO 2981023 --- [host-check-agent-status2,c294b31261a53078,5a9e10ebaf2faa0d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:14:46.034 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:14:46.034 INFO 2981023 --- [host-check-agent-status1,c294b31261a53078,7992f8c8b485b711] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:14:46.035 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:14:46.035 INFO 2981023 --- [host-check-agent-status4,c294b31261a53078,280ad83d14dd3d6e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:14:46.038 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:46.041 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:46.041 INFO 2981023 --- [host-check-clockdiff3,9d916a59269dd7b8,9a9bc1b2255344e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049686 0 0 , error= 2024-09-11 18:14:46.042 INFO 2981023 --- [host-check-agent-status3,c294b31261a53078,fbb52e9f17111b28] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:46.043 INFO 2981023 --- [host-check-clockdiff2,9d916a59269dd7b8,64a8b9d581d2cb57] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049686 5 5 , error= 2024-09-11 18:14:46.044 INFO 2981023 --- [host-check-agent-status3,c294b31261a53078,fbb52e9f17111b28] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:46.044 INFO 2981023 --- [host-check-agent-status2,c294b31261a53078,5a9e10ebaf2faa0d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:14:46.046 INFO 2981023 --- [host-check-agent-status2,c294b31261a53078,5a9e10ebaf2faa0d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:14:46.047 INFO 2981023 --- [host-check-agent-status1,c294b31261a53078,7992f8c8b485b711] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:46.048 INFO 2981023 --- [host-check-agent-status4,c294b31261a53078,280ad83d14dd3d6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:14:46.048 INFO 2981023 --- [host-check-agent-status1,c294b31261a53078,7992f8c8b485b711] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:46.049 INFO 2981023 --- [host-check-agent-status4,c294b31261a53078,280ad83d14dd3d6e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:14:46.061 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:14:46.063 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:14:46.069157226+08:00 2024-09-11 18:14:46.065 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:14:46.063+08:00 2024-09-11 18:14:46.068 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:14:46.063+08:00 and status AVAILABLE 2024-09-11 18:14:46.075 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:14:46.076 INFO 2981023 --- [host-check-agent-status4,c294b31261a53078,280ad83d14dd3d6e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:14:46.079 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:14:46.079 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:14:46.078533477+08:00 2024-09-11 18:14:46.080 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:14:46.079+08:00 2024-09-11 18:14:46.080 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:14:46.080640747+08:00 2024-09-11 18:14:46.081 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:14:46.081 INFO 2981023 --- [host-check-agent-status4,c294b31261a53078,280ad83d14dd3d6e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:14:46.081 INFO 2981023 --- [host-check-agent-status4,c294b31261a53078,280ad83d14dd3d6e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:14:46.081 INFO 2981023 --- [host-check-agent-status4,c294b31261a53078,280ad83d14dd3d6e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:14:46.081 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:14:46.082 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:14:46.080+08:00 2024-09-11 18:14:46.082 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:14:46.088089682+08:00 2024-09-11 18:14:46.083 INFO 2981023 --- [host-check-agent-status2,c294b31261a53078,5a9e10ebaf2faa0d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:14:46.084 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:14:46.080+08:00 and status AVAILABLE 2024-09-11 18:14:46.085 INFO 2981023 --- [host-check-agent-status2,c294b31261a53078,5a9e10ebaf2faa0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:14:46.085 INFO 2981023 --- [host-check-agent-status2,c294b31261a53078,5a9e10ebaf2faa0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:14:46.085 INFO 2981023 --- [host-check-agent-status2,c294b31261a53078,5a9e10ebaf2faa0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:14:46.086 INFO 2981023 --- [host-check-agent-status1,c294b31261a53078,7992f8c8b485b711] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:14:46.088 INFO 2981023 --- [host-check-agent-status1,c294b31261a53078,7992f8c8b485b711] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:14:46.088 INFO 2981023 --- [host-check-agent-status1,c294b31261a53078,7992f8c8b485b711] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:14:46.089 INFO 2981023 --- [host-check-agent-status1,c294b31261a53078,7992f8c8b485b711] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:14:46.094 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:14:46.079+08:00 and status AVAILABLE 2024-09-11 18:14:46.101 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:14:46.103 INFO 2981023 --- [host-check-agent-status3,c294b31261a53078,fbb52e9f17111b28] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:14:46.105 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:14:46.083+08:00 2024-09-11 18:14:46.108 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:14:46.083+08:00 and status AVAILABLE 2024-09-11 18:14:46.109 INFO 2981023 --- [host-check-agent-status3,c294b31261a53078,fbb52e9f17111b28] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:14:46.109 INFO 2981023 --- [host-check-agent-status3,c294b31261a53078,fbb52e9f17111b28] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:14:46.109 INFO 2981023 --- [host-check-agent-status3,c294b31261a53078,fbb52e9f17111b28] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:14:46.115 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:14:46.120 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:46.121 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:46.122 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:14:46.123 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:14:46.130 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:14:46.137 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:14:46.140 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:14:46.151 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:46.152 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:14:46.154 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:14:46.154 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:46.158 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:14:46.159 INFO 2981023 --- [ocp-schedules-27,c580da329012243c,741ac2dd00088814] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:14:46.162 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:14:46.169 INFO 2981023 --- [ocp-schedules-27,c580da329012243c,741ac2dd00088814] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:14:46.176 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:14:46.182 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:14:46.190 INFO 2981023 --- [ocp-schedules-5,3081aa6ceb55514b,4da7fe4a330bd7ee] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:14:46.190 INFO 2981023 --- [ocp-schedules-14,734be1c5f211d424,0efa05a366a7db30] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:14:46.190 INFO 2981023 --- [ocp-schedules-24,38ee1c9c22a938bd,8d003b16e8f3d80d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:14:46.190 INFO 2981023 --- [ocp-schedules-22,c2033c5395d7709a,18387d4157ad8bac] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:14:46.192 INFO 2981023 --- [ocp-schedules-14,734be1c5f211d424,0efa05a366a7db30] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:14:46.201 INFO 2981023 --- [ocp-schedules-5,3081aa6ceb55514b,4da7fe4a330bd7ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:14:46.209 INFO 2981023 --- [ocp-schedules-24,38ee1c9c22a938bd,8d003b16e8f3d80d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:14:46.217 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:14:46.217 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:14:46.242 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:14:46.247 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049642972000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:14:46.248 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:14:46.248 INFO 2981023 --- [ocp-schedules-14,734be1c5f211d424,0efa05a366a7db30] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:14:46.249 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:14:46.253 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:14:46.257 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.258 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:14:46.267 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:14:46.268 INFO 2981023 --- [ocp-schedules-30,3f36eb37de02f46a,32c7c3413450919c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:14:46.268 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.268 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:14:46.273 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:14:46.276 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.277 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:14:46.278 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:14:46.280 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:14:46.280 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.282 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:14:46.283 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:14:46.285 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:14:46.286 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:14:46.289 INFO 2981023 --- [ocp-schedules-5,3081aa6ceb55514b,4da7fe4a330bd7ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:14:46.293 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.294 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:46.301 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.302 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:46.313 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:14:46.313 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:14:46.320 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.323 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:46.330 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:14:46.347 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:14:46.370 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:14:46.366554+08:00, lastProcessTime=2024-09-11T18:14:36.337225+08:00 2024-09-11 18:14:46.371 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:14:46.378 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:14:46.378 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:14:46.387 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.388 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:14:46.393 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.394 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.394 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.394 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:14:46.395 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:14:46.395 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:14:46.396 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:14:46.398 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.403 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:14:46.413 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:14:46.421 INFO 2981023 --- [host-check-host-status4,5236c8c9b92b7efc,82a4ddb0eeac17a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:14:46.421 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:14:46.421 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049642972000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:14:46.423 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:14:46.425 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.428 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:14:46.429 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.429 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:46.430 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:14:46.430 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.431 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:14:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:14:46.433 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:14:46.436 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:14:46.439 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:14:46.452 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:14:46.452 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:14:46.457 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:14:46.466 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:14:46.469 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:14:46.477 INFO 2981023 --- [host-check-host-status2,5236c8c9b92b7efc,9ea927e0bbf57bbb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:14:46.485 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.486 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.489 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:14:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.490 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:14:46.491 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:14:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:14:46.498 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:14:46.498 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.499 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.499 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:14:46.500 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:14:46.500 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.501 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:14:46.501 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:14:46.501 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.504 INFO 2981023 --- [host-check-host-status1,5236c8c9b92b7efc,06211d12dc84cb9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:14:46.507 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:14:46.508 INFO 2981023 --- [host-check-host-status3,5236c8c9b92b7efc,79906d6ce9e1c20b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:14:46.508 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.508 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.509 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:14:46.509 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:14:46.510 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:14:46.527 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:14:46.530 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:14:46.540 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.541 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:14:46.541 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:14:46.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.544 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:14:46.545 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:14:46.553 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.554 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:14:46.554 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:14:46.554 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.554 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:14:46.555 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:14:46.555 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.571 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:14:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:14:46.582 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.582 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.583 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:14:46.583 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:14:46.588 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:14:46.591 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:14:46.591 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:14:46.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:14:46.593 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:14:46.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.600 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:14:46.609 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.609 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:14:46.610 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.610 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:14:46.611 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:14:46.611 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.611 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:14:46.611 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:14:46.641 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:14:46.649 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:46.649 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:46.650 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:14:46.660 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:14:46.660 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:14:46.660 INFO 2981023 --- [ocp-schedules-24,38ee1c9c22a938bd,8d003b16e8f3d80d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:14:46.664 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:14:46.666 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:14:46.366554+08:00 2024-09-11 18:14:46.679 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:14:46.685 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:14:46.688 INFO 2981023 --- [ocp-schedules-7,89f654f0c9b52acd,f8518fdeda6af828] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:14:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,a8d06a2404abae05,5abea258e3563650] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:47.319 INFO 2981023 --- [prometheus-minute-schedule-1,a8d06a2404abae05,5abea258e3563650] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:47.419 INFO 2981023 --- [ocp-async-4,8f2a611a7e6c2276,33a6f23ccb5255db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:47.787 INFO 2981023 --- [ocp-async-9,058e5fcd78a7f84f,ec17c852f2285ffb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:14:48.069 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:48.070 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:49.182 INFO 2981023 --- [host-check-clockdiff1,9d916a59269dd7b8,560c89e2da0fabc5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049689 5 5 , error= 2024-09-11 18:14:49.419 INFO 2981023 --- [ocp-async-6,bcdc14e5a8914cf4,f7f9f08ad9fb02be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:14:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:14:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,19ebcea4b2b19d99,a76a40bd50bd0938] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,19ebcea4b2b19d99,a76a40bd50bd0938] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:50.332 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:14:50.337 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.338 INFO 2981023 --- [ocp-schedules-9,a9473ac7bd1c3cba,053bb44e77393415] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:14:50.341 INFO 2981023 --- [ocp-schedules-23,9e5021e502c3a20e,22f4aa37c90ccdb2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:14:51.070 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:51.072 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:51.419 INFO 2981023 --- [ocp-async-2,77a2d694b407c2b0,3dfca102caee6098] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:51.617 INFO 2981023 --- [ocp-schedules-8,1fff588e3b80cfe1,bf28cbd19819143e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:14:51.618 INFO 2981023 --- [ocp-schedules-8,1fff588e3b80cfe1,bf28cbd19819143e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:14:51.618 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:14:51.618 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:14:51.618 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:14:51.619 INFO 2981023 --- [ocp-schedules-8,1fff588e3b80cfe1,bf28cbd19819143e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:14:51.974 INFO 2981023 --- [host-check-clockdiff4,9d916a59269dd7b8,36360f3a0c3ffdb8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049691 0 0 , error= 2024-09-11 18:14:52.628 INFO 2981023 --- [ocp-schedules-23,61f66c568b1b3aa4,7a9e75c7f06a254f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:14:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,0f793febb25e3d2a,db5ae7e7e982aa83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:53.313 INFO 2981023 --- [ocp-schedules-6,d5e309bb51b8e60c,b22cfb3f8cbd692c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:14:53.313 INFO 2981023 --- [ocp-schedules-6,d5e309bb51b8e60c,b22cfb3f8cbd692c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:14:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,0f793febb25e3d2a,db5ae7e7e982aa83] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:53.419 INFO 2981023 --- [ocp-async-10,ae20be5cc8795147,1ed9d2efb941bf29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:53.878 INFO 2981023 --- [ocp-schedules-11,78256de53f568c6c,c1f1ff0ada7aac48] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:14:53.941 WARN 2981023 --- [scanner-13632431-1,78256de53f568c6c,9b52fcc5cc0a29c8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:14:54.072 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:54.072 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:54.333 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3f8bfc8dd97e910d,fc0a8533969bca27] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=3f8bfc8dd97e910d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:14:54.333 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,6a258c9ac4ecbd00,b936cdf88677cf6f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=6a258c9ac4ecbd00, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:14:54.334 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,6a258c9ac4ecbd00,b936cdf88677cf6f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=6a258c9ac4ecbd00, duration=4 ms] 2024-09-11 18:14:54.337 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,3f8bfc8dd97e910d,fc0a8533969bca27] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=3f8bfc8dd97e910d, duration=8 ms] 2024-09-11 18:14:54.343 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,cf69ddbbddb02b74,8e23ebda0ecb399d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=cf69ddbbddb02b74, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:14:54.344 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,cf69ddbbddb02b74,8e23ebda0ecb399d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=cf69ddbbddb02b74, duration=2 ms] 2024-09-11 18:14:54.438 INFO 2981023 --- [ocp-schedules-29,c0d0cc21055c9049,bc6e9aece7b93c4d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:14:54.793 INFO 2981023 --- [ocp-schedules-3,4959738a299d55a5,ba520e182280afc3] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:14:54.811 WARN 2981023 --- [ocp-schedules-3,4959738a299d55a5,ba520e182280afc3] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:14:55.400 INFO 2981023 --- [ocp-schedules-15,d31003cb615eee08,5e96a8bd7fdacbec] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:14:55.409 INFO 2981023 --- [ocp-schedules-15,d31003cb615eee08,5e96a8bd7fdacbec] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:14:55.409 INFO 2981023 --- [ocp-schedules-15,d31003cb615eee08,5e96a8bd7fdacbec] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:14:55.419 INFO 2981023 --- [ocp-async-5,4b66da191c7ab9a8,e55d19a1bf0174ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:55.872 INFO 2981023 --- [ocp-schedules-23,d9affc7c654274d6,fc606bb5c20929b3] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:14:55.872 INFO 2981023 --- [ocp-schedules-23,d9affc7c654274d6,fc606bb5c20929b3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:14:55.872 INFO 2981023 --- [ocp-schedules-23,d9affc7c654274d6,fc606bb5c20929b3] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:14:55.872 INFO 2981023 --- [ocp-schedules-23,d9affc7c654274d6,fc606bb5c20929b3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:14:55.873 INFO 2981023 --- [ocp-schedules-23,d9affc7c654274d6,fc606bb5c20929b3] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:14:55.873 INFO 2981023 --- [ocp-schedules-23,d9affc7c654274d6,fc606bb5c20929b3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:14:56.058 INFO 2981023 --- [ocp-schedules-20,c74815ef28193ea6,4b75eb63ff7073ff] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:14:56.065 INFO 2981023 --- [ocp-schedules-20,c74815ef28193ea6,4b75eb63ff7073ff] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:14:56.065 INFO 2981023 --- [ocp-schedules-20,c74815ef28193ea6,4b75eb63ff7073ff] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:14:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,61500c00eac869dd,48c38838bc6225cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,61500c00eac869dd,48c38838bc6225cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:14:56.351102+08:00, lastProcessTime=2024-09-11T18:14:46.366554+08:00 2024-09-11 18:14:56.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:14:56.351102+08:00 2024-09-11 18:14:57.072 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:14:57.073 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:14:57.420 INFO 2981023 --- [ocp-async-7,a55a6d21e3aa15aa,305e8f08f93479fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:57.787 INFO 2981023 --- [ocp-async-3,c45fc6a878f4db12,9fdb5ec7d01f5655] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:14:58.027 INFO 2981023 --- [ocp-schedules-27,b2aa7cb1b5bf075a,ad3cdd4fdd71f7f8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:14:58.027 INFO 2981023 --- [ocp-schedules-27,b2aa7cb1b5bf075a,ad3cdd4fdd71f7f8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:14:58.048 INFO 2981023 --- [Collect all tenants major compactions1,b2aa7cb1b5bf075a,4f7c8f8bc6e07b2b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:14:58.049 INFO 2981023 --- [Collect all tenants major compactions3,b2aa7cb1b5bf075a,8bc38ce215c85fa6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:14:58.050 INFO 2981023 --- [Collect all tenants major compactions6,b2aa7cb1b5bf075a,a9cbe6343f4f8dbc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:14:58.050 INFO 2981023 --- [Collect all tenants major compactions5,b2aa7cb1b5bf075a,1ee4ecf650b2750d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:14:58.051 INFO 2981023 --- [Collect all tenants major compactions4,b2aa7cb1b5bf075a,85ae485348a19f6f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:14:58.052 INFO 2981023 --- [Collect all tenants major compactions2,b2aa7cb1b5bf075a,a17b63dead9726a7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:14:58.066 INFO 2981023 --- [Collect all tenants major compactions1,b2aa7cb1b5bf075a,4f7c8f8bc6e07b2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:58.067 INFO 2981023 --- [Collect all tenants major compactions6,b2aa7cb1b5bf075a,a9cbe6343f4f8dbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:58.067 INFO 2981023 --- [Collect all tenants major compactions1,b2aa7cb1b5bf075a,4f7c8f8bc6e07b2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:14:58.069 INFO 2981023 --- [Collect all tenants major compactions6,b2aa7cb1b5bf075a,a9cbe6343f4f8dbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:14:58.069 INFO 2981023 --- [Collect all tenants major compactions5,b2aa7cb1b5bf075a,1ee4ecf650b2750d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:58.069 INFO 2981023 --- [Collect all tenants major compactions4,b2aa7cb1b5bf075a,85ae485348a19f6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:58.070 INFO 2981023 --- [Collect all tenants major compactions3,b2aa7cb1b5bf075a,8bc38ce215c85fa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:58.071 INFO 2981023 --- [Collect all tenants major compactions5,b2aa7cb1b5bf075a,1ee4ecf650b2750d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:14:58.071 INFO 2981023 --- [Collect all tenants major compactions2,b2aa7cb1b5bf075a,a17b63dead9726a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:58.071 INFO 2981023 --- [Collect all tenants major compactions4,b2aa7cb1b5bf075a,85ae485348a19f6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:14:58.071 INFO 2981023 --- [Collect all tenants major compactions1,b2aa7cb1b5bf075a,4f7c8f8bc6e07b2b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:14:58.071 INFO 2981023 --- [Collect all tenants major compactions1,b2aa7cb1b5bf075a,4f7c8f8bc6e07b2b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:14:58.072 INFO 2981023 --- [Collect all tenants major compactions2,b2aa7cb1b5bf075a,a17b63dead9726a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:14:58.072 INFO 2981023 --- [Collect all tenants major compactions3,b2aa7cb1b5bf075a,8bc38ce215c85fa6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:14:58.073 INFO 2981023 --- [Collect all tenants major compactions4,b2aa7cb1b5bf075a,85ae485348a19f6f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:14:58.073 INFO 2981023 --- [Collect all tenants major compactions5,b2aa7cb1b5bf075a,1ee4ecf650b2750d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:14:58.073 INFO 2981023 --- [Collect all tenants major compactions4,b2aa7cb1b5bf075a,85ae485348a19f6f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:14:58.073 INFO 2981023 --- [Collect all tenants major compactions5,b2aa7cb1b5bf075a,1ee4ecf650b2750d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:14:58.074 INFO 2981023 --- [Collect all tenants major compactions6,b2aa7cb1b5bf075a,a9cbe6343f4f8dbc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:14:58.074 INFO 2981023 --- [Collect all tenants major compactions6,b2aa7cb1b5bf075a,a9cbe6343f4f8dbc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:14:58.074 INFO 2981023 --- [Collect all tenants major compactions2,b2aa7cb1b5bf075a,a17b63dead9726a7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:14:58.074 INFO 2981023 --- [Collect all tenants major compactions2,b2aa7cb1b5bf075a,a17b63dead9726a7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:14:58.074 INFO 2981023 --- [Collect all tenants major compactions3,b2aa7cb1b5bf075a,8bc38ce215c85fa6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:14:58.074 INFO 2981023 --- [Collect all tenants major compactions3,b2aa7cb1b5bf075a,8bc38ce215c85fa6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:14:58.075 INFO 2981023 --- [ocp-schedules-27,b2aa7cb1b5bf075a,ad3cdd4fdd71f7f8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:14:58.833 INFO 2981023 --- [ocp-schedules-9,c72d4c0b9eca05f0,fd066e6ad9624053] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:14:58.833+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:14:58.834 INFO 2981023 --- [ocp-schedules-9,c72d4c0b9eca05f0,fd066e6ad9624053] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:14:58.873 INFO 2981023 --- [ocp-schedules-9,c72d4c0b9eca05f0,fd066e6ad9624053] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 38 to SqlAggregateData 2024-09-11 18:14:58.875 INFO 2981023 --- [ocp-schedules-9,c72d4c0b9eca05f0,fd066e6ad9624053] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:14:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,8e7bfcc0de2c53dd,b0af3b6ebf41786a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049700, interval=60 2024-09-11 18:14:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,8e7bfcc0de2c53dd,b0af3b6ebf41786a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:14:59.420 INFO 2981023 --- [ocp-async-10,1fcd076b1db46a92,49f935e1a60016ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:14:59.753 INFO 2981023 --- [ocp-schedules-22,ba72eeab106151fb,2b36a2def92bee74] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:14:59.754 INFO 2981023 --- [ocp-schedules-22,ba72eeab106151fb,2b36a2def92bee74] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:14:59.783 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:14:59.783 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:14:59.784 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.784 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.787 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:14:59.787 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:14:59.788 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:14:59.788 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.791 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:14:59.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.791 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.792 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.794 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.795 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.804 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.804 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:14:59.807 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.807 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.812 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.813 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:14:59.816 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.816 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.816 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:14:59.817 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5351D74B-0-0] 2024-09-11 18:14:59.817 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:14:59.817 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.817 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:14:59.819 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.819 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:14:59.819 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.820 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.826 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.827 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:14:59.830 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.830 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:14:59.831 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.832 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:14:59.832 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.834 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:14:59.835 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.836 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:14:59.836 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:14:59.840 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:14:59.843 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.844 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:14:59.844 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:14:59.844 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.845 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5351D755-0-0] 2024-09-11 18:14:59.845 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:14:59.845 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:14:59.846 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.846 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:14:59.847 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.847 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:14:59.848 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.849 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:14:59.849 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.850 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:14:59.850 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.851 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:14:59.852 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.852 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:14:59.853 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:14:59.853 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.854 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.854 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:14:59.854 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:14:59.854 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.855 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.855 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:14:59.856 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:14:59.856 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.856 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:14:59.856 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:14:59.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.858 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:14:59.858 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.858 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:14:59.859 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:14:59.859 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:14:59.862 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:59.862 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:14:59.864 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.864 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.871 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:59.872 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.873 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:14:59.877 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:14:59.879 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.880 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:14:59.880 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:14:59.881 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:14:59.883 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:14:59.883 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:14:59.883 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:14:59.883 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:14:59.883 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.883 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:14:59.888 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:14:59.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:14:59.890 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:14:59.890 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.890 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.891 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:14:59.892 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:14:59.892 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:14:59.896 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:14:59.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:14:59.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:14:59.897 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:14:59.904 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:14:59.905 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:14:59.905 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:14:59.905 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:14:59.905 INFO 2981023 --- [ocp-schedules-22,ba72eeab106151fb,2b36a2def92bee74] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:15:00.073 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:00.073 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:00.733 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:10:01.0, nextRun=2024-09-11 18:15:00.0, currentDate=Wed Sep 11 18:15:00 HKT 2024 2024-09-11 18:15:00.733 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:10:01.0, nextRunTime=2024-09-11 18:15:00.0} 2024-09-11 18:15:00.737 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 18:15:00.737 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:15:00.737 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:15:00.737+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:15:00.749 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:10:01.0, nextRun=2024-09-11 18:15:00.0, currentDate=Wed Sep 11 18:15:00 HKT 2024 2024-09-11 18:15:00.749 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:10:01.0, nextRunTime=2024-09-11 18:15:00.0} 2024-09-11 18:15:00.762 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 18:15:00.769 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 18:15:00.769 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 18:15:00.769 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:15:00.769 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,1fbca268fe9e1705] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T18:15:00.769+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:15:01.420 INFO 2981023 --- [ocp-async-3,11eb34cca01beb54,4fd052f3c7acc375] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:01.707 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,32a380b1cd565652] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048245, operation:EXECUTE, state: RUNNING 2024-09-11 18:15:01.720 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,32a380b1cd565652] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048246, operation:EXECUTE, state: RUNNING 2024-09-11 18:15:01.733 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048245, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048215, task_operation=execute, latest_execution_start_time=2024-09-11T18:15:01.699+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048245}, listMap={}}, executor=10.31.35.10 2024-09-11 18:15:01.738 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048246, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048216, task_operation=execute, latest_execution_start_time=2024-09-11T18:15:01.718+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048246}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 18:15:01.740 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 18:15:01.750 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:15:01.759 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 18:15:01.762 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 18:15:01.773 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 18:15:01.774 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 18:15:01.774 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 18:15:01.787 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:15:01.790 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 18:15:01.796 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 18:15:01.806 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,a9e2073123df36ce] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048246, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:15:01.851 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:15:01.877 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:15:01.910 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:15:01.916 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 18:15:01.917 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 18:15:01.923 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:15:02.011 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:15:02.040 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048216, status:SUCCESS 2024-09-11 18:15:02.043 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048216, status:SUCCESS 2024-09-11 18:15:02.059 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:15:02.076 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:15:02.093 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:15:02.128 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:15:02.142 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:15:02.167 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:15:02.180 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 18:15:02.180 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3395.459579986], b:[-5.807309276841022E12] 2024-09-11 18:15:02.185 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:15:02.268 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:15:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,e8ca18f689ebdf9d,4341f8dd2086ff9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,e8ca18f689ebdf9d,4341f8dd2086ff9d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:02.325 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:15:02.338 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:15:02.345 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:15:02.358 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:15:02.364 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:15:02.370 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:15:02.372 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 18:15:02.372 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:15:02.390 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:15:02.399 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:15:02.413 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,79dcd05ea0055a5a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048245, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:15:02.630 INFO 2981023 --- [ocp-schedules-24,e56838cbe15469bb,15341d8fe191b407] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:15:03.054 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048215, status:SUCCESS 2024-09-11 18:15:03.055 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048215, status:SUCCESS 2024-09-11 18:15:03.073 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:03.074 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:03.420 INFO 2981023 --- [ocp-async-9,ee502bf69cd8a093,5fed346c91e1ac13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:04.069 INFO 2981023 --- [ocp-schedules-7,d58822176dd10c63,ad120ebcda71655c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:15:04.122 WARN 2981023 --- [scanner-13632819-1,d58822176dd10c63,8e7a664f16c33d48] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:15:04.128 INFO 2981023 --- [alarm-detect-17,d58822176dd10c63,fb37baa2c85edfb2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645733, data is empty. 2024-09-11 18:15:04.134 INFO 2981023 --- [alarm-detect-17,d58822176dd10c63,fb37baa2c85edfb2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645771, data is empty. 2024-09-11 18:15:04.138 INFO 2981023 --- [alarm-detect-17,d58822176dd10c63,fb37baa2c85edfb2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645771, data is empty. 2024-09-11 18:15:04.166 INFO 2981023 --- [alarm-detect-9,d58822176dd10c63,7aee2e669c2cd673] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645733, data is empty. 2024-09-11 18:15:04.168 INFO 2981023 --- [alarm-detect-1,d58822176dd10c63,03fcb5da45ed9995] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645733, data is empty. 2024-09-11 18:15:04.169 INFO 2981023 --- [alarm-detect-9,d58822176dd10c63,7aee2e669c2cd673] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645771, data is empty. 2024-09-11 18:15:04.173 INFO 2981023 --- [alarm-detect-9,d58822176dd10c63,7aee2e669c2cd673] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645771, data is empty. 2024-09-11 18:15:04.173 INFO 2981023 --- [alarm-detect-1,d58822176dd10c63,03fcb5da45ed9995] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645771, data is empty. 2024-09-11 18:15:04.177 INFO 2981023 --- [alarm-detect-1,d58822176dd10c63,03fcb5da45ed9995] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645771, data is empty. 2024-09-11 18:15:04.280 INFO 2981023 --- [alarm-detect-12,d58822176dd10c63,4b968eea1f5dd160] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645733, data is empty. 2024-09-11 18:15:04.287 INFO 2981023 --- [alarm-detect-12,d58822176dd10c63,4b968eea1f5dd160] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645771, data is empty. 2024-09-11 18:15:04.289 INFO 2981023 --- [alarm-detect-12,d58822176dd10c63,4b968eea1f5dd160] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645771, data is empty. 2024-09-11 18:15:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,fc94a076852a24e0,c7d1f4ab37ca7ff9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,fc94a076852a24e0,c7d1f4ab37ca7ff9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:05.420 INFO 2981023 --- [ocp-async-1,42eed75ec6d9190a,00ff3273a9a358c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:05.680 INFO 2981023 --- [ocp-schedules-4,eff1bd414c593a2b,b8d12d1be315e8ba] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:15:06.074 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:06.076 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:15:06.352957+08:00, lastProcessTime=2024-09-11T18:14:56.351102+08:00 2024-09-11 18:15:06.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:15:06.352957+08:00 2024-09-11 18:15:07.420 INFO 2981023 --- [ocp-async-10,c61edfb1e0ce9405,3dffb1ed539c0ab2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:07.787 INFO 2981023 --- [ocp-async-4,97b3cf496f86ad6a,4eeb1866d352c838] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:15:07.965 INFO 2981023 --- [ocp-schedules-2,3357384a220ad8af,5c38864fa0d98d79] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:15:08.076 INFO 2981023 --- [ocp-schedules-27,09f543b13ef5130b,3b989a1d78995b4c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:15:08.076+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:15:08.077 INFO 2981023 --- [ocp-schedules-27,09f543b13ef5130b,3b989a1d78995b4c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:15:08.078 INFO 2981023 --- [ocp-schedules-27,09f543b13ef5130b,3b989a1d78995b4c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:15:08.079 INFO 2981023 --- [ocp-schedules-27,09f543b13ef5130b,3b989a1d78995b4c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:15:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,c5305865b6f9827b,554c67e46a8cc3c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,c5305865b6f9827b,554c67e46a8cc3c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:09.076 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:09.076 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:09.420 INFO 2981023 --- [ocp-async-7,18896f0d14e139e6,a0996c955a084fae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:10.003 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:15:10.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:15:10.189 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:11:09.814, end=2024-09-11T18:13:10.186} 2024-09-11 18:15:10.192 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obC 2024-09-11 18:15:10.192 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:15:10.193 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:09.814, end=2024-09-11T18:13:10.186},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:15:10.193 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:15:10.193 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:15:10.194 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:15:10.200 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:09.814, end=2024-09-11T18:13:10.186},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:15:10.200 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:15:10.200 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:15:10.209 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:15:10.226 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310226000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.226Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.228 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:15:10.228 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.233 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.233 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.234 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.234 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.239 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.239 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.240 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:15:10.240 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.226, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.241 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:15:10.243 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:09.814, end=2024-09-11T18:13:10.186},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:15:10.243 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:15:10.243 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:15:10.283 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310283000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.283Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.289 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310289000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.289Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.293 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310293000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.293Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.297 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310297000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.297Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.298 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310298000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.298Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.299 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310299000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.299Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.299 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310299000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.299Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.299 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310299000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.299Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.305 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.305 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.305 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:15:10.305 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.312 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.312 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.314 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.314 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.317 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.317 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.319 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:15:10.319 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.322 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 67 2024-09-11 18:15:10.322 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.322 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.322 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.324 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.324 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.327 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.327 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.328 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.328 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.330 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.330 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.330 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:15:10.330 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.331 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.331 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.331 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.331 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.336 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.336 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.336 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.336 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.337 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.337 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.338 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.338 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.339 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.339 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.341 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.341 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.341 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.343 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.343 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.345 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.345 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.346 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.347 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.347 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.347 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.348 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.283, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.350 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.350 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.350 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.350 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.352 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.353 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.353 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.357 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.357 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.358 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.359 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.360 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.360 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.360 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.363 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.363 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.364 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.366 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.299, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.366 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.293, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.368 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.368 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.372 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.372 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.299, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.372 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.289, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.374 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.375 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.297, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.378 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310378000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.378Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.379 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.298, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.382 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.299, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.397 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,72db552d82825a64,6cf47bfb091af473] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=72db552d82825a64, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:15:10.402 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f8f1e586f4795bb9,8893492c5f6ec80f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=f8f1e586f4795bb9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:15:10.405 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.405 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.407 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,7d3a5b4ac98ecfca,3bb91e556d680b5c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=7d3a5b4ac98ecfca, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:15:10.410 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,7d3a5b4ac98ecfca,3bb91e556d680b5c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=7d3a5b4ac98ecfca, duration=8 ms] 2024-09-11 18:15:10.410 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f8f1e586f4795bb9,8893492c5f6ec80f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=f8f1e586f4795bb9, duration=20 ms] 2024-09-11 18:15:10.411 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,72db552d82825a64,6cf47bfb091af473] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=72db552d82825a64, duration=21 ms] 2024-09-11 18:15:10.413 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310413000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.413Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.421 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310421000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=215632585E2CA5F0C915A1E0D150020F, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.421Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.421 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.421 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.422 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310421000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=2E0351B66C4102B9EDA81C0E5D431120, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.421Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.424 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310423000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.423Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.426 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:15:10.426 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.428 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310428000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.428Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.428 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.428 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.430 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310430000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.430Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.434 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 77 2024-09-11 18:15:10.434 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.434 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 95 2024-09-11 18:15:10.434 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.437 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310437000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.437Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.442 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.442 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.443 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.443 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.443 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.443 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.445 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 95 2024-09-11 18:15:10.445 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.448 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.448 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=215632585E2CA5F0C915A1E0D150020F, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.448 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:15:10.448 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.451 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.454 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.454 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.455 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.455 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.455 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.455 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.457 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.457 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.459 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.459 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.459 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.459 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=2E0351B66C4102B9EDA81C0E5D431120, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.459 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.459 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.461 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.378, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.461 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.461 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.462 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.462 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.464 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.464 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.468 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.468 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.468 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.468 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.469 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.469 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=215632585E2CA5F0C915A1E0D150020F, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.471 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.472 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.472 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.472 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.473 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.473 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.473 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.473 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=2E0351B66C4102B9EDA81C0E5D431120, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.474 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.476 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.476 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.476 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.477 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.477 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.479 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.413, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.481 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.482 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.421, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=215632585E2CA5F0C915A1E0D150020F, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.483 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.423, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.483 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.421, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=2E0351B66C4102B9EDA81C0E5D431120, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.487 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.487 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.491 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.428, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=7B1FDC6912D46A43A8DF7CB6E11EA072, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.492 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.495 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.495 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.496 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310496000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.496Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.497 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.503 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.43, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.507 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.507 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.507 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.437, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.513 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310513000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6861114A054F03969A56C5F7382C049A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.513Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.513 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310513000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.513Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.513 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310513000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:15:10.513Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:15:10.516 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.516 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.520 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.520 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.520 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.520 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.521 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.521 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.528 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.528 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6861114A054F03969A56C5F7382C049A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.528 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.528 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.529 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.529 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.529 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:15:10.529 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.531 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.531 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.531 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.531 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.531 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.535 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.496, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.536 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.536 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.537 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.537 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.538 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.538 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.539 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.539 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6861114A054F03969A56C5F7382C049A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.540 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.540 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.543 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.543 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.545 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:15:10.546 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.513, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.546 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.513, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=6861114A054F03969A56C5F7382C049A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.549 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.513, end=2024-09-11T18:13:10.186}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.549 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:15:10.550 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:09.814, end=2024-09-11T18:13:10.186},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:15:10.550 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:15:10.550 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:15:10.551 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:15:10.551 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:09.814, end=2024-09-11T18:13:10.186},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:15:10.551 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:15:10.551 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:15:10.552 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:15:10.554 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:09.814, end=2024-09-11T18:13:10.186},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:15:10.554 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:15:10.554 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:15:10.565 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:15:10.583 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963310583000, endTimeUs=1726049590186000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=BD5DCCE7A516959882352D2E7F3E65DA, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T10:15:10.583Z, endTime=2024-09-11T10:13:10.186Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000029, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000010, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 18:15:10.585 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:15:10.585 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:15:10.591 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.591 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=BD5DCCE7A516959882352D2E7F3E65DA, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.592 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:15:10.592 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:15:10.599 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:15:10.599 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049590186000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=BD5DCCE7A516959882352D2E7F3E65DA, sqlIdList=null, dynamicSql=false) 2024-09-11 18:15:10.600 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:15:10.600 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:15:10.583, end=2024-09-11T18:13:10.186}, clusterId=1000006, tenantId=1000006, sqlId=BD5DCCE7A516959882352D2E7F3E65DA, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:15:10.600 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:15:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:15:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:15:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,2869a3f380d64118,4ac0de1750cd1d0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,2869a3f380d64118,4ac0de1750cd1d0d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:11.420 INFO 2981023 --- [ocp-async-4,9fe5a1e25e68792c,df6e299d22c889f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:12.076 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:12.078 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:12.632 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049640, nodeCount=1000, affectRows=1000 2024-09-11 18:15:12.633 INFO 2981023 --- [ocp-schedules-26,9c9a4386564e57a2,e90fdf460487934e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:15:12.635 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049640, nodeCount=1000, validCount=1000 2024-09-11 18:15:12.643 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049640, nodeCount=1000, affectRows=1000 2024-09-11 18:15:12.646 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049640, nodeCount=999, validCount=999 2024-09-11 18:15:12.646 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049640, nodeCount=48, affectRows=48 2024-09-11 18:15:12.646 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049640, nodeCount=46, validCount=46 2024-09-11 18:15:13.421 INFO 2981023 --- [ocp-async-6,ac4de9b00e4e20dd,390c3b88ecc1ab85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:13.961 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049640, nodeCount=480, affectRows=480 2024-09-11 18:15:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,0be58d7b91d0b7b8,d786a1fd1ee28304] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,7768ed27def73261,f5461c007c167a5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:15:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,0be58d7b91d0b7b8,d786a1fd1ee28304] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:14.327 INFO 2981023 --- [ocp-schedules-25,b5ba09c0efb8e858,56020beb20ae9613] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:15:14.372 WARN 2981023 --- [scanner-13634616-1,b5ba09c0efb8e858,6024aacb2d77ff9f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:15:15.078 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:15.079 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:15.411 INFO 2981023 --- [ocp-schedules-1,c8bc35a633808761,92f93f620266e48c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:15:15.419 INFO 2981023 --- [ocp-schedules-1,c8bc35a633808761,92f93f620266e48c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:15:15.419 INFO 2981023 --- [ocp-schedules-1,c8bc35a633808761,92f93f620266e48c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:15:15.421 INFO 2981023 --- [ocp-async-3,eafbd0c97157a9a1,684e8e3c286b0ebf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:15.927 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049640, nodeCount=480, validCount=480 2024-09-11 18:15:16.021 INFO 2981023 --- [ocp-schedules-30,a19fcf0641333a9f,40382620c1947559] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:15:16.021 INFO 2981023 --- [ocp-schedules-21,8ab684ce54a84c33,d33acbdafe9602a6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:15:16.021 INFO 2981023 --- [host-check-clockdiff1,8ab684ce54a84c33,dc0f6647cf3e8797] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:15:16.022 INFO 2981023 --- [host-check-agent-status1,a19fcf0641333a9f,33b7e519c74a465f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:15:16.023 INFO 2981023 --- [ocp-schedules-4,2c82d02c05aaa100,b1a2ae1aed613f32] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:15:16.023 INFO 2981023 --- [host-check-agent-status2,a19fcf0641333a9f,16e293e7422f1503] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:15:16.024 INFO 2981023 --- [host-check-clockdiff3,8ab684ce54a84c33,60c5b6fbca8af2d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:15:16.024 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:15:16.024 INFO 2981023 --- [host-check-clockdiff4,8ab684ce54a84c33,0bc5c9ea8d803f20] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:15:16.024 INFO 2981023 --- [host-check-clockdiff2,8ab684ce54a84c33,8bd23207418fe850] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:15:16.024 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:15:16.026 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:15:16.027 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:15:16.036 INFO 2981023 --- [host-check-agent-status3,a19fcf0641333a9f,a44b07f3964a91a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:15:16.043 INFO 2981023 --- [host-check-agent-status4,a19fcf0641333a9f,b6dab0b564426565] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:15:16.044 INFO 2981023 --- [host-check-clockdiff1,8ab684ce54a84c33,dc0f6647cf3e8797] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049716 6 6 , error= 2024-09-11 18:15:16.048 INFO 2981023 --- [host-check-clockdiff3,8ab684ce54a84c33,60c5b6fbca8af2d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049716 0 0 , error= 2024-09-11 18:15:16.058 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:15:16.062 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:15:16.064 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:15:16.065 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:15:16.070 INFO 2981023 --- [host-check-agent-status3,a19fcf0641333a9f,a44b07f3964a91a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:15:16.070 INFO 2981023 --- [host-check-agent-status1,a19fcf0641333a9f,33b7e519c74a465f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:15:16.088 INFO 2981023 --- [host-check-agent-status3,a19fcf0641333a9f,a44b07f3964a91a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:15:16.092 INFO 2981023 --- [host-check-agent-status1,a19fcf0641333a9f,33b7e519c74a465f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:15:16.093 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:15:16.093 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:15:16.098 INFO 2981023 --- [host-check-agent-status2,a19fcf0641333a9f,16e293e7422f1503] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:15:16.173 INFO 2981023 --- [host-check-agent-status2,a19fcf0641333a9f,16e293e7422f1503] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:15:16.174 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:15:16.175 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:15:16.176 INFO 2981023 --- [host-check-agent-status4,a19fcf0641333a9f,b6dab0b564426565] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:15:16.184 INFO 2981023 --- [host-check-agent-status4,a19fcf0641333a9f,b6dab0b564426565] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:15:16.192 INFO 2981023 --- [ocp-schedules-16,856015cb288bb40b,842940eb264e9bc9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:15:16.195 INFO 2981023 --- [ocp-schedules-7,dbe4e0ff02cfe1b7,bb220a3091148c1d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:15:16.199 INFO 2981023 --- [ocp-schedules-15,2c583b8ae9c571b9,eae95168542c0377] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:15:16.199 INFO 2981023 --- [ocp-schedules-5,01834a4fc70998bd,147b3fc67c2a8417] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:15:16.201 INFO 2981023 --- [ocp-schedules-16,856015cb288bb40b,842940eb264e9bc9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:15:16.242 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049642972000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:15:16.266 INFO 2981023 --- [ocp-schedules-7,dbe4e0ff02cfe1b7,bb220a3091148c1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:15:16.267 INFO 2981023 --- [ocp-schedules-15,2c583b8ae9c571b9,eae95168542c0377] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:15:16.301 INFO 2981023 --- [ocp-schedules-20,0585c3881047b4e1,b3f60f3747152376] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:15:16.307 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:15:16.308 INFO 2981023 --- [host-check-agent-status4,a19fcf0641333a9f,b6dab0b564426565] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:15:16.308 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:15:16.309 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:15:16.314471212+08:00 2024-09-11 18:15:16.310 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:15:16.316070121+08:00 2024-09-11 18:15:16.312 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:15:16.309+08:00 2024-09-11 18:15:16.313 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:15:16.313 INFO 2981023 --- [host-check-agent-status4,a19fcf0641333a9f,b6dab0b564426565] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:15:16.313 INFO 2981023 --- [host-check-agent-status4,a19fcf0641333a9f,b6dab0b564426565] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:15:16.313 INFO 2981023 --- [host-check-agent-status4,a19fcf0641333a9f,b6dab0b564426565] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:15:16.314 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:15:16.310+08:00 2024-09-11 18:15:16.314 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:15:16.315 INFO 2981023 --- [host-check-agent-status2,a19fcf0641333a9f,16e293e7422f1503] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:15:16.315 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:15:16.315135847+08:00 2024-09-11 18:15:16.315 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:15:16.309+08:00 and status AVAILABLE 2024-09-11 18:15:16.316 INFO 2981023 --- [host-check-agent-status1,a19fcf0641333a9f,33b7e519c74a465f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:15:16.316 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:15:16.310+08:00 and status AVAILABLE 2024-09-11 18:15:16.316 INFO 2981023 --- [host-check-agent-status3,a19fcf0641333a9f,a44b07f3964a91a5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:15:16.317 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:15:16.317019239+08:00 2024-09-11 18:15:16.317 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:15:16.315+08:00 2024-09-11 18:15:16.318 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:15:16.317+08:00 2024-09-11 18:15:16.326 INFO 2981023 --- [ocp-schedules-7,dbe4e0ff02cfe1b7,bb220a3091148c1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:15:16.326 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:15:16.317+08:00 and status AVAILABLE 2024-09-11 18:15:16.327 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:15:16.315+08:00 and status AVAILABLE 2024-09-11 18:15:16.327 INFO 2981023 --- [host-check-agent-status1,a19fcf0641333a9f,33b7e519c74a465f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:15:16.327 INFO 2981023 --- [host-check-agent-status1,a19fcf0641333a9f,33b7e519c74a465f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:15:16.327 INFO 2981023 --- [host-check-agent-status1,a19fcf0641333a9f,33b7e519c74a465f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:15:16.328 INFO 2981023 --- [host-check-agent-status2,a19fcf0641333a9f,16e293e7422f1503] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:15:16.328 INFO 2981023 --- [host-check-agent-status2,a19fcf0641333a9f,16e293e7422f1503] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:15:16.328 INFO 2981023 --- [host-check-agent-status2,a19fcf0641333a9f,16e293e7422f1503] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:15:16.330 INFO 2981023 --- [host-check-agent-status3,a19fcf0641333a9f,a44b07f3964a91a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:15:16.330 INFO 2981023 --- [host-check-agent-status3,a19fcf0641333a9f,a44b07f3964a91a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:15:16.330 INFO 2981023 --- [host-check-agent-status3,a19fcf0641333a9f,a44b07f3964a91a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:15:16.331 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:15:16.354 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:15:16.354 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:15:16.355 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:15:16.373 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.373 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:15:16.374 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:15:16.375 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.376 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:15:16.376 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:15:16.377 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:15:16.377 INFO 2981023 --- [ocp-schedules-16,856015cb288bb40b,842940eb264e9bc9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:15:16.378 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:15:16.380 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:15:16.381 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:15:16.383 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:15:16.383 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:15:16.385 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:15:16.383691+08:00, lastProcessTime=2024-09-11T18:15:06.352957+08:00 2024-09-11 18:15:16.386 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:15:16.388 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:15:16.406 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.406 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:16.408 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:15:16.409 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:15:16.412 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:15:16.414 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:15:16.415 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.415 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:16.422 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.422 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:16.429 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:15:16.431 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:15:16.437 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:15:16.439 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:15:16.470 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:15:16.471 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:15:16.472 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:15:16.473 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:15:16.486 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:15:16.501 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:15:16.509 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:15:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:16.516 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:15:16.516 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:15:16.516 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.517 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:15:16.519 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:15:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:15:16.536 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:15:16.536 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:15:16.537 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:15:16.538 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:15:16.542 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:15:16.546 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:15:16.557 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:15:16.557 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:15:16.559 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:15:16.559 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049642972000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:15:16.561 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:15:16.561 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:15:16.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:15:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:15:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:15:16.579 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:15:16.579 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:15:16.582 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:15:16.583 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:15:16.592 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:15:16.592 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:15:16.599 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:15:16.602 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:15:16.608 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:15:16.612 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:15:16.617 INFO 2981023 --- [host-check-host-status2,2c82d02c05aaa100,388fc587f7b04b2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:15:16.620 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:15:16.623 INFO 2981023 --- [host-check-host-status4,2c82d02c05aaa100,829dbccb5d749a4e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:15:16.643 INFO 2981023 --- [host-check-host-status1,2c82d02c05aaa100,1914ef497be82aac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:15:16.647 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:15:16.656 INFO 2981023 --- [host-check-host-status3,2c82d02c05aaa100,912e9b183d553598] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:15:16.661 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:16.661 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.662 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:15:16.682 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:15:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:16.687 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:15:16.687 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.688 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:16.689 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.689 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:15:16.690 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:15:16.690 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:15:16.696 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:15:16.696 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:15:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:15:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:15:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:15:16.713 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:16.713 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.714 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:15:16.714 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:15:16.717 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:15:16.719 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:15:16.720 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:15:16.722 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:15:16.383691+08:00 2024-09-11 18:15:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:15:16.760 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:15:16.766 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:16.766 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:16.767 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:15:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:15:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:15:16.772 INFO 2981023 --- [ocp-schedules-15,2c583b8ae9c571b9,eae95168542c0377] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:15:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,e7230fd260aa7839,4abe4623c65f999f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,e7230fd260aa7839,4abe4623c65f999f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:17.421 INFO 2981023 --- [ocp-async-9,23d14500d8a8ef85,a38cb7b3450a565f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:17.787 INFO 2981023 --- [ocp-async-2,f1b345e97ef94eba,ad60983295642743] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:15:17.812 INFO 2981023 --- [host-check-clockdiff2,8ab684ce54a84c33,8bd23207418fe850] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049717 6 6 , error= 2024-09-11 18:15:18.079 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:18.079 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:19.421 INFO 2981023 --- [ocp-async-5,b0975cd07acde626,ef812f8df9146b95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,9ff232fe6d8a5b6e,fa5389d781e1a76c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,9ff232fe6d8a5b6e,fa5389d781e1a76c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:21.080 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:21.082 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:21.421 INFO 2981023 --- [ocp-async-2,ffdd5676d02666fc,9f24bb1b98677c05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:21.883 INFO 2981023 --- [host-check-clockdiff4,8ab684ce54a84c33,0bc5c9ea8d803f20] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049721 0 0 , error= 2024-09-11 18:15:22.637 INFO 2981023 --- [ocp-schedules-3,d7dfb663b117d53a,47d30a4530f232ef] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:15:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,1e13ff3d30804f08,d16366fca02e3683] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,1e13ff3d30804f08,d16366fca02e3683] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:23.421 INFO 2981023 --- [ocp-async-8,eeebca436a42aa6d,fbb70288811b34c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:24.082 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:24.082 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:24.441 INFO 2981023 --- [ocp-schedules-26,c92a857fef2ac456,75de7af6a3a36d7e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:15:24.547 INFO 2981023 --- [ocp-schedules-6,0db1d012cb619bc2,f2feb75963acee81] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:15:24.600 WARN 2981023 --- [scanner-13634837-1,0db1d012cb619bc2,30b7f0d752f5f787] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:15:24.625 INFO 2981023 --- [alarm-detect-5,0db1d012cb619bc2,cd648602e2add893] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645771, data is empty. 2024-09-11 18:15:24.643 INFO 2981023 --- [alarm-detect-5,0db1d012cb619bc2,cd648602e2add893] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645771, data is empty. 2024-09-11 18:15:24.646 INFO 2981023 --- [alarm-detect-5,0db1d012cb619bc2,cd648602e2add893] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645771, data is empty. 2024-09-11 18:15:24.651 INFO 2981023 --- [alarm-detect-16,0db1d012cb619bc2,531942662bcb4b38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645771, data is empty. 2024-09-11 18:15:24.652 INFO 2981023 --- [alarm-detect-1,0db1d012cb619bc2,97efd0509865f64c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645771, data is empty. 2024-09-11 18:15:24.655 INFO 2981023 --- [alarm-detect-16,0db1d012cb619bc2,531942662bcb4b38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645771, data is empty. 2024-09-11 18:15:24.655 INFO 2981023 --- [alarm-detect-1,0db1d012cb619bc2,97efd0509865f64c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645771, data is empty. 2024-09-11 18:15:24.660 INFO 2981023 --- [alarm-detect-16,0db1d012cb619bc2,531942662bcb4b38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645771, data is empty. 2024-09-11 18:15:24.668 INFO 2981023 --- [alarm-detect-1,0db1d012cb619bc2,97efd0509865f64c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645771, data is empty. 2024-09-11 18:15:24.719 INFO 2981023 --- [alarm-detect-6,0db1d012cb619bc2,e2caa9c1cb89952e] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007109] will be resolved. 2024-09-11 18:15:24.719 INFO 2981023 --- [alarm-detect-6,0db1d012cb619bc2,e2caa9c1cb89952e] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007109] will be resolved. 2024-09-11 18:15:24.756 INFO 2981023 --- [alarm-detect-6,0db1d012cb619bc2,e2caa9c1cb89952e] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:15:24.780 INFO 2981023 --- [alarm-detect-5,0db1d012cb619bc2,b672cacedf2febe2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645771, data is empty. 2024-09-11 18:15:24.787 INFO 2981023 --- [alarm-detect-5,0db1d012cb619bc2,b672cacedf2febe2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645771, data is empty. 2024-09-11 18:15:24.790 INFO 2981023 --- [alarm-detect-5,0db1d012cb619bc2,b672cacedf2febe2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645771, data is empty. 2024-09-11 18:15:25.421 INFO 2981023 --- [ocp-async-1,18ce7a69d74d99cd,c01c6f8bd304c43c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:26.127 INFO 2981023 --- [ocp-schedules-25,2eed18bf3458368a,51db34593556eb65] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:15:26.127 INFO 2981023 --- [ocp-schedules-25,2eed18bf3458368a,51db34593556eb65] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:15:26.127 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:15:26.127 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:15:26.127 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:15:26.131 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:15:26.131 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:15:26.135 INFO 2981023 --- [ocp-schedules-25,2eed18bf3458368a,51db34593556eb65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:15:26.135 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:15:26.135 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:15:26.139 INFO 2981023 --- [ocp-schedules-25,2eed18bf3458368a,51db34593556eb65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:15:26.139 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:15:26.140 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:15:26.160 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:15:26.161 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:15:26.164 INFO 2981023 --- [ocp-schedules-25,2eed18bf3458368a,51db34593556eb65] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:15:26.168 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:15:26.170 INFO 2981023 --- [ocp-schedules-25,2eed18bf3458368a,51db34593556eb65] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:15:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,4c4c57f4fd47ba36,e747987fcc013d50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,4c4c57f4fd47ba36,e747987fcc013d50] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:26.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:15:26.350691+08:00, lastProcessTime=2024-09-11T18:15:16.383691+08:00 2024-09-11 18:15:26.477 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:15:26.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:15:26.350691+08:00 2024-09-11 18:15:27.082 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:27.083 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:27.421 INFO 2981023 --- [ocp-async-10,3897f9c990f021ca,0b28a9422b514537] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:27.787 INFO 2981023 --- [ocp-async-6,0a588a2788a73b8a,9545ecd574e6cf39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:15:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,e6f324c6b36341f5,1ce0563ec4d0595d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,e6f324c6b36341f5,1ce0563ec4d0595d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:29.422 INFO 2981023 --- [ocp-async-9,d84d306d9b18b32b,26caaddc3546a274] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:30.083 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:30.083 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:31.422 INFO 2981023 --- [ocp-async-6,9cd256d0df4f20ff,4cdf3f5094532746] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,9444c20b2d124bdb,1518ee119aa0b157] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,9444c20b2d124bdb,1518ee119aa0b157] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:32.639 INFO 2981023 --- [ocp-schedules-24,e1ea0feb099da371,22badb42694b3d0b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:15:33.084 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:33.084 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:33.422 INFO 2981023 --- [ocp-async-3,134b448a56bdb2b7,f815ed9da23a70c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:34.826 INFO 2981023 --- [ocp-schedules-2,998d9344b406ae3f,27b4fa2574558577] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:15:34.864 WARN 2981023 --- [scanner-13635142-1,998d9344b406ae3f,d194eb5f63ed84f6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:15:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,e96dba4581f94024,cb7f80383b1ca15c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,e96dba4581f94024,cb7f80383b1ca15c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:35.422 INFO 2981023 --- [ocp-async-7,25df33ab8a3a3610,017f4009b9cafba9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:35.422 INFO 2981023 --- [ocp-schedules-27,ec1b1caa6c7e3895,f9054194facbada5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:15:35.432 INFO 2981023 --- [ocp-schedules-27,ec1b1caa6c7e3895,f9054194facbada5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:15:35.432 INFO 2981023 --- [ocp-schedules-27,ec1b1caa6c7e3895,f9054194facbada5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:15:36.084 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:36.085 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:15:36.337920+08:00, lastProcessTime=2024-09-11T18:15:26.350691+08:00 2024-09-11 18:15:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:15:36.337920+08:00 2024-09-11 18:15:37.314 INFO 2981023 --- [prometheus-cache-monitor1,7b6cafe74d7a7aa5,1f29f5ce0be50ba6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:15:37.315 INFO 2981023 --- [prometheus-cache-monitor1,7b6cafe74d7a7aa5,1f29f5ce0be50ba6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:15:37.422 INFO 2981023 --- [ocp-async-4,8c8b4ceb56013ee7,14b238d1e7a03176] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:37.793 INFO 2981023 --- [ocp-async-8,939c05a724caf153,6c1a6363120c2ddc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:15:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,cadec4ffff9e8c66,15fe8fef629f1a1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,cadec4ffff9e8c66,15fe8fef629f1a1e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:39.085 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:39.086 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:39.422 INFO 2981023 --- [ocp-async-10,d38cad169397964f,e97f62ff868ce42d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:40.004 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:15:40.005 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:15:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,8c24027b1a633b5f,2d32dcc70b38b146] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,8c24027b1a633b5f,2d32dcc70b38b146] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:41.422 INFO 2981023 --- [ocp-async-8,f259fab531dd7c5d,a844e65a63fcaf22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:41.936 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:15:41.936 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:15:41.939 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:15:41.939 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:15:41.939 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:15:41.939 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:15:41.941 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:15:41.941 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:15:41.941 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:15:41.941 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:15:41.962 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:15:41.966 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:15:41.966 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:15:41.966 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:15:41.966 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:15:41.966 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:15:41.970 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:15:42.086 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:42.087 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:42.206 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:15:42.490 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.54904058S 2024-09-11 18:15:42.490 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:15:42.537 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:15:42.537 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:15:42.552 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:15:42.584 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:15:42.584 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:15:42.585 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:15:42.585 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:15:42.585 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:15:42.585 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:15:42.590 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:15:42.642 INFO 2981023 --- [ocp-schedules-22,e46b302e47d91de6,8425723be99c2caa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:15:42.669 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] o.s.boot.SpringApplication : Started application in 0.729 seconds (JVM running for 85542.541) 2024-09-11 18:15:42.669 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.73007283S 2024-09-11 18:15:42.863 INFO 2981023 --- [ocp-schedules-5,da1344e8c53c0b1b,729ad2e26d3726cc] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:15:43.422 INFO 2981023 --- [ocp-async-1,eef12186b562945b,d9a1576c422d1e32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,15ba8ed7ef41cc6c,7fc7c3b213bd9fc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,15ba8ed7ef41cc6c,7fc7c3b213bd9fc5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:45.013 INFO 2981023 --- [ocp-schedules-16,fc37874a71453092,8db5c6a2d76712b0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:15:45.056 WARN 2981023 --- [scanner-13635396-1,fc37874a71453092,25e4f0cf1eca9f34] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:15:45.082 INFO 2981023 --- [alarm-detect-6,fc37874a71453092,b286adef21a275a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645771, data is empty. 2024-09-11 18:15:45.086 INFO 2981023 --- [alarm-detect-6,fc37874a71453092,b286adef21a275a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645771, data is empty. 2024-09-11 18:15:45.087 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:45.088 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:45.091 INFO 2981023 --- [alarm-detect-6,fc37874a71453092,b286adef21a275a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645771, data is empty. 2024-09-11 18:15:45.101 INFO 2981023 --- [prometheus-second-schedule-1,ad7ef242702adcfc,43796ba844538da9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:15:45.105 INFO 2981023 --- [alarm-detect-6,fc37874a71453092,f4cbff20516810a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645771, data is empty. 2024-09-11 18:15:45.105 INFO 2981023 --- [alarm-detect-12,fc37874a71453092,06c2172ec762bddc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645771, data is empty. 2024-09-11 18:15:45.113 INFO 2981023 --- [alarm-detect-6,fc37874a71453092,f4cbff20516810a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645771, data is empty. 2024-09-11 18:15:45.115 INFO 2981023 --- [alarm-detect-6,fc37874a71453092,f4cbff20516810a4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645771, data is empty. 2024-09-11 18:15:45.120 INFO 2981023 --- [alarm-detect-12,fc37874a71453092,06c2172ec762bddc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645771, data is empty. 2024-09-11 18:15:45.123 INFO 2981023 --- [alarm-detect-12,fc37874a71453092,06c2172ec762bddc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645771, data is empty. 2024-09-11 18:15:45.228 INFO 2981023 --- [alarm-detect-1,fc37874a71453092,78c6765ef9919ec1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645771, data is empty. 2024-09-11 18:15:45.232 INFO 2981023 --- [alarm-detect-1,fc37874a71453092,78c6765ef9919ec1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645771, data is empty. 2024-09-11 18:15:45.235 INFO 2981023 --- [alarm-detect-1,fc37874a71453092,78c6765ef9919ec1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645771, data is empty. 2024-09-11 18:15:45.422 INFO 2981023 --- [ocp-async-5,c1b137e3e344146b,1bb191b832cce0dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:45.878 INFO 2981023 --- [ocp-schedules-21,d81bcd5cc53d8118,ce612e6cac86dea1] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:15:45.905 WARN 2981023 --- [ocp-schedules-21,d81bcd5cc53d8118,ce612e6cac86dea1] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:15:46.019 INFO 2981023 --- [ocp-schedules-30,4bfeb9447a2871f1,f6611e75f3a67911] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:15:46.020 INFO 2981023 --- [ocp-schedules-25,1503babc79c27673,a05ee778b0bca0bc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:15:46.022 INFO 2981023 --- [host-check-clockdiff1,4bfeb9447a2871f1,6ca4a3293d8ec860] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:15:46.024 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:15:46.025 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:15:46.025 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:15:46.026 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:15:46.027 INFO 2981023 --- [host-check-clockdiff2,4bfeb9447a2871f1,8cdb7145767c6d68] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:15:46.027 INFO 2981023 --- [host-check-clockdiff3,4bfeb9447a2871f1,95c33f8c4b0d8c8e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:15:46.031 INFO 2981023 --- [host-check-clockdiff4,4bfeb9447a2871f1,0ca4e71dabfaaa5c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:15:46.034 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:15:46.038 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:15:46.038 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:15:46.040 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:15:46.040 INFO 2981023 --- [ocp-schedules-22,114d69303ea709b1,8995b3a0d269c075] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:15:46.040 INFO 2981023 --- [host-check-clockdiff3,4bfeb9447a2871f1,95c33f8c4b0d8c8e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049746 0 0 , error= 2024-09-11 18:15:46.040 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:15:46.040 INFO 2981023 --- [host-check-agent-status2,114d69303ea709b1,62fb64e2ab6392d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:15:46.041 INFO 2981023 --- [host-check-agent-status1,114d69303ea709b1,31d11677adcbdd52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:15:46.041 INFO 2981023 --- [host-check-agent-status3,114d69303ea709b1,bf52081e4312af8a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:15:46.042 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:15:46.043 INFO 2981023 --- [host-check-agent-status4,114d69303ea709b1,e72fc8d9156c0aeb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:15:46.044 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:15:46.046 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:15:46.051 INFO 2981023 --- [host-check-agent-status3,114d69303ea709b1,bf52081e4312af8a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:15:46.051 INFO 2981023 --- [host-check-agent-status2,114d69303ea709b1,62fb64e2ab6392d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:15:46.051 INFO 2981023 --- [host-check-agent-status1,114d69303ea709b1,31d11677adcbdd52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:15:46.053 INFO 2981023 --- [host-check-agent-status2,114d69303ea709b1,62fb64e2ab6392d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:15:46.054 INFO 2981023 --- [host-check-agent-status3,114d69303ea709b1,bf52081e4312af8a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:15:46.055 INFO 2981023 --- [host-check-agent-status1,114d69303ea709b1,31d11677adcbdd52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:15:46.057 INFO 2981023 --- [host-check-agent-status4,114d69303ea709b1,e72fc8d9156c0aeb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:15:46.060 INFO 2981023 --- [host-check-agent-status4,114d69303ea709b1,e72fc8d9156c0aeb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:15:46.070 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:15:46.072 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:15:46.073 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:15:46.078247832+08:00 2024-09-11 18:15:46.073 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:15:46.073+08:00 2024-09-11 18:15:46.074 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:15:46.074214697+08:00 2024-09-11 18:15:46.075 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:15:46.075 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:15:46.074+08:00 2024-09-11 18:15:46.077 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:15:46.082163132+08:00 2024-09-11 18:15:46.077 INFO 2981023 --- [host-check-agent-status2,114d69303ea709b1,62fb64e2ab6392d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:15:46.079 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:15:46.073+08:00 and status AVAILABLE 2024-09-11 18:15:46.080 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:15:46.074+08:00 and status AVAILABLE 2024-09-11 18:15:46.081 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:15:46.077+08:00 2024-09-11 18:15:46.087 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:15:46.077+08:00 and status AVAILABLE 2024-09-11 18:15:46.087 INFO 2981023 --- [host-check-agent-status1,114d69303ea709b1,31d11677adcbdd52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:15:46.092 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:15:46.095 INFO 2981023 --- [host-check-agent-status2,114d69303ea709b1,62fb64e2ab6392d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:15:46.095 INFO 2981023 --- [host-check-agent-status2,114d69303ea709b1,62fb64e2ab6392d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:15:46.095 INFO 2981023 --- [host-check-agent-status2,114d69303ea709b1,62fb64e2ab6392d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:15:46.095 INFO 2981023 --- [host-check-agent-status1,114d69303ea709b1,31d11677adcbdd52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:15:46.095 INFO 2981023 --- [host-check-agent-status1,114d69303ea709b1,31d11677adcbdd52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:15:46.095 INFO 2981023 --- [host-check-agent-status1,114d69303ea709b1,31d11677adcbdd52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:15:46.097 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:15:46.117 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:15:46.121 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:15:46.122 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:15:46.124 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:15:46.125 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:15:46.125 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:15:46.127 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:15:46.147 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:15:46.147 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:15:46.149 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:15:46.154 INFO 2981023 --- [host-check-agent-status3,114d69303ea709b1,bf52081e4312af8a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:15:46.157 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:15:46.159 INFO 2981023 --- [ocp-schedules-3,0e40fd0fe8a8ac6c,4cb2d6b35c573c86] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:15:46.160 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:15:46.165 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:15:46.164529492+08:00 2024-09-11 18:15:46.167 INFO 2981023 --- [host-check-agent-status3,114d69303ea709b1,bf52081e4312af8a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:15:46.167 INFO 2981023 --- [host-check-agent-status3,114d69303ea709b1,bf52081e4312af8a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:15:46.167 INFO 2981023 --- [host-check-agent-status3,114d69303ea709b1,bf52081e4312af8a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:15:46.169 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:15:46.170 INFO 2981023 --- [ocp-schedules-3,0e40fd0fe8a8ac6c,4cb2d6b35c573c86] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:15:46.175 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:15:46.165+08:00 2024-09-11 18:15:46.178 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:15:46.180 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:15:46.183 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:15:46.165+08:00 and status AVAILABLE 2024-09-11 18:15:46.185 INFO 2981023 --- [host-check-agent-status4,114d69303ea709b1,e72fc8d9156c0aeb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:15:46.187 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:15:46.188 INFO 2981023 --- [ocp-schedules-10,c98e254472cbd17e,f7a8ffca8330e973] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:15:46.188 INFO 2981023 --- [host-check-agent-status4,114d69303ea709b1,e72fc8d9156c0aeb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:15:46.188 INFO 2981023 --- [host-check-agent-status4,114d69303ea709b1,e72fc8d9156c0aeb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:15:46.188 INFO 2981023 --- [host-check-agent-status4,114d69303ea709b1,e72fc8d9156c0aeb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:15:46.191 INFO 2981023 --- [ocp-schedules-6,5cf81f0ff7ede0a6,414cd45d8f0a3f94] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:15:46.192 INFO 2981023 --- [ocp-schedules-29,3bb915cdc34c4d9d,9094afcbb8b3763c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:15:46.192 INFO 2981023 --- [ocp-schedules-18,7023915896eee65b,5ea1faca40239d7b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:15:46.192 INFO 2981023 --- [ocp-schedules-29,3bb915cdc34c4d9d,9094afcbb8b3763c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:15:46.194 INFO 2981023 --- [ocp-schedules-10,c98e254472cbd17e,f7a8ffca8330e973] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:15:46.197 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:15:46.197 INFO 2981023 --- [ocp-schedules-6,5cf81f0ff7ede0a6,414cd45d8f0a3f94] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:15:46.208 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:15:46.213 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:15:46.232 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:15:46.240 INFO 2981023 --- [ocp-schedules-2,6c5ed5cd4d40be1b,d4eadcbd7bbddb52] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:15:46.248 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:15:46.250 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:15:46.295 INFO 2981023 --- [ocp-schedules-6,5cf81f0ff7ede0a6,414cd45d8f0a3f94] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:15:46.295 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049642972000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:15:46.297 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.298 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:15:46.298 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:15:46.301 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.301 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:15:46.303 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.303 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:15:46.304 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.304 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:15:46.304 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.304 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:15:46.304 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:46.313 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:15:46.318 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:15:46.319 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:15:46.320 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.321 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:15:46.322 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:15:46.323 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:15:46.323 INFO 2981023 --- [ocp-schedules-10,c98e254472cbd17e,f7a8ffca8330e973] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:15:46.324 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:15:46.324 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.325 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:46.328 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.329 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:46.334 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:15:46.346 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:15:46.348 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:15:46.347915+08:00, lastProcessTime=2024-09-11T18:15:36.337920+08:00 2024-09-11 18:15:46.349 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:15:46.357 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:15:46.364 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:15:46.376 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:15:46.377 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.380 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.381 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:15:46.385 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:15:46.390 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.391 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.391 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:15:46.392 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:15:46.392 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.397 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:15:46.398 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:15:46.405 INFO 2981023 --- [host-check-host-status4,1503babc79c27673,e18ba1e17cb8139b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:15:46.406 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:15:46.416 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:15:46.419 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:15:46.419 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049642972000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:15:46.423 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:15:46.428 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:15:46.447 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.448 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.448 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:46.449 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:15:46.449 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.452 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.455 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:15:46.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:15:46.457 INFO 2981023 --- [host-check-host-status1,1503babc79c27673,db2ecc58cd279266] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:15:46.458 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:15:46.468 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.469 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.469 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:15:46.470 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:15:46.473 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.473 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:15:46.473 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:15:46.473 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:15:46.473 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:15:46.474 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:15:46.478 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.484 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:15:46.487 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:15:46.491 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.491 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.492 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:15:46.494 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:15:46.497 INFO 2981023 --- [host-check-host-status2,1503babc79c27673,b2bb056e3754830e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:15:46.500 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:15:46.503 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:15:46.511 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.511 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.512 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:15:46.513 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:15:46.517 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:15:46.517 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:15:46.518 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:15:46.520 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:15:46.520 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.523 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:15:46.524 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:15:46.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:15:46.526 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.526 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:15:46.527 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.527 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:15:46.528 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:15:46.528 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.534 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:15:46.534 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:15:46.545 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:15:46.550 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:15:46.552 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:15:46.557 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:15:46.560 INFO 2981023 --- [host-check-host-status3,1503babc79c27673,3d852e754c957838] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:15:46.564 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.565 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.566 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:15:46.566 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:15:46.567 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:15:46.570 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:15:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:15:46.575 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:15:46.587 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.588 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.588 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:15:46.589 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:15:46.589 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:15:46.620 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:15:46.627 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.630 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:15:46.631 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.631 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:15:46.639 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:15:46.640 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:46.640 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:15:46.643 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:15:46.643 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:15:46.643 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:15:46.647 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:15:46.647 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:15:46.648 INFO 2981023 --- [ocp-schedules-29,3bb915cdc34c4d9d,9094afcbb8b3763c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:15:46.650 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:15:46.347915+08:00 2024-09-11 18:15:46.658 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:15:46.664 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:15:46.667 INFO 2981023 --- [ocp-schedules-7,54031e49a7815253,bdf3882c85dfa9ae] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:15:46.845 INFO 2981023 --- [ocp-schedules-26,bb30645fd96ae43f,f0c6e3c67b78640d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8196,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:15:46.845 INFO 2981023 --- [ocp-schedules-26,bb30645fd96ae43f,f0c6e3c67b78640d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:15:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,57282142a07b8291,ebbdf4381e3a8f0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,57282142a07b8291,ebbdf4381e3a8f0a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:47.423 INFO 2981023 --- [ocp-async-8,e2b8311f75f1e48b,2fc6a49d5f7b8fa9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:47.787 INFO 2981023 --- [ocp-async-10,3eb070849a08b2d5,102662a1723bf79d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:15:47.817 INFO 2981023 --- [host-check-clockdiff2,4bfeb9447a2871f1,8cdb7145767c6d68] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049747 5 5 , error= 2024-09-11 18:15:48.089 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:48.089 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:49.423 INFO 2981023 --- [ocp-async-6,1d1897bdaadd1bec,88ea81ab23ac7ec8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:15:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:15:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,8ae094abf10b0554,d38e676a78d7edc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,8ae094abf10b0554,d38e676a78d7edc8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:50.344 INFO 2981023 --- [ocp-schedules-29,b8d8f80e303a3d8c,72ca206840487d84] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:15:51.089 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:51.091 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:51.262 INFO 2981023 --- [host-check-clockdiff1,4bfeb9447a2871f1,6ca4a3293d8ec860] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049751 5 5 , error= 2024-09-11 18:15:51.423 INFO 2981023 --- [ocp-async-10,19e96f5eb3edd928,814d2553e9bfe0b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:51.619 INFO 2981023 --- [ocp-schedules-6,4bb09dd331a4e966,c4e333fc589d03d9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:15:51.620 INFO 2981023 --- [ocp-schedules-6,4bb09dd331a4e966,c4e333fc589d03d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:15:51.620 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:15:51.620 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:15:51.620 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:15:51.620 INFO 2981023 --- [ocp-schedules-6,4bb09dd331a4e966,c4e333fc589d03d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:15:52.016 INFO 2981023 --- [host-check-clockdiff4,4bfeb9447a2871f1,0ca4e71dabfaaa5c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049752 0 0 , error= 2024-09-11 18:15:52.645 INFO 2981023 --- [ocp-schedules-29,2c154666cbd13b9b,8f02f6d18f7cef2b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:15:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,d33ae06255686d4c,a5ff3a2b6d7c4d2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,d33ae06255686d4c,a5ff3a2b6d7c4d2e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:53.323 INFO 2981023 --- [ocp-schedules-11,363add65f89d2352,abc36dc162fc3208] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:15:53.323 INFO 2981023 --- [ocp-schedules-11,363add65f89d2352,abc36dc162fc3208] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:15:53.423 INFO 2981023 --- [ocp-async-9,0b568dc051a49898,87415ab30a1cd0c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:54.091 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:54.092 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:54.444 INFO 2981023 --- [ocp-schedules-5,04a7828f69abb248,d3877ebfb5693f77] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:15:54.848 INFO 2981023 --- [ocp-schedules-18,cf708968c87e9142,e1c4621ce7bbd6ea] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:15:54.868 WARN 2981023 --- [ocp-schedules-18,cf708968c87e9142,e1c4621ce7bbd6ea] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:15:55.059 INFO 2981023 --- [ocp-schedules-26,8f6429c22e635cba,640273009176d5f7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:15:55.059 INFO 2981023 --- [ocp-schedules-26,8f6429c22e635cba,640273009176d5f7] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:15:55.064 INFO 2981023 --- [ocp-schedules-26,8f6429c22e635cba,640273009176d5f7] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:15:55.064 INFO 2981023 --- [ocp-schedules-26,8f6429c22e635cba,640273009176d5f7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:15:55.064 INFO 2981023 --- [ocp-schedules-26,8f6429c22e635cba,640273009176d5f7] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:15:55.074 INFO 2981023 --- [ocp-schedules-26,8f6429c22e635cba,640273009176d5f7] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:15:55.074 INFO 2981023 --- [ocp-schedules-26,8f6429c22e635cba,640273009176d5f7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:15:55.074 INFO 2981023 --- [ocp-schedules-26,8f6429c22e635cba,640273009176d5f7] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:15:55.086 INFO 2981023 --- [ocp-schedules-26,8f6429c22e635cba,640273009176d5f7] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:15:55.271 INFO 2981023 --- [ocp-schedules-19,ab109d6b7a6078dc,e85fb7a87f146571] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:15:55.321 WARN 2981023 --- [scanner-13635705-1,ab109d6b7a6078dc,3a7c2d57516bf9ce] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:15:55.423 INFO 2981023 --- [ocp-async-1,ee3c11a30afa97fa,9d3c075feb458f29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:55.435 INFO 2981023 --- [ocp-schedules-22,fe569fb550151b99,30919329c7087e75] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:15:55.454 INFO 2981023 --- [ocp-schedules-22,fe569fb550151b99,30919329c7087e75] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:15:55.454 INFO 2981023 --- [ocp-schedules-22,fe569fb550151b99,30919329c7087e75] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:15:55.873 INFO 2981023 --- [ocp-schedules-3,c7c3933ed7b2f7b8,2879cbf33c28298b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:15:55.873 INFO 2981023 --- [ocp-schedules-3,c7c3933ed7b2f7b8,2879cbf33c28298b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:15:55.876 INFO 2981023 --- [ocp-schedules-3,c7c3933ed7b2f7b8,2879cbf33c28298b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:15:55.876 INFO 2981023 --- [ocp-schedules-3,c7c3933ed7b2f7b8,2879cbf33c28298b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:15:55.878 INFO 2981023 --- [ocp-schedules-3,c7c3933ed7b2f7b8,2879cbf33c28298b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:15:55.878 INFO 2981023 --- [ocp-schedules-3,c7c3933ed7b2f7b8,2879cbf33c28298b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:15:56.065 INFO 2981023 --- [ocp-schedules-5,56f1c83e83e8e5ac,2f60db9f1a54adba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:15:56.076 INFO 2981023 --- [ocp-schedules-5,56f1c83e83e8e5ac,2f60db9f1a54adba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:15:56.076 INFO 2981023 --- [ocp-schedules-5,56f1c83e83e8e5ac,2f60db9f1a54adba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:15:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,ae183ea7aafeb2f5,22d65b9eb2b4fece] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,ae183ea7aafeb2f5,22d65b9eb2b4fece] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:15:56.352422+08:00, lastProcessTime=2024-09-11T18:15:46.347915+08:00 2024-09-11 18:15:56.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:15:56.352422+08:00 2024-09-11 18:15:57.093 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:15:57.093 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:15:57.423 INFO 2981023 --- [ocp-async-2,e5616c943f549cad,945f3215e131c219] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:57.788 INFO 2981023 --- [ocp-async-5,de36627585dee0b2,30517a659e74ef0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:15:58.075 INFO 2981023 --- [ocp-schedules-20,17a159904805a88a,80e08eaa38b63633] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:15:58.075 INFO 2981023 --- [ocp-schedules-20,17a159904805a88a,80e08eaa38b63633] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:15:58.105 INFO 2981023 --- [Collect all tenants major compactions2,17a159904805a88a,238cfbb9b5fb521a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:15:58.106 INFO 2981023 --- [Collect all tenants major compactions3,17a159904805a88a,5417911c71cabc05] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:15:58.106 INFO 2981023 --- [Collect all tenants major compactions5,17a159904805a88a,b597bf879dc462f0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:15:58.111 INFO 2981023 --- [Collect all tenants major compactions6,17a159904805a88a,e6787c62c13c8216] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:15:58.111 INFO 2981023 --- [Collect all tenants major compactions1,17a159904805a88a,510842cfa7898fe5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:15:58.112 INFO 2981023 --- [Collect all tenants major compactions4,17a159904805a88a,98416ff8cf6c61b3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:15:58.133 INFO 2981023 --- [Collect all tenants major compactions5,17a159904805a88a,b597bf879dc462f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:58.133 INFO 2981023 --- [Collect all tenants major compactions2,17a159904805a88a,238cfbb9b5fb521a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:58.133 INFO 2981023 --- [Collect all tenants major compactions3,17a159904805a88a,5417911c71cabc05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:58.134 INFO 2981023 --- [Collect all tenants major compactions5,17a159904805a88a,b597bf879dc462f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:15:58.134 INFO 2981023 --- [Collect all tenants major compactions3,17a159904805a88a,5417911c71cabc05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:15:58.135 INFO 2981023 --- [Collect all tenants major compactions1,17a159904805a88a,510842cfa7898fe5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:58.137 INFO 2981023 --- [Collect all tenants major compactions5,17a159904805a88a,b597bf879dc462f0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:15:58.138 INFO 2981023 --- [Collect all tenants major compactions6,17a159904805a88a,e6787c62c13c8216] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:58.139 INFO 2981023 --- [Collect all tenants major compactions6,17a159904805a88a,e6787c62c13c8216] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:15:58.139 INFO 2981023 --- [Collect all tenants major compactions4,17a159904805a88a,98416ff8cf6c61b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:58.140 INFO 2981023 --- [Collect all tenants major compactions5,17a159904805a88a,b597bf879dc462f0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:15:58.140 INFO 2981023 --- [Collect all tenants major compactions3,17a159904805a88a,5417911c71cabc05] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:15:58.140 INFO 2981023 --- [Collect all tenants major compactions3,17a159904805a88a,5417911c71cabc05] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:15:58.141 INFO 2981023 --- [Collect all tenants major compactions4,17a159904805a88a,98416ff8cf6c61b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:15:58.141 INFO 2981023 --- [Collect all tenants major compactions2,17a159904805a88a,238cfbb9b5fb521a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:15:58.142 INFO 2981023 --- [Collect all tenants major compactions6,17a159904805a88a,e6787c62c13c8216] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:15:58.142 INFO 2981023 --- [Collect all tenants major compactions6,17a159904805a88a,e6787c62c13c8216] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:15:58.143 INFO 2981023 --- [Collect all tenants major compactions1,17a159904805a88a,510842cfa7898fe5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:15:58.143 INFO 2981023 --- [Collect all tenants major compactions2,17a159904805a88a,238cfbb9b5fb521a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:15:58.143 INFO 2981023 --- [Collect all tenants major compactions2,17a159904805a88a,238cfbb9b5fb521a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:15:58.143 INFO 2981023 --- [Collect all tenants major compactions4,17a159904805a88a,98416ff8cf6c61b3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:15:58.143 INFO 2981023 --- [Collect all tenants major compactions4,17a159904805a88a,98416ff8cf6c61b3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:15:58.144 INFO 2981023 --- [Collect all tenants major compactions1,17a159904805a88a,510842cfa7898fe5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:15:58.144 INFO 2981023 --- [Collect all tenants major compactions1,17a159904805a88a,510842cfa7898fe5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:15:58.145 INFO 2981023 --- [ocp-schedules-20,17a159904805a88a,80e08eaa38b63633] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:15:58.888 INFO 2981023 --- [ocp-schedules-2,51baf461636218a1,7abe69b436ace142] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:15:58.888+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:15:58.889 INFO 2981023 --- [ocp-schedules-2,51baf461636218a1,7abe69b436ace142] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:15:58.993 INFO 2981023 --- [ocp-schedules-2,51baf461636218a1,7abe69b436ace142] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 19 to SqlAggregateData 2024-09-11 18:15:58.994 INFO 2981023 --- [ocp-schedules-2,51baf461636218a1,7abe69b436ace142] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:15:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,676696f7ee37b392,705935946b51355d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049760, interval=60 2024-09-11 18:15:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,676696f7ee37b392,705935946b51355d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:15:59.423 INFO 2981023 --- [ocp-async-9,45de2d49f4e1abf3,1aab8735b3bcbaa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:15:59.905 INFO 2981023 --- [ocp-schedules-28,8f25d7f6bdd04fc3,8f719a35c2b2ba41] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:15:59.906 INFO 2981023 --- [ocp-schedules-28,8f25d7f6bdd04fc3,8f719a35c2b2ba41] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:15:59.937 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:15:59.937 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.938 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:15:59.938 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:15:59.938 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.938 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.940 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:15:59.943 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:15:59.943 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:15:59.950 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.950 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.950 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:15:59.951 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:15:59.951 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.952 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:15:59.958 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.959 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:15:59.963 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.963 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:15:59.965 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.965 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:15:59.965 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.965 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:15:59.973 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.973 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:15:59.975 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.975 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:15:59.976 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:15:59.976 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5431D6F3-0-0] 2024-09-11 18:15:59.976 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:15:59.976 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.976 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:15:59.979 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.980 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:15:59.980 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.980 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:15:59.988 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.988 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:15:59.992 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.992 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:15:59.997 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.998 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:15:59.998 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:15:59.999 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:15:59.999 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.000 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:16:00.000 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:16:00.000 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.001 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:16:00.001 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:16:00.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5431D6FD-0-0] 2024-09-11 18:16:00.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:16:00.008 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:16:00.010 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.010 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:16:00.011 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.011 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:16:00.012 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.013 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:16:00.014 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.014 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.015 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:16:00.015 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:16:00.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.016 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:16:00.016 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:16:00.017 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.017 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.017 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:16:00.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:16:00.018 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.018 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.019 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:16:00.019 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:16:00.020 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.020 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.020 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:16:00.021 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:16:00.021 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.021 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:16:00.022 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.022 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:16:00.022 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:16:00.022 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:16:00.024 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:16:00.025 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:16:00.028 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:00.031 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.031 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:16:00.037 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:00.039 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.039 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:00.040 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:16:00.043 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.044 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:16:00.046 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:16:00.049 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:16:00.050 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:16:00.051 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.051 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:16:00.051 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:16:00.051 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:16:00.051 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:16:00.051 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:16:00.053 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:16:00.053 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.053 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:16:00.057 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:16:00.059 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:16:00.059 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:00.060 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:16:00.065 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:16:00.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:16:00.067 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:16:00.067 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:16:00.067 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:16:00.067 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:16:00.067 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:16:00.067 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:16:00.068 INFO 2981023 --- [ocp-schedules-28,8f25d7f6bdd04fc3,8f719a35c2b2ba41] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:16:00.093 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:00.094 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:01.423 INFO 2981023 --- [ocp-async-1,eabffb444505a910,e9f26a9a9ea363ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,b49303d3259d0535,13a311299031949f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,b49303d3259d0535,13a311299031949f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:02.648 INFO 2981023 --- [ocp-schedules-25,eaa4ed09d146c52a,764c188756fdb867] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:16:02.981 INFO 2981023 --- [ocp-schedules-2,1b63ed3348454f4d,227ff726b5112fde] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:16:02.982 INFO 2981023 --- [ocp-schedules-2,1b63ed3348454f4d,227ff726b5112fde] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:16:03.002 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:03.003 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:16:03.008 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:03.010 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:16:03.011 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049763011000} 2024-09-11 18:16:03.021 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:16:03.027 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000232, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:16:03.027 INFO 2981023 --- [ocp-schedules-2,1b63ed3348454f4d,227ff726b5112fde] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:16:03.094 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:03.094 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:03.423 INFO 2981023 --- [ocp-async-10,c901499a8c2bb493,58b47bd320e5c5f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:03.456 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:16:03.456 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:16:03.456 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:16:03.460 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000232, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:16:03.461 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:16:04.221 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000232 2024-09-11 18:16:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,8867a740c16a18de,b17d60e63e951ca9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,8867a740c16a18de,b17d60e63e951ca9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:05.424 INFO 2981023 --- [ocp-async-7,9b2a4261ba2c3dd8,3a3b768a64956c27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:05.469 INFO 2981023 --- [ocp-schedules-18,e40027673859bb0e,70f994758d5b2646] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:16:05.512 WARN 2981023 --- [scanner-13635882-1,e40027673859bb0e,479352d2329f2890] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:16:05.530 INFO 2981023 --- [alarm-detect-1,e40027673859bb0e,19aadde158c79b9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645771, data is empty. 2024-09-11 18:16:05.535 INFO 2981023 --- [alarm-detect-1,e40027673859bb0e,19aadde158c79b9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645790, data is empty. 2024-09-11 18:16:05.540 INFO 2981023 --- [alarm-detect-1,e40027673859bb0e,19aadde158c79b9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645790, data is empty. 2024-09-11 18:16:05.555 INFO 2981023 --- [alarm-detect-11,e40027673859bb0e,23d1d1e0d2a3de0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645771, data is empty. 2024-09-11 18:16:05.557 INFO 2981023 --- [alarm-detect-13,e40027673859bb0e,3b20801d581d26fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645771, data is empty. 2024-09-11 18:16:05.560 INFO 2981023 --- [alarm-detect-11,e40027673859bb0e,23d1d1e0d2a3de0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645790, data is empty. 2024-09-11 18:16:05.562 INFO 2981023 --- [alarm-detect-13,e40027673859bb0e,3b20801d581d26fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645790, data is empty. 2024-09-11 18:16:05.562 INFO 2981023 --- [alarm-detect-11,e40027673859bb0e,23d1d1e0d2a3de0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645790, data is empty. 2024-09-11 18:16:05.567 INFO 2981023 --- [alarm-detect-13,e40027673859bb0e,3b20801d581d26fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645790, data is empty. 2024-09-11 18:16:05.681 INFO 2981023 --- [alarm-detect-11,e40027673859bb0e,ef0ba5298c4af48a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645771, data is empty. 2024-09-11 18:16:05.690 INFO 2981023 --- [alarm-detect-11,e40027673859bb0e,ef0ba5298c4af48a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645790, data is empty. 2024-09-11 18:16:05.695 INFO 2981023 --- [alarm-detect-11,e40027673859bb0e,ef0ba5298c4af48a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645790, data is empty. 2024-09-11 18:16:05.984 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:05.985 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:16:05.986 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:05.986 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:16:05.986 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:16:06.000 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:16:06.005 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:16:06.010 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:16:06.010 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:16:06.010 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:16:06.012 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:06.013 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:16:06.014 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:06.014 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:16:06.056 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:13:41, start=2024-09-11T18:10:11, end=2024-09-11T18:12:41, updates=308, granule=120000ms, delay=210000ms 2024-09-11 18:16:06.061 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:13:41, target=2024-09-11T18:12:41, granule=120000ms, delay=60000ms) 2024-09-11 18:16:06.061 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:12:41, target=2024-09-11T18:00:51, granule=600000ms, delay=710000ms) 2024-09-11 18:16:06.061 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=75ms, period=120000ms 2024-09-11 18:16:06.094 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:06.095 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:06.134 INFO 2981023 --- [ocp-schedules-15,070d6908b155b9a4,1ae52d592f2f508b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:16:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:16:06.337236+08:00, lastProcessTime=2024-09-11T18:15:56.352422+08:00 2024-09-11 18:16:06.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:16:06.337236+08:00 2024-09-11 18:16:07.424 INFO 2981023 --- [ocp-async-9,5d08e0123e4b0174,852d8d92e1e61451] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:07.788 INFO 2981023 --- [ocp-async-8,2364c5bd65011a48,f6eaed761ca77366] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:16:07.979 INFO 2981023 --- [ocp-schedules-30,a453c240ecf265aa,a6f7ab8496ee6211] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:16:08.093 INFO 2981023 --- [ocp-schedules-17,019a75771a346cfd,2e0ef438d21762eb] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:16:08.093+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:16:08.094 INFO 2981023 --- [ocp-schedules-17,019a75771a346cfd,2e0ef438d21762eb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:16:08.095 INFO 2981023 --- [ocp-schedules-17,019a75771a346cfd,2e0ef438d21762eb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:16:08.095 INFO 2981023 --- [ocp-schedules-17,019a75771a346cfd,2e0ef438d21762eb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:16:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,454bc99c7707f5a8,b97bb717ae4fb4e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:08.315 INFO 2981023 --- [prometheus-minute-schedule-1,454bc99c7707f5a8,b97bb717ae4fb4e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:09.095 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:09.096 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:09.424 INFO 2981023 --- [ocp-async-3,ee54ec7c3d1f0e3a,ef3ac6220333f4d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:10.002 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:16:10.002 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:16:10.330 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,cb30cf90aced38fd,c99adeca74a319c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=cb30cf90aced38fd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:16:10.330 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,af7933562b0373aa,06a3b53dd6b67724] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=af7933562b0373aa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:16:10.331 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,cb30cf90aced38fd,c99adeca74a319c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=cb30cf90aced38fd, duration=5 ms] 2024-09-11 18:16:10.336 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,af7933562b0373aa,06a3b53dd6b67724] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=af7933562b0373aa, duration=11 ms] 2024-09-11 18:16:10.340 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,64b33ac7ceb96f5f,58a2e04713dbd57c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=64b33ac7ceb96f5f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:16:10.341 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,64b33ac7ceb96f5f,58a2e04713dbd57c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=64b33ac7ceb96f5f, duration=4 ms] 2024-09-11 18:16:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:16:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:16:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,60669a8cd65b7fe8,b82cb3bb55c6bd71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,60669a8cd65b7fe8,b82cb3bb55c6bd71] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:11.424 INFO 2981023 --- [ocp-async-8,b9cacdaacf47c8ca,d81276fd96430298] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:12.096 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:12.096 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:12.650 INFO 2981023 --- [ocp-schedules-19,a955cb9210b7fb58,4ce7d1103ef0357d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:16:12.728 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049700, nodeCount=1000, affectRows=1000 2024-09-11 18:16:12.730 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049700, nodeCount=1000, validCount=1000 2024-09-11 18:16:12.739 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049700, nodeCount=48, affectRows=48 2024-09-11 18:16:12.739 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049700, nodeCount=46, validCount=46 2024-09-11 18:16:12.743 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049700, nodeCount=1000, affectRows=1000 2024-09-11 18:16:12.745 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049700, nodeCount=999, validCount=999 2024-09-11 18:16:13.424 INFO 2981023 --- [ocp-async-5,c6f9133562403388,da01a75d839fd811] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:14.314 INFO 2981023 --- [prometheus-minute-schedule-1,6cf54a3d4dc66eee,452c0dbd61cb5995] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,7f23aa717776773c,7bb2a020462bd6fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:16:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,6cf54a3d4dc66eee,452c0dbd61cb5995] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:15.097 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:15.097 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:15.424 INFO 2981023 --- [ocp-async-1,defe708e11b1f67c,bdb906274cd4858e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:15.455 INFO 2981023 --- [ocp-schedules-9,179d89f88b01e5d6,0318a53851f17b02] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:16:15.460 INFO 2981023 --- [ocp-schedules-9,179d89f88b01e5d6,0318a53851f17b02] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:16:15.460 INFO 2981023 --- [ocp-schedules-9,179d89f88b01e5d6,0318a53851f17b02] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:16:15.717 INFO 2981023 --- [ocp-schedules-21,2ffbac07cdca129a,a73fbc898aab91c8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:16:15.762 WARN 2981023 --- [scanner-13637746-1,2ffbac07cdca129a,6da11eb96a341b5c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:16:15.984 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049700, nodeCount=480, affectRows=480 2024-09-11 18:16:16.019 INFO 2981023 --- [ocp-schedules-20,194d944eea586f30,f2f82930bc590475] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:16:16.019 INFO 2981023 --- [host-check-clockdiff1,194d944eea586f30,4952e84b4582a357] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:16:16.020 INFO 2981023 --- [host-check-clockdiff2,194d944eea586f30,6d9b5fa9a6130cbe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:16:16.021 INFO 2981023 --- [host-check-clockdiff3,194d944eea586f30,eec1b2f0c1d2881b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:16:16.023 INFO 2981023 --- [host-check-clockdiff4,194d944eea586f30,8ef0570a5e1bf95d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:16:16.031 INFO 2981023 --- [ocp-schedules-4,acb2bfb45cdb0248,8ebc4df23ceb494f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:16:16.038 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:16:16.038 INFO 2981023 --- [ocp-schedules-6,0c3499cb78caffe8,9d62229c5d63e520] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:16:16.038 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:16:16.038 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:16:16.040 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:16:16.041 INFO 2981023 --- [host-check-agent-status1,0c3499cb78caffe8,e8354e04fe93e4f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:16:16.043 INFO 2981023 --- [host-check-agent-status2,0c3499cb78caffe8,8ad325f6757f0f59] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:16:16.043 INFO 2981023 --- [host-check-agent-status3,0c3499cb78caffe8,5b10a98354addf58] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:16:16.052 INFO 2981023 --- [host-check-agent-status4,0c3499cb78caffe8,bffa2178340b29f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:16:16.062 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:16:16.066 INFO 2981023 --- [host-check-clockdiff2,194d944eea586f30,6d9b5fa9a6130cbe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049776 5 5 , error= 2024-09-11 18:16:16.067 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:16:16.073 INFO 2981023 --- [host-check-clockdiff3,194d944eea586f30,eec1b2f0c1d2881b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049776 0 0 , error= 2024-09-11 18:16:16.078 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:16:16.079 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:16:16.082 INFO 2981023 --- [host-check-agent-status2,0c3499cb78caffe8,8ad325f6757f0f59] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:16:16.082 INFO 2981023 --- [host-check-agent-status1,0c3499cb78caffe8,e8354e04fe93e4f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:16:16.086 INFO 2981023 --- [host-check-agent-status3,0c3499cb78caffe8,5b10a98354addf58] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:16:16.097 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:16:16.097 INFO 2981023 --- [host-check-agent-status3,0c3499cb78caffe8,5b10a98354addf58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:16:16.098 INFO 2981023 --- [host-check-agent-status2,0c3499cb78caffe8,8ad325f6757f0f59] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:16:16.102 INFO 2981023 --- [host-check-agent-status1,0c3499cb78caffe8,e8354e04fe93e4f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:16:16.159 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:16:16.184 INFO 2981023 --- [host-check-agent-status4,0c3499cb78caffe8,bffa2178340b29f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:16:16.191 INFO 2981023 --- [ocp-schedules-18,eebee2ca197a5a99,fd31d7dec504d24a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:16:16.203 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:16:16.203 INFO 2981023 --- [ocp-schedules-17,d7296ea14865f856,54579bc35b4bf3aa] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:16:16.203 INFO 2981023 --- [ocp-schedules-18,eebee2ca197a5a99,fd31d7dec504d24a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:16:16.203 INFO 2981023 --- [ocp-schedules-13,2e3b9ba1cca6f354,b84a43446d634f81] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:16:16.203 INFO 2981023 --- [ocp-schedules-22,a85fa6f6a5ee7db7,b56df634dbd66013] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:16:16.206 INFO 2981023 --- [host-check-agent-status4,0c3499cb78caffe8,bffa2178340b29f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:16:16.227 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049763011000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:16:16.257 INFO 2981023 --- [ocp-schedules-13,2e3b9ba1cca6f354,b84a43446d634f81] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:16:16.258 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:16:16.264 INFO 2981023 --- [ocp-schedules-17,d7296ea14865f856,54579bc35b4bf3aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:16:16.300 INFO 2981023 --- [ocp-schedules-17,d7296ea14865f856,54579bc35b4bf3aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:16:16.308 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.308 INFO 2981023 --- [ocp-schedules-27,78e31436a7184db9,071d0ccc13198328] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:16:16.309 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:16:16.314 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:16:16.324 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:16:16.325 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:16:16.328 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:16:16.329 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:16:16.333947538+08:00 2024-09-11 18:16:16.329 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:16:16.330 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:16:16.329+08:00 2024-09-11 18:16:16.331 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:16:16.331475920+08:00 2024-09-11 18:16:16.331 INFO 2981023 --- [host-check-agent-status1,0c3499cb78caffe8,e8354e04fe93e4f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:16:16.333 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:16:16.338045092+08:00 2024-09-11 18:16:16.333 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:16:16.331+08:00 2024-09-11 18:16:16.333 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:16:16.329+08:00 and status AVAILABLE 2024-09-11 18:16:16.334 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:16:16.331402756+08:00 2024-09-11 18:16:16.336 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:16:16.333+08:00 2024-09-11 18:16:16.337 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:16:16.331+08:00 and status AVAILABLE 2024-09-11 18:16:16.338 INFO 2981023 --- [host-check-agent-status1,0c3499cb78caffe8,e8354e04fe93e4f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:16:16.338 INFO 2981023 --- [host-check-agent-status1,0c3499cb78caffe8,e8354e04fe93e4f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:16:16.338 INFO 2981023 --- [host-check-agent-status1,0c3499cb78caffe8,e8354e04fe93e4f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:16:16.338 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:16:16.334+08:00 2024-09-11 18:16:16.339 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:16:16.333+08:00 and status AVAILABLE 2024-09-11 18:16:16.341 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:16:16.334+08:00 and status AVAILABLE 2024-09-11 18:16:16.342 INFO 2981023 --- [host-check-agent-status3,0c3499cb78caffe8,5b10a98354addf58] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:16:16.344 INFO 2981023 --- [host-check-agent-status3,0c3499cb78caffe8,5b10a98354addf58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:16:16.344 INFO 2981023 --- [host-check-agent-status3,0c3499cb78caffe8,5b10a98354addf58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:16:16.344 INFO 2981023 --- [host-check-agent-status3,0c3499cb78caffe8,5b10a98354addf58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:16:16.345 INFO 2981023 --- [host-check-agent-status4,0c3499cb78caffe8,bffa2178340b29f6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:16:16.347 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.348 INFO 2981023 --- [host-check-agent-status2,0c3499cb78caffe8,8ad325f6757f0f59] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:16:16.348 INFO 2981023 --- [host-check-agent-status4,0c3499cb78caffe8,bffa2178340b29f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:16:16.348 INFO 2981023 --- [host-check-agent-status4,0c3499cb78caffe8,bffa2178340b29f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:16:16.348 INFO 2981023 --- [host-check-agent-status4,0c3499cb78caffe8,bffa2178340b29f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:16:16.350 INFO 2981023 --- [host-check-agent-status2,0c3499cb78caffe8,8ad325f6757f0f59] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:16:16.350 INFO 2981023 --- [host-check-agent-status2,0c3499cb78caffe8,8ad325f6757f0f59] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:16:16.350 INFO 2981023 --- [host-check-agent-status2,0c3499cb78caffe8,8ad325f6757f0f59] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:16:16.350 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:16:16.350 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:16:16.349161+08:00, lastProcessTime=2024-09-11T18:16:06.337236+08:00 2024-09-11 18:16:16.351 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:16:16.351 INFO 2981023 --- [ocp-schedules-18,eebee2ca197a5a99,fd31d7dec504d24a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:16:16.358 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.359 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:16:16.360 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:16.361 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:16:16.361 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:16:16.368 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:16:16.397 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:16:16.397 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:16:16.399 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:16:16.399 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:16:16.399 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:16:16.400 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:16:16.401 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:16:16.404 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:16:16.405 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.406 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:16.408 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.410 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:16.426 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:16:16.430 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:16:16.431 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:16:16.432 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:16.433 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.433 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:16:16.440 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:16:16.456 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:16:16.457 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:16:16.465 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:16:16.469 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:16:16.485 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:16:16.496 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:16:16.498 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:16:16.498 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049763011000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:16:16.500 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:16:16.503 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:16:16.506 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:16.508 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:16.511 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:16:16.512 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:16:16.516 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:16:16.523 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:16:16.530 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:16:16.531 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:16:16.535 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:16:16.540 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:16:16.545 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:16:16.556 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:16:16.558 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:16:16.560 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:16:16.562 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:16:16.577 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:16:16.580 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:16:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:16:16.587 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:16:16.592 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:16:16.595 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:16:16.597 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:16:16.597 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:16:16.604 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:16:16.605 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.606 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:16:16.606 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:16:16.610 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:16:16.611 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:16:16.616 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:16:16.618 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:16:16.627 INFO 2981023 --- [host-check-host-status4,acb2bfb45cdb0248,015900b616ee0d27] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:16:16.632 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:16:16.633 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:16.635 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:16:16.636 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.640 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:16.642 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:16:16.643 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.644 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:16:16.646 INFO 2981023 --- [host-check-host-status1,acb2bfb45cdb0248,e551859d4e5464bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:16:16.646 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:16:16.647 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:16:16.660 INFO 2981023 --- [host-check-host-status2,acb2bfb45cdb0248,e01973c951fab815] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:16:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:16:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:16:16.668 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:16:16.691 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:16:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:16.704 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:16:16.704 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.705 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:16:16.705 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:16:16.731 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:16:16.731 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:16:16.735 INFO 2981023 --- [host-check-host-status3,acb2bfb45cdb0248,eb5c3c5409bf4bce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:16:16.737 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:16:16.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:16:16.742 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:16.743 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.744 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:16:16.753 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:16:16.753 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:16:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:16:16.756 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:16:16.349161+08:00 2024-09-11 18:16:16.782 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:16:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:16.788 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:16.789 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:16:16.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:16:16.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:16:16.794 INFO 2981023 --- [ocp-schedules-13,2e3b9ba1cca6f354,b84a43446d634f81] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:16:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,a136df30d86241aa,d46d8f81d795dcf8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,a136df30d86241aa,d46d8f81d795dcf8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:17.424 INFO 2981023 --- [ocp-async-3,88e1a0c3fc0bc70c,4f0b3369cb5e8c03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:17.788 INFO 2981023 --- [ocp-async-10,d3bc2efd66982cb3,bf85b03a31fa55c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:16:18.097 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:18.098 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:18.255 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049700, nodeCount=480, validCount=480 2024-09-11 18:16:18.550 INFO 2981023 --- [host-check-clockdiff1,194d944eea586f30,4952e84b4582a357] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049778 5 5 , error= 2024-09-11 18:16:19.424 INFO 2981023 --- [ocp-async-9,1876efe61e166419,b7534ddecab50896] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,85fec224cbc41843,d504843485b2c592] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,85fec224cbc41843,d504843485b2c592] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:21.098 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:21.100 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:21.425 INFO 2981023 --- [ocp-async-10,8dca62aad15e144d,8a18f583152d0901] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:22.041 INFO 2981023 --- [host-check-clockdiff4,194d944eea586f30,8ef0570a5e1bf95d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049782 0 0 , error= 2024-09-11 18:16:22.658 INFO 2981023 --- [ocp-schedules-25,f9f2db87dc64066e,67ff5a1192779dd0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:16:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,8d4b8e8993c48e11,d9d78b65a6e7a600] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,8d4b8e8993c48e11,d9d78b65a6e7a600] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:23.425 INFO 2981023 --- [ocp-async-7,485be144b81bf12d,f9b4f63c013eae84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:24.100 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:24.101 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:24.447 INFO 2981023 --- [ocp-schedules-6,4f6d118d387c7cb9,9adb6439addc8986] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:16:25.425 INFO 2981023 --- [ocp-async-8,64604f2bbdb1efba,aedb599fe825717e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:25.922 INFO 2981023 --- [ocp-schedules-6,291150984341d284,620fa9a25ae9562b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:16:25.988 WARN 2981023 --- [scanner-13637962-1,291150984341d284,f9025765454ee920] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:16:26.000 INFO 2981023 --- [alarm-detect-6,291150984341d284,c2492f4251862623] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645790, data is empty. 2024-09-11 18:16:26.003 INFO 2981023 --- [alarm-detect-6,291150984341d284,c2492f4251862623] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645790, data is empty. 2024-09-11 18:16:26.005 INFO 2981023 --- [alarm-detect-6,291150984341d284,c2492f4251862623] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645790, data is empty. 2024-09-11 18:16:26.020 INFO 2981023 --- [alarm-detect-11,291150984341d284,6d3e8596730311a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645790, data is empty. 2024-09-11 18:16:26.021 INFO 2981023 --- [alarm-detect-9,291150984341d284,1661b4e1b2ae8143] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645790, data is empty. 2024-09-11 18:16:26.023 INFO 2981023 --- [alarm-detect-11,291150984341d284,6d3e8596730311a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645790, data is empty. 2024-09-11 18:16:26.026 INFO 2981023 --- [alarm-detect-9,291150984341d284,1661b4e1b2ae8143] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645790, data is empty. 2024-09-11 18:16:26.027 INFO 2981023 --- [alarm-detect-11,291150984341d284,6d3e8596730311a8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645790, data is empty. 2024-09-11 18:16:26.029 INFO 2981023 --- [alarm-detect-9,291150984341d284,1661b4e1b2ae8143] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645790, data is empty. 2024-09-11 18:16:26.148 INFO 2981023 --- [alarm-detect-20,291150984341d284,98ec2e739f1366d9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645790, data is empty. 2024-09-11 18:16:26.153 INFO 2981023 --- [alarm-detect-20,291150984341d284,98ec2e739f1366d9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645790, data is empty. 2024-09-11 18:16:26.156 INFO 2981023 --- [alarm-detect-20,291150984341d284,98ec2e739f1366d9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645790, data is empty. 2024-09-11 18:16:26.190 INFO 2981023 --- [ocp-schedules-23,86a6796b42e4beba,e9c5b0932fb68ce2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:16:26.190 INFO 2981023 --- [ocp-schedules-23,86a6796b42e4beba,e9c5b0932fb68ce2] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:16:26.190 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:16:26.190 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:16:26.190 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:16:26.216 INFO 2981023 --- [ocp-schedules-23,86a6796b42e4beba,e9c5b0932fb68ce2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:16:26.218 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:16:26.222 INFO 2981023 --- [ocp-schedules-23,86a6796b42e4beba,e9c5b0932fb68ce2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:16:26.223 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:16:26.223 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:16:26.224 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:16:26.225 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:16:26.226 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:16:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,ae1e73fde5f37129,56c5d6b9e90232ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:26.363 INFO 2981023 --- [prometheus-minute-schedule-1,ae1e73fde5f37129,56c5d6b9e90232ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:26.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:16:26.462543+08:00, lastProcessTime=2024-09-11T18:16:16.349161+08:00 2024-09-11 18:16:26.477 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:16:26.478 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:16:26.488 INFO 2981023 --- [ocp-schedules-23,86a6796b42e4beba,e9c5b0932fb68ce2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:16:26.495 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:16:26.497 INFO 2981023 --- [ocp-schedules-23,86a6796b42e4beba,e9c5b0932fb68ce2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:16:26.609 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:16:26.462543+08:00 2024-09-11 18:16:27.102 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:27.102 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:27.425 INFO 2981023 --- [ocp-async-5,04f6a5403a09b99a,ac5525044f4ddb76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:27.788 INFO 2981023 --- [ocp-async-6,59b1be6bff7bf8ee,00444c37ed4ff103] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:16:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,20e77f5f4acf85d1,85550d41882917e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,20e77f5f4acf85d1,85550d41882917e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:29.425 INFO 2981023 --- [ocp-async-3,3a5104b231775cb4,5494b1bb744321fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:30.105 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:30.106 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:31.425 INFO 2981023 --- [ocp-async-6,483f1dcf51454996,fb9f684aa2bac56f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,664e12a1791da5f4,85d9573ad3f841de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,664e12a1791da5f4,85d9573ad3f841de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:32.661 INFO 2981023 --- [ocp-schedules-25,e557a48990e55c57,b8816cef556d167f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:16:33.106 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:33.106 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:33.425 INFO 2981023 --- [ocp-async-2,f13835e55e5527c3,746fd375bf24d765] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,1ab6f174aaf5de90,3476013b80e8134e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,1ab6f174aaf5de90,3476013b80e8134e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:35.425 INFO 2981023 --- [ocp-async-1,90fd3cd01834246e,fedcc89a2ef9f4d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:35.463 INFO 2981023 --- [ocp-schedules-29,a62043d6b499c529,99a219241f2169f6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:16:35.472 INFO 2981023 --- [ocp-schedules-29,a62043d6b499c529,99a219241f2169f6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:16:35.472 INFO 2981023 --- [ocp-schedules-29,a62043d6b499c529,99a219241f2169f6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:16:36.106 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:36.107 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:16:36.337099+08:00, lastProcessTime=2024-09-11T18:16:26.462543+08:00 2024-09-11 18:16:36.392 INFO 2981023 --- [ocp-schedules-5,47659a66d6609ad6,7148fddcb988adf9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:16:36.435 WARN 2981023 --- [scanner-13638273-1,47659a66d6609ad6,3b8c9ba30f5ef302] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:16:36.544 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:16:36.337099+08:00 2024-09-11 18:16:37.314 INFO 2981023 --- [prometheus-cache-monitor1,a039b17817c72b57,289d550cda363458] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2050,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:16:37.315 INFO 2981023 --- [prometheus-cache-monitor1,a039b17817c72b57,289d550cda363458] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:16:37.426 INFO 2981023 --- [ocp-async-4,d27763c218c53b90,b8c8facf1af66a17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:37.788 INFO 2981023 --- [ocp-async-7,f0f48a77b07e6020,5cbf0ed3e81f7897] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:16:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,1e7b004334bc065c,0e42c849e43f0615] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,1e7b004334bc065c,0e42c849e43f0615] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:39.107 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:39.107 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:39.426 INFO 2981023 --- [ocp-async-5,9578312ef3380d25,aab633707b9399be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:40.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:16:40.003 INFO 2981023 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:16:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,01cfe3ebe021059f,6bd93a40afd9f65a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,01cfe3ebe021059f,6bd93a40afd9f65a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:41.426 INFO 2981023 --- [ocp-async-7,92450d1640be03b0,c34588654aab9a99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:42.108 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:42.108 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:42.663 INFO 2981023 --- [ocp-schedules-27,fa11ec848a755e8f,a4af79f8e7caa868] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:16:43.426 INFO 2981023 --- [ocp-async-8,6943cf7d3b5d5d54,4eaad28e750dffcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,a3ad8b8fb4e0c620,010fc82644e7208e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:44.317 INFO 2981023 --- [prometheus-minute-schedule-1,a3ad8b8fb4e0c620,010fc82644e7208e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:45.101 INFO 2981023 --- [prometheus-second-schedule-1,b4cb385c3cb4a1a1,b47f56760c12aa6d] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:16:45.108 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:45.109 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:45.426 INFO 2981023 --- [ocp-async-9,e71bb716a1f549a1,37a54575ac6f53fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:46.027 INFO 2981023 --- [ocp-schedules-19,433ca517cb3503fe,507a832db37880bf] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:16:46.028 INFO 2981023 --- [ocp-schedules-12,f9d9b5fd37b0eed3,2a0adabd6618c3c3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:16:46.028 INFO 2981023 --- [ocp-schedules-4,aa9574fa01b6bfcc,2deea5334063b288] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:16:46.028 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:16:46.028 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:16:46.029 INFO 2981023 --- [host-check-agent-status1,433ca517cb3503fe,ae7838bc89bba783] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:16:46.029 INFO 2981023 --- [host-check-agent-status2,433ca517cb3503fe,ba6761d1626e6733] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:16:46.029 INFO 2981023 --- [host-check-agent-status3,433ca517cb3503fe,28a128a56d30f779] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:16:46.031 INFO 2981023 --- [host-check-agent-status4,433ca517cb3503fe,eed5d26c595c5174] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:16:46.033 INFO 2981023 --- [host-check-clockdiff1,aa9574fa01b6bfcc,d5138958a6829bdb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:16:46.033 INFO 2981023 --- [host-check-clockdiff2,aa9574fa01b6bfcc,c7c31d686eed2828] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:16:46.034 INFO 2981023 --- [host-check-clockdiff3,aa9574fa01b6bfcc,7d1e5dec291faa10] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:16:46.035 INFO 2981023 --- [host-check-agent-status1,433ca517cb3503fe,ae7838bc89bba783] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:16:46.037 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:16:46.037 INFO 2981023 --- [host-check-agent-status1,433ca517cb3503fe,ae7838bc89bba783] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:16:46.038 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:16:46.038 INFO 2981023 --- [host-check-agent-status2,433ca517cb3503fe,ba6761d1626e6733] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:16:46.039 INFO 2981023 --- [host-check-clockdiff4,aa9574fa01b6bfcc,fbefb1393c3676d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:16:46.039 INFO 2981023 --- [host-check-agent-status2,433ca517cb3503fe,ba6761d1626e6733] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:16:46.039 INFO 2981023 --- [host-check-agent-status3,433ca517cb3503fe,28a128a56d30f779] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:16:46.040 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:16:46.040 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:16:46.040 INFO 2981023 --- [host-check-agent-status4,433ca517cb3503fe,eed5d26c595c5174] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:16:46.041 INFO 2981023 --- [host-check-agent-status3,433ca517cb3503fe,28a128a56d30f779] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:16:46.042 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:16:46.042 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:16:46.048 INFO 2981023 --- [host-check-clockdiff3,aa9574fa01b6bfcc,7d1e5dec291faa10] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049806 0 0 , error= 2024-09-11 18:16:46.053 INFO 2981023 --- [host-check-agent-status4,433ca517cb3503fe,eed5d26c595c5174] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:16:46.067 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:16:46.068 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:16:46.077 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:16:46.078 INFO 2981023 --- [host-check-agent-status2,433ca517cb3503fe,ba6761d1626e6733] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:16:46.081 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:16:46.085 INFO 2981023 --- [host-check-agent-status1,433ca517cb3503fe,ae7838bc89bba783] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:16:46.089 INFO 2981023 --- [host-check-agent-status1,433ca517cb3503fe,ae7838bc89bba783] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:16:46.089 INFO 2981023 --- [host-check-agent-status1,433ca517cb3503fe,ae7838bc89bba783] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:16:46.089 INFO 2981023 --- [host-check-agent-status1,433ca517cb3503fe,ae7838bc89bba783] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:16:46.093 INFO 2981023 --- [host-check-agent-status2,433ca517cb3503fe,ba6761d1626e6733] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:16:46.093 INFO 2981023 --- [host-check-agent-status2,433ca517cb3503fe,ba6761d1626e6733] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:16:46.093 INFO 2981023 --- [host-check-agent-status2,433ca517cb3503fe,ba6761d1626e6733] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:16:46.093 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:16:46.096 INFO 2981023 --- [host-check-agent-status4,433ca517cb3503fe,eed5d26c595c5174] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:16:46.100 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:16:46.099523385+08:00 2024-09-11 18:16:46.102 INFO 2981023 --- [host-check-agent-status4,433ca517cb3503fe,eed5d26c595c5174] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:16:46.102 INFO 2981023 --- [host-check-agent-status4,433ca517cb3503fe,eed5d26c595c5174] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:16:46.102 INFO 2981023 --- [host-check-agent-status4,433ca517cb3503fe,eed5d26c595c5174] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:16:46.105 INFO 2981023 --- [ocp-schedules-14,6e16c570700e57e0,0dc40a9e4004415f] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:16:46.109 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:16:46.100+08:00 2024-09-11 18:16:46.116 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:16:46.100+08:00 and status AVAILABLE 2024-09-11 18:16:46.124 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:16:46.131 WARN 2981023 --- [ocp-schedules-14,6e16c570700e57e0,0dc40a9e4004415f] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:16:46.158 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:16:46.176 INFO 2981023 --- [ocp-schedules-3,2489c6fa5cf85589,7109895b419054b8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:16:46.191 INFO 2981023 --- [ocp-schedules-30,fb1e69678129fe1e,1c7560199a411e43] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:16:46.191 INFO 2981023 --- [ocp-schedules-7,1b58339a6a50f55a,1c93ab5b254818e8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:16:46.202 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:16:46.202 INFO 2981023 --- [ocp-schedules-21,e25603adc1063c6f,448133fc4c391779] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:16:46.202 INFO 2981023 --- [ocp-schedules-29,a4d00becbb941e72,82146b8b90cd87fe] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:16:46.203 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:16:46.208065856+08:00 2024-09-11 18:16:46.203 INFO 2981023 --- [ocp-schedules-7,1b58339a6a50f55a,1c93ab5b254818e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:16:46.203 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:16:46.205 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:16:46.207 INFO 2981023 --- [ocp-schedules-30,fb1e69678129fe1e,1c7560199a411e43] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:16:46.208 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:16:46.203+08:00 2024-09-11 18:16:46.209 INFO 2981023 --- [host-check-agent-status3,433ca517cb3503fe,28a128a56d30f779] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:16:46.211 INFO 2981023 --- [ocp-schedules-21,e25603adc1063c6f,448133fc4c391779] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:16:46.217 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:16:46.203+08:00 and status AVAILABLE 2024-09-11 18:16:46.219 INFO 2981023 --- [host-check-agent-status3,433ca517cb3503fe,28a128a56d30f779] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:16:46.219 INFO 2981023 --- [host-check-agent-status3,433ca517cb3503fe,28a128a56d30f779] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:16:46.219 INFO 2981023 --- [host-check-agent-status3,433ca517cb3503fe,28a128a56d30f779] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:16:46.219 INFO 2981023 --- [ocp-schedules-3,2489c6fa5cf85589,7109895b419054b8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:16:46.228 INFO 2981023 --- [ocp-schedules-8,6ae27691b013284f,f3fbd0dceeecb79d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:16:46.232 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:16:46.233 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:16:46.232956869+08:00 2024-09-11 18:16:46.233 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:16:46.233+08:00 2024-09-11 18:16:46.235 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:16:46.233+08:00 and status AVAILABLE 2024-09-11 18:16:46.244 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:16:46.244 INFO 2981023 --- [ocp-schedules-17,3f7db5e38c2228f8,f89d8ab0c04c6158] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 18:16:46.248 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:16:46.248 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:16:46.252 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:16:46.249408537+08:00 2024-09-11 18:16:46.256 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:16:46.252+08:00 2024-09-11 18:16:46.257 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:16:46.257 INFO 2981023 --- [ocp-schedules-17,3f7db5e38c2228f8,f89d8ab0c04c6158] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:16:46.267 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:16:46.252+08:00 and status AVAILABLE 2024-09-11 18:16:46.267 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049763011000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:16:46.268 INFO 2981023 --- [ocp-schedules-28,e74daa4481a3a07f,4f001594ce6b55a8] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 18:16:46.268 INFO 2981023 --- [ocp-schedules-28,e74daa4481a3a07f,4f001594ce6b55a8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 18:16:46.272 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:16:46.277 INFO 2981023 --- [ocp-schedules-30,fb1e69678129fe1e,1c7560199a411e43] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:16:46.283 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:16:46.284 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:16:46.287 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:16:46.291 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:16:46.295 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:16:46.298 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:16:46.299 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.299 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:16:46.308 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:16:46.309 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.309 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:16:46.310 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:16:46.320 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ad87dabb892dfa60,3ce81a1c99482115] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=ad87dabb892dfa60, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 18:16:46.320 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,ad87dabb892dfa60,3ce81a1c99482115] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=ad87dabb892dfa60, duration=0 ms] 2024-09-11 18:16:46.321 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:16:46.329 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.329 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:16:46.330 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:16:46.335 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.338 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 18:16:46.342 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.342 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.342 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:46.343 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:16:46.343 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:16:46.346 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.347 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:16:46.352 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c55e797e8fc79f77,2f284373b08550bd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=c55e797e8fc79f77, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 18:16:46.352 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,c55e797e8fc79f77,2f284373b08550bd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=c55e797e8fc79f77, duration=0 ms] 2024-09-11 18:16:46.354 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:16:46.354 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.354 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.355 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:16:46.355 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:46.359 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.359 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:16:46.360 INFO 2981023 --- [ocp-schedules-7,1b58339a6a50f55a,1c93ab5b254818e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:16:46.360 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:16:46.360 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.360 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.361 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:16:46.363 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 18:16:46.367 INFO 2981023 --- [ocp-schedules-17,3f7db5e38c2228f8,f89d8ab0c04c6158] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 18:16:46.371 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:16:46.373 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:16:46.372469+08:00, lastProcessTime=2024-09-11T18:16:36.337099+08:00 2024-09-11 18:16:46.374 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:16:46.375 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:16:46.390 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.392 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:46.396 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:16:46.397 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:16:46.402 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:16:46.404 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:16:46.418 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:16:46.420 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:16:46.431 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:16:46.438 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:16:46.444 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.446 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:16:46.447 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.449 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:16:46.453 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:16:46.455 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:16:46.468 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.469 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.470 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:16:46.470 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:16:46.470 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.473 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:16:46.480 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:16:46.480 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.482 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:16:46.483 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:16:46.483 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:16:46.483 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049763011000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:16:46.484 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.484 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:46.487 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:16:46.493 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:16:46.503 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:16:46.510 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:16:46.511 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:16:46.512 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:16:46.522 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:16:46.525 INFO 2981023 --- [pool-10-thread-7,472da0ed9ba9ca58,87c841c10ec3bb52] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 18:16:46.527 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:16:46.532 INFO 2981023 --- [host-check-host-status1,f9d9b5fd37b0eed3,e779b97a55834da7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:16:46.535 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:16:46.535 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:16:46.544 INFO 2981023 --- [pool-10-thread-6,472da0ed9ba9ca58,e8d9258de077d7cf] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 18:16:46.549 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:16:46.550 INFO 2981023 --- [pool-10-thread-7,472da0ed9ba9ca58,87c841c10ec3bb52] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 18:16:46.550 INFO 2981023 --- [pool-10-thread-7,472da0ed9ba9ca58,87c841c10ec3bb52] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 18:16:46.553 INFO 2981023 --- [pool-10-thread-5,472da0ed9ba9ca58,cfdb72b697ac1ee3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 18:16:46.553 INFO 2981023 --- [pool-10-thread-5,472da0ed9ba9ca58,cfdb72b697ac1ee3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 18:16:46.555 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 18:16:46.555 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 18:16:46.561 INFO 2981023 --- [ForkJoinPool-6-worker-2,472da0ed9ba9ca58,ffcab985e93e03da] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 18:16:46.561 INFO 2981023 --- [ForkJoinPool-6-worker-2,472da0ed9ba9ca58,ffcab985e93e03da] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 18:16:46.562 INFO 2981023 --- [pool-10-thread-8,472da0ed9ba9ca58,17ccb37f0a42344e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 18:16:46.563 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:16:46.568 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 18:16:46.568 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 18:16:46.578 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.588 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:16:46.592 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:16:46.603 INFO 2981023 --- [ocp-schedules-26,73cbdd8e783a7722,662ddbba9e1e5537] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:16:46.606 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.606 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.607 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:16:46.611 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:16:46.612 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.612 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:16:46.613 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:16:46.613 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.619 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:16:46.620 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:16:46.621 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:16:46.626 WARN 2981023 --- [ForkJoinPool-6-worker-2,472da0ed9ba9ca58,ffcab985e93e03da] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 18:16:46.637 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.642 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:16:46.646 INFO 2981023 --- [host-check-host-status2,f9d9b5fd37b0eed3,8f8b87bcc2511063] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:16:46.649 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.651 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.651 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:16:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:16:46.657 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.658 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:16:46.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:16:46.659 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:16:46.664 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.665 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:16:46.666 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:16:46.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:16:46.668 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.669 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:16:46.670 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.670 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:16:46.671 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 18:16:46.674 INFO 2981023 --- [host-check-host-status4,f9d9b5fd37b0eed3,df82f23d7e1d2cf6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:16:46.681 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.683 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:16:46.692 INFO 2981023 --- [ForkJoinPool-6-worker-6,472da0ed9ba9ca58,480492e95afeae37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.693 INFO 2981023 --- [host-check-host-status3,f9d9b5fd37b0eed3,36025e9821b6d25b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:16:46.694 INFO 2981023 --- [ForkJoinPool-6-worker-6,472da0ed9ba9ca58,480492e95afeae37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:16:46.695 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:16:46.695 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.696 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:16:46.696 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.697 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:16:46.697 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:16:46.714 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.715 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:16:46.716 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.717 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:16:46.718 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:16:46.719 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 18:16:46.728 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:16:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.730 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.731 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:16:46.733 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.734 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:16:46.746 INFO 2981023 --- [ForkJoinPool-6-worker-6,472da0ed9ba9ca58,480492e95afeae37] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:16:46.747 INFO 2981023 --- [ForkJoinPool-6-worker-6,472da0ed9ba9ca58,480492e95afeae37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.747 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:16:46.749 INFO 2981023 --- [ForkJoinPool-6-worker-6,472da0ed9ba9ca58,480492e95afeae37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:16:46.750 INFO 2981023 --- [ForkJoinPool-6-worker-6,472da0ed9ba9ca58,480492e95afeae37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.751 INFO 2981023 --- [ForkJoinPool-6-worker-6,472da0ed9ba9ca58,480492e95afeae37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:16:46.752 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:16:46.765 WARN 2981023 --- [scanner-13638502-1,73cbdd8e783a7722,b8e3573eb5d58fe0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:16:46.793 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.793 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.793 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.794 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:16:46.794 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:16:46.794 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.798 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:16:46.798 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 18:16:46.799 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:16:46.809 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:16:46.812 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.813 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.814 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:16:46, 2024-09-11 18:16:46] 2024-09-11 18:16:46.816 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:16:46, 2024-09-11 18:16:46] 2024-09-11 18:16:46.818 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 18:16:46.818 INFO 2981023 --- [ForkJoinPool-6-worker-11,472da0ed9ba9ca58,bddcda15b549d9da] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 18:16:46.822 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.824 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:16:46.828 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:16:46.828 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:16:46.832 INFO 2981023 --- [alarm-detect-15,73cbdd8e783a7722,ceb0ca62f67fca49] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645790, data is empty. 2024-09-11 18:16:46.846 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:16:46.847 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:16:46.848 INFO 2981023 --- [alarm-detect-15,73cbdd8e783a7722,ceb0ca62f67fca49] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645790, data is empty. 2024-09-11 18:16:46.848 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.851 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:16:46, 2024-09-11 18:16:46] 2024-09-11 18:16:46.853 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 18:16:46.853 INFO 2981023 --- [ForkJoinPool-6-worker-9,472da0ed9ba9ca58,c871653e8c2f457c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 18:16:46.857 INFO 2981023 --- [alarm-detect-15,73cbdd8e783a7722,ceb0ca62f67fca49] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645790, data is empty. 2024-09-11 18:16:46.863 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:16:46.865 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.866 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:16:46.867 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.867 INFO 2981023 --- [ForkJoinPool-6-worker-4,472da0ed9ba9ca58,630f7654410389dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:16:46.876 INFO 2981023 --- [pool-10-thread-6,472da0ed9ba9ca58,e8d9258de077d7cf] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 18:16:46.877 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:16:46.885 INFO 2981023 --- [alarm-detect-18,73cbdd8e783a7722,836944afbbd05992] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645790, data is empty. 2024-09-11 18:16:46.890 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:16:46.891 INFO 2981023 --- [alarm-detect-18,73cbdd8e783a7722,836944afbbd05992] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645790, data is empty. 2024-09-11 18:16:46.892 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.893 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.893 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:16:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:16:46.895 INFO 2981023 --- [alarm-detect-18,73cbdd8e783a7722,836944afbbd05992] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645790, data is empty. 2024-09-11 18:16:46.901 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:16:46.902 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.904 INFO 2981023 --- [alarm-detect-16,73cbdd8e783a7722,f0e870fa6785d337] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645790, data is empty. 2024-09-11 18:16:46.905 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.905 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:16:46.909 INFO 2981023 --- [alarm-detect-16,73cbdd8e783a7722,f0e870fa6785d337] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645790, data is empty. 2024-09-11 18:16:46.911 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:16:46.913 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:16:46.915 INFO 2981023 --- [alarm-detect-16,73cbdd8e783a7722,f0e870fa6785d337] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645790, data is empty. 2024-09-11 18:16:46.920 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:16:46.920 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:16:46.933 INFO 2981023 --- [pool-10-thread-8,472da0ed9ba9ca58,17ccb37f0a42344e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 18:16:46.948 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:16:46.960 INFO 2981023 --- [pool-10-thread-5,472da0ed9ba9ca58,cfdb72b697ac1ee3] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 18:16:46.963 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.964 INFO 2981023 --- [pool-10-thread-5,472da0ed9ba9ca58,cfdb72b697ac1ee3] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 18:16:46.967 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.967 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.969 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.969 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:16:46.970 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:16:46.970 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.972 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 18:16:46.978 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:16:46.991 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:16:46.992 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:46.992 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:16:47.000 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:16:47.000 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:16:47.022 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:16:47.030 INFO 2981023 --- [ocp-schedules-21,e25603adc1063c6f,448133fc4c391779] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:16:47.050 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:16:46.372469+08:00 2024-09-11 18:16:47.061 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:16:47.072 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:16:47.079 INFO 2981023 --- [host-check-clockdiff2,aa9574fa01b6bfcc,c7c31d686eed2828] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049806 6 6 , error= 2024-09-11 18:16:47.109 INFO 2981023 --- [ocp-schedules-27,9afd2fea66a68664,003f535b95386113] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:16:47.114 INFO 2981023 --- [alarm-detect-18,73cbdd8e783a7722,f49c970600d32ab9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645790, data is empty. 2024-09-11 18:16:47.118 INFO 2981023 --- [alarm-detect-18,73cbdd8e783a7722,f49c970600d32ab9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645790, data is empty. 2024-09-11 18:16:47.124 INFO 2981023 --- [alarm-detect-18,73cbdd8e783a7722,f49c970600d32ab9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645790, data is empty. 2024-09-11 18:16:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,3e9e18f3c0247d34,842ff7a427d7eed5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,3e9e18f3c0247d34,842ff7a427d7eed5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:47.426 INFO 2981023 --- [ocp-async-7,98eb8a771d40b2a7,a97b60c52964c00a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:47.788 INFO 2981023 --- [ocp-async-2,64cad920a75b70f6,5391d2072c7e9b0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:16:48.109 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:48.109 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:48.560 INFO 2981023 --- [host-check-clockdiff1,aa9574fa01b6bfcc,d5138958a6829bdb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049808 5 5 , error= 2024-09-11 18:16:49.426 INFO 2981023 --- [ocp-async-6,fec42865f518d423,ee099241f7bed94e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:16:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:16:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,db3b57a39aace9b5,4423accb54b68780] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,db3b57a39aace9b5,4423accb54b68780] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:50.346 INFO 2981023 --- [ocp-schedules-18,0621c7e14cf6fb1a,082a7bb508443050] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:16:51.109 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:51.110 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:51.426 INFO 2981023 --- [ocp-async-9,9cd8f03b212f4ad5,da138c6426267d15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:51.620 INFO 2981023 --- [ocp-schedules-24,9f1b8e803257daf3,f24a87b3e7a19963] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:16:51.622 INFO 2981023 --- [ocp-schedules-24,9f1b8e803257daf3,f24a87b3e7a19963] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:16:51.622 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:16:51.622 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:16:51.622 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:16:51.622 INFO 2981023 --- [ocp-schedules-24,9f1b8e803257daf3,f24a87b3e7a19963] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:16:52.043 INFO 2981023 --- [host-check-clockdiff4,aa9574fa01b6bfcc,fbefb1393c3676d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049812 0 0 , error= 2024-09-11 18:16:52.666 INFO 2981023 --- [ocp-schedules-5,73802051da9c2b9e,dd30f979a268cdac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:16:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,ca24b20d4fbbeaf7,a741027e0c1dcb81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,ca24b20d4fbbeaf7,a741027e0c1dcb81] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:53.334 INFO 2981023 --- [ocp-schedules-27,ea95c2feb757ea02,ecdd8f01af46ce9b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:16:53.334 INFO 2981023 --- [ocp-schedules-27,ea95c2feb757ea02,ecdd8f01af46ce9b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:16:53.426 INFO 2981023 --- [ocp-async-3,58d960f0bfb0b954,d543c5d63204d825] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:54.114 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:54.114 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:54.449 INFO 2981023 --- [ocp-schedules-30,3ffc79efa0658779,e71006f463290397] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:16:54.894 INFO 2981023 --- [ocp-schedules-9,e7abbf5d1be9d88c,5321f7361196fc4b] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:16:54.912 WARN 2981023 --- [ocp-schedules-9,e7abbf5d1be9d88c,5321f7361196fc4b] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:16:55.427 INFO 2981023 --- [ocp-async-8,ce9a4ee0bd78ad46,d4aeae85e28a87b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:55.473 INFO 2981023 --- [ocp-schedules-16,d2a09bfb1f97dc9e,db78bfc450a9c911] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:16:55.478 INFO 2981023 --- [ocp-schedules-16,d2a09bfb1f97dc9e,db78bfc450a9c911] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:16:55.478 INFO 2981023 --- [ocp-schedules-16,d2a09bfb1f97dc9e,db78bfc450a9c911] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:16:55.880 INFO 2981023 --- [ocp-schedules-12,878fe03caf61e71a,78903255b64a287d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:16:55.880 INFO 2981023 --- [ocp-schedules-12,878fe03caf61e71a,78903255b64a287d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:16:55.881 INFO 2981023 --- [ocp-schedules-12,878fe03caf61e71a,78903255b64a287d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:16:55.881 INFO 2981023 --- [ocp-schedules-12,878fe03caf61e71a,78903255b64a287d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:16:55.881 INFO 2981023 --- [ocp-schedules-12,878fe03caf61e71a,78903255b64a287d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:16:55.881 INFO 2981023 --- [ocp-schedules-12,878fe03caf61e71a,78903255b64a287d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:16:56.076 INFO 2981023 --- [ocp-schedules-17,70ee515832255156,9c67123787a3f388] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:16:56.083 INFO 2981023 --- [ocp-schedules-17,70ee515832255156,9c67123787a3f388] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:16:56.083 INFO 2981023 --- [ocp-schedules-17,70ee515832255156,9c67123787a3f388] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:16:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,0c5d37e584029bf9,9c12bb35937c255c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,0c5d37e584029bf9,9c12bb35937c255c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:16:56.353406+08:00, lastProcessTime=2024-09-11T18:16:46.372469+08:00 2024-09-11 18:16:56.479 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:16:56.353406+08:00 2024-09-11 18:16:57.114 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:16:57.115 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:16:57.150 INFO 2981023 --- [ocp-schedules-8,3d607ef12417c5a8,a597552f06d53301] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:16:57.195 WARN 2981023 --- [scanner-13638815-1,3d607ef12417c5a8,1aab56ea9e65722b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:16:57.427 INFO 2981023 --- [ocp-async-6,4fb37547bb8af6d4,d21d147eed6e6c5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:16:57.788 INFO 2981023 --- [ocp-async-5,4728b8a1d970c7ad,a74f3c2123424b7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:16:58.145 INFO 2981023 --- [ocp-schedules-24,b7a3e45f51a60911,4cfecd9bb4c30483] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:16:58.145 INFO 2981023 --- [ocp-schedules-24,b7a3e45f51a60911,4cfecd9bb4c30483] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:16:58.160 INFO 2981023 --- [Collect all tenants major compactions1,b7a3e45f51a60911,67242fc0194e7aa4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:16:58.161 INFO 2981023 --- [Collect all tenants major compactions2,b7a3e45f51a60911,f30e5cd3ac549fb2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:16:58.162 INFO 2981023 --- [Collect all tenants major compactions3,b7a3e45f51a60911,a3faa94d233ea371] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:16:58.162 INFO 2981023 --- [Collect all tenants major compactions4,b7a3e45f51a60911,c9e12cdcea8c3562] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:16:58.162 INFO 2981023 --- [Collect all tenants major compactions5,b7a3e45f51a60911,f378805be54e014f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:16:58.164 INFO 2981023 --- [Collect all tenants major compactions6,b7a3e45f51a60911,b5078ea4d2f9205e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:16:58.172 INFO 2981023 --- [Collect all tenants major compactions1,b7a3e45f51a60911,67242fc0194e7aa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:58.173 INFO 2981023 --- [Collect all tenants major compactions2,b7a3e45f51a60911,f30e5cd3ac549fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:58.173 INFO 2981023 --- [Collect all tenants major compactions5,b7a3e45f51a60911,f378805be54e014f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:58.174 INFO 2981023 --- [Collect all tenants major compactions2,b7a3e45f51a60911,f30e5cd3ac549fb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:16:58.174 INFO 2981023 --- [Collect all tenants major compactions5,b7a3e45f51a60911,f378805be54e014f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:16:58.174 INFO 2981023 --- [Collect all tenants major compactions1,b7a3e45f51a60911,67242fc0194e7aa4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:16:58.174 INFO 2981023 --- [Collect all tenants major compactions3,b7a3e45f51a60911,a3faa94d233ea371] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:58.175 INFO 2981023 --- [Collect all tenants major compactions3,b7a3e45f51a60911,a3faa94d233ea371] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:16:58.177 INFO 2981023 --- [Collect all tenants major compactions4,b7a3e45f51a60911,c9e12cdcea8c3562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:58.178 INFO 2981023 --- [Collect all tenants major compactions5,b7a3e45f51a60911,f378805be54e014f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:16:58.178 INFO 2981023 --- [Collect all tenants major compactions5,b7a3e45f51a60911,f378805be54e014f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:16:58.178 INFO 2981023 --- [Collect all tenants major compactions2,b7a3e45f51a60911,f30e5cd3ac549fb2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:16:58.178 INFO 2981023 --- [Collect all tenants major compactions2,b7a3e45f51a60911,f30e5cd3ac549fb2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:16:58.178 INFO 2981023 --- [Collect all tenants major compactions3,b7a3e45f51a60911,a3faa94d233ea371] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:16:58.178 INFO 2981023 --- [Collect all tenants major compactions3,b7a3e45f51a60911,a3faa94d233ea371] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:16:58.178 INFO 2981023 --- [Collect all tenants major compactions1,b7a3e45f51a60911,67242fc0194e7aa4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:16:58.178 INFO 2981023 --- [Collect all tenants major compactions1,b7a3e45f51a60911,67242fc0194e7aa4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:16:58.179 INFO 2981023 --- [Collect all tenants major compactions4,b7a3e45f51a60911,c9e12cdcea8c3562] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:16:58.180 INFO 2981023 --- [Collect all tenants major compactions4,b7a3e45f51a60911,c9e12cdcea8c3562] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:16:58.180 INFO 2981023 --- [Collect all tenants major compactions4,b7a3e45f51a60911,c9e12cdcea8c3562] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:16:58.183 INFO 2981023 --- [Collect all tenants major compactions6,b7a3e45f51a60911,b5078ea4d2f9205e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:16:58.184 INFO 2981023 --- [Collect all tenants major compactions6,b7a3e45f51a60911,b5078ea4d2f9205e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:16:58.187 INFO 2981023 --- [Collect all tenants major compactions6,b7a3e45f51a60911,b5078ea4d2f9205e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:16:58.187 INFO 2981023 --- [Collect all tenants major compactions6,b7a3e45f51a60911,b5078ea4d2f9205e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:16:58.187 INFO 2981023 --- [ocp-schedules-24,b7a3e45f51a60911,4cfecd9bb4c30483] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:16:59.010 INFO 2981023 --- [ocp-schedules-3,88098157d1494fc1,80f5835ca3c89e76] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:16:59.010+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:16:59.025 INFO 2981023 --- [ocp-schedules-3,88098157d1494fc1,80f5835ca3c89e76] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:16:59.189 INFO 2981023 --- [ocp-schedules-3,88098157d1494fc1,80f5835ca3c89e76] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 47 to SqlAggregateData 2024-09-11 18:16:59.190 INFO 2981023 --- [ocp-schedules-3,88098157d1494fc1,80f5835ca3c89e76] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:16:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,2632624e7fcd8ddd,99175dcaf74deff6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049820, interval=60 2024-09-11 18:16:59.317 INFO 2981023 --- [prometheus-minute-schedule-1,2632624e7fcd8ddd,99175dcaf74deff6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:16:59.427 INFO 2981023 --- [ocp-async-7,fc5a25041f586908,bd6e4c6f7dc08025] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:00.068 INFO 2981023 --- [ocp-schedules-12,402d9c4fdd143058,c5db905d6f7da391] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:17:00.069 INFO 2981023 --- [ocp-schedules-12,402d9c4fdd143058,c5db905d6f7da391] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:17:00.109 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:17:00.109 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:17:00.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.113 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:17:00.114 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:17:00.115 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:00.117 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.118 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.118 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:00.119 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.122 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.124 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.124 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.126 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:17:00.126 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.129 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:17:00.131 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.131 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:17:00.133 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.134 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.135 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.135 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.137 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.138 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:17:00.144 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.145 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:17:00.146 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.147 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.147 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:17:00.147 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.147 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5551D6DB-0-0] 2024-09-11 18:17:00.148 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:17:00.148 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.148 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.148 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:17:00.152 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.152 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:17:00.162 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.162 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.164 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.164 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:17:00.165 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.165 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:17:00.166 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.166 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:17:00.167 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.167 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.168 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:17:00.168 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.169 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:17:00.169 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.170 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:17:00.170 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.171 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:17:00.171 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:17:00.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.173 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:17:00.173 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:17:00.174 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:17:00.177 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.177 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:17:00.178 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:17:00.178 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.179 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:17:00.180 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.180 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:17:00.181 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:17:00.183 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:17:00.187 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.187 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:17:00.188 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:17:00.188 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5551D6E5-0-0] 2024-09-11 18:17:00.188 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:17:00.190 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:00.192 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.193 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:17:00.195 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.195 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:17:00.196 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.197 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:17:00.197 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.198 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:17:00.199 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.199 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:17:00.200 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:17:00.200 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:00.200 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.201 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:17:00.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.202 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:17:00.202 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.202 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:17:00.203 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:17:00.203 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.203 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.203 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:17:00.204 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:17:00.209 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:17:00.216 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:00.217 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.218 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:17:00.219 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:17:00.220 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:17:00.221 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.221 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:17:00.221 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:17:00.221 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:17:00.221 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.227 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:17:00.229 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:17:00.229 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:17:00.229 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:17:00.229 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:17:00.230 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:17:00.231 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:00.231 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:17:00.242 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:17:00.243 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:17:00.243 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:17:00.243 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:17:00.244 INFO 2981023 --- [ocp-schedules-12,402d9c4fdd143058,c5db905d6f7da391] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:17:01.427 INFO 2981023 --- [ocp-async-5,d6b5133c3799116e,c0be197965954b7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,de9c2a6301d4ad42,411c8bfaff32d07f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,de9c2a6301d4ad42,411c8bfaff32d07f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:02.668 INFO 2981023 --- [ocp-schedules-2,ca83846af1dfaa11,85a1dd6afca26d1a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:17:03.118 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:03.119 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:03.427 INFO 2981023 --- [ocp-async-10,3c1e0aed1709eac0,17796b3bdcd36d4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,83a2151a51755a96,af518a8d15b337ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,83a2151a51755a96,af518a8d15b337ee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:05.427 INFO 2981023 --- [ocp-async-1,317d21a180092ff9,c40a361d2b3a29df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:06.119 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:06.120 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:06.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:17:06.353565+08:00, lastProcessTime=2024-09-11T18:16:56.353406+08:00 2024-09-11 18:17:06.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:17:06.353565+08:00 2024-09-11 18:17:06.604 INFO 2981023 --- [ocp-schedules-26,1bd6059f39a3d4c6,eccbf4ecf0918e51] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:17:07.333 INFO 2981023 --- [ocp-schedules-10,aea79ba7d1b441ce,7cb0011759be7acd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:17:07.401 WARN 2981023 --- [scanner-13639038-1,aea79ba7d1b441ce,cba484252d196bfb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:17:07.414 INFO 2981023 --- [alarm-detect-5,aea79ba7d1b441ce,7f0ac7a2344f59f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645790, data is empty. 2024-09-11 18:17:07.421 INFO 2981023 --- [alarm-detect-5,aea79ba7d1b441ce,7f0ac7a2344f59f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645837, data is empty. 2024-09-11 18:17:07.422 INFO 2981023 --- [alarm-detect-17,aea79ba7d1b441ce,3d8e2510f94ef8f4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645790, data is empty. 2024-09-11 18:17:07.423 INFO 2981023 --- [alarm-detect-5,aea79ba7d1b441ce,7f0ac7a2344f59f7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645837, data is empty. 2024-09-11 18:17:07.426 INFO 2981023 --- [alarm-detect-3,aea79ba7d1b441ce,b24505ca34ec5863] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645790, data is empty. 2024-09-11 18:17:07.427 INFO 2981023 --- [alarm-detect-17,aea79ba7d1b441ce,3d8e2510f94ef8f4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645837, data is empty. 2024-09-11 18:17:07.429 INFO 2981023 --- [ocp-async-4,29bd42b55bc034ce,58599b86a4cc27f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:07.430 INFO 2981023 --- [alarm-detect-3,aea79ba7d1b441ce,b24505ca34ec5863] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645837, data is empty. 2024-09-11 18:17:07.432 INFO 2981023 --- [alarm-detect-17,aea79ba7d1b441ce,3d8e2510f94ef8f4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645837, data is empty. 2024-09-11 18:17:07.436 INFO 2981023 --- [alarm-detect-3,aea79ba7d1b441ce,b24505ca34ec5863] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645837, data is empty. 2024-09-11 18:17:07.547 INFO 2981023 --- [alarm-detect-18,aea79ba7d1b441ce,12a281f1b7e30b0b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645790, data is empty. 2024-09-11 18:17:07.555 INFO 2981023 --- [alarm-detect-18,aea79ba7d1b441ce,12a281f1b7e30b0b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645837, data is empty. 2024-09-11 18:17:07.559 INFO 2981023 --- [alarm-detect-18,aea79ba7d1b441ce,12a281f1b7e30b0b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645837, data is empty. 2024-09-11 18:17:07.788 INFO 2981023 --- [ocp-async-3,61d710ff0fb058ee,f049770ced594ddb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:17:08.011 INFO 2981023 --- [ocp-schedules-17,908644c35a24c43d,b946978165296358] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:17:08.108 INFO 2981023 --- [ocp-schedules-19,60ade8c5ca49f20f,cb8458e125489ef8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:17:08.108+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:17:08.109 INFO 2981023 --- [ocp-schedules-19,60ade8c5ca49f20f,cb8458e125489ef8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:17:08.111 INFO 2981023 --- [ocp-schedules-19,60ade8c5ca49f20f,cb8458e125489ef8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:17:08.111 INFO 2981023 --- [ocp-schedules-19,60ade8c5ca49f20f,cb8458e125489ef8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:17:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,0d59dc34a0ae6caa,6157d4c86b7ce539] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,0d59dc34a0ae6caa,6157d4c86b7ce539] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:09.120 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:09.120 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:09.429 INFO 2981023 --- [ocp-async-6,88a18de657be8d56,3dafed05699e0c00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:10.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:17:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:17:10.378 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ad055f5ed8ad18ff,fbf856cc84714631] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=ad055f5ed8ad18ff, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:17:10.378 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a0d1554c86113348,53a44d1f2251ab26] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=a0d1554c86113348, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:17:10.379 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,a0d1554c86113348,53a44d1f2251ab26] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=a0d1554c86113348, duration=5 ms] 2024-09-11 18:17:10.382 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,ad055f5ed8ad18ff,fbf856cc84714631] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=ad055f5ed8ad18ff, duration=9 ms] 2024-09-11 18:17:10.389 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0142d070ee3c28c4,6b6c2e546daafd37] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=0142d070ee3c28c4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:17:10.390 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,0142d070ee3c28c4,6b6c2e546daafd37] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=0142d070ee3c28c4, duration=3 ms] 2024-09-11 18:17:10.625 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:13:10.186, end=2024-09-11T18:15:10.621} 2024-09-11 18:17:10.627 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=17 2024-09-11 18:17:10.628 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:17:10.632 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:13:10.186, end=2024-09-11T18:15:10.621},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:17:10.632 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:17:10.632 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:17:10.682 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430682000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.682Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.684 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430684000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.684Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.688 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430688000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.688Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.688 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430688000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.688Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.693 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430693000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.693Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.698 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430698000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.698Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.699 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430699000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.699Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.703 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430703000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.703Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.707 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.707 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.708 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:17:10.708 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.712 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 68 2024-09-11 18:17:10.712 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.717 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.717 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.721 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.721 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.723 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.723 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.723 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:17:10.723 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.726 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.726 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.730 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:17:10.730 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.730 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.730 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.732 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.732 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.733 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.733 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.734 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.734 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.734 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.734 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.736 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.736 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.737 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.737 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.738 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.738 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.741 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.741 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.742 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.742 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.742 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.742 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.743 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.743 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.744 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.744 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.746 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.746 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.746 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.746 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.748 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.748 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.749 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.749 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.751 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.751 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.751 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.752 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.752 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.752 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.754 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.755 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.755 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.756 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.757 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.757 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.758 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.759 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.759 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.760 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.760 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.760 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.761 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.682, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.763 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.764 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.693, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.765 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.684, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.766 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.688, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.768 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.688, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.772 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.778 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.699, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.779 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.698, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.781 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.703, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.796 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430796000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.796Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.807 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430807000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.807Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.808 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.808 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.811 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430811000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.811Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.815 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430815000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.815Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.820 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.820 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.821 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430821000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.821Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.823 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430823000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.823Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.829 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.829 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.830 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.830 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.830 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.830 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.833 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430833000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6861114A054F03969A56C5F7382C049A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.833Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.837 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:17:10.837 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.841 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:17:10.841 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.845 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.845 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.847 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.849 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.849 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.853 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.853 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.858 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.858 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.858 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.796, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.860 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430860000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.860Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.862 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.862 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.862 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.862 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.862 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.862 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.863 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.863 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.867 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.867 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.870 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.870 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.871 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.871 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.873 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.873 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6861114A054F03969A56C5F7382C049A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.873 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.873 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.876 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.876 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.878 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.878 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.880 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.880 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.888 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.888 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.891 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.891 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.891 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.891 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.892 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.892 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.892 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.892 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.892 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.893 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430893000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=C0995E24A01F869BEB311E8C790E6C86, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.893Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.894 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.894 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.894 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.894 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6861114A054F03969A56C5F7382C049A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.895 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.895 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.895 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.896 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.896 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.901 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.902 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.902 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.807, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.904 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:17:10.904 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.904 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.815, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.905 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.811, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.905 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.905 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.906 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.823, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.908 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.908 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.821, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.910 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.833, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=6861114A054F03969A56C5F7382C049A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.914 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.914 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:17:10.918 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.918 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.919 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.86, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.923 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:17:10.926 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.926 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=C0995E24A01F869BEB311E8C790E6C86, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.927 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:17:10.931 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.893, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=6, sqlId=C0995E24A01F869BEB311E8C790E6C86, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.931 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:17:10.936 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:13:10.186, end=2024-09-11T18:15:10.621},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:17:10.937 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:17:10.937 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:17:10.946 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:17:10.959 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963430959000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:17:10.959Z, endTime=2024-09-11T10:15:10.621Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:17:10.961 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:17:10.961 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:17:10.966 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.966 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.968 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:17:10.968 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:17:10.973 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:17:10.973 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049710621000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:17:10.974 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:17:10.974 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:17:10.959, end=2024-09-11T18:15:10.621}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:17:10.975 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:17:10.976 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:13:10.186, end=2024-09-11T18:15:10.621},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:17:10.976 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:17:10.976 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:17:10.977 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:17:10.978 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:13:10.186, end=2024-09-11T18:15:10.621},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:17:10.978 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:17:10.978 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:17:10.979 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:17:10.980 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:13:10.186, end=2024-09-11T18:15:10.621},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:17:10.980 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:17:10.980 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:17:10.991 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:17:10.991 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:17:10.991 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:13:10.186, end=2024-09-11T18:15:10.621},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:17:10.991 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:17:10.991 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:17:10.992 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:17:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,8a78b0a51e8dcc66,4744b0ab7c9d8281] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,8a78b0a51e8dcc66,4744b0ab7c9d8281] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:11.429 INFO 2981023 --- [ocp-async-3,137b8bd7c5baa574,f51cc1dbbf01f0af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:12.121 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:12.121 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:12.671 INFO 2981023 --- [ocp-schedules-27,0b8d5fdbe1192e45,22b1a55c2d66a4ea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:17:12.972 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049760, nodeCount=1000, affectRows=1000 2024-09-11 18:17:12.974 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049760, nodeCount=1000, validCount=1000 2024-09-11 18:17:12.983 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049760, nodeCount=1000, affectRows=1000 2024-09-11 18:17:12.985 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049760, nodeCount=999, validCount=999 2024-09-11 18:17:12.986 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049760, nodeCount=50, affectRows=50 2024-09-11 18:17:12.986 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049760, nodeCount=48, validCount=48 2024-09-11 18:17:13.347 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049760, nodeCount=480, affectRows=480 2024-09-11 18:17:13.405 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 18:17:13.420 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:13.423 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:13.429 INFO 2981023 --- [ocp-async-8,07d21aa5f69311dd,3bd423f9e3a1f903] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:13.495 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:17:13.496 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:17:13.508 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:13.509 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:13.521 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:17:13.522 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:17:13.533 INFO 2981023 --- [ocp-schedules-1,c35bd43d8a2ba817,0e6a6eea19e075ff] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2024-09-11 18:17:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,ad895a0caefd5875,c39fd18090132efb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,87bd771d97294d15,fa5bc46c41c1135d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:17:14.318 INFO 2981023 --- [prometheus-minute-schedule-1,ad895a0caefd5875,c39fd18090132efb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:15.115 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049760, nodeCount=480, validCount=480 2024-09-11 18:17:15.121 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:15.123 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:15.429 INFO 2981023 --- [ocp-async-2,f4baac12adba8a62,51c5b6811dbcc50c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:15.481 INFO 2981023 --- [ocp-schedules-27,81eadc54c0e1fc1a,5a3f84fdd3e2a405] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:17:15.489 INFO 2981023 --- [ocp-schedules-27,81eadc54c0e1fc1a,5a3f84fdd3e2a405] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:17:15.489 INFO 2981023 --- [ocp-schedules-27,81eadc54c0e1fc1a,5a3f84fdd3e2a405] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:17:16.020 INFO 2981023 --- [ocp-schedules-3,1aa845635e872cf9,39a28e969c91234f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:17:16.021 INFO 2981023 --- [host-check-clockdiff1,1aa845635e872cf9,c332c8186887d649] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:17:16.021 INFO 2981023 --- [host-check-clockdiff3,1aa845635e872cf9,bc81f66241e224dd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:17:16.021 INFO 2981023 --- [host-check-clockdiff2,1aa845635e872cf9,cf2726f68165d4eb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:17:16.022 INFO 2981023 --- [ocp-schedules-23,4f1d6261cb0ef7d8,748986fd63176068] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:17:16.022 INFO 2981023 --- [host-check-clockdiff4,1aa845635e872cf9,7bcfe0ccc7f43a93] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:17:16.023 INFO 2981023 --- [ocp-schedules-29,beadb07ad274f8f3,66fe491a596137d1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:17:16.024 INFO 2981023 --- [host-check-agent-status1,4f1d6261cb0ef7d8,fee9838b36035440] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:17:16.025 INFO 2981023 --- [host-check-agent-status2,4f1d6261cb0ef7d8,5b7e3e23645293b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:17:16.025 INFO 2981023 --- [host-check-agent-status3,4f1d6261cb0ef7d8,8171cca7495619bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:17:16.025 INFO 2981023 --- [host-check-agent-status4,4f1d6261cb0ef7d8,37d9057976ddf9ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:17:16.033 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:17:16.033 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:17:16.033 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:17:16.043 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:17:16.043 INFO 2981023 --- [host-check-clockdiff2,1aa845635e872cf9,cf2726f68165d4eb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049836 6 6 , error= 2024-09-11 18:17:16.044 INFO 2981023 --- [host-check-clockdiff3,1aa845635e872cf9,bc81f66241e224dd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049836 0 0 , error= 2024-09-11 18:17:16.065 INFO 2981023 --- [host-check-agent-status2,4f1d6261cb0ef7d8,5b7e3e23645293b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:17:16.066 INFO 2981023 --- [host-check-agent-status3,4f1d6261cb0ef7d8,8171cca7495619bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:16.069 INFO 2981023 --- [host-check-agent-status3,4f1d6261cb0ef7d8,8171cca7495619bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:16.069 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:17:16.070 INFO 2981023 --- [host-check-agent-status2,4f1d6261cb0ef7d8,5b7e3e23645293b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:17:16.077 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:16.078 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:17:16.079 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:17:16.080 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:17:16.080 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:16.086 INFO 2981023 --- [host-check-agent-status1,4f1d6261cb0ef7d8,fee9838b36035440] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:16.087 INFO 2981023 --- [host-check-agent-status1,4f1d6261cb0ef7d8,fee9838b36035440] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:16.088 INFO 2981023 --- [host-check-agent-status4,4f1d6261cb0ef7d8,37d9057976ddf9ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:17:16.089 INFO 2981023 --- [host-check-agent-status4,4f1d6261cb0ef7d8,37d9057976ddf9ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:17:16.090 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:16.093 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:16.172 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:17:16.174 INFO 2981023 --- [host-check-agent-status3,4f1d6261cb0ef7d8,8171cca7495619bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:17:16.175 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:17:16.174878005+08:00 2024-09-11 18:17:16.177 INFO 2981023 --- [host-check-agent-status2,4f1d6261cb0ef7d8,5b7e3e23645293b9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:17:16.177 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:17:16.178 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:17:16.175+08:00 2024-09-11 18:17:16.180 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:17:16.185237708+08:00 2024-09-11 18:17:16.181 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:17:16.175+08:00 and status AVAILABLE 2024-09-11 18:17:16.181 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:17:16.180+08:00 2024-09-11 18:17:16.183 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:17:16.180+08:00 and status AVAILABLE 2024-09-11 18:17:16.186 INFO 2981023 --- [host-check-agent-status4,4f1d6261cb0ef7d8,37d9057976ddf9ba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:17:16.187 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:17:16.187 INFO 2981023 --- [host-check-agent-status1,4f1d6261cb0ef7d8,fee9838b36035440] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:17:16.187 INFO 2981023 --- [host-check-agent-status2,4f1d6261cb0ef7d8,5b7e3e23645293b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:17:16.187 INFO 2981023 --- [host-check-agent-status2,4f1d6261cb0ef7d8,5b7e3e23645293b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:17:16.187 INFO 2981023 --- [host-check-agent-status2,4f1d6261cb0ef7d8,5b7e3e23645293b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:17:16.190 INFO 2981023 --- [host-check-agent-status4,4f1d6261cb0ef7d8,37d9057976ddf9ba] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:17:16.190 INFO 2981023 --- [host-check-agent-status4,4f1d6261cb0ef7d8,37d9057976ddf9ba] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:17:16.190 INFO 2981023 --- [host-check-agent-status4,4f1d6261cb0ef7d8,37d9057976ddf9ba] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:17:16.191 INFO 2981023 --- [host-check-agent-status1,4f1d6261cb0ef7d8,fee9838b36035440] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:17:16.191 INFO 2981023 --- [host-check-agent-status1,4f1d6261cb0ef7d8,fee9838b36035440] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:17:16.191 INFO 2981023 --- [host-check-agent-status1,4f1d6261cb0ef7d8,fee9838b36035440] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:17:16.192 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:17:16.193 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:17:16.193022716+08:00 2024-09-11 18:17:16.193 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:17:16.193+08:00 2024-09-11 18:17:16.194 INFO 2981023 --- [ocp-schedules-2,c4b9cfde957ec3b5,d4d916f9e84ea534] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:17:16.194 INFO 2981023 --- [ocp-schedules-15,0ef7dcfbe1acfdee,eca5de4443ba028d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:17:16.194 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:17:16.193+08:00 and status AVAILABLE 2024-09-11 18:17:16.195 INFO 2981023 --- [ocp-schedules-19,da289a9350a9f4c9,fbc463efb38f394a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:17:16.195 INFO 2981023 --- [ocp-schedules-7,9af379bb4cc34c4a,9f8c073e11262efd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:17:16.196 INFO 2981023 --- [host-check-agent-status3,4f1d6261cb0ef7d8,8171cca7495619bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:17:16.196 INFO 2981023 --- [host-check-agent-status3,4f1d6261cb0ef7d8,8171cca7495619bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:17:16.196 INFO 2981023 --- [host-check-agent-status3,4f1d6261cb0ef7d8,8171cca7495619bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:17:16.200 INFO 2981023 --- [ocp-schedules-19,da289a9350a9f4c9,fbc463efb38f394a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:17:16.201 INFO 2981023 --- [ocp-schedules-15,0ef7dcfbe1acfdee,eca5de4443ba028d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:17:16.203 INFO 2981023 --- [ocp-schedules-15,0ef7dcfbe1acfdee,eca5de4443ba028d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:17:16.203 INFO 2981023 --- [ocp-schedules-2,c4b9cfde957ec3b5,d4d916f9e84ea534] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:17:16.205 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:17:16.194097551+08:00 2024-09-11 18:17:16.207 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049763011000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:17:16.210 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:17:16.205+08:00 2024-09-11 18:17:16.213 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:17:16.214 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:17:16.205+08:00 and status AVAILABLE 2024-09-11 18:17:16.215 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:17:16.218 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:17:16.234 INFO 2981023 --- [ocp-schedules-14,458c9bc9402ff2d6,a10c625d026a7049] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:17:16.239 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:16.239 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:17:16.240 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:16.241 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:17:16.258 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:17:16.260 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:17:16.264 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:17:16.268 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:17:16.282 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:17:16.284 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:16.286 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:16.288 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:17:16.300 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.301 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:17:16.301 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:17:16.302 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:17:16.316 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:17:16.321 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:17:16.324 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:17:16.324 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:17:16.327 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.327 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.328 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:17:16.328 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:16.328 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:17:16.329 INFO 2981023 --- [ocp-schedules-2,c4b9cfde957ec3b5,d4d916f9e84ea534] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:17:16.330 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.331 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:16.336 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.336 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:17:16.336 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:16.337 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:17:16.350 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:17:16.361 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:17:16.360111+08:00, lastProcessTime=2024-09-11T18:17:06.353565+08:00 2024-09-11 18:17:16.362 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:17:16.381 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:17:16.384 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:17:16.386 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:16.387 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.387 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:17:16.391 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:17:16.393 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:17:16.411 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:17:16.413 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:17:16.414 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:17:16.414 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049763011000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:17:16.417 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:17:16.419 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:17:16.421 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:17:16.423 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:16.423 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:17:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:16.425 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:17:16.426 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.426 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:17:16.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:17:16.429 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:17:16.431 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:17:16.440 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:17:16.440 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:17:16.451 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:17:16.457 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:17:16.458 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:17:16.463 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:17:16.465 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:16.465 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:17:16.467 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:17:16.468 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:17:16.475 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:17:16.479 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:17:16.479 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:17:16.482 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:16.483 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:16.487 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:17:16.496 INFO 2981023 --- [host-check-host-status2,beadb07ad274f8f3,fc40840bc5549f25] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:17:16.500 INFO 2981023 --- [host-check-host-status4,beadb07ad274f8f3,efe3a358af30e412] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:17:16.501 INFO 2981023 --- [host-check-host-status1,beadb07ad274f8f3,3052c7b5e48866e7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:17:16.504 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:17:16.518 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:16.522 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.522 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:17:16.533 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:17:16.542 INFO 2981023 --- [host-check-host-status3,beadb07ad274f8f3,53fa38ac67810b24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:17:16.550 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:16.551 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.552 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:16.552 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:17:16.552 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.555 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:17:16.556 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:17:16.564 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:17:16.564 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:17:16.576 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:17:16.585 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:16.586 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.587 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:17:16.587 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:17:16.587 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:17:16.590 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:17:16.594 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:16.594 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:16.595 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:16.595 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.596 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:17:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:17:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:17:16.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:17:16.638 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:17:16.640 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:17:16.360111+08:00 2024-09-11 18:17:16.643 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:16.644 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:16.645 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:17:16.650 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:17:16.650 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:17:16.650 INFO 2981023 --- [ocp-schedules-19,da289a9350a9f4c9,fbc463efb38f394a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:17:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,9211e8f416549d3f,8fd9235d205765c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,9211e8f416549d3f,8fd9235d205765c7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:17.430 INFO 2981023 --- [ocp-async-9,6717d33a0bc9d8d2,df657b38221b8c82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:17.600 INFO 2981023 --- [ocp-schedules-14,8f78351d0d253bb4,4c003e17d6ec0144] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:17:17.653 WARN 2981023 --- [scanner-13640818-1,8f78351d0d253bb4,ffe522e35a7e58d5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:17:17.714 INFO 2981023 --- [alarm-detect-9,8f78351d0d253bb4,7cd3a70afa383f58] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726049805 ~ 1726049827 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10}}, data=[ValueData(timestamp=1726049820, value=1.54)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.54, values={1726049820=1.54}, valueCount=1, labels={app=HOST, host=10.31.35.10, host_name=vm-eas-prd-rtd-ocp, ob_cluster_id=1719994437, ob_cluster_name=linkprdocp, obzone=zone1, svr_ip=10.31.35.10})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:17:17.714 INFO 2981023 --- [alarm-detect-9,8f78351d0d253bb4,7cd3a70afa383f58] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.10-3, currentState=Pending 2024-09-11 18:17:17.789 INFO 2981023 --- [ocp-async-5,06bb41e2a7e5fa39,82b72528fa982984] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:17:18.123 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:18.123 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:19.430 INFO 2981023 --- [ocp-async-4,e725f1ed96d4326a,3ac34cf36fc64b99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:19.744 INFO 2981023 --- [host-check-clockdiff1,1aa845635e872cf9,c332c8186887d649] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049839 5 5 , error= 2024-09-11 18:17:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,6335c606f3855516,ba52e0cff63a53f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,6335c606f3855516,ba52e0cff63a53f1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:21.123 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:21.125 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:21.430 INFO 2981023 --- [ocp-async-5,3be8fddc478485fc,13726b73e2424f00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:22.019 INFO 2981023 --- [host-check-clockdiff4,1aa845635e872cf9,7bcfe0ccc7f43a93] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049842 0 0 , error= 2024-09-11 18:17:22.674 INFO 2981023 --- [ocp-schedules-26,1d93623c55bbfb1c,628da903bdc40795] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:17:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,6ff8138c3816096c,9bbd684d1c8876f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:23.315 INFO 2981023 --- [prometheus-minute-schedule-1,6ff8138c3816096c,9bbd684d1c8876f8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:23.430 INFO 2981023 --- [ocp-async-1,328355787aedea9e,770429417cc4fd47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:24.125 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:24.126 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:24.452 INFO 2981023 --- [ocp-schedules-18,b80785b0e8a1774c,2dd51dc6ab0e7c30] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:17:25.430 INFO 2981023 --- [ocp-async-7,8c5c666c124161d4,7674008aea4cdd59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,da53c0326efa0aa1,f0914cdfc762103e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,da53c0326efa0aa1,f0914cdfc762103e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:26.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:17:26.354011+08:00, lastProcessTime=2024-09-11T18:17:16.360111+08:00 2024-09-11 18:17:26.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:17:26.354011+08:00 2024-09-11 18:17:26.500 INFO 2981023 --- [ocp-schedules-16,e165848b149b4664,190cc1a53b11991c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:17:26.501 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:17:26.501 INFO 2981023 --- [ocp-schedules-16,e165848b149b4664,190cc1a53b11991c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:17:26.501 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:17:26.501 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:17:26.522 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:17:26.525 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:26.526 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:17:26.527 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:17:26.529 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:26.530 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:17:26.538 INFO 2981023 --- [ocp-schedules-16,e165848b149b4664,190cc1a53b11991c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:26.539 INFO 2981023 --- [ocp-schedules-16,e165848b149b4664,190cc1a53b11991c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:26.768 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:17:26.769 INFO 2981023 --- [ocp-schedules-16,e165848b149b4664,190cc1a53b11991c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:17:26.770 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:17:26.772 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:17:26.773 INFO 2981023 --- [ocp-schedules-16,e165848b149b4664,190cc1a53b11991c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:17:27.126 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:27.126 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:27.430 INFO 2981023 --- [ocp-async-10,45a130cb0960f3b2,6cd490ff8e8c22cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:27.789 INFO 2981023 --- [ocp-async-3,0d1a809d37913331,e824019be5fd05c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:17:27.798 INFO 2981023 --- [ocp-schedules-28,ec85017f35c74bef,b94a92e9e1c13b56] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:17:27.819 WARN 2981023 --- [scanner-13641027-1,ec85017f35c74bef,4f5b673dc5cf179b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:17:27.881 INFO 2981023 --- [alarm-detect-9,ec85017f35c74bef,fa56c589ed56d4a3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645837, data is empty. 2024-09-11 18:17:27.886 INFO 2981023 --- [alarm-detect-9,ec85017f35c74bef,fa56c589ed56d4a3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645837, data is empty. 2024-09-11 18:17:27.892 INFO 2981023 --- [alarm-detect-9,ec85017f35c74bef,fa56c589ed56d4a3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645837, data is empty. 2024-09-11 18:17:27.898 INFO 2981023 --- [alarm-detect-9,ec85017f35c74bef,dcbe0c7921d24517] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645837, data is empty. 2024-09-11 18:17:27.900 INFO 2981023 --- [alarm-detect-9,ec85017f35c74bef,dcbe0c7921d24517] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645837, data is empty. 2024-09-11 18:17:27.902 INFO 2981023 --- [alarm-detect-9,ec85017f35c74bef,dcbe0c7921d24517] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645837, data is empty. 2024-09-11 18:17:27.908 INFO 2981023 --- [alarm-detect-13,ec85017f35c74bef,9b35e949b972176d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645837, data is empty. 2024-09-11 18:17:27.912 INFO 2981023 --- [alarm-detect-13,ec85017f35c74bef,9b35e949b972176d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645837, data is empty. 2024-09-11 18:17:27.916 INFO 2981023 --- [alarm-detect-13,ec85017f35c74bef,9b35e949b972176d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645837, data is empty. 2024-09-11 18:17:28.025 INFO 2981023 --- [alarm-detect-12,ec85017f35c74bef,b20a6be1a4396448] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645837, data is empty. 2024-09-11 18:17:28.030 INFO 2981023 --- [alarm-detect-12,ec85017f35c74bef,b20a6be1a4396448] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645837, data is empty. 2024-09-11 18:17:28.034 INFO 2981023 --- [alarm-detect-12,ec85017f35c74bef,b20a6be1a4396448] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645837, data is empty. 2024-09-11 18:17:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,b858e9e49cf1e614,1085c635b0ede0bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,b858e9e49cf1e614,1085c635b0ede0bb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:29.430 INFO 2981023 --- [ocp-async-9,e3cc464671fbd6a7,274eaeca63928eb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:30.127 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:30.127 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:31.430 INFO 2981023 --- [ocp-async-3,99716d0985730c82,27f93a40d28da7d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,8b50c70c62a0405c,00da83cffbfc7b47] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,8b50c70c62a0405c,00da83cffbfc7b47] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:32.676 INFO 2981023 --- [ocp-schedules-4,fa8c7094dd8e3bbd,6674b2b9206391b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:17:33.127 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:33.128 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:33.430 INFO 2981023 --- [ocp-async-2,1796f33b842404ed,5b21385789a72e19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,5b76c8f4f8485208,e359a1b5cfa3710d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,5b76c8f4f8485208,e359a1b5cfa3710d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:35.430 INFO 2981023 --- [ocp-async-6,ef850640a3478cce,0077998314bac9f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:35.492 INFO 2981023 --- [ocp-schedules-10,51cc2c0aa2f7804b,7c3355f072762199] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:17:35.500 INFO 2981023 --- [ocp-schedules-10,51cc2c0aa2f7804b,7c3355f072762199] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:17:35.501 INFO 2981023 --- [ocp-schedules-10,51cc2c0aa2f7804b,7c3355f072762199] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:17:36.128 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:36.128 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:17:36.337621+08:00, lastProcessTime=2024-09-11T18:17:26.354011+08:00 2024-09-11 18:17:36.465 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:17:36.337621+08:00 2024-09-11 18:17:37.314 INFO 2981023 --- [prometheus-cache-monitor1,1be3913255840a6b,eb83332f58d5e693] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:17:37.315 INFO 2981023 --- [prometheus-cache-monitor1,1be3913255840a6b,eb83332f58d5e693] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:17:37.430 INFO 2981023 --- [ocp-async-8,60847cfffcca8fe0,a667307584815b31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:37.789 INFO 2981023 --- [ocp-async-5,937cd31b22b74b91,a1260a2ad229d915] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:17:38.078 INFO 2981023 --- [ocp-schedules-20,e6ffdaded83caff3,e030ed4bbc7b7fb8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:17:38.118 WARN 2981023 --- [scanner-13641365-1,e6ffdaded83caff3,88ea37318a7b45ac] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:17:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,3e6d863318871398,2d57b156bf713c14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,3e6d863318871398,2d57b156bf713c14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:39.128 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:39.129 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:39.431 INFO 2981023 --- [ocp-async-10,62c42238036fb837,30efaa967187147b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:40.003 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:17:40.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:17:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,bd9d4a5e013697ba,485d2dfba0b7412f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,bd9d4a5e013697ba,485d2dfba0b7412f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:41.431 INFO 2981023 --- [ocp-async-5,78833e2aa1a821f6,64826d5f0ddf5e43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:42.129 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:42.129 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:42.679 INFO 2981023 --- [ocp-schedules-11,e9d22c7838448b6a,8379e2d17d802f15] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:17:43.431 INFO 2981023 --- [ocp-async-9,a086c3253ed01110,9bfb9808a7ffcaa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,c3cd0df12c1cf6e8,9927ba474aef164e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,c3cd0df12c1cf6e8,9927ba474aef164e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:45.101 INFO 2981023 --- [prometheus-second-schedule-1,6d5747227a61aac6,a759e5d35953afba] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:17:45.130 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:45.130 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:45.431 INFO 2981023 --- [ocp-async-3,565b8c8318b55662,67aec7fb7956e89c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:46.015 INFO 2981023 --- [ocp-schedules-17,976e34cc40b949b9,1ad2c08c71d6cc68] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:17:46.015 INFO 2981023 --- [host-check-clockdiff1,976e34cc40b949b9,7b1fb8e48b3d7502] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:17:46.016 INFO 2981023 --- [host-check-clockdiff2,976e34cc40b949b9,ab806f6615eb0155] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:17:46.018 INFO 2981023 --- [host-check-clockdiff4,976e34cc40b949b9,90ed5b35889490d7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:17:46.018 INFO 2981023 --- [host-check-clockdiff3,976e34cc40b949b9,2bc4e4e95e6da1f6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:17:46.027 INFO 2981023 --- [ocp-schedules-8,236449904843f8a0,2abd962463d4858e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:17:46.028 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:17:46.028 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:17:46.029 INFO 2981023 --- [ocp-schedules-13,d32c43a5b20f621d,e3908a0241293390] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:17:46.029 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:17:46.029 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:17:46.030 INFO 2981023 --- [host-check-agent-status1,d32c43a5b20f621d,feb81e79e918a2f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:17:46.033 INFO 2981023 --- [host-check-agent-status4,d32c43a5b20f621d,e2b99292bfd0fef8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:17:46.034 INFO 2981023 --- [host-check-agent-status2,d32c43a5b20f621d,cc831d1bacebebdd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:17:46.034 INFO 2981023 --- [host-check-agent-status3,d32c43a5b20f621d,6e602bf948301ddd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:17:46.035 INFO 2981023 --- [host-check-clockdiff3,976e34cc40b949b9,2bc4e4e95e6da1f6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049866 0 0 , error= 2024-09-11 18:17:46.036 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:17:46.036 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:46.036 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:46.037 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:17:46.037 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:17:46.037 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:46.038 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:46.039 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:17:46.040 INFO 2981023 --- [host-check-agent-status3,d32c43a5b20f621d,6e602bf948301ddd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:46.042 INFO 2981023 --- [host-check-agent-status3,d32c43a5b20f621d,6e602bf948301ddd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:46.043 INFO 2981023 --- [host-check-agent-status4,d32c43a5b20f621d,e2b99292bfd0fef8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:17:46.045 INFO 2981023 --- [host-check-agent-status4,d32c43a5b20f621d,e2b99292bfd0fef8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:17:46.046 INFO 2981023 --- [host-check-agent-status1,d32c43a5b20f621d,feb81e79e918a2f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:46.047 INFO 2981023 --- [host-check-agent-status2,d32c43a5b20f621d,cc831d1bacebebdd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:17:46.048 INFO 2981023 --- [host-check-agent-status2,d32c43a5b20f621d,cc831d1bacebebdd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:17:46.048 INFO 2981023 --- [host-check-agent-status1,d32c43a5b20f621d,feb81e79e918a2f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:46.064 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:17:46.065 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:17:46.066 INFO 2981023 --- [host-check-agent-status3,d32c43a5b20f621d,6e602bf948301ddd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:17:46.066 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:17:46.066367176+08:00 2024-09-11 18:17:46.067 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:17:46.072133665+08:00 2024-09-11 18:17:46.067 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:17:46.067+08:00 2024-09-11 18:17:46.068 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:17:46.066+08:00 2024-09-11 18:17:46.070 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:17:46.067+08:00 and status AVAILABLE 2024-09-11 18:17:46.070 INFO 2981023 --- [host-check-agent-status3,d32c43a5b20f621d,6e602bf948301ddd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:17:46.070 INFO 2981023 --- [host-check-agent-status3,d32c43a5b20f621d,6e602bf948301ddd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:17:46.070 INFO 2981023 --- [host-check-agent-status3,d32c43a5b20f621d,6e602bf948301ddd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:17:46.070 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:17:46.066+08:00 and status AVAILABLE 2024-09-11 18:17:46.072 INFO 2981023 --- [host-check-agent-status4,d32c43a5b20f621d,e2b99292bfd0fef8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:17:46.077 INFO 2981023 --- [host-check-agent-status4,d32c43a5b20f621d,e2b99292bfd0fef8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:17:46.077 INFO 2981023 --- [host-check-agent-status4,d32c43a5b20f621d,e2b99292bfd0fef8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:17:46.077 INFO 2981023 --- [host-check-agent-status4,d32c43a5b20f621d,e2b99292bfd0fef8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:17:46.077 INFO 2981023 --- [host-check-agent-status2,d32c43a5b20f621d,cc831d1bacebebdd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:17:46.077 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:17:46.078 INFO 2981023 --- [host-check-agent-status1,d32c43a5b20f621d,feb81e79e918a2f9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:17:46.079 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:17:46.084496431+08:00 2024-09-11 18:17:46.079 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:17:46.079+08:00 2024-09-11 18:17:46.079 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:17:46.080 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:17:46.080388220+08:00 2024-09-11 18:17:46.081 INFO 2981023 --- [host-check-agent-status2,d32c43a5b20f621d,cc831d1bacebebdd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:17:46.081 INFO 2981023 --- [host-check-agent-status2,d32c43a5b20f621d,cc831d1bacebebdd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:17:46.081 INFO 2981023 --- [host-check-agent-status2,d32c43a5b20f621d,cc831d1bacebebdd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:17:46.081 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:17:46.079+08:00 and status AVAILABLE 2024-09-11 18:17:46.081 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:17:46.081 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:17:46.080+08:00 2024-09-11 18:17:46.081 INFO 2981023 --- [host-check-agent-status1,d32c43a5b20f621d,feb81e79e918a2f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:17:46.082 INFO 2981023 --- [host-check-agent-status1,d32c43a5b20f621d,feb81e79e918a2f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:17:46.082 INFO 2981023 --- [host-check-agent-status1,d32c43a5b20f621d,feb81e79e918a2f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:17:46.088 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:17:46.080+08:00 and status AVAILABLE 2024-09-11 18:17:46.090 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:17:46.106 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:17:46.111 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:46.112 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:46.114 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:17:46.115 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:17:46.117 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:46.118 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:46.121 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:17:46.138 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:17:46.140 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:17:46.143 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:17:46.145 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:17:46.146 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:17:46.163 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:17:46.171 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:17:46.172 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:17:46.172 INFO 2981023 --- [ocp-schedules-24,3699802f8a0571bf,2051ffbc05495435] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:17:46.187 INFO 2981023 --- [ocp-schedules-30,1ec77c26edfe6b09,4de26e6c2053a36d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:17:46.187 INFO 2981023 --- [ocp-schedules-2,45662eb5e458fd98,872ad2a7d64312ba] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:17:46.188 INFO 2981023 --- [ocp-schedules-28,497a86290409eacd,ea964c4a5f510ee7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:17:46.188 INFO 2981023 --- [ocp-schedules-6,0e07b0ea63fb6b2c,30d3dbde2c2bae0b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:17:46.193 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:17:46.194 INFO 2981023 --- [ocp-schedules-28,497a86290409eacd,ea964c4a5f510ee7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:17:46.197 INFO 2981023 --- [ocp-schedules-2,45662eb5e458fd98,872ad2a7d64312ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:17:46.199 INFO 2981023 --- [ocp-schedules-2,45662eb5e458fd98,872ad2a7d64312ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:17:46.201 INFO 2981023 --- [ocp-schedules-30,1ec77c26edfe6b09,4de26e6c2053a36d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:17:46.202 INFO 2981023 --- [ocp-schedules-24,3699802f8a0571bf,2051ffbc05495435] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:17:46.207 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049763011000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:17:46.210 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:17:46.210 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:17:46.212 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:17:46.216 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:17:46.227 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:17:46.262 INFO 2981023 --- [ocp-schedules-1,e14abd53e009d2a6,ae86563a48caf3aa] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:17:46.263 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:17:46.264 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:17:46.289 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.290 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:17:46.291 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:17:46.294 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:17:46.296 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.296 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:46.301 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:17:46.303 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.303 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:46.304 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.306 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:46.313 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.314 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:17:46.314 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.315 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:17:46.315 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.315 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:17:46.315 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:17:46.316 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:17:46.316 INFO 2981023 --- [ocp-schedules-30,1ec77c26edfe6b09,4de26e6c2053a36d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:17:46.319 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:17:46.328 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:17:46.328 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:17:46.329 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:17:46.336 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:17:46.339 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:17:46.345 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:17:46.341774+08:00, lastProcessTime=2024-09-11T18:17:36.337621+08:00 2024-09-11 18:17:46.347 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.349 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.349 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:17:46.366 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:17:46.367 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:17:46.399 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:17:46.399 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049763011000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:17:46.405 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:17:46.407 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:17:46.416 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:17:46.422 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.423 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.423 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.423 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:17:46.423 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.423 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:17:46.424 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:17:46.424 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:46.425 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:17:46.426 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.426 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:17:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:17:46.428 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:17:46.432 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:17:46.437 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:17:46.445 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:17:46.445 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:17:46.448 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:17:46.453 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:17:46.456 INFO 2981023 --- [host-check-host-status1,236449904843f8a0,4562a94b0d953658] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:17:46.460 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:17:46.460 INFO 2981023 --- [host-check-host-status4,236449904843f8a0,42c79e3c221ee2f3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:17:46.465 INFO 2981023 --- [host-check-host-status2,236449904843f8a0,f9ee36780601b8a3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:17:46.471 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.472 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.472 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:17:46.483 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:17:46.483 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:17:46.493 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.495 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.496 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:17:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:17:46.502 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:17:46.503 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.507 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:17:46.508 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:17:46.512 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.515 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.516 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:17:46.516 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.516 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.517 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:17:46.517 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.517 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:17:46.519 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:17:46.519 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:17:46.520 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.520 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:17:46.520 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:17:46.520 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:17:46.522 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:17:46.522 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.528 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:17:46.528 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:17:46.530 INFO 2981023 --- [host-check-host-status3,236449904843f8a0,5339726b2a2c9169] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:17:46.537 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:17:46.537 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:17:46.549 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:17:46.561 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:17:46.574 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.574 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.574 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:17:46.575 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.575 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.575 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:17:46.575 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:17:46.575 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.577 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:17:46.592 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:17:46.593 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:17:46.600 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.600 INFO 2981023 --- [ocp-schedules-3,05c5e37d946ef59e,092cb3407962a16d] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:17:46.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.601 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:17:46.608 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:17:46.610 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:17:46.610 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:17:46.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:17:46.611 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:17:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:17:46.628 WARN 2981023 --- [ocp-schedules-3,05c5e37d946ef59e,092cb3407962a16d] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:17:46.632 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:17:46.633 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.634 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.634 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:17:46.635 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:17:46.635 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.661 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:17:46.667 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:17:46.668 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:46.668 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:17:46.674 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:17:46.674 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:17:46.675 INFO 2981023 --- [ocp-schedules-28,497a86290409eacd,ea964c4a5f510ee7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:17:46.678 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:17:46.341774+08:00 2024-09-11 18:17:46.686 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:17:46.702 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:17:46.707 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:17:46.710 INFO 2981023 --- [ocp-schedules-12,dd7017298bb02a28,82f27eed88dd5719] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:17:46.845 INFO 2981023 --- [ocp-schedules-16,695704b89544a85e,4caba453901f1f39] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:17:46.845 INFO 2981023 --- [ocp-schedules-16,695704b89544a85e,4caba453901f1f39] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:17:46.964 INFO 2981023 --- [host-check-clockdiff2,976e34cc40b949b9,ab806f6615eb0155] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049866 5 5 , error= 2024-09-11 18:17:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,ccd05d01877bbbb3,d566e094a2e3606c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,ccd05d01877bbbb3,d566e094a2e3606c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:47.431 INFO 2981023 --- [ocp-async-6,d1eec0ac78bca4c5,58e5ba6f788a3eb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:47.789 INFO 2981023 --- [ocp-async-10,3a64551d795950b0,7df107fb68b7a3f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:17:47.812 INFO 2981023 --- [host-check-clockdiff1,976e34cc40b949b9,7b1fb8e48b3d7502] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049867 5 5 , error= 2024-09-11 18:17:48.130 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:48.131 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:48.266 INFO 2981023 --- [ocp-schedules-2,9f1b0eb29868f991,d6e963eadf59389b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:17:48.310 WARN 2981023 --- [scanner-13641618-1,9f1b0eb29868f991,298cc77207518d29] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:17:48.330 INFO 2981023 --- [alarm-detect-19,9f1b0eb29868f991,a576785450fba194] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645837, data is empty. 2024-09-11 18:17:48.336 INFO 2981023 --- [alarm-detect-19,9f1b0eb29868f991,a576785450fba194] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645837, data is empty. 2024-09-11 18:17:48.339 INFO 2981023 --- [alarm-detect-19,9f1b0eb29868f991,a576785450fba194] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645837, data is empty. 2024-09-11 18:17:48.359 INFO 2981023 --- [alarm-detect-17,9f1b0eb29868f991,9be2a71303e34ac2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645837, data is empty. 2024-09-11 18:17:48.366 INFO 2981023 --- [alarm-detect-4,9f1b0eb29868f991,8e863656269c230f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645837, data is empty. 2024-09-11 18:17:48.371 INFO 2981023 --- [alarm-detect-17,9f1b0eb29868f991,9be2a71303e34ac2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645837, data is empty. 2024-09-11 18:17:48.371 INFO 2981023 --- [alarm-detect-4,9f1b0eb29868f991,8e863656269c230f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645837, data is empty. 2024-09-11 18:17:48.405 INFO 2981023 --- [alarm-detect-4,9f1b0eb29868f991,8e863656269c230f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645837, data is empty. 2024-09-11 18:17:48.411 INFO 2981023 --- [alarm-detect-17,9f1b0eb29868f991,9be2a71303e34ac2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645837, data is empty. 2024-09-11 18:17:48.495 INFO 2981023 --- [alarm-detect-20,9f1b0eb29868f991,7a51eca492d4021a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645837, data is empty. 2024-09-11 18:17:48.498 INFO 2981023 --- [alarm-detect-20,9f1b0eb29868f991,7a51eca492d4021a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645837, data is empty. 2024-09-11 18:17:48.503 INFO 2981023 --- [alarm-detect-20,9f1b0eb29868f991,7a51eca492d4021a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645837, data is empty. 2024-09-11 18:17:49.431 INFO 2981023 --- [ocp-async-9,ce447ead81b8a514,51f5dc4b0a5d6e15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:17:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:17:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,338788c0f26dafca,bb50bae11255d5fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,338788c0f26dafca,bb50bae11255d5fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:50.340 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:17:50.344 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.345 INFO 2981023 --- [ocp-schedules-13,dcadb4ddc3be1d8c,70c848d0405e44eb] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:17:50.349 INFO 2981023 --- [ocp-schedules-8,cfa958359b3c7bdc,6f71f85646c837c3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:17:51.131 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:51.132 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:51.431 INFO 2981023 --- [ocp-async-10,18b5ed8d5f1f501c,c9e92c40ccf89fc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:51.622 INFO 2981023 --- [ocp-schedules-22,b43e9f8c12de17d0,34c7c3c752a1decb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:17:51.623 INFO 2981023 --- [ocp-schedules-22,b43e9f8c12de17d0,34c7c3c752a1decb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:17:51.624 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:17:51.624 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:17:51.624 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:17:51.624 INFO 2981023 --- [ocp-schedules-22,b43e9f8c12de17d0,34c7c3c752a1decb] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:17:51.999 INFO 2981023 --- [host-check-clockdiff4,976e34cc40b949b9,90ed5b35889490d7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049871 0 0 , error= 2024-09-11 18:17:52.681 INFO 2981023 --- [ocp-schedules-29,30e016980a71ed54,8c80db8d4e9ccdc1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:17:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,af4a4859ecec3dff,ff1fa328537d2857] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,af4a4859ecec3dff,ff1fa328537d2857] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:53.337 INFO 2981023 --- [ocp-schedules-15,05138d9df9b6252a,f06754c3f3f74bbc] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:17:53.337 INFO 2981023 --- [ocp-schedules-15,05138d9df9b6252a,f06754c3f3f74bbc] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:17:53.432 INFO 2981023 --- [ocp-async-5,980fe21b4d47c67f,e1623e61f9768384] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:54.132 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:54.134 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:54.455 INFO 2981023 --- [ocp-schedules-28,a473aed3ea3e25d7,469e3c5535822742] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:17:54.949 INFO 2981023 --- [ocp-schedules-1,4cf20c9befbc36c8,1473b8dc785fe86d] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:17:54.969 WARN 2981023 --- [ocp-schedules-1,4cf20c9befbc36c8,1473b8dc785fe86d] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:17:55.432 INFO 2981023 --- [ocp-async-8,a2180c314bef9d14,37c6e1d9ad92acae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:55.503 INFO 2981023 --- [ocp-schedules-29,24e82e5877d1fd81,b973a2d75b77dcbb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:17:55.514 INFO 2981023 --- [ocp-schedules-29,24e82e5877d1fd81,b973a2d75b77dcbb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:17:55.514 INFO 2981023 --- [ocp-schedules-29,24e82e5877d1fd81,b973a2d75b77dcbb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:17:55.882 INFO 2981023 --- [ocp-schedules-16,507d83706d343ab7,70f0742a6056f3da] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:17:55.882 INFO 2981023 --- [ocp-schedules-16,507d83706d343ab7,70f0742a6056f3da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:17:55.882 INFO 2981023 --- [ocp-schedules-16,507d83706d343ab7,70f0742a6056f3da] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:17:55.882 INFO 2981023 --- [ocp-schedules-16,507d83706d343ab7,70f0742a6056f3da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:17:55.883 INFO 2981023 --- [ocp-schedules-16,507d83706d343ab7,70f0742a6056f3da] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:17:55.883 INFO 2981023 --- [ocp-schedules-16,507d83706d343ab7,70f0742a6056f3da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:17:56.083 INFO 2981023 --- [ocp-schedules-19,1cd24124aa03979d,655e9628e018d764] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:17:56.089 INFO 2981023 --- [ocp-schedules-19,1cd24124aa03979d,655e9628e018d764] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:17:56.089 INFO 2981023 --- [ocp-schedules-19,1cd24124aa03979d,655e9628e018d764] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:17:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,5ea27e3185b2dd20,7f59d9769a5879d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,5ea27e3185b2dd20,7f59d9769a5879d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:17:56.353159+08:00, lastProcessTime=2024-09-11T18:17:46.341774+08:00 2024-09-11 18:17:56.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:17:56.353159+08:00 2024-09-11 18:17:57.134 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:17:57.135 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:17:57.432 INFO 2981023 --- [ocp-async-3,baac58670afaaf50,bfbc60d9fe9fee03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:17:57.789 INFO 2981023 --- [ocp-async-8,454a567400788975,7ca7342dfb486d76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:17:58.187 INFO 2981023 --- [ocp-schedules-29,ece3bf8279238b96,9008b1e17bdeeec4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:17:58.187 INFO 2981023 --- [ocp-schedules-29,ece3bf8279238b96,9008b1e17bdeeec4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:17:58.219 INFO 2981023 --- [Collect all tenants major compactions1,ece3bf8279238b96,1df79454edf656ad] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:17:58.219 INFO 2981023 --- [Collect all tenants major compactions5,ece3bf8279238b96,5259a660f8dd6c90] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:17:58.219 INFO 2981023 --- [Collect all tenants major compactions4,ece3bf8279238b96,75d122f1d9f63165] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:17:58.220 INFO 2981023 --- [Collect all tenants major compactions2,ece3bf8279238b96,d59ae70925b4f4ce] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:17:58.220 INFO 2981023 --- [Collect all tenants major compactions3,ece3bf8279238b96,ce1348fb25757a1d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:17:58.226 INFO 2981023 --- [Collect all tenants major compactions6,ece3bf8279238b96,5b0804f3a8d9fe5e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:17:58.241 INFO 2981023 --- [Collect all tenants major compactions5,ece3bf8279238b96,5259a660f8dd6c90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:58.242 INFO 2981023 --- [Collect all tenants major compactions5,ece3bf8279238b96,5259a660f8dd6c90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:17:58.243 INFO 2981023 --- [Collect all tenants major compactions2,ece3bf8279238b96,d59ae70925b4f4ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:58.243 INFO 2981023 --- [Collect all tenants major compactions3,ece3bf8279238b96,ce1348fb25757a1d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:58.243 INFO 2981023 --- [Collect all tenants major compactions1,ece3bf8279238b96,1df79454edf656ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:58.244 INFO 2981023 --- [Collect all tenants major compactions2,ece3bf8279238b96,d59ae70925b4f4ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:17:58.244 INFO 2981023 --- [Collect all tenants major compactions3,ece3bf8279238b96,ce1348fb25757a1d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:17:58.246 INFO 2981023 --- [Collect all tenants major compactions1,ece3bf8279238b96,1df79454edf656ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:17:58.247 INFO 2981023 --- [Collect all tenants major compactions5,ece3bf8279238b96,5259a660f8dd6c90] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:17:58.247 INFO 2981023 --- [Collect all tenants major compactions5,ece3bf8279238b96,5259a660f8dd6c90] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:17:58.247 INFO 2981023 --- [Collect all tenants major compactions2,ece3bf8279238b96,d59ae70925b4f4ce] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:17:58.247 INFO 2981023 --- [Collect all tenants major compactions2,ece3bf8279238b96,d59ae70925b4f4ce] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:17:58.247 INFO 2981023 --- [Collect all tenants major compactions3,ece3bf8279238b96,ce1348fb25757a1d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:17:58.247 INFO 2981023 --- [Collect all tenants major compactions3,ece3bf8279238b96,ce1348fb25757a1d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:17:58.248 INFO 2981023 --- [Collect all tenants major compactions4,ece3bf8279238b96,75d122f1d9f63165] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:58.248 INFO 2981023 --- [Collect all tenants major compactions1,ece3bf8279238b96,1df79454edf656ad] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:17:58.248 INFO 2981023 --- [Collect all tenants major compactions1,ece3bf8279238b96,1df79454edf656ad] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:17:58.248 INFO 2981023 --- [Collect all tenants major compactions4,ece3bf8279238b96,75d122f1d9f63165] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:17:58.250 INFO 2981023 --- [Collect all tenants major compactions6,ece3bf8279238b96,5b0804f3a8d9fe5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:17:58.250 INFO 2981023 --- [Collect all tenants major compactions4,ece3bf8279238b96,75d122f1d9f63165] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:17:58.250 INFO 2981023 --- [Collect all tenants major compactions4,ece3bf8279238b96,75d122f1d9f63165] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:17:58.251 INFO 2981023 --- [Collect all tenants major compactions6,ece3bf8279238b96,5b0804f3a8d9fe5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:17:58.253 INFO 2981023 --- [Collect all tenants major compactions6,ece3bf8279238b96,5b0804f3a8d9fe5e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:17:58.253 INFO 2981023 --- [Collect all tenants major compactions6,ece3bf8279238b96,5b0804f3a8d9fe5e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:17:58.253 INFO 2981023 --- [ocp-schedules-29,ece3bf8279238b96,9008b1e17bdeeec4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:17:58.549 INFO 2981023 --- [ocp-schedules-17,7636d29a0ee9a86d,e345e9aba560a2f1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:17:58.596 WARN 2981023 --- [scanner-13641901-1,7636d29a0ee9a86d,30876eabff2fb3be] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:17:59.082 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.085 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, 2024-09-11 18:17:59.088 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.091 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.094 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.097 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.098 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.098 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.099 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.100 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.100 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.101 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.102 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.103 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.103 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.104 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.105 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.106 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.108 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.111 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.113 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.115 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.118 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.121 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.123 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.126 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:11:58.976, end=2024-09-11T18:16:59.077} 2024-09-11 18:17:59.126 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 18:17:59.204 INFO 2981023 --- [ocp-schedules-19,775b939d7b7a748b,b7b74fdac5137614] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:17:59.204+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:17:59.207 INFO 2981023 --- [ocp-schedules-19,775b939d7b7a748b,b7b74fdac5137614] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:17:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,cd62de3e00da9d0d,3be99ef053dce9d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049880, interval=60 2024-09-11 18:17:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,cd62de3e00da9d0d,3be99ef053dce9d4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:17:59.372 INFO 2981023 --- [ocp-schedules-19,775b939d7b7a748b,b7b74fdac5137614] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 63 to SqlAggregateData 2024-09-11 18:17:59.373 INFO 2981023 --- [ocp-schedules-19,775b939d7b7a748b,b7b74fdac5137614] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:17:59.432 INFO 2981023 --- [ocp-async-5,388428b6b3b8d96b,26bbb0672e49824f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:00.135 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:00.135 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:00.244 INFO 2981023 --- [ocp-schedules-30,cf70bff1936bfcec,a0d0ffa2c4e53b67] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:18:00.245 INFO 2981023 --- [ocp-schedules-30,cf70bff1936bfcec,a0d0ffa2c4e53b67] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:18:00.274 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:18:00.274 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.275 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:18:00.275 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.278 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:18:00.279 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:18:00.282 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.282 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.282 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:18:00.283 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.285 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:18:00.286 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.286 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.287 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.287 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.289 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.290 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.292 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.293 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:18:00.298 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.299 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.299 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.299 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:18:00.303 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.303 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.305 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.306 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:18:00.312 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.312 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.314 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:18:00.314 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE56A1D6DC-0-0] 2024-09-11 18:18:00.314 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:18:00.314 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.315 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:18:00.315 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.316 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:18:00.316 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.317 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.327 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.328 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:18:00.331 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.331 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:18:00.332 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.332 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:18:00.333 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.333 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:18:00.334 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.334 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:18:00.335 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.335 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:18:00.336 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.336 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:18:00.337 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.337 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:18:00.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:18:00.341 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.341 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:18:00.342 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:18:00.342 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:18:00.342 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:18:00.343 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE56A1D6E6-0-0] 2024-09-11 18:18:00.343 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:18:00.344 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.345 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:18:00.345 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.346 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:18:00.346 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:18:00.347 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.347 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:18:00.348 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:18:00.350 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:18:00.350 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.351 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:18:00.351 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.352 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:18:00.353 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.354 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:18:00.355 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.355 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:18:00.356 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.357 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:18:00.357 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.358 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:18:00.358 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:18:00.359 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:00.361 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:18:00.361 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.362 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:18:00.369 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:18:00.370 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:00.372 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.372 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.372 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:18:00.372 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.373 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:18:00.379 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:18:00.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:00.381 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:18:00.382 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.382 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:18:00.383 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.383 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.385 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:18:00.387 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:18:00.387 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:18:00.387 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:18:00.389 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:18:00.390 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:18:00.390 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:18:00.390 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:18:00.398 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:18:00.399 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:18:00.400 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:00.400 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:18:00.411 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:18:00.412 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:18:00.412 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:18:00.412 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:18:00.412 INFO 2981023 --- [ocp-schedules-30,cf70bff1936bfcec,a0d0ffa2c4e53b67] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:18:01.432 INFO 2981023 --- [ocp-async-8,6a66b00155165ae2,9fa620b52c42440e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:02.259 INFO 2981023 --- [ocp-schedules-17,ecc5ae24571b017c,6d47567dbd853b47] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:18:02.259 INFO 2981023 --- [ocp-schedules-17,ecc5ae24571b017c,6d47567dbd853b47] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:18:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,098fe0619f7c9aa7,40ad9e819686688f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,098fe0619f7c9aa7,40ad9e819686688f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:02.684 INFO 2981023 --- [ocp-schedules-24,8b3e2706b83480e3,e5a00f002f10ebcd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:18:03.027 INFO 2981023 --- [ocp-schedules-6,6ecba3f3ff0ccca1,4613b7161e642321] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:18:03.028 INFO 2981023 --- [ocp-schedules-6,6ecba3f3ff0ccca1,4613b7161e642321] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:18:03.050 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:03.050 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:18:03.056 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:03.058 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:18:03.058 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049883058000} 2024-09-11 18:18:03.064 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:18:03.068 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000233, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:18:03.068 INFO 2981023 --- [ocp-schedules-6,6ecba3f3ff0ccca1,4613b7161e642321] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:18:03.136 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:03.136 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:03.432 INFO 2981023 --- [ocp-async-10,c55d95897f402ca2,86413fb006ad9de9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:03.592 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:18:03.592 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:18:03.592 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:18:03.597 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:18:03.607 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000233, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:18:04.240 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000233 2024-09-11 18:18:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,2f285fc1a3ce5b65,aaaaa781834e1862] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,2f285fc1a3ce5b65,aaaaa781834e1862] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:05.432 INFO 2981023 --- [ocp-async-6,9068052511c8ac84,4c6971ea83deb2b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:05.968 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:18:05.982 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:18:05.987 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:18:05.992 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:18:05.992 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:18:05.992 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:18:05.994 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:05.995 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:18:05.995 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:05.996 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:18:06.028 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:15:41, start=2024-09-11T18:12:41, end=2024-09-11T18:15:11, updates=269, granule=120000ms, delay=180000ms 2024-09-11 18:18:06.034 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:15:41, target=2024-09-11T18:15:11, granule=120000ms, delay=30000ms) 2024-09-11 18:18:06.122 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T18:15:11, start=2024-09-11T18:00:51, end=2024-09-11T18:10:51, updates=370, granule=600000ms, delay=860000ms 2024-09-11 18:18:06.126 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T18:15:11, target=2024-09-11T18:10:51, granule=600000ms, delay=260000ms) 2024-09-11 18:18:06.126 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=158ms, period=120000ms 2024-09-11 18:18:06.136 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:06.136 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:18:06.336764+08:00, lastProcessTime=2024-09-11T18:17:56.353159+08:00 2024-09-11 18:18:06.465 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:18:06.336764+08:00 2024-09-11 18:18:07.058 INFO 2981023 --- [ocp-schedules-8,131ad05b92b5a98f,2ed0c964492663e7] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:18:07.432 INFO 2981023 --- [ocp-async-2,ef02394db3b4ce78,c985d88056d4d442] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:07.789 INFO 2981023 --- [ocp-async-3,06b88e7bb6606908,351c158cd3de1bfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:18:08.024 INFO 2981023 --- [ocp-schedules-23,acbde17374056ff6,0b2750921ca40dc1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:18:08.125 INFO 2981023 --- [ocp-schedules-24,0d3afc213edbc654,3602a1de8fa4603b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:18:08.125+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:18:08.125 INFO 2981023 --- [ocp-schedules-24,0d3afc213edbc654,3602a1de8fa4603b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:18:08.127 INFO 2981023 --- [ocp-schedules-24,0d3afc213edbc654,3602a1de8fa4603b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:18:08.127 INFO 2981023 --- [ocp-schedules-24,0d3afc213edbc654,3602a1de8fa4603b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:18:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,1c4961d92d3384bd,2a37cf67105ef678] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,1c4961d92d3384bd,2a37cf67105ef678] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:08.744 INFO 2981023 --- [ocp-schedules-14,19d24b2d07746307,a977061b1290f2c9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:18:08.804 INFO 2981023 --- [alarm-detect-7,19d24b2d07746307,6c77bb44d44038cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645837, data is empty. 2024-09-11 18:18:08.808 INFO 2981023 --- [alarm-detect-7,19d24b2d07746307,6c77bb44d44038cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645900, data is empty. 2024-09-11 18:18:08.811 INFO 2981023 --- [alarm-detect-7,19d24b2d07746307,6c77bb44d44038cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645900, data is empty. 2024-09-11 18:18:08.816 WARN 2981023 --- [scanner-13642150-1,19d24b2d07746307,076839aa98307e26] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:18:08.839 INFO 2981023 --- [alarm-detect-2,19d24b2d07746307,368234b191b6cfe1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645837, data is empty. 2024-09-11 18:18:08.845 INFO 2981023 --- [alarm-detect-2,19d24b2d07746307,368234b191b6cfe1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645900, data is empty. 2024-09-11 18:18:08.848 INFO 2981023 --- [alarm-detect-4,19d24b2d07746307,a96e5bd8fb690408] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645837, data is empty. 2024-09-11 18:18:08.850 INFO 2981023 --- [alarm-detect-2,19d24b2d07746307,368234b191b6cfe1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645900, data is empty. 2024-09-11 18:18:08.855 INFO 2981023 --- [alarm-detect-4,19d24b2d07746307,a96e5bd8fb690408] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645900, data is empty. 2024-09-11 18:18:08.859 INFO 2981023 --- [alarm-detect-4,19d24b2d07746307,a96e5bd8fb690408] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645900, data is empty. 2024-09-11 18:18:08.961 INFO 2981023 --- [alarm-detect-10,19d24b2d07746307,b65f6721b77ccf27] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645837, data is empty. 2024-09-11 18:18:08.967 INFO 2981023 --- [alarm-detect-10,19d24b2d07746307,b65f6721b77ccf27] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645900, data is empty. 2024-09-11 18:18:08.971 INFO 2981023 --- [alarm-detect-10,19d24b2d07746307,b65f6721b77ccf27] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645900, data is empty. 2024-09-11 18:18:09.137 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:09.137 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:09.433 INFO 2981023 --- [ocp-async-1,d1fbad95c3a2a158,17ae2a52b9b44b3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:10.003 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:18:10.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:18:10.413 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c6f015aa981a2c91,a9c18b23c0f67c25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=c6f015aa981a2c91, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:18:10.413 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,cb70c5597c480795,16517487a0a2c3d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=cb70c5597c480795, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:18:10.414 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,c6f015aa981a2c91,a9c18b23c0f67c25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=c6f015aa981a2c91, duration=2 ms] 2024-09-11 18:18:10.419 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,cb70c5597c480795,16517487a0a2c3d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=cb70c5597c480795, duration=10 ms] 2024-09-11 18:18:10.424 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1d7eff521d9b1e5a,3ecca0fd94d72e9d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=1d7eff521d9b1e5a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:18:10.426 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,1d7eff521d9b1e5a,3ecca0fd94d72e9d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=1d7eff521d9b1e5a, duration=4 ms] 2024-09-11 18:18:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:18:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:18:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,bf2dfcb92bc8b2e3,0de3317b5a7be273] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:11.319 INFO 2981023 --- [prometheus-minute-schedule-1,bf2dfcb92bc8b2e3,0de3317b5a7be273] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:11.433 INFO 2981023 --- [ocp-async-3,643d1150a1c3d44a,f3385e4cebe1162a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:12.137 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:12.137 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:12.687 INFO 2981023 --- [ocp-schedules-30,5bd95ee133a835dd,c14064b89acaee24] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:18:13.152 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049820, nodeCount=1000, affectRows=1000 2024-09-11 18:18:13.155 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049820, nodeCount=999, validCount=999 2024-09-11 18:18:13.156 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049820, nodeCount=1000, affectRows=1000 2024-09-11 18:18:13.158 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049820, nodeCount=1000, validCount=1000 2024-09-11 18:18:13.161 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049820, nodeCount=49, affectRows=49 2024-09-11 18:18:13.161 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049820, nodeCount=47, validCount=47 2024-09-11 18:18:13.433 INFO 2981023 --- [ocp-async-9,c65ca49f150767a9,5609182c93b5267b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,71d102d0a050b723,83d164fb7173d134] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,43616929256dbff4,533aeb8a7a74ee76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:18:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,71d102d0a050b723,83d164fb7173d134] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:15.138 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:15.138 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:15.207 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049820, nodeCount=480, affectRows=480 2024-09-11 18:18:15.433 INFO 2981023 --- [ocp-async-4,e0ec27d928f61b8f,625b97900434e519] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:15.515 INFO 2981023 --- [ocp-schedules-16,f32b925be325a7e1,844276307c6bd3e2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:18:15.520 INFO 2981023 --- [ocp-schedules-16,f32b925be325a7e1,844276307c6bd3e2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:18:15.520 INFO 2981023 --- [ocp-schedules-16,f32b925be325a7e1,844276307c6bd3e2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:18:15.997 INFO 2981023 --- [ocp-schedules-17,2a9cc48a4078e079,cad43c3b9ae5c3ce] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:18:15.998 INFO 2981023 --- [host-check-clockdiff1,2a9cc48a4078e079,8e6ffc9a74deb638] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:18:15.998 INFO 2981023 --- [host-check-clockdiff2,2a9cc48a4078e079,c3f416eb0fb0cc8f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:18:15.999 INFO 2981023 --- [host-check-clockdiff3,2a9cc48a4078e079,ce3b767cab894dec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:18:16.000 INFO 2981023 --- [host-check-clockdiff4,2a9cc48a4078e079,94c9eb9e397dd7b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:18:16.012 INFO 2981023 --- [ocp-schedules-10,415a2c1b06b94b10,b63557adffb81937] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:18:16.016 INFO 2981023 --- [host-check-clockdiff3,2a9cc48a4078e079,ce3b767cab894dec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049896 0 0 , error= 2024-09-11 18:18:16.016 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:18:16.016 INFO 2981023 --- [ocp-schedules-28,830e3611a23eddd8,b5114951dbdb23b1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:18:16.016 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:18:16.016 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:18:16.017 INFO 2981023 --- [host-check-agent-status1,830e3611a23eddd8,e0cea7ba7699576d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:18:16.017 INFO 2981023 --- [host-check-agent-status3,830e3611a23eddd8,736007cb1d3df834] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:18:16.017 INFO 2981023 --- [host-check-agent-status4,830e3611a23eddd8,a35aa5b1a584070c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:18:16.018 INFO 2981023 --- [host-check-agent-status2,830e3611a23eddd8,205e6a1dda1d8d00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:18:16.040 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:18:16.041 INFO 2981023 --- [host-check-agent-status4,830e3611a23eddd8,a35aa5b1a584070c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:18:16.044 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:18:16.045 INFO 2981023 --- [host-check-agent-status2,830e3611a23eddd8,205e6a1dda1d8d00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:18:16.046 INFO 2981023 --- [host-check-agent-status4,830e3611a23eddd8,a35aa5b1a584070c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:18:16.046 INFO 2981023 --- [host-check-agent-status2,830e3611a23eddd8,205e6a1dda1d8d00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:18:16.048 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:18:16.049 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:18:16.049 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:18:16.051 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:18:16.052 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:18:16.054 INFO 2981023 --- [host-check-agent-status3,830e3611a23eddd8,736007cb1d3df834] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:18:16.055 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:18:16.056 INFO 2981023 --- [host-check-agent-status1,830e3611a23eddd8,e0cea7ba7699576d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:18:16.067 INFO 2981023 --- [host-check-agent-status3,830e3611a23eddd8,736007cb1d3df834] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:18:16.088 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:18:16.093 INFO 2981023 --- [host-check-agent-status1,830e3611a23eddd8,e0cea7ba7699576d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:18:16.190 INFO 2981023 --- [ocp-schedules-8,26ea520ab97b64c7,a9d9e9e4041ff53e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:18:16.190 INFO 2981023 --- [ocp-schedules-24,a89a6bed4a52dda6,006c063f6c9b6104] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:18:16.191 INFO 2981023 --- [ocp-schedules-8,26ea520ab97b64c7,a9d9e9e4041ff53e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:18:16.192 INFO 2981023 --- [ocp-schedules-27,4d8b8f7064ea1f53,adc45b242e982f3d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:18:16.193 INFO 2981023 --- [ocp-schedules-1,4266bd49841c00db,fcfd7a1f779c53e4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:18:16.194 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049883058000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:18:16.201 INFO 2981023 --- [ocp-schedules-27,4d8b8f7064ea1f53,adc45b242e982f3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:18:16.202 INFO 2981023 --- [ocp-schedules-24,a89a6bed4a52dda6,006c063f6c9b6104] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:18:16.215 INFO 2981023 --- [ocp-schedules-24,a89a6bed4a52dda6,006c063f6c9b6104] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:18:16.299 INFO 2981023 --- [host-check-agent-status3,830e3611a23eddd8,736007cb1d3df834] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:18:16.300 INFO 2981023 --- [ocp-schedules-29,d902212aa778cb3a,eb39f3dc527825c7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:18:16.304 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:18:16.309 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:18:16.311 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:18:16.313 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.313 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:18:16.314 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:18:16.313454804+08:00 2024-09-11 18:18:16.314 INFO 2981023 --- [host-check-agent-status3,830e3611a23eddd8,736007cb1d3df834] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:18:16.314 INFO 2981023 --- [host-check-agent-status3,830e3611a23eddd8,736007cb1d3df834] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:18:16.314 INFO 2981023 --- [host-check-agent-status3,830e3611a23eddd8,736007cb1d3df834] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:18:16.314 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:18:16.314 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:18:16.313897858+08:00 2024-09-11 18:18:16.315 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:18:16.314+08:00 2024-09-11 18:18:16.317 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:18:16.314+08:00 and status AVAILABLE 2024-09-11 18:18:16.317 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.317 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:18:16.314+08:00 2024-09-11 18:18:16.318 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:18:16.319 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:18:16.319 INFO 2981023 --- [ocp-schedules-8,26ea520ab97b64c7,a9d9e9e4041ff53e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:18:16.320 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:18:16.323830870+08:00 2024-09-11 18:18:16.322 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:18:16.314+08:00 and status AVAILABLE 2024-09-11 18:18:16.322 INFO 2981023 --- [host-check-agent-status2,830e3611a23eddd8,205e6a1dda1d8d00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:18:16.322 INFO 2981023 --- [host-check-agent-status4,830e3611a23eddd8,a35aa5b1a584070c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:18:16.322 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:18:16.320+08:00 2024-09-11 18:18:16.324 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:18:16.320+08:00 and status AVAILABLE 2024-09-11 18:18:16.325 INFO 2981023 --- [host-check-agent-status2,830e3611a23eddd8,205e6a1dda1d8d00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:18:16.325 INFO 2981023 --- [host-check-agent-status2,830e3611a23eddd8,205e6a1dda1d8d00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:18:16.325 INFO 2981023 --- [host-check-agent-status2,830e3611a23eddd8,205e6a1dda1d8d00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:18:16.325 INFO 2981023 --- [host-check-agent-status4,830e3611a23eddd8,a35aa5b1a584070c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:18:16.325 INFO 2981023 --- [host-check-agent-status4,830e3611a23eddd8,a35aa5b1a584070c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:18:16.325 INFO 2981023 --- [host-check-agent-status4,830e3611a23eddd8,a35aa5b1a584070c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:18:16.330 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:18:16.331 INFO 2981023 --- [host-check-agent-status1,830e3611a23eddd8,e0cea7ba7699576d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:18:16.335 INFO 2981023 --- [host-check-agent-status1,830e3611a23eddd8,e0cea7ba7699576d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:18:16.335 INFO 2981023 --- [host-check-agent-status1,830e3611a23eddd8,e0cea7ba7699576d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:18:16.335 INFO 2981023 --- [host-check-agent-status1,830e3611a23eddd8,e0cea7ba7699576d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:18:16.337 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:18:16.339 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.339 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:16.346 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:18:16.347 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:18:16.352164186+08:00 2024-09-11 18:18:16.350 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:18:16.361 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:18:16.347+08:00 2024-09-11 18:18:16.365 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:18:16.347+08:00 and status AVAILABLE 2024-09-11 18:18:16.367 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:18:16.366228+08:00, lastProcessTime=2024-09-11T18:18:06.336764+08:00 2024-09-11 18:18:16.367 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:18:16.368 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:18:16.369 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:18:16.370 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:18:16.374 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:18:16.375 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:18:16.376 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:18:16.384 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.384 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:16.391 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.392 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:16.394 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:18:16.394 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:18:16.396 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:18:16.397 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:18:16.402 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:18:16.414 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:18:16.426 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:18:16.427 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:16.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.430 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:18:16.430 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:18:16.435 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:18:16.436 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:18:16.440 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:18:16.456 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:18:16.459 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:18:16.471 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:18:16.483 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:18:16.503 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:18:16.507 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:18:16.508 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:16.511 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.511 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:16.512 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:18:16.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.514 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:18:16.514 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049883058000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:18:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:18:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:18:16.524 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:18:16.532 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:18:16.532 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:18:16.532 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:18:16.533 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:18:16.543 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:18:16.555 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:18:16.561 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:18:16.572 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:18:16.579 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:18:16.587 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:18:16.587 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:18:16.593 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:18:16.593 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:16.594 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.594 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:18:16.595 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:18:16.597 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:16.598 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:18:16.598 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.598 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:18:16.599 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:18:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:18:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:18:16.606 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:18:16.608 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:18:16.626 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:18:16.628 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:18:16.656 INFO 2981023 --- [host-check-host-status4,415a2c1b06b94b10,548a9d34e1bb0b02] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:18:16.663 INFO 2981023 --- [host-check-host-status2,415a2c1b06b94b10,c6eb45466950507c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:18:16.669 INFO 2981023 --- [host-check-host-status3,415a2c1b06b94b10,98adcdc5403f6f11] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:18:16.674 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:18:16.686 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:18:16.688 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:16.689 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.691 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:16.692 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:18:16.693 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:18:16.698 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:18:16.711 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:18:16.712 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:18:16.712 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:18:16.725 INFO 2981023 --- [host-check-host-status1,415a2c1b06b94b10,59f7f489a8883120] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:18:16.736 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:18:16.748 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:18:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:16.754 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.754 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:18:16.755 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:18:16.759 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:18:16.761 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:16.762 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.762 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:18:16.763 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:18:16.763 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:18:16.773 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:18:16.773 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:18:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:18:16.778 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:18:16.366228+08:00 2024-09-11 18:18:16.808 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:18:16.813 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:16.813 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:16.814 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:18:16.819 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:18:16.819 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:18:16.819 INFO 2981023 --- [ocp-schedules-27,4d8b8f7064ea1f53,adc45b242e982f3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:18:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,c6242c1bd0a75621,25ee543845156eed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,c6242c1bd0a75621,25ee543845156eed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:17.433 INFO 2981023 --- [ocp-async-7,8e27db2c85e4a03f,88f381f639e2e072] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:17.514 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049820, nodeCount=480, validCount=480 2024-09-11 18:18:17.789 INFO 2981023 --- [ocp-async-8,aabc578b87094dd7,5f3a8a58a1e4d934] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:18:18.138 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:18.139 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:19.007 INFO 2981023 --- [ocp-schedules-29,93f8e500f0550cea,b055c3a89b7a54cb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:18:19.054 WARN 2981023 --- [scanner-13643912-1,93f8e500f0550cea,a35fa0756ceeac85] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:18:19.433 INFO 2981023 --- [ocp-async-2,747853cb4a294fbb,4a8d2c612208b3f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:20.136 INFO 2981023 --- [host-check-clockdiff2,2a9cc48a4078e079,c3f416eb0fb0cc8f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049900 6 6 , error= 2024-09-11 18:18:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,317a86953a692dbb,1aad63c8f43251dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,317a86953a692dbb,1aad63c8f43251dc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:21.139 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:21.141 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:21.255 INFO 2981023 --- [host-check-clockdiff1,2a9cc48a4078e079,8e6ffc9a74deb638] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049901 5 5 , error= 2024-09-11 18:18:21.433 INFO 2981023 --- [ocp-async-10,88d17b4f15ebdd8c,a152a1d0c4ff8503] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:21.957 INFO 2981023 --- [host-check-clockdiff4,2a9cc48a4078e079,94c9eb9e397dd7b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049901 0 0 , error= 2024-09-11 18:18:22.689 INFO 2981023 --- [ocp-schedules-6,fdb08048b10fce60,fda88e803ad59698] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:18:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,eec73a92c4ef2815,d3ec1bc38ac7dd62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,eec73a92c4ef2815,d3ec1bc38ac7dd62] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:23.433 INFO 2981023 --- [ocp-async-6,6c3ccb371c5bb4bc,f925a6967d6739fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:23.603 INFO 2981023 --- [ocp-schedules-7,280965bd69ba4012,1cd1dd4d6b0e6314] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 18:18:23.603 INFO 2981023 --- [ocp-schedules-7,280965bd69ba4012,1cd1dd4d6b0e6314] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 18:18:23.625 INFO 2981023 --- [pool-599-thread-1,280965bd69ba4012,259aa68493c9480b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:18:23.625 INFO 2981023 --- [pool-599-thread-2,280965bd69ba4012,ed49a4b4a9ab6a37] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:18:23.626 INFO 2981023 --- [pool-599-thread-3,280965bd69ba4012,815cc25ffa455199] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:18:23.648 INFO 2981023 --- [pool-599-thread-2,280965bd69ba4012,ed49a4b4a9ab6a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:23.648 INFO 2981023 --- [pool-599-thread-1,280965bd69ba4012,259aa68493c9480b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:23.649 INFO 2981023 --- [pool-599-thread-3,280965bd69ba4012,815cc25ffa455199] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:23.651 INFO 2981023 --- [pool-599-thread-2,280965bd69ba4012,ed49a4b4a9ab6a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049603000000, 1726049903000000] 2024-09-11 18:18:23.652 INFO 2981023 --- [pool-599-thread-3,280965bd69ba4012,815cc25ffa455199] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049603000000, 1726049903000000] 2024-09-11 18:18:23.654 INFO 2981023 --- [pool-599-thread-1,280965bd69ba4012,259aa68493c9480b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049603000000, 1726049903000000] 2024-09-11 18:18:23.659 INFO 2981023 --- [pool-599-thread-3,280965bd69ba4012,815cc25ffa455199] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:18:23.730 INFO 2981023 --- [pool-599-thread-1,280965bd69ba4012,259aa68493c9480b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:18:23.791 INFO 2981023 --- [pool-599-thread-2,280965bd69ba4012,ed49a4b4a9ab6a37] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:18:23.792 INFO 2981023 --- [ocp-schedules-7,280965bd69ba4012,1cd1dd4d6b0e6314] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 18:18:24.141 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:24.142 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:24.458 INFO 2981023 --- [ocp-schedules-3,a3e04058a899e2e4,1b1ee0c072a0d3a9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:18:25.433 INFO 2981023 --- [ocp-async-5,4d449b5754b02692,e2cb58143dc7652d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,4c83c2c6dfc6be00,3f077f6832bd44dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,4c83c2c6dfc6be00,3f077f6832bd44dc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:18:26.352163+08:00, lastProcessTime=2024-09-11T18:18:16.366228+08:00 2024-09-11 18:18:26.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:18:26.352163+08:00 2024-09-11 18:18:26.784 INFO 2981023 --- [ocp-schedules-20,39d0fb139b712717,972ba4288192fce6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:18:26.784 INFO 2981023 --- [ocp-schedules-20,39d0fb139b712717,972ba4288192fce6] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:18:26.785 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:18:26.785 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:18:26.785 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:18:26.810 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:18:26.811 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:18:26.811 INFO 2981023 --- [ocp-schedules-20,39d0fb139b712717,972ba4288192fce6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:18:26.811 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:18:26.814 INFO 2981023 --- [ocp-schedules-20,39d0fb139b712717,972ba4288192fce6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:18:26.814 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:18:26.814 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:18:26.866 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:18:27.063 INFO 2981023 --- [ocp-schedules-20,39d0fb139b712717,972ba4288192fce6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:18:27.064 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:18:27.066 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:18:27.076 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:18:27.078 INFO 2981023 --- [ocp-schedules-20,39d0fb139b712717,972ba4288192fce6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:18:27.143 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:27.143 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:27.433 INFO 2981023 --- [ocp-async-8,f091b381c84d1619,19cbb73902916739] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:27.789 INFO 2981023 --- [ocp-async-3,b636fdad6e445e74,5ee2726027810404] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:18:28.513 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.515 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obCluster 2024-09-11 18:18:28.520 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.520 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:18:28.523 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.526 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.534 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.534 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.535 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.536 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.537 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.537 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.538 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.538 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:18:28.539 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.539 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.540 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.543 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.544 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:18:28.545 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.549 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.552 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.553 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.554 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.554 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.555 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.556 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.560 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.585 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 31 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.633 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 10 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.643 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 165 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.658 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:12:28.311, end=2024-09-11T18:17:28.509} 2024-09-11 18:18:28.659 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 4 dubious sqls 2024-09-11 18:18:29.205 INFO 2981023 --- [ocp-schedules-7,8be9283375cb7d22,06f70e383a518263] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:18:29.261 WARN 2981023 --- [scanner-13644116-1,8be9283375cb7d22,1e08529bae191af8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:18:29.291 INFO 2981023 --- [alarm-detect-13,8be9283375cb7d22,10bacbe207b02baf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645900, data is empty. 2024-09-11 18:18:29.296 INFO 2981023 --- [alarm-detect-13,8be9283375cb7d22,10bacbe207b02baf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645900, data is empty. 2024-09-11 18:18:29.299 INFO 2981023 --- [alarm-detect-13,8be9283375cb7d22,10bacbe207b02baf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645900, data is empty. 2024-09-11 18:18:29.306 INFO 2981023 --- [alarm-detect-13,8be9283375cb7d22,7236216b243b99cc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645900, data is empty. 2024-09-11 18:18:29.310 INFO 2981023 --- [alarm-detect-17,8be9283375cb7d22,3c1093b63c7a8dec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645900, data is empty. 2024-09-11 18:18:29.310 INFO 2981023 --- [alarm-detect-13,8be9283375cb7d22,7236216b243b99cc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645900, data is empty. 2024-09-11 18:18:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,95812ee1e836e30d,0201beaf7210c660] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:29.314 INFO 2981023 --- [alarm-detect-17,8be9283375cb7d22,3c1093b63c7a8dec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645900, data is empty. 2024-09-11 18:18:29.314 INFO 2981023 --- [alarm-detect-13,8be9283375cb7d22,7236216b243b99cc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645900, data is empty. 2024-09-11 18:18:29.322 INFO 2981023 --- [alarm-detect-17,8be9283375cb7d22,3c1093b63c7a8dec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645900, data is empty. 2024-09-11 18:18:29.322 INFO 2981023 --- [prometheus-minute-schedule-1,95812ee1e836e30d,0201beaf7210c660] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:29.434 INFO 2981023 --- [ocp-async-1,7de205140aec4917,40224902fd4c42f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:29.443 INFO 2981023 --- [alarm-detect-14,8be9283375cb7d22,9799b39458974c3e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645900, data is empty. 2024-09-11 18:18:29.448 INFO 2981023 --- [alarm-detect-14,8be9283375cb7d22,9799b39458974c3e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645900, data is empty. 2024-09-11 18:18:29.453 INFO 2981023 --- [alarm-detect-14,8be9283375cb7d22,9799b39458974c3e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645900, data is empty. 2024-09-11 18:18:30.143 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:30.144 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:31.434 INFO 2981023 --- [ocp-async-3,288d3b690b48bbd3,be0d52fa8046451a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,3a3ed04b9572b4ae,a159b78ae621b618] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,3a3ed04b9572b4ae,a159b78ae621b618] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:32.692 INFO 2981023 --- [ocp-schedules-4,4846f4d49608fa69,13d3317f281f0b6e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:18:33.144 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:33.145 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:33.434 INFO 2981023 --- [ocp-async-2,8e05bd6dec5be513,9892f12fb7485bab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,fe808a973c83e578,917dca92aefa48d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,fe808a973c83e578,917dca92aefa48d6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:35.434 INFO 2981023 --- [ocp-async-10,ef38d3bee8d4fae6,9e0cc8cebc892ec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:35.523 INFO 2981023 --- [ocp-schedules-30,daf0b32417219b48,c1f2a10e9a349347] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:18:35.532 INFO 2981023 --- [ocp-schedules-30,daf0b32417219b48,c1f2a10e9a349347] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:18:35.532 INFO 2981023 --- [ocp-schedules-30,daf0b32417219b48,c1f2a10e9a349347] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:18:36.146 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:36.146 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:18:36.336618+08:00, lastProcessTime=2024-09-11T18:18:26.352163+08:00 2024-09-11 18:18:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:18:36.336618+08:00 2024-09-11 18:18:37.314 INFO 2981023 --- [prometheus-cache-monitor1,b851470a96aabda5,1d1af152abf65c94] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:18:37.315 INFO 2981023 --- [prometheus-cache-monitor1,b851470a96aabda5,1d1af152abf65c94] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:18:37.434 INFO 2981023 --- [ocp-async-1,ada055e9b5cf76dc,40eb2165594d643c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:37.789 INFO 2981023 --- [ocp-async-7,d608e882cd20adbf,9f8aabb8604e560a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:18:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,c914ff3df8b1cf7a,af97b935fef7d70a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,c914ff3df8b1cf7a,af97b935fef7d70a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:39.146 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:39.147 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:39.434 INFO 2981023 --- [ocp-async-8,21a241fbf687258a,a18176e2a7189938] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:39.495 INFO 2981023 --- [ocp-schedules-30,375ea60916876db5,99e03d63a7cde788] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:18:39.533 WARN 2981023 --- [scanner-13644491-1,375ea60916876db5,cf4fef0cde0ac710] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:18:40.003 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:18:40.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:18:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,e12053e54591af95,0e81372b82b21797] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,e12053e54591af95,0e81372b82b21797] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:41.434 INFO 2981023 --- [ocp-async-7,8f7e3d703d61b11b,3687ef68dc5b882b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:42.147 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:42.148 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:42.694 INFO 2981023 --- [ocp-schedules-9,963495ae26d9b945,5ee0676331f487b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:18:42.864 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:18:42.864 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:18:42.866 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:18:42.866 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:18:42.866 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:18:42.866 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:18:42.868 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:18:42.868 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:18:42.868 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:18:42.868 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:18:42.894 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:18:42.898 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:18:42.899 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:18:42.899 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:18:42.899 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:18:42.899 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:18:42.903 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:18:43.160 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:18:43.406 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.537653793S 2024-09-11 18:18:43.406 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:18:43.435 INFO 2981023 --- [ocp-async-5,9be20e136df3514d,79da537ebd13de98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:43.459 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:18:43.459 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:18:43.475 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:18:43.508 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:18:43.508 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:18:43.508 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:18:43.508 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:18:43.508 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:18:43.508 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:18:43.513 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:18:43.592 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] o.s.boot.SpringApplication : Started application in 0.726 seconds (JVM running for 85723.465) 2024-09-11 18:18:43.592 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.726513909S 2024-09-11 18:18:43.792 INFO 2981023 --- [ocp-schedules-22,d98cca2c3b634cf4,c878ef0ade98e834] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:18:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,3a4566070886ec17,68352ededd4603a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,3a4566070886ec17,68352ededd4603a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:45.101 INFO 2981023 --- [prometheus-second-schedule-1,1edd3ed424d0c2db,6408244e20b11179] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:18:45.148 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:45.148 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:45.435 INFO 2981023 --- [ocp-async-3,f3c6fee194e5e092,62c7bfc3068744b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:46.023 INFO 2981023 --- [ocp-schedules-1,0636476191224bd4,655a3c84d8bcc4c3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:18:46.023 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:18:46.026 INFO 2981023 --- [ocp-schedules-11,bde26c22e0c37c08,be3b4a863c0d5e45] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:18:46.028 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:18:46.029 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:18:46.031 INFO 2981023 --- [ocp-schedules-13,4bab0c0fd667602b,0f215e6118500a7b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:18:46.033 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:18:46.037 INFO 2981023 --- [host-check-agent-status1,4bab0c0fd667602b,565a043c6f9fdab7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:18:46.037 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:18:46.038 INFO 2981023 --- [host-check-agent-status2,4bab0c0fd667602b,3a355b151c9bef5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:18:46.039 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:18:46.040 INFO 2981023 --- [host-check-agent-status3,4bab0c0fd667602b,4d2b5905c7e196bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:18:46.040 INFO 2981023 --- [host-check-agent-status4,4bab0c0fd667602b,c3ae07f0794e8ed1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:18:46.042 INFO 2981023 --- [host-check-clockdiff1,bde26c22e0c37c08,75474c1acccd6728] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:18:46.042 INFO 2981023 --- [host-check-clockdiff2,bde26c22e0c37c08,7e8c751bed0f26bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:18:46.044 INFO 2981023 --- [host-check-agent-status1,4bab0c0fd667602b,565a043c6f9fdab7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:18:46.046 INFO 2981023 --- [host-check-agent-status1,4bab0c0fd667602b,565a043c6f9fdab7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:18:46.049 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:18:46.049 INFO 2981023 --- [host-check-agent-status2,4bab0c0fd667602b,3a355b151c9bef5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:18:46.050 INFO 2981023 --- [host-check-agent-status4,4bab0c0fd667602b,c3ae07f0794e8ed1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:18:46.050 INFO 2981023 --- [host-check-agent-status3,4bab0c0fd667602b,4d2b5905c7e196bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:18:46.051 INFO 2981023 --- [host-check-agent-status2,4bab0c0fd667602b,3a355b151c9bef5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:18:46.051 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:18:46.056163120+08:00 2024-09-11 18:18:46.052 INFO 2981023 --- [host-check-agent-status4,4bab0c0fd667602b,c3ae07f0794e8ed1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:18:46.052 INFO 2981023 --- [host-check-clockdiff3,bde26c22e0c37c08,e57505fa7f6ab3f8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:18:46.054 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:18:46.051+08:00 2024-09-11 18:18:46.054 INFO 2981023 --- [host-check-agent-status3,4bab0c0fd667602b,4d2b5905c7e196bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:18:46.055 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:18:46.060 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:18:46.061 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:18:46.063 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:18:46.063 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:18:46.051+08:00 and status AVAILABLE 2024-09-11 18:18:46.071 INFO 2981023 --- [host-check-clockdiff4,bde26c22e0c37c08,bdfcba15292d51f6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:18:46.081 INFO 2981023 --- [host-check-clockdiff3,bde26c22e0c37c08,e57505fa7f6ab3f8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049926 0 0 , error= 2024-09-11 18:18:46.081 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:18:46.088 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:18:46.098 INFO 2981023 --- [host-check-clockdiff1,bde26c22e0c37c08,75474c1acccd6728] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049926 5 5 , error= 2024-09-11 18:18:46.102 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:18:46.092368752+08:00 2024-09-11 18:18:46.108 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:18:46.109 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:18:46.102+08:00 2024-09-11 18:18:46.110 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:18:46.114 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:18:46.115 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:18:46.102+08:00 and status AVAILABLE 2024-09-11 18:18:46.117 INFO 2981023 --- [host-check-agent-status1,4bab0c0fd667602b,565a043c6f9fdab7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:18:46.118 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:18:46.123280186+08:00 2024-09-11 18:18:46.120 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:18:46.118+08:00 2024-09-11 18:18:46.122 INFO 2981023 --- [host-check-agent-status1,4bab0c0fd667602b,565a043c6f9fdab7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:18:46.122 INFO 2981023 --- [host-check-agent-status1,4bab0c0fd667602b,565a043c6f9fdab7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:18:46.122 INFO 2981023 --- [host-check-agent-status1,4bab0c0fd667602b,565a043c6f9fdab7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:18:46.122 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:18:46.118+08:00 and status AVAILABLE 2024-09-11 18:18:46.124 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:18:46.124 INFO 2981023 --- [host-check-agent-status4,4bab0c0fd667602b,c3ae07f0794e8ed1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:18:46.125 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:18:46.126 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:18:46.129 INFO 2981023 --- [host-check-agent-status4,4bab0c0fd667602b,c3ae07f0794e8ed1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:18:46.129 INFO 2981023 --- [host-check-agent-status4,4bab0c0fd667602b,c3ae07f0794e8ed1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:18:46.129 INFO 2981023 --- [host-check-agent-status4,4bab0c0fd667602b,c3ae07f0794e8ed1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:18:46.131 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:18:46.134 INFO 2981023 --- [host-check-agent-status2,4bab0c0fd667602b,3a355b151c9bef5d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:18:46.135 INFO 2981023 --- [host-check-agent-status3,4bab0c0fd667602b,4d2b5905c7e196bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:18:46.138 INFO 2981023 --- [host-check-agent-status3,4bab0c0fd667602b,4d2b5905c7e196bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:18:46.138 INFO 2981023 --- [host-check-agent-status3,4bab0c0fd667602b,4d2b5905c7e196bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:18:46.138 INFO 2981023 --- [host-check-agent-status3,4bab0c0fd667602b,4d2b5905c7e196bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:18:46.139 INFO 2981023 --- [host-check-agent-status2,4bab0c0fd667602b,3a355b151c9bef5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:18:46.139 INFO 2981023 --- [host-check-agent-status2,4bab0c0fd667602b,3a355b151c9bef5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:18:46.139 INFO 2981023 --- [host-check-agent-status2,4bab0c0fd667602b,3a355b151c9bef5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:18:46.144 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:18:46.145 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:18:46.146 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:18:46.147 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:18:46.150 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:18:46.151 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:18:46.150822558+08:00 2024-09-11 18:18:46.151 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:18:46.151+08:00 2024-09-11 18:18:46.152 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:18:46.154 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:18:46.151+08:00 and status AVAILABLE 2024-09-11 18:18:46.162 INFO 2981023 --- [ocp-schedules-26,dc645e1911c894df,30fbe22a4cd9d039] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:18:46.164 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:18:46.173 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:18:46.178 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:18:46.178 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:18:46.187 INFO 2981023 --- [ocp-schedules-19,7bdae00509e99d4a,2be748227caf67d4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:18:46.187 INFO 2981023 --- [ocp-schedules-24,6e98ce7918a8d602,914e10e59d037ab5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:18:46.192 INFO 2981023 --- [ocp-schedules-5,66b450007d2bf723,65a3b945039e66db] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:18:46.192 INFO 2981023 --- [ocp-schedules-28,9f92c1c6850b23bd,b30f944668301d66] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:18:46.193 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:18:46.195 INFO 2981023 --- [ocp-schedules-19,7bdae00509e99d4a,2be748227caf67d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:18:46.196 INFO 2981023 --- [ocp-schedules-26,dc645e1911c894df,30fbe22a4cd9d039] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:18:46.196 INFO 2981023 --- [ocp-schedules-24,6e98ce7918a8d602,914e10e59d037ab5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:18:46.196 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049883058000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:18:46.197 INFO 2981023 --- [ocp-schedules-24,6e98ce7918a8d602,914e10e59d037ab5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:18:46.205 INFO 2981023 --- [ocp-schedules-5,66b450007d2bf723,65a3b945039e66db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:18:46.207 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:18:46.211 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:18:46.219 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:18:46.224 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:18:46.226 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:18:46.228 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:18:46.234 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:18:46.238 INFO 2981023 --- [ocp-schedules-22,38b4ec118d1fe4a3,faaf7c9755b89983] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:18:46.247 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:18:46.262 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.263 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:18:46.263 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:18:46.269 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:18:46.271 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.271 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:18:46.273 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:18:46.274 INFO 2981023 --- [ocp-schedules-19,7bdae00509e99d4a,2be748227caf67d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:18:46.276 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:18:46.279 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:18:46.287 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:18:46.292 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.293 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:46.294 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:18:46.296 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:18:46.307 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.308 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:46.309 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:18:46.314 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:18:46.315 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.315 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:46.319 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:18:46.332 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.332 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:18:46.333 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:18:46.333 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:18:46.333 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:18:46.333 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.341 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:18:46.333027+08:00, lastProcessTime=2024-09-11T18:18:36.336618+08:00 2024-09-11 18:18:46.342 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:18:46.350 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:18:46.357 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:18:46.362 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:18:46.366 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.366 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:18:46.370 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.370 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:18:46.384 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:18:46.384 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:18:46.394 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:18:46.400 INFO 2981023 --- [host-check-host-status1,0636476191224bd4,cb7478954b6c9bd2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:18:46.407 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:18:46.418 INFO 2981023 --- [host-check-host-status2,0636476191224bd4,2a0bbc868a382bae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:18:46.419 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:18:46.422 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:18:46.423 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.423 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:18:46.423 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049883058000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:18:46.423 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.424 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:18:46.424 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:18:46.425 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.429 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.430 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.430 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:46.431 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:18:46.435 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.437 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:18:46.437 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:18:46.446 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:18:46.447 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:18:46.447 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:18:46.452 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:18:46.458 INFO 2981023 --- [host-check-host-status4,0636476191224bd4,2521304166d5f32e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:18:46.464 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:18:46.468 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:18:46.478 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.479 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.481 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:18:46.481 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:18:46.482 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:18:46.485 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:18:46.485 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.489 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:18:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:18:46.494 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.495 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.495 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:18:46.496 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:18:46.499 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.499 INFO 2981023 --- [host-check-host-status3,0636476191224bd4,53e450eb7717c90e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:18:46.500 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:18:46.500 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:18:46.501 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.507 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:18:46.508 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:18:46.509 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.510 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.510 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:18:46.530 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:18:46.544 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.544 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.545 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:18:46.545 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:18:46.546 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.548 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:18:46.549 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:18:46.549 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.550 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.550 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:18:46.551 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:18:46.551 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.565 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:18:46.565 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:18:46.566 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:18:46.572 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:18:46.580 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:18:46.581 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.585 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.585 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:18:46.586 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:18:46.590 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.591 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.591 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:18:46.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:18:46.595 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:18:46.596 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:18:46.596 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:18:46.598 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:18:46.599 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:18:46.603 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.604 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.604 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:18:46.605 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:18:46.605 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.612 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:18:46.621 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:18:46.333027+08:00 2024-09-11 18:18:46.641 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:18:46.648 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:18:46.648 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:46.649 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:18:46.649 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:18:46.654 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:18:46.654 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:18:46.654 INFO 2981023 --- [ocp-schedules-5,66b450007d2bf723,65a3b945039e66db] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:18:46.665 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:18:46.672 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:18:46.675 INFO 2981023 --- [ocp-schedules-25,6a164e54a830cc50,f4aa8a168530b45e] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:18:46.766 INFO 2981023 --- [ocp-schedules-10,568e5e9894e25f0e,aea9870fee79be60] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:18:46.785 WARN 2981023 --- [ocp-schedules-10,568e5e9894e25f0e,aea9870fee79be60] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:18:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,15fab0f735a00f93,f58a47513fa58e25] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,15fab0f735a00f93,f58a47513fa58e25] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:47.435 INFO 2981023 --- [ocp-async-7,3d31b2da2c65da65,fb7ce2d17f89593a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:47.789 INFO 2981023 --- [ocp-async-4,82aad6d91b6bd54d,e2a7a673cf49fe2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:18:47.833 INFO 2981023 --- [host-check-clockdiff2,bde26c22e0c37c08,7e8c751bed0f26bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049927 5 5 , error= 2024-09-11 18:18:48.149 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:48.149 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:49.435 INFO 2981023 --- [ocp-async-1,86529dd8e23192b9,0ee47af3137591fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:49.732 INFO 2981023 --- [ocp-schedules-21,f623426a65cc29f8,aeebdc3848416118] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:18:49.811 WARN 2981023 --- [scanner-13644723-1,f623426a65cc29f8,42224139e657c896] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:18:49.835 INFO 2981023 --- [alarm-detect-8,f623426a65cc29f8,e27a5cc74cafcc79] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645900, data is empty. 2024-09-11 18:18:49.842 INFO 2981023 --- [alarm-detect-8,f623426a65cc29f8,e27a5cc74cafcc79] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645900, data is empty. 2024-09-11 18:18:49.850 INFO 2981023 --- [alarm-detect-8,f623426a65cc29f8,e27a5cc74cafcc79] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645900, data is empty. 2024-09-11 18:18:49.858 INFO 2981023 --- [alarm-detect-4,f623426a65cc29f8,1402c6ba732f0c6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645900, data is empty. 2024-09-11 18:18:49.859 INFO 2981023 --- [alarm-detect-5,f623426a65cc29f8,9242a132a3662902] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645900, data is empty. 2024-09-11 18:18:49.862 INFO 2981023 --- [alarm-detect-4,f623426a65cc29f8,1402c6ba732f0c6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645900, data is empty. 2024-09-11 18:18:49.867 INFO 2981023 --- [alarm-detect-5,f623426a65cc29f8,9242a132a3662902] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645900, data is empty. 2024-09-11 18:18:49.870 INFO 2981023 --- [alarm-detect-4,f623426a65cc29f8,1402c6ba732f0c6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645900, data is empty. 2024-09-11 18:18:49.873 INFO 2981023 --- [alarm-detect-5,f623426a65cc29f8,9242a132a3662902] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645900, data is empty. 2024-09-11 18:18:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:18:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:18:49.964 INFO 2981023 --- [alarm-detect-11,f623426a65cc29f8,db6843cf5fe109b9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645900, data is empty. 2024-09-11 18:18:49.968 INFO 2981023 --- [alarm-detect-11,f623426a65cc29f8,db6843cf5fe109b9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645900, data is empty. 2024-09-11 18:18:49.972 INFO 2981023 --- [alarm-detect-11,f623426a65cc29f8,db6843cf5fe109b9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645900, data is empty. 2024-09-11 18:18:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,2aaf3ca7c23f1114,a2af3326639bb5cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,2aaf3ca7c23f1114,a2af3326639bb5cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:50.351 INFO 2981023 --- [ocp-schedules-24,f91e54e713f91c6b,072c2096688a3928] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:18:51.149 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:51.150 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:51.435 INFO 2981023 --- [ocp-async-4,9a91c14fba6c7f11,1098bfc2a588b615] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:51.624 INFO 2981023 --- [ocp-schedules-10,06aad6fbc2e130f2,04f3169cc088191f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:18:51.625 INFO 2981023 --- [ocp-schedules-10,06aad6fbc2e130f2,04f3169cc088191f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:18:51.625 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:18:51.625 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:18:51.625 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:18:51.626 INFO 2981023 --- [ocp-schedules-10,06aad6fbc2e130f2,04f3169cc088191f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:18:52.061 INFO 2981023 --- [host-check-clockdiff4,bde26c22e0c37c08,bdfcba15292d51f6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049932 0 0 , error= 2024-09-11 18:18:52.697 INFO 2981023 --- [ocp-schedules-4,08afa52db7de751a,d731c09cce08a26d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:18:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,f178521302b5d71f,a8443ef515c06411] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,f178521302b5d71f,a8443ef515c06411] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:53.344 INFO 2981023 --- [ocp-schedules-8,3244e82725956525,12e36127ae1172ce] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:18:53.344 INFO 2981023 --- [ocp-schedules-8,3244e82725956525,12e36127ae1172ce] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:18:53.435 INFO 2981023 --- [ocp-async-10,b68bf3214b0e6421,bd8adc7100b84ead] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:54.150 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:54.150 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:54.460 INFO 2981023 --- [ocp-schedules-24,ebadc486c2c9e092,0bf439996fd7de95] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:18:54.934 INFO 2981023 --- [ocp-schedules-20,83f16822f5f935f0,b3dab9a491ce9d7c] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T18:18:54.934+08:00, tenantSize=6 2024-09-11 18:18:54.938 INFO 2981023 --- [ocp-schedules-20,83f16822f5f935f0,b3dab9a491ce9d7c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726049934935000 2024-09-11 18:18:54.938 INFO 2981023 --- [ocp-schedules-20,83f16822f5f935f0,b3dab9a491ce9d7c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726049634897000 2024-09-11 18:18:54.954 INFO 2981023 --- [ocp-schedules-20,83f16822f5f935f0,b3dab9a491ce9d7c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 18:18:54.997 INFO 2981023 --- [ocp-schedules-9,70141d1e21f7cc83,24d69e3b084364d7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:18:55.014 WARN 2981023 --- [ocp-schedules-9,70141d1e21f7cc83,24d69e3b084364d7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:18:55.086 INFO 2981023 --- [ocp-schedules-27,9ad37524fbab8722,9f75dcb8c186d3e8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:18:55.086 INFO 2981023 --- [ocp-schedules-27,9ad37524fbab8722,9f75dcb8c186d3e8] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:18:55.089 INFO 2981023 --- [ocp-schedules-27,9ad37524fbab8722,9f75dcb8c186d3e8] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:18:55.089 INFO 2981023 --- [ocp-schedules-27,9ad37524fbab8722,9f75dcb8c186d3e8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:18:55.089 INFO 2981023 --- [ocp-schedules-27,9ad37524fbab8722,9f75dcb8c186d3e8] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:18:55.092 INFO 2981023 --- [ocp-schedules-27,9ad37524fbab8722,9f75dcb8c186d3e8] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:18:55.092 INFO 2981023 --- [ocp-schedules-27,9ad37524fbab8722,9f75dcb8c186d3e8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:18:55.092 INFO 2981023 --- [ocp-schedules-27,9ad37524fbab8722,9f75dcb8c186d3e8] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:18:55.100 INFO 2981023 --- [ocp-schedules-27,9ad37524fbab8722,9f75dcb8c186d3e8] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:18:55.435 INFO 2981023 --- [ocp-async-5,79c5dc151b91e892,d40a3c76a4723467] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:55.533 INFO 2981023 --- [ocp-schedules-14,a4291b06da8edeb7,5930cf40a20c2a83] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:18:55.538 INFO 2981023 --- [ocp-schedules-14,a4291b06da8edeb7,5930cf40a20c2a83] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:18:55.538 INFO 2981023 --- [ocp-schedules-14,a4291b06da8edeb7,5930cf40a20c2a83] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:18:55.883 INFO 2981023 --- [ocp-schedules-28,491383439a7eee9c,f1776f68dd870a04] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:18:55.883 INFO 2981023 --- [ocp-schedules-28,491383439a7eee9c,f1776f68dd870a04] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:18:55.886 INFO 2981023 --- [ocp-schedules-28,491383439a7eee9c,f1776f68dd870a04] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:18:55.886 INFO 2981023 --- [ocp-schedules-28,491383439a7eee9c,f1776f68dd870a04] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:18:55.888 INFO 2981023 --- [ocp-schedules-28,491383439a7eee9c,f1776f68dd870a04] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:18:55.888 INFO 2981023 --- [ocp-schedules-28,491383439a7eee9c,f1776f68dd870a04] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:18:56.089 INFO 2981023 --- [ocp-schedules-22,0629dbec7f2f38ff,dfc9448b5c95071f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:18:56.096 INFO 2981023 --- [ocp-schedules-22,0629dbec7f2f38ff,dfc9448b5c95071f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:18:56.096 INFO 2981023 --- [ocp-schedules-22,0629dbec7f2f38ff,dfc9448b5c95071f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:18:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,49555b59bdfc3fb4,c0edeec5a1efa454] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,49555b59bdfc3fb4,c0edeec5a1efa454] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:56.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:18:56.337348+08:00, lastProcessTime=2024-09-11T18:18:46.333027+08:00 2024-09-11 18:18:56.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:18:56.337348+08:00 2024-09-11 18:18:57.150 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:18:57.151 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:18:57.435 INFO 2981023 --- [ocp-async-2,e1b5ccc7ab14d046,8c4ec4a4a04303b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:57.789 INFO 2981023 --- [ocp-async-5,525fa29d53264b97,2e66064a1feeae5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:18:58.254 INFO 2981023 --- [ocp-schedules-15,c7ea8b507ba47f23,042f7e6e0c7d606d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:18:58.254 INFO 2981023 --- [ocp-schedules-15,c7ea8b507ba47f23,042f7e6e0c7d606d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:18:58.269 INFO 2981023 --- [Collect all tenants major compactions1,c7ea8b507ba47f23,229615761cac8894] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:18:58.270 INFO 2981023 --- [Collect all tenants major compactions5,c7ea8b507ba47f23,1ab541d281c6d0ce] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:18:58.271 INFO 2981023 --- [Collect all tenants major compactions6,c7ea8b507ba47f23,d689b47c4fc56b87] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:18:58.272 INFO 2981023 --- [Collect all tenants major compactions2,c7ea8b507ba47f23,2cf34d4d53b0b5a7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:18:58.272 INFO 2981023 --- [Collect all tenants major compactions3,c7ea8b507ba47f23,a257d9f732edf6a0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:18:58.272 INFO 2981023 --- [Collect all tenants major compactions4,c7ea8b507ba47f23,c65a9fc8cd15ba3f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:18:58.287 INFO 2981023 --- [Collect all tenants major compactions5,c7ea8b507ba47f23,1ab541d281c6d0ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:58.288 INFO 2981023 --- [Collect all tenants major compactions1,c7ea8b507ba47f23,229615761cac8894] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:58.288 INFO 2981023 --- [Collect all tenants major compactions5,c7ea8b507ba47f23,1ab541d281c6d0ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:18:58.288 INFO 2981023 --- [Collect all tenants major compactions3,c7ea8b507ba47f23,a257d9f732edf6a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:58.289 INFO 2981023 --- [Collect all tenants major compactions1,c7ea8b507ba47f23,229615761cac8894] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:18:58.289 INFO 2981023 --- [Collect all tenants major compactions3,c7ea8b507ba47f23,a257d9f732edf6a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:18:58.290 INFO 2981023 --- [Collect all tenants major compactions4,c7ea8b507ba47f23,c65a9fc8cd15ba3f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:58.292 INFO 2981023 --- [Collect all tenants major compactions4,c7ea8b507ba47f23,c65a9fc8cd15ba3f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:18:58.292 INFO 2981023 --- [Collect all tenants major compactions5,c7ea8b507ba47f23,1ab541d281c6d0ce] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:18:58.292 INFO 2981023 --- [Collect all tenants major compactions3,c7ea8b507ba47f23,a257d9f732edf6a0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:18:58.292 INFO 2981023 --- [Collect all tenants major compactions6,c7ea8b507ba47f23,d689b47c4fc56b87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:58.293 INFO 2981023 --- [Collect all tenants major compactions2,c7ea8b507ba47f23,2cf34d4d53b0b5a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:18:58.294 INFO 2981023 --- [Collect all tenants major compactions6,c7ea8b507ba47f23,d689b47c4fc56b87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:18:58.294 INFO 2981023 --- [Collect all tenants major compactions4,c7ea8b507ba47f23,c65a9fc8cd15ba3f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:18:58.295 INFO 2981023 --- [Collect all tenants major compactions2,c7ea8b507ba47f23,2cf34d4d53b0b5a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:18:58.295 INFO 2981023 --- [Collect all tenants major compactions5,c7ea8b507ba47f23,1ab541d281c6d0ce] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:18:58.296 INFO 2981023 --- [Collect all tenants major compactions2,c7ea8b507ba47f23,2cf34d4d53b0b5a7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:18:58.296 INFO 2981023 --- [Collect all tenants major compactions2,c7ea8b507ba47f23,2cf34d4d53b0b5a7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:18:58.297 INFO 2981023 --- [Collect all tenants major compactions6,c7ea8b507ba47f23,d689b47c4fc56b87] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:18:58.297 INFO 2981023 --- [Collect all tenants major compactions6,c7ea8b507ba47f23,d689b47c4fc56b87] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:18:58.297 INFO 2981023 --- [Collect all tenants major compactions3,c7ea8b507ba47f23,a257d9f732edf6a0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:18:58.298 INFO 2981023 --- [Collect all tenants major compactions1,c7ea8b507ba47f23,229615761cac8894] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:18:58.298 INFO 2981023 --- [Collect all tenants major compactions1,c7ea8b507ba47f23,229615761cac8894] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:18:58.299 INFO 2981023 --- [Collect all tenants major compactions4,c7ea8b507ba47f23,c65a9fc8cd15ba3f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:18:58.300 INFO 2981023 --- [ocp-schedules-15,c7ea8b507ba47f23,042f7e6e0c7d606d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:18:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,1ddd8b217aa795b7,9dad593610a596eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726049940, interval=60 2024-09-11 18:18:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,1ddd8b217aa795b7,9dad593610a596eb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:18:59.387 INFO 2981023 --- [ocp-schedules-14,ea1696b3ca355d86,369f6a06923453ac] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:18:59.387+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:18:59.391 INFO 2981023 --- [ocp-schedules-14,ea1696b3ca355d86,369f6a06923453ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:18:59.436 INFO 2981023 --- [ocp-async-10,cd68075d8c7459a9,4d7be63ae33aa01d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:18:59.526 INFO 2981023 --- [ocp-schedules-14,ea1696b3ca355d86,369f6a06923453ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 55 to SqlAggregateData 2024-09-11 18:18:59.528 INFO 2981023 --- [ocp-schedules-14,ea1696b3ca355d86,369f6a06923453ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:19:00.018 INFO 2981023 --- [ocp-schedules-2,333447db960821b8,049e1260b62cb87c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:19:00.055 WARN 2981023 --- [scanner-13645024-1,333447db960821b8,153c6a3393f6e151] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:19:00.151 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:00.154 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:00.413 INFO 2981023 --- [ocp-schedules-29,90b89c162da5a965,8775af349000d7c5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:19:00.414 INFO 2981023 --- [ocp-schedules-29,90b89c162da5a965,8775af349000d7c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:19:00.446 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:19:00.447 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.448 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:19:00.448 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.451 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:19:00.451 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:19:00.451 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.452 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:19:00.454 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:19:00.461 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.462 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.466 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.467 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.468 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.468 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.469 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.470 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.481 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.482 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.482 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.482 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:19:00.485 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.485 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.489 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.489 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:19:00.494 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.494 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:19:00.499 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.499 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.500 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.500 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.500 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.500 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:19:00.501 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:19:00.501 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5791D734-0-0] 2024-09-11 18:19:00.501 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:19:00.501 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.502 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:19:00.511 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.512 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:19:00.513 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.513 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.514 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:19:00.514 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:19:00.514 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.515 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.515 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:19:00.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.515 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:19:00.515 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:19:00.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:19:00.516 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.517 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:19:00.518 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.519 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:19:00.522 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:19:00.526 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.526 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:19:00.527 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.527 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.527 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:19:00.528 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:19:00.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.528 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:19:00.528 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5791D73E-0-0] 2024-09-11 18:19:00.529 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:19:00.529 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:19:00.529 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.530 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:19:00.530 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.531 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:19:00.532 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:19:00.534 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:19:00.536 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.537 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:19:00.537 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.538 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:00.538 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:19:00.538 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.539 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:19:00.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.540 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:19:00.540 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.541 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.541 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:19:00.542 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.543 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:19:00.543 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:19:00.545 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:19:00.548 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:00.550 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.551 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:19:00.557 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:19:00.557 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:19:00.558 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:00.560 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:19:00.560 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:19:00.560 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:19:00.560 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:19:00.560 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.560 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.560 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:19:00.561 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:19:00.566 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:19:00.571 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:19:00.572 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:19:00.572 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.573 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:19:00.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:00.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:19:00.578 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:19:00.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:19:00.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:19:00.582 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:19:00.584 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:19:00.586 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:19:00.586 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:19:00.586 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:19:00.586 INFO 2981023 --- [ocp-schedules-29,90b89c162da5a965,8775af349000d7c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:19:01.436 INFO 2981023 --- [ocp-async-5,81798d3c7c02ea85,a326d7a1d2cefbeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,c1fe1ec0e3961b63,60a9ef5c7113a0bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,c1fe1ec0e3961b63,60a9ef5c7113a0bb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:02.699 INFO 2981023 --- [ocp-schedules-1,63aa54e67f7301fe,356ea038b12889a4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:19:03.154 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:03.155 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:03.436 INFO 2981023 --- [ocp-async-4,f0aeda04e2929161,add04483c7108d75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,6bc6c8be6d1ffc4c,b61f0bcb0c916c54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,6bc6c8be6d1ffc4c,b61f0bcb0c916c54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:05.436 INFO 2981023 --- [ocp-async-7,246ab4f132cb71d9,3337ee36a5054102] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:06.155 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:06.156 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:19:06.353144+08:00, lastProcessTime=2024-09-11T18:18:56.337348+08:00 2024-09-11 18:19:06.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:19:06.353144+08:00 2024-09-11 18:19:07.436 INFO 2981023 --- [ocp-async-8,d090b664bf7298f2,b24d397982b87c3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:07.512 INFO 2981023 --- [ocp-schedules-5,60e7c1be55740d9f,a16c1fe850765a95] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:19:07.790 INFO 2981023 --- [ocp-async-6,38d561f49d8abb95,acdd5f846ead5beb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:19:08.046 INFO 2981023 --- [ocp-schedules-24,2d1fd22f5b1f001d,57deed8646baaaa6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:19:08.140 INFO 2981023 --- [ocp-schedules-15,a3fb9215eb87dda1,0adcbc233289248d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:19:08.140+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:19:08.141 INFO 2981023 --- [ocp-schedules-15,a3fb9215eb87dda1,0adcbc233289248d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:19:08.143 INFO 2981023 --- [ocp-schedules-15,a3fb9215eb87dda1,0adcbc233289248d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:19:08.143 INFO 2981023 --- [ocp-schedules-15,a3fb9215eb87dda1,0adcbc233289248d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:19:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,b5e18b7815668f09,4e0489aa28f222e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,b5e18b7815668f09,4e0489aa28f222e4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:09.157 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:09.157 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:09.436 INFO 2981023 --- [ocp-async-2,f451b36cf2b4938d,95a77658967a7c1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:10.003 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:19:10.004 INFO 2981023 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:19:10.227 INFO 2981023 --- [ocp-schedules-11,377f23e40ae7cdf6,db2ce7c73768c465] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:19:10.254 WARN 2981023 --- [scanner-13645239-1,377f23e40ae7cdf6,fb8f2ed84e3107b6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:19:10.287 INFO 2981023 --- [alarm-detect-3,377f23e40ae7cdf6,9d22e9ff3d956896] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645900, data is empty. 2024-09-11 18:19:10.289 INFO 2981023 --- [alarm-detect-3,377f23e40ae7cdf6,9d22e9ff3d956896] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645955, data is empty. 2024-09-11 18:19:10.293 INFO 2981023 --- [alarm-detect-3,377f23e40ae7cdf6,9d22e9ff3d956896] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645955, data is empty. 2024-09-11 18:19:10.318 INFO 2981023 --- [alarm-detect-8,377f23e40ae7cdf6,e9b3ab46302b3721] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645900, data is empty. 2024-09-11 18:19:10.320 INFO 2981023 --- [alarm-detect-14,377f23e40ae7cdf6,481875f9f325d693] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645900, data is empty. 2024-09-11 18:19:10.324 INFO 2981023 --- [alarm-detect-14,377f23e40ae7cdf6,481875f9f325d693] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645955, data is empty. 2024-09-11 18:19:10.324 INFO 2981023 --- [alarm-detect-8,377f23e40ae7cdf6,e9b3ab46302b3721] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645955, data is empty. 2024-09-11 18:19:10.329 INFO 2981023 --- [alarm-detect-14,377f23e40ae7cdf6,481875f9f325d693] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645955, data is empty. 2024-09-11 18:19:10.329 INFO 2981023 --- [alarm-detect-8,377f23e40ae7cdf6,e9b3ab46302b3721] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645955, data is empty. 2024-09-11 18:19:10.373 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,80e95f33bb8f2fee,e8f42e5f5939b7d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=80e95f33bb8f2fee, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:19:10.387 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6e322cb660a0e7b8,15858293a51cd3ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=6e322cb660a0e7b8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:19:10.388 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,2b3c942aa79927e5,091dcabedd0a321a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=2b3c942aa79927e5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:19:10.389 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6e322cb660a0e7b8,15858293a51cd3ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=6e322cb660a0e7b8, duration=21 ms] 2024-09-11 18:19:10.419 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,80e95f33bb8f2fee,e8f42e5f5939b7d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=80e95f33bb8f2fee, duration=54 ms] 2024-09-11 18:19:10.425 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,2b3c942aa79927e5,091dcabedd0a321a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=2b3c942aa79927e5, duration=39 ms] 2024-09-11 18:19:10.437 INFO 2981023 --- [alarm-detect-4,377f23e40ae7cdf6,54ccc64347273560] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645900, data is empty. 2024-09-11 18:19:10.442 INFO 2981023 --- [alarm-detect-4,377f23e40ae7cdf6,54ccc64347273560] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645955, data is empty. 2024-09-11 18:19:10.448 INFO 2981023 --- [alarm-detect-4,377f23e40ae7cdf6,54ccc64347273560] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645955, data is empty. 2024-09-11 18:19:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:19:10.920 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:19:11.013 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:15:10.621, end=2024-09-11T18:17:11.008} 2024-09-11 18:19:11.018 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprd 2024-09-11 18:19:11.018 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:19:11.019 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:15:10.621, end=2024-09-11T18:17:11.008},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:19:11.019 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:19:11.019 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:19:11.020 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:19:11.032 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:15:10.621, end=2024-09-11T18:17:11.008},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:19:11.032 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:19:11.033 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:19:11.084 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551084000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.084Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.091 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551091000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.091Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.092 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551092000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.092Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.096 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551096000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.096Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.097 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551097000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.097Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.099 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551099000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.099Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.099 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551099000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.099Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.099 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551099000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.099Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.106 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.106 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.107 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:19:11.107 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:19:11.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.115 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.115 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.120 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.120 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.121 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.121 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.121 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 68 2024-09-11 18:19:11.121 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.130 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:19:11.130 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.130 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.130 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.130 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.130 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.131 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.131 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.132 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.132 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.135 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.135 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.136 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.136 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.136 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.136 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.137 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.137 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.138 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.138 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.140 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.140 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.141 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.141 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.141 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.141 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.145 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.145 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.145 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.145 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.146 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.146 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.148 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.148 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.150 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.150 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.150 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.150 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.151 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.151 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.151 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.155 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.156 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.156 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.157 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.158 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.158 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.158 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.161 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.161 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.161 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.161 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.097, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.162 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.162 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.165 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.084, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.166 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.166 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.092, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.166 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.091, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.168 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.168 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.168 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.170 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.172 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.099, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.176 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.096, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.176 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.099, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.179 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.099, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.205 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551205000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.205Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.205 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551205000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.205Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.207 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551207000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.207Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.212 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551212000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.212Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.214 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551214000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.214Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.220 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.220 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.221 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551221000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.221Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.222 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.222 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.224 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.224 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.224 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551224000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.224Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.228 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.228 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.228 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551228000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.228Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.236 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.236 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.236 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.236 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.236 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.236 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.239 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.239 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.239 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.239 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.239 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.239 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.242 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.242 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.244 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:19:11.244 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.246 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.246 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.247 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.247 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.251 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.251 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.252 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.252 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.253 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.253 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.254 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.254 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.255 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.255 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.257 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.257 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.258 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.258 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.260 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.260 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.262 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.262 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.263 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.263 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.264 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.264 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.264 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.264 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.267 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.269 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.269 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.270 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.270 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.270 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.212, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.270 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.270 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.272 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.272 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.272 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.205, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=D00A63B6841FCB92DF44E69D8615BA2B, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.273 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.274 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.205, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.274 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.274 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.275 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.275 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.278 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.278 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.207, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.279 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.221, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.280 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.224, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.280 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.280 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.281 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.282 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.214, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.285 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.228, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.285 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:19:11.285 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:15:10.621, end=2024-09-11T18:17:11.008},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:19:11.286 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:19:11.286 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:19:11.287 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:19:11.287 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:15:10.621, end=2024-09-11T18:17:11.008},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:19:11.287 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:19:11.287 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:19:11.288 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:19:11.291 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:15:10.621, end=2024-09-11T18:17:11.008},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:19:11.291 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:19:11.291 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:19:11.301 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:19:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,b199793a7feab0b3,9f2303d814509bd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,b199793a7feab0b3,9f2303d814509bd0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:11.322 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551322000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.322Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.328 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:19:11.328 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.330 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963551330000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=AD63995D4A6E8C3787D81BDA88E06774, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:19:11.330Z, endTime=2024-09-11T10:17:11.008Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:19:11.337 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.337 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.339 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.339 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.345 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.345 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.346 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:19:11.346 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.322, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.352 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 260 2024-09-11 18:19:11.352 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:19:11.358 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.358 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.361 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:19:11.361 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:19:11.366 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:19:11.366 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049831008000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 18:19:11.367 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:19:11.372 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:19:11.33, end=2024-09-11T18:17:11.008}, clusterId=2, tenantId=4, sqlId=AD63995D4A6E8C3787D81BDA88E06774, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:19:11.372 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:19:11.374 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:15:10.621, end=2024-09-11T18:17:11.008},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:19:11.374 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:19:11.374 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:19:11.385 INFO 2981023 --- [plan_change_detector13,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:19:11.385 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:19:11.436 INFO 2981023 --- [ocp-async-6,6d02e5da1f2cd8cd,e36db89b2186e925] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:12.157 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:12.158 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:12.593 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049880, nodeCount=480, affectRows=480 2024-09-11 18:19:12.702 INFO 2981023 --- [ocp-schedules-20,4e2321c67f74d74a,971063a18763ae3e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:19:13.328 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049880, nodeCount=1000, affectRows=1000 2024-09-11 18:19:13.331 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049880, nodeCount=1000, affectRows=1000 2024-09-11 18:19:13.331 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049880, nodeCount=999, validCount=999 2024-09-11 18:19:13.333 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049880, nodeCount=1000, validCount=1000 2024-09-11 18:19:13.339 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049880, nodeCount=48, affectRows=48 2024-09-11 18:19:13.339 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049880, nodeCount=46, validCount=46 2024-09-11 18:19:13.436 INFO 2981023 --- [ocp-async-3,35ac36e163c48d50,d49ef01fa445d783] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,9be6c05baf9eb689,21d883470cd54228] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,13c95af1e5aa66f7,e736ccbb95aa38c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:19:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,9be6c05baf9eb689,21d883470cd54228] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:14.556 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049880, nodeCount=480, validCount=480 2024-09-11 18:19:15.158 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:15.159 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:15.436 INFO 2981023 --- [ocp-async-1,713d42bfb364fb9f,31cf8cf0f550b1aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:15.541 INFO 2981023 --- [ocp-schedules-20,6e1661d7dc28f2b2,4814d5508f30e7d1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:19:15.550 INFO 2981023 --- [ocp-schedules-20,6e1661d7dc28f2b2,4814d5508f30e7d1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:19:15.550 INFO 2981023 --- [ocp-schedules-20,6e1661d7dc28f2b2,4814d5508f30e7d1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:19:16.021 INFO 2981023 --- [ocp-schedules-21,727175ed499c57dc,8c0957fee25fb317] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:19:16.021 INFO 2981023 --- [host-check-clockdiff1,727175ed499c57dc,e3285df63fee06ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:19:16.022 INFO 2981023 --- [host-check-clockdiff2,727175ed499c57dc,d7029060c6127e43] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:19:16.022 INFO 2981023 --- [host-check-clockdiff3,727175ed499c57dc,313ce37f6a08e67c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:19:16.023 INFO 2981023 --- [host-check-clockdiff4,727175ed499c57dc,9fe50897004686ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:19:16.025 INFO 2981023 --- [ocp-schedules-9,fa83310b407277b7,fcf04927464614ad] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:19:16.025 INFO 2981023 --- [ocp-schedules-29,217a347703439cb9,51f235d8bd21f146] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:19:16.025 INFO 2981023 --- [host-check-agent-status1,fa83310b407277b7,433a0d4f921fcd71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:19:16.026 INFO 2981023 --- [host-check-agent-status2,fa83310b407277b7,03e88755576e4d14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:19:16.029 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:19:16.031 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:19:16.031 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:19:16.044 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:19:16.045 INFO 2981023 --- [host-check-agent-status3,fa83310b407277b7,22cf28253341bdc5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:19:16.045 INFO 2981023 --- [host-check-agent-status4,fa83310b407277b7,b24621abf08f4990] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:19:16.058 INFO 2981023 --- [host-check-agent-status2,fa83310b407277b7,03e88755576e4d14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:19:16.063 INFO 2981023 --- [host-check-agent-status2,fa83310b407277b7,03e88755576e4d14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:19:16.064 INFO 2981023 --- [host-check-agent-status1,fa83310b407277b7,433a0d4f921fcd71] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:19:16.066 INFO 2981023 --- [host-check-agent-status3,fa83310b407277b7,22cf28253341bdc5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:19:16.066 INFO 2981023 --- [host-check-agent-status1,fa83310b407277b7,433a0d4f921fcd71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:19:16.067 INFO 2981023 --- [host-check-clockdiff3,727175ed499c57dc,313ce37f6a08e67c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049956 0 0 , error= 2024-09-11 18:19:16.067 INFO 2981023 --- [host-check-agent-status4,fa83310b407277b7,b24621abf08f4990] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:19:16.067 INFO 2981023 --- [host-check-agent-status3,fa83310b407277b7,22cf28253341bdc5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:19:16.069 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:19:16.069 INFO 2981023 --- [host-check-agent-status4,fa83310b407277b7,b24621abf08f4990] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:19:16.069 INFO 2981023 --- [host-check-clockdiff2,727175ed499c57dc,d7029060c6127e43] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049956 5 5 , error= 2024-09-11 18:19:16.071 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:19:16.073 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:19:16.074 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:19:16.075 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:19:16.079 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:19:16.080 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:19:16.088 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:19:16.196 INFO 2981023 --- [ocp-schedules-15,fe13dbfafc1fb2cc,c230b147f8ba6c9b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:19:16.197 INFO 2981023 --- [ocp-schedules-1,6cfe22b24357c05d,e4071d7854d02518] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:19:16.199 INFO 2981023 --- [ocp-schedules-27,0ad335ad7bdd6b91,06df0d61ce64c7e0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:19:16.199 INFO 2981023 --- [ocp-schedules-13,741bed42eec68d8e,d69f712e87bbbe4a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:19:16.207 INFO 2981023 --- [ocp-schedules-15,fe13dbfafc1fb2cc,c230b147f8ba6c9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:19:16.211 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049883058000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:19:16.260 INFO 2981023 --- [ocp-schedules-27,0ad335ad7bdd6b91,06df0d61ce64c7e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:19:16.270 INFO 2981023 --- [ocp-schedules-1,6cfe22b24357c05d,e4071d7854d02518] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:19:16.293 INFO 2981023 --- [host-check-agent-status4,fa83310b407277b7,b24621abf08f4990] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:19:16.293 INFO 2981023 --- [host-check-agent-status1,fa83310b407277b7,433a0d4f921fcd71] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:19:16.295 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:19:16.297 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:19:16.302793036+08:00 2024-09-11 18:19:16.298 INFO 2981023 --- [host-check-agent-status4,fa83310b407277b7,b24621abf08f4990] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:19:16.298 INFO 2981023 --- [host-check-agent-status4,fa83310b407277b7,b24621abf08f4990] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:19:16.298 INFO 2981023 --- [host-check-agent-status4,fa83310b407277b7,b24621abf08f4990] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:19:16.298 INFO 2981023 --- [host-check-agent-status1,fa83310b407277b7,433a0d4f921fcd71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:19:16.298 INFO 2981023 --- [host-check-agent-status1,fa83310b407277b7,433a0d4f921fcd71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:19:16.298 INFO 2981023 --- [host-check-agent-status1,fa83310b407277b7,433a0d4f921fcd71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:19:16.300 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.300 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:19:16.301 INFO 2981023 --- [host-check-agent-status3,fa83310b407277b7,22cf28253341bdc5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:19:16.301 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:19:16.301 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:19:16.297+08:00 2024-09-11 18:19:16.305 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:19:16.297+08:00 and status AVAILABLE 2024-09-11 18:19:16.305 INFO 2981023 --- [ocp-schedules-22,a60a44fe0f66aeb9,246fffd80ea84b8d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:19:16.306 INFO 2981023 --- [host-check-agent-status3,fa83310b407277b7,22cf28253341bdc5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:19:16.306 INFO 2981023 --- [host-check-agent-status3,fa83310b407277b7,22cf28253341bdc5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:19:16.306 INFO 2981023 --- [host-check-agent-status3,fa83310b407277b7,22cf28253341bdc5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:19:16.306 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:19:16.307 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:19:16.306864092+08:00 2024-09-11 18:19:16.307 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.311 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:19:16.311 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:19:16.307+08:00 2024-09-11 18:19:16.311 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:19:16.314 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:19:16.307+08:00 and status AVAILABLE 2024-09-11 18:19:16.318 INFO 2981023 --- [ocp-schedules-15,fe13dbfafc1fb2cc,c230b147f8ba6c9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:19:16.318 INFO 2981023 --- [ocp-schedules-1,6cfe22b24357c05d,e4071d7854d02518] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:19:16.322 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:19:16.322 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:19:16.327965778+08:00 2024-09-11 18:19:16.353 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:19:16.364 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:19:16.323+08:00 2024-09-11 18:19:16.367 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:19:16.368 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:19:16.366872+08:00, lastProcessTime=2024-09-11T18:19:06.353144+08:00 2024-09-11 18:19:16.369 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:19:16.323+08:00 and status AVAILABLE 2024-09-11 18:19:16.385 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:19:16.386 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:19:16.389 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:19:16.390 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:19:16.391 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:19:16.391385668+08:00 2024-09-11 18:19:16.393 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:19:16.391+08:00 2024-09-11 18:19:16.395 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:19:16.391+08:00 and status AVAILABLE 2024-09-11 18:19:16.396 INFO 2981023 --- [host-check-agent-status2,fa83310b407277b7,03e88755576e4d14] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:19:16.398 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:19:16.399 INFO 2981023 --- [host-check-agent-status2,fa83310b407277b7,03e88755576e4d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:19:16.399 INFO 2981023 --- [host-check-agent-status2,fa83310b407277b7,03e88755576e4d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:19:16.399 INFO 2981023 --- [host-check-agent-status2,fa83310b407277b7,03e88755576e4d14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:19:16.400 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:19:16.405 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:19:16.406 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:19:16.408 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:19:16.408 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.409 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.409 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:16.410 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:16.410 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.412 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:16.459 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:19:16.460 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:19:16.463 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:19:16.464 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:19:16.481 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:19:16.488 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:19:16.493 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:16.494 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:19:16.494 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:19:16.496 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.497 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:19:16.506 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:19:16.508 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:19:16.511 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:19:16.518 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:19:16.520 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:19:16.520 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049883058000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:19:16.523 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:19:16.525 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:19:16.535 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:16.536 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.536 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:16.537 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:19:16.537 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.540 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:19:16.541 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:19:16.559 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:19:16.561 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:19:16.562 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:19:16.562 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:19:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:19:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:19:16.582 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:19:16.583 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:19:16.586 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:19:16.588 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:19:16.590 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:19:16.591 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:16.593 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.594 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:19:16.595 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:19:16.595 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:19:16.595 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:19:16.599 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:19:16.600 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:19:16.600 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:19:16.605 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:16.605 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.606 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:19:16.606 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:19:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:19:16.617 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:19:16.622 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:19:16.624 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:19:16.624 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:19:16.625 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:19:16.638 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:19:16.647 INFO 2981023 --- [host-check-host-status4,217a347703439cb9,f9fa96587f6b6a5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:19:16.647 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:19:16.648 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:16.649 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.651 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:19:16.652 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:16.653 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:19:16.653 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.656 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:19:16.656 INFO 2981023 --- [host-check-host-status1,217a347703439cb9,d4c0e3019360ea0e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:19:16.658 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:19:16.661 INFO 2981023 --- [host-check-host-status2,217a347703439cb9,91f549927a866169] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:19:16.668 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:19:16.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:19:16.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:19:16.685 INFO 2981023 --- [host-check-host-status3,217a347703439cb9,912b5c7909ee571d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:19:16.693 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:19:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.704 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:19:16.705 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:19:16.709 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:19:16.714 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:19:16.715 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:19:16.717 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:19:16.366872+08:00 2024-09-11 18:19:16.719 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:19:16.724 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:16.725 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.726 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:19:16.727 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:19:16.731 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:19:16.731 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:19:16.761 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:19:16.767 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:16.767 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:16.768 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:19:16.773 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:19:16.773 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:19:16.773 INFO 2981023 --- [ocp-schedules-27,0ad335ad7bdd6b91,06df0d61ce64c7e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:19:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,ebf43052f32cde74,b7c6dd011b72927e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,ebf43052f32cde74,b7c6dd011b72927e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:17.436 INFO 2981023 --- [ocp-async-9,6aa9438a9cab3fa9,8413d665f09d71cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:17.790 INFO 2981023 --- [ocp-async-5,cd2f359eb6eaa80c,72b923be82f0f1c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:19:18.159 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:18.160 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:19.437 INFO 2981023 --- [ocp-async-10,311ff1f5acd1e1da,ec4e1b1d45f768c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:20.167 INFO 2981023 --- [host-check-clockdiff1,727175ed499c57dc,e3285df63fee06ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049960 5 5 , error= 2024-09-11 18:19:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,587bdd8eca1da0e2,06105bb6029e26ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,587bdd8eca1da0e2,06105bb6029e26ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:20.494 INFO 2981023 --- [ocp-schedules-23,d60c442595b0b526,ca5b9ba9d3735bff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:19:20.535 WARN 2981023 --- [scanner-13647054-1,d60c442595b0b526,d6a656f623044e0d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:19:21.160 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:21.161 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:21.437 INFO 2981023 --- [ocp-async-5,2aaf9bb5cccde7f8,83f7f5ce038db9fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:22.001 INFO 2981023 --- [host-check-clockdiff4,727175ed499c57dc,9fe50897004686ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049961 0 0 , error= 2024-09-11 18:19:22.705 INFO 2981023 --- [ocp-schedules-19,28714de5d3dbf107,9257cb5d6baba841] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:19:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,11de5faf028efbc1,08aa4e03b9533c5e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,11de5faf028efbc1,08aa4e03b9533c5e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:23.437 INFO 2981023 --- [ocp-async-2,db294bf4b3586b9a,68d1196dca85be44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:24.161 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:24.161 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:24.463 INFO 2981023 --- [ocp-schedules-8,0e2ed9e7049a11d4,aa999ec98d01f6a9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:19:25.437 INFO 2981023 --- [ocp-async-6,2a83da1c8a6b8d6d,62d3af129239ccab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,c23966159a6716a3,0bf974542914dbcb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,c23966159a6716a3,0bf974542914dbcb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:19:26.337281+08:00, lastProcessTime=2024-09-11T18:19:16.366872+08:00 2024-09-11 18:19:26.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:19:26.337281+08:00 2024-09-11 18:19:27.082 INFO 2981023 --- [ocp-schedules-11,28a70b02e9b26b54,430d909fc9799541] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:19:27.082 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:19:27.082 INFO 2981023 --- [ocp-schedules-11,28a70b02e9b26b54,430d909fc9799541] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:19:27.082 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:19:27.082 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:19:27.087 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:19:27.087 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:19:27.087 INFO 2981023 --- [ocp-schedules-11,28a70b02e9b26b54,430d909fc9799541] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:19:27.088 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:19:27.088 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:19:27.088 INFO 2981023 --- [ocp-schedules-11,28a70b02e9b26b54,430d909fc9799541] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:19:27.088 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:19:27.089 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:19:27.104 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:19:27.106 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:19:27.107 INFO 2981023 --- [ocp-schedules-11,28a70b02e9b26b54,430d909fc9799541] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:19:27.108 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:19:27.110 INFO 2981023 --- [ocp-schedules-11,28a70b02e9b26b54,430d909fc9799541] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:19:27.162 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:27.162 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:27.437 INFO 2981023 --- [ocp-async-10,bd02f2e88c4ee6b9,20b7397ac4be4b16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:27.790 INFO 2981023 --- [ocp-async-3,e49b77b1e9667807,7ae7492b4f62ef4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:19:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,695eab96cfeaa33f,ea089ab32a3892a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,695eab96cfeaa33f,ea089ab32a3892a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:29.437 INFO 2981023 --- [ocp-async-9,5bf378a4c586220f,2bd664d5a47ff3f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:30.162 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:30.163 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:30.684 INFO 2981023 --- [ocp-schedules-11,248ae14d9b42dd00,999eba01eb885e3f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:19:30.702 WARN 2981023 --- [scanner-13647221-1,248ae14d9b42dd00,f2436366e61bb45d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:19:30.761 INFO 2981023 --- [alarm-detect-14,248ae14d9b42dd00,8e01cdea3c1cf158] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645955, data is empty. 2024-09-11 18:19:30.767 INFO 2981023 --- [alarm-detect-14,248ae14d9b42dd00,8e01cdea3c1cf158] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645955, data is empty. 2024-09-11 18:19:30.771 INFO 2981023 --- [alarm-detect-14,248ae14d9b42dd00,8e01cdea3c1cf158] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645955, data is empty. 2024-09-11 18:19:30.782 INFO 2981023 --- [alarm-detect-13,248ae14d9b42dd00,73207894548ef817] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645955, data is empty. 2024-09-11 18:19:30.789 INFO 2981023 --- [alarm-detect-13,248ae14d9b42dd00,73207894548ef817] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645955, data is empty. 2024-09-11 18:19:30.790 INFO 2981023 --- [alarm-detect-14,248ae14d9b42dd00,65e096089ac48add] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645955, data is empty. 2024-09-11 18:19:30.795 INFO 2981023 --- [alarm-detect-14,248ae14d9b42dd00,65e096089ac48add] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645955, data is empty. 2024-09-11 18:19:30.797 INFO 2981023 --- [alarm-detect-13,248ae14d9b42dd00,73207894548ef817] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645955, data is empty. 2024-09-11 18:19:30.799 INFO 2981023 --- [alarm-detect-14,248ae14d9b42dd00,65e096089ac48add] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645955, data is empty. 2024-09-11 18:19:30.944 INFO 2981023 --- [alarm-detect-17,248ae14d9b42dd00,11be3e9ed4846757] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645955, data is empty. 2024-09-11 18:19:30.950 INFO 2981023 --- [alarm-detect-17,248ae14d9b42dd00,11be3e9ed4846757] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645955, data is empty. 2024-09-11 18:19:30.953 INFO 2981023 --- [alarm-detect-17,248ae14d9b42dd00,11be3e9ed4846757] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645955, data is empty. 2024-09-11 18:19:31.437 INFO 2981023 --- [ocp-async-3,ee7338752be98ed2,e0205421c2cf6a7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,34557366c67ca438,104a5d3885cbc510] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,34557366c67ca438,104a5d3885cbc510] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:32.707 INFO 2981023 --- [ocp-schedules-10,bfb0614deba91aad,4daa86d87c78912f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:19:33.163 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:33.164 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:33.437 INFO 2981023 --- [ocp-async-4,ce022c73d91e8335,1faf86ac0912df99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,3cc891a4cea6c8c7,694af8167bf6467e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,3cc891a4cea6c8c7,694af8167bf6467e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:35.438 INFO 2981023 --- [ocp-async-5,0f2560034ca07bd9,8667cf6010b43305] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:35.551 INFO 2981023 --- [ocp-schedules-3,6fc15b8bceb409df,501a455b180a1d53] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:19:35.556 INFO 2981023 --- [ocp-schedules-3,6fc15b8bceb409df,501a455b180a1d53] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:19:35.556 INFO 2981023 --- [ocp-schedules-3,6fc15b8bceb409df,501a455b180a1d53] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:19:36.164 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:36.164 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:19:36.336557+08:00, lastProcessTime=2024-09-11T18:19:26.337281+08:00 2024-09-11 18:19:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:19:36.336557+08:00 2024-09-11 18:19:37.314 INFO 2981023 --- [prometheus-cache-monitor1,5c5a4c78e8c95b7b,82e8e05faf4f9baa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:19:37.315 INFO 2981023 --- [prometheus-cache-monitor1,5c5a4c78e8c95b7b,82e8e05faf4f9baa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:19:37.438 INFO 2981023 --- [ocp-async-9,a6a479f2bc3ae70e,ab8cc2e631ac5897] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:37.790 INFO 2981023 --- [ocp-async-7,17020d53ba7683d2,07dbfd5eb5af9311] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:19:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,30d7ccc7ed4d231f,7b0c2583bdf529b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,30d7ccc7ed4d231f,7b0c2583bdf529b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:39.164 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:39.165 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:39.438 INFO 2981023 --- [ocp-async-1,db16ddc57ec06f08,425120ada3b3dc7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:40.002 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:19:40.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:19:41.018 INFO 2981023 --- [ocp-schedules-9,2c30cfaf253c91c6,a1b706cfe3b5ba73] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:19:41.067 WARN 2981023 --- [scanner-13647611-1,2c30cfaf253c91c6,23913858be0e8880] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:19:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,1965b4e7acd0a775,16ed6580b722b672] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,1965b4e7acd0a775,16ed6580b722b672] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:41.438 INFO 2981023 --- [ocp-async-7,a1080e32b6779e64,eb75405c44c628da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:42.165 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:42.166 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:42.709 INFO 2981023 --- [ocp-schedules-9,8686c5cb10437d9e,680452d2eec43a7c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:19:43.438 INFO 2981023 --- [ocp-async-8,71d06e4b98e4de7f,c4a293032bcb4cdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,0d1b99fe7e0fd8cc,cee9686216e50d9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,0d1b99fe7e0fd8cc,cee9686216e50d9d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:45.101 INFO 2981023 --- [prometheus-second-schedule-1,c5215f2341cc2a8b,a89ea6e9e2cca1a2] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:19:45.166 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:45.167 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:45.438 INFO 2981023 --- [ocp-async-6,ca626e7f7a761abf,357b8d458d6a3097] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:45.997 INFO 2981023 --- [ocp-schedules-3,b312f33aebfc811f,3b4012d5036b5c4a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:19:45.998 INFO 2981023 --- [host-check-clockdiff1,b312f33aebfc811f,4f8f5bb90cdfa845] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:19:45.998 INFO 2981023 --- [host-check-clockdiff2,b312f33aebfc811f,6a88e986b6fe161e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:19:45.998 INFO 2981023 --- [host-check-clockdiff4,b312f33aebfc811f,05a16ede7d7f4c9c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:19:45.998 INFO 2981023 --- [host-check-clockdiff3,b312f33aebfc811f,5b94acb5badb7536] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:19:46.015 INFO 2981023 --- [ocp-schedules-21,7e418ecdbbaad0c9,fb97a4bf72c04716] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:19:46.015 INFO 2981023 --- [ocp-schedules-20,e6d3677ce6d14135,9e73ee581db89518] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:19:46.017 INFO 2981023 --- [host-check-agent-status1,7e418ecdbbaad0c9,39ca3c79426f17df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:19:46.017 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:19:46.018 INFO 2981023 --- [host-check-agent-status2,7e418ecdbbaad0c9,9fa7aeb6215d7edc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:19:46.018 INFO 2981023 --- [host-check-agent-status3,7e418ecdbbaad0c9,c05e630e2064d796] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:19:46.021 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:19:46.022 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:19:46.023 INFO 2981023 --- [host-check-agent-status1,7e418ecdbbaad0c9,39ca3c79426f17df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:19:46.024 INFO 2981023 --- [host-check-agent-status2,7e418ecdbbaad0c9,9fa7aeb6215d7edc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:19:46.024 INFO 2981023 --- [host-check-agent-status1,7e418ecdbbaad0c9,39ca3c79426f17df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:19:46.025 INFO 2981023 --- [host-check-agent-status3,7e418ecdbbaad0c9,c05e630e2064d796] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:19:46.025 INFO 2981023 --- [host-check-agent-status2,7e418ecdbbaad0c9,9fa7aeb6215d7edc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:19:46.027 INFO 2981023 --- [host-check-agent-status4,7e418ecdbbaad0c9,cdb1e09f25968ecb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:19:46.027 INFO 2981023 --- [host-check-agent-status3,7e418ecdbbaad0c9,c05e630e2064d796] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:19:46.028 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:19:46.029 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:19:46.029 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:19:46.038 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:19:46.040 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:19:46.042 INFO 2981023 --- [host-check-agent-status4,7e418ecdbbaad0c9,cdb1e09f25968ecb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:19:46.043 INFO 2981023 --- [host-check-agent-status4,7e418ecdbbaad0c9,cdb1e09f25968ecb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:19:46.045 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:19:46.045 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:19:46.046 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:19:46.047 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:19:46.050 INFO 2981023 --- [host-check-agent-status2,7e418ecdbbaad0c9,9fa7aeb6215d7edc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:19:46.050 INFO 2981023 --- [host-check-agent-status1,7e418ecdbbaad0c9,39ca3c79426f17df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:19:46.051 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:19:46.052 INFO 2981023 --- [host-check-agent-status3,7e418ecdbbaad0c9,c05e630e2064d796] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:19:46.053 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:19:46.058013762+08:00 2024-09-11 18:19:46.054 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:19:46.053+08:00 2024-09-11 18:19:46.054 INFO 2981023 --- [host-check-agent-status1,7e418ecdbbaad0c9,39ca3c79426f17df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:19:46.054 INFO 2981023 --- [host-check-agent-status1,7e418ecdbbaad0c9,39ca3c79426f17df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:19:46.054 INFO 2981023 --- [host-check-agent-status1,7e418ecdbbaad0c9,39ca3c79426f17df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:19:46.056 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:19:46.053+08:00 and status AVAILABLE 2024-09-11 18:19:46.058 INFO 2981023 --- [host-check-agent-status2,7e418ecdbbaad0c9,9fa7aeb6215d7edc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:19:46.058 INFO 2981023 --- [host-check-agent-status2,7e418ecdbbaad0c9,9fa7aeb6215d7edc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:19:46.058 INFO 2981023 --- [host-check-agent-status2,7e418ecdbbaad0c9,9fa7aeb6215d7edc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:19:46.059 INFO 2981023 --- [host-check-agent-status3,7e418ecdbbaad0c9,c05e630e2064d796] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:19:46.059 INFO 2981023 --- [host-check-agent-status3,7e418ecdbbaad0c9,c05e630e2064d796] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:19:46.059 INFO 2981023 --- [host-check-agent-status3,7e418ecdbbaad0c9,c05e630e2064d796] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:19:46.062 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:19:46.063 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:19:46.064 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:19:46.069622140+08:00 2024-09-11 18:19:46.064 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:19:46.064+08:00 2024-09-11 18:19:46.066 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:19:46.064+08:00 and status AVAILABLE 2024-09-11 18:19:46.068 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:19:46.068 INFO 2981023 --- [host-check-agent-status4,7e418ecdbbaad0c9,cdb1e09f25968ecb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:19:46.069 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:19:46.070 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:19:46.070 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:19:46.070 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:19:46.070506807+08:00 2024-09-11 18:19:46.071 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:19:46.070+08:00 2024-09-11 18:19:46.072 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:19:46.070+08:00 and status AVAILABLE 2024-09-11 18:19:46.073 INFO 2981023 --- [host-check-agent-status4,7e418ecdbbaad0c9,cdb1e09f25968ecb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:19:46.073 INFO 2981023 --- [host-check-agent-status4,7e418ecdbbaad0c9,cdb1e09f25968ecb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:19:46.073 INFO 2981023 --- [host-check-agent-status4,7e418ecdbbaad0c9,cdb1e09f25968ecb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:19:46.077 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:19:46.079 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:19:46.080 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:19:46.079958572+08:00 2024-09-11 18:19:46.081 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:19:46.080+08:00 2024-09-11 18:19:46.084 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:19:46.084 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:19:46.085 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:19:46.088 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:19:46.080+08:00 and status AVAILABLE 2024-09-11 18:19:46.093 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:19:46.096 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:19:46.097 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:19:46.101 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:19:46.102 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:19:46.109 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:19:46.115 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:19:46.118 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:19:46.120 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:19:46.126 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:19:46.139 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:19:46.144 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:19:46.158 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:19:46.159 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:19:46.160 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:19:46.160 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:19:46.195 INFO 2981023 --- [ocp-schedules-24,07c51a2a616f8dee,5a57c713d1390593] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:19:46.195 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:19:46.195 INFO 2981023 --- [ocp-schedules-11,7617c0f54e797818,828d8b5866643bec] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:19:46.196 INFO 2981023 --- [ocp-schedules-15,a803b7df4bfb59ba,e451ab0fd1952f1a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:19:46.196 INFO 2981023 --- [ocp-schedules-2,242ba8c26ec3470d,a5feb4b22af4d0c1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:19:46.198 INFO 2981023 --- [ocp-schedules-11,7617c0f54e797818,828d8b5866643bec] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:19:46.198 INFO 2981023 --- [ocp-schedules-13,6dfffae4f6d71030,73a16273812253b7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:19:46.199 INFO 2981023 --- [ocp-schedules-15,a803b7df4bfb59ba,e451ab0fd1952f1a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:19:46.202 INFO 2981023 --- [ocp-schedules-11,7617c0f54e797818,828d8b5866643bec] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:19:46.206 INFO 2981023 --- [ocp-schedules-24,07c51a2a616f8dee,5a57c713d1390593] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:19:46.210 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:19:46.215 INFO 2981023 --- [ocp-schedules-13,6dfffae4f6d71030,73a16273812253b7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:19:46.224 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:19:46.227 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049883058000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:19:46.227 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:19:46.235 INFO 2981023 --- [ocp-schedules-29,b8523348ace00b0a,1e5db4858a27f32b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:19:46.244 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:19:46.261 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:19:46.265 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:19:46.275 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:19:46.278 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.278 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:19:46.279 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.279 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:19:46.279 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:46.281 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.281 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:46.282 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:19:46.294 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:19:46.302 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:19:46.309 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:19:46.312 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.316 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.316 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.317 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:19:46.318 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:19:46.319 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:19:46.319 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:19:46.321 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:19:46.321 INFO 2981023 --- [ocp-schedules-24,07c51a2a616f8dee,5a57c713d1390593] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:19:46.328 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:19:46.338 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:19:46.340 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:19:46.340045+08:00, lastProcessTime=2024-09-11T18:19:36.336557+08:00 2024-09-11 18:19:46.347 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.348 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:46.352 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:19:46.356 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:19:46.357 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.357 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.358 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:19:46.359 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:19:46.359 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:46.359 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:19:46.360 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.363 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:19:46.364 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:19:46.373 INFO 2981023 --- [host-check-host-status1,e6d3677ce6d14135,40383ec937838365] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:19:46.376 INFO 2981023 --- [host-check-host-status4,e6d3677ce6d14135,f297ecd844700b6f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:19:46.385 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:19:46.392 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:19:46.392 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:19:46.392 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.393 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.393 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.393 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:19:46.393 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:19:46.399 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:19:46.399 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049883058000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:19:46.402 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:19:46.403 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.409 INFO 2981023 --- [host-check-host-status2,e6d3677ce6d14135,5dab643c6d516736] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:19:46.414 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:19:46.418 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:19:46.422 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:19:46.423 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.424 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:19:46.425 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:19:46.425 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.426 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:19:46.426 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:19:46.427 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.429 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:19:46.430 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:19:46.431 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:19:46.431 INFO 2981023 --- [host-check-host-status3,e6d3677ce6d14135,9087d9762c052b7f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:19:46.434 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:19:46.435 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:19:46.441 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:19:46.441 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:19:46.455 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:19:46.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:19:46.465 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.465 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.466 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:19:46.467 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:19:46.470 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:19:46.473 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.474 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:19:46.475 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:19:46.475 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.475 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:19:46.476 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:19:46.476 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.488 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:19:46.500 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:19:46.516 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:19:46.524 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:19:46.533 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.534 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.534 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:19:46.539 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.540 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:19:46.543 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:19:46.543 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049412655000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:19:46.547 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.547 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.548 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.548 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.548 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:19:46.548 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:19:46.549 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:19:46.550 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.550 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:19:46.551 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:19:46.551 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.554 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:19:46.554 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049389296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:19:46.558 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:19:46.558 INFO 2981023 --- [ocp-schedules-15,a803b7df4bfb59ba,e451ab0fd1952f1a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:19:46.576 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:19:46.586 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:19:46.340045+08:00 2024-09-11 18:19:46.588 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.589 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.589 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:19:46.590 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:19:46.590 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.601 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:19:46.613 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:19:46.624 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:19:46.625 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.625 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:19:46.625 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:19:46.626 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:46.666 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:19:46.682 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:19:46.688 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:19:46.691 INFO 2981023 --- [ocp-schedules-26,1d7f743571e6c542,66a9ea48dc85af5b] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:19:46.846 INFO 2981023 --- [ocp-schedules-14,1381b50091b397da,c4abed96afaef0c5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:19:46.846 INFO 2981023 --- [ocp-schedules-14,1381b50091b397da,c4abed96afaef0c5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:19:46.913 INFO 2981023 --- [ocp-schedules-17,edf7552f639d6d53,4c5fa0e8201f2f48] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:19:46.930 WARN 2981023 --- [ocp-schedules-17,edf7552f639d6d53,4c5fa0e8201f2f48] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:19:46.945 INFO 2981023 --- [host-check-clockdiff3,b312f33aebfc811f,5b94acb5badb7536] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049986 0 0 , error= 2024-09-11 18:19:46.947 INFO 2981023 --- [host-check-clockdiff1,b312f33aebfc811f,4f8f5bb90cdfa845] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049986 5 5 , error= 2024-09-11 18:19:46.948 INFO 2981023 --- [host-check-clockdiff2,b312f33aebfc811f,6a88e986b6fe161e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049986 6 6 , error= 2024-09-11 18:19:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,e6be701843670aed,f2d9ca1c0ba93e11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,e6be701843670aed,f2d9ca1c0ba93e11] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:47.438 INFO 2981023 --- [ocp-async-7,72bd0ebe01c06e43,5b99d876eb9d5172] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:47.790 INFO 2981023 --- [ocp-async-6,f7fa1b25238cb0f7,8303799250204193] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:19:48.167 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:48.167 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:49.295 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c42ca34fa3c12318,9c71026486f8e820] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&version=2&ObClusterId=1719994437, client=10.31.35.10, traceId=c42ca34fa3c12318, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkprdocp,1719994437,2,{"ObRegion":"linkprdocp","ObCluster":"linkprdocp","ObRegionId":1719994437,"ObClusterId":1719994437,"Type":"PRIMARY","timestamp":1726049989294382,"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:19:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c42ca34fa3c12318,9c71026486f8e820] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkprdocp, obClusterId=1719994437, rootServerJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, groupName=linkprdocp, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:19:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c42ca34fa3c12318,9c71026486f8e820] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:19:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c42ca34fa3c12318,9c71026486f8e820] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkprdocp, clusterId=1719994437, affectRows=1 2024-09-11 18:19:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c42ca34fa3c12318,9c71026486f8e820] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.10, traceId=c42ca34fa3c12318, duration=9 ms] 2024-09-11 18:19:49.439 INFO 2981023 --- [ocp-async-3,6810f52d8fdd3bb3,52d93b865f5e037f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:19:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:19:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,3380929d10506ce0,09ae86ee3b46391d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,3380929d10506ce0,09ae86ee3b46391d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:50.353 INFO 2981023 --- [ocp-schedules-19,e0924fadff484956,17e282c5ee52bd76] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:19:51.167 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:51.168 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:51.243 INFO 2981023 --- [ocp-schedules-25,e7d250ba6af6a7b8,440c98fe67c4ea00] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:19:51.283 WARN 2981023 --- [scanner-13647801-1,e7d250ba6af6a7b8,e13766f2b51428f7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:19:51.319 INFO 2981023 --- [alarm-detect-2,e7d250ba6af6a7b8,756ac0dc0e4acc85] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645955, data is empty. 2024-09-11 18:19:51.326 INFO 2981023 --- [alarm-detect-2,e7d250ba6af6a7b8,756ac0dc0e4acc85] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645955, data is empty. 2024-09-11 18:19:51.330 INFO 2981023 --- [alarm-detect-2,e7d250ba6af6a7b8,756ac0dc0e4acc85] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645955, data is empty. 2024-09-11 18:19:51.357 INFO 2981023 --- [alarm-detect-2,e7d250ba6af6a7b8,82ec8bb523fc9d35] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645955, data is empty. 2024-09-11 18:19:51.358 INFO 2981023 --- [alarm-detect-14,e7d250ba6af6a7b8,a5b927d4adc72329] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645955, data is empty. 2024-09-11 18:19:51.365 INFO 2981023 --- [alarm-detect-2,e7d250ba6af6a7b8,82ec8bb523fc9d35] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645955, data is empty. 2024-09-11 18:19:51.367 INFO 2981023 --- [alarm-detect-14,e7d250ba6af6a7b8,a5b927d4adc72329] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645955, data is empty. 2024-09-11 18:19:51.371 INFO 2981023 --- [alarm-detect-2,e7d250ba6af6a7b8,82ec8bb523fc9d35] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645955, data is empty. 2024-09-11 18:19:51.373 INFO 2981023 --- [alarm-detect-14,e7d250ba6af6a7b8,a5b927d4adc72329] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645955, data is empty. 2024-09-11 18:19:51.439 INFO 2981023 --- [ocp-async-6,2e5aaee0b40c16bf,408c27fe188fcb6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:51.465 INFO 2981023 --- [alarm-detect-11,e7d250ba6af6a7b8,c0271ea4c520169f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645955, data is empty. 2024-09-11 18:19:51.470 INFO 2981023 --- [alarm-detect-11,e7d250ba6af6a7b8,c0271ea4c520169f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645955, data is empty. 2024-09-11 18:19:51.475 INFO 2981023 --- [alarm-detect-11,e7d250ba6af6a7b8,c0271ea4c520169f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645955, data is empty. 2024-09-11 18:19:51.626 INFO 2981023 --- [ocp-schedules-13,8cdbdda2daadf9ff,c18087afa0a2e5c4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:19:51.627 INFO 2981023 --- [ocp-schedules-13,8cdbdda2daadf9ff,c18087afa0a2e5c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:19:51.627 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:19:51.627 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:19:51.627 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:19:51.632 INFO 2981023 --- [ocp-schedules-13,8cdbdda2daadf9ff,c18087afa0a2e5c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:19:52.007 INFO 2981023 --- [host-check-clockdiff4,b312f33aebfc811f,05a16ede7d7f4c9c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726049992 0 0 , error= 2024-09-11 18:19:52.712 INFO 2981023 --- [ocp-schedules-16,229f7257feb69945,815f7f438c0d316f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:19:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,f8c55c4692643ece,b9231238f0e3202a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,f8c55c4692643ece,b9231238f0e3202a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:53.347 INFO 2981023 --- [ocp-schedules-28,f926828fa1510f52,0aa281f33d9a5d59] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:19:53.347 INFO 2981023 --- [ocp-schedules-28,f926828fa1510f52,0aa281f33d9a5d59] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:19:53.439 INFO 2981023 --- [ocp-async-2,25db30f03c6397a8,c9d38ca03d8938ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:54.168 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:54.169 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:54.465 INFO 2981023 --- [ocp-schedules-27,0a0c1d0227eeb398,0981559d2a1b3c2e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:19:55.039 INFO 2981023 --- [ocp-schedules-22,da22a0215db661a1,d9df644208d71c63] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:19:55.057 WARN 2981023 --- [ocp-schedules-22,da22a0215db661a1,d9df644208d71c63] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:19:55.439 INFO 2981023 --- [ocp-async-8,7c936bcb645bf663,f877c220a28f0fef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:55.559 INFO 2981023 --- [ocp-schedules-7,9a75020bf457aea5,fcaf0ebb59ec9a23] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:19:55.568 INFO 2981023 --- [ocp-schedules-7,9a75020bf457aea5,fcaf0ebb59ec9a23] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:19:55.568 INFO 2981023 --- [ocp-schedules-7,9a75020bf457aea5,fcaf0ebb59ec9a23] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:19:55.890 INFO 2981023 --- [ocp-schedules-25,7a7cb94f9aab5307,b8e7ff9de7202e89] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:19:55.890 INFO 2981023 --- [ocp-schedules-25,7a7cb94f9aab5307,b8e7ff9de7202e89] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:19:55.891 INFO 2981023 --- [ocp-schedules-25,7a7cb94f9aab5307,b8e7ff9de7202e89] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:19:55.891 INFO 2981023 --- [ocp-schedules-25,7a7cb94f9aab5307,b8e7ff9de7202e89] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:19:55.893 INFO 2981023 --- [ocp-schedules-25,7a7cb94f9aab5307,b8e7ff9de7202e89] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:19:55.893 INFO 2981023 --- [ocp-schedules-25,7a7cb94f9aab5307,b8e7ff9de7202e89] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:19:56.096 INFO 2981023 --- [ocp-schedules-27,27159c92d5bb2b10,d99690daa2f63235] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:19:56.102 INFO 2981023 --- [ocp-schedules-27,27159c92d5bb2b10,d99690daa2f63235] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:19:56.102 INFO 2981023 --- [ocp-schedules-27,27159c92d5bb2b10,d99690daa2f63235] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:19:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,2e34f8c7ebb1057e,e520e3976b73a317] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,2e34f8c7ebb1057e,e520e3976b73a317] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:19:56.351125+08:00, lastProcessTime=2024-09-11T18:19:46.340045+08:00 2024-09-11 18:19:56.480 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:19:56.351125+08:00 2024-09-11 18:19:57.169 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:19:57.169 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:19:57.439 INFO 2981023 --- [ocp-async-4,9b363bc19cf375b2,7e9ff3f8f562395c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:57.790 INFO 2981023 --- [ocp-async-1,7e82e80ec1d4f1d4,708f51f3272f6764] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:19:58.300 INFO 2981023 --- [ocp-schedules-22,ff4dc6aa060a2d63,d05d856e83827e6e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:19:58.300 INFO 2981023 --- [ocp-schedules-22,ff4dc6aa060a2d63,d05d856e83827e6e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:19:58.315 INFO 2981023 --- [Collect all tenants major compactions1,ff4dc6aa060a2d63,234b0693ce6fcd71] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:19:58.316 INFO 2981023 --- [Collect all tenants major compactions3,ff4dc6aa060a2d63,18a0c66c6ce4cff4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:19:58.316 INFO 2981023 --- [Collect all tenants major compactions2,ff4dc6aa060a2d63,0bb47dd13347b2bc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:19:58.317 INFO 2981023 --- [Collect all tenants major compactions4,ff4dc6aa060a2d63,4f67af17125ca6d6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:19:58.318 INFO 2981023 --- [Collect all tenants major compactions5,ff4dc6aa060a2d63,e0cbfcead6d0c5e7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:19:58.318 INFO 2981023 --- [Collect all tenants major compactions6,ff4dc6aa060a2d63,2af835a762c4a021] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:19:58.330 INFO 2981023 --- [Collect all tenants major compactions2,ff4dc6aa060a2d63,0bb47dd13347b2bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:58.330 INFO 2981023 --- [Collect all tenants major compactions1,ff4dc6aa060a2d63,234b0693ce6fcd71] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:58.330 INFO 2981023 --- [Collect all tenants major compactions3,ff4dc6aa060a2d63,18a0c66c6ce4cff4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:58.331 INFO 2981023 --- [Collect all tenants major compactions2,ff4dc6aa060a2d63,0bb47dd13347b2bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:19:58.332 INFO 2981023 --- [Collect all tenants major compactions1,ff4dc6aa060a2d63,234b0693ce6fcd71] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:19:58.332 INFO 2981023 --- [Collect all tenants major compactions3,ff4dc6aa060a2d63,18a0c66c6ce4cff4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:19:58.332 INFO 2981023 --- [Collect all tenants major compactions5,ff4dc6aa060a2d63,e0cbfcead6d0c5e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:58.333 INFO 2981023 --- [Collect all tenants major compactions4,ff4dc6aa060a2d63,4f67af17125ca6d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:58.334 INFO 2981023 --- [Collect all tenants major compactions5,ff4dc6aa060a2d63,e0cbfcead6d0c5e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:19:58.334 INFO 2981023 --- [Collect all tenants major compactions4,ff4dc6aa060a2d63,4f67af17125ca6d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:19:58.335 INFO 2981023 --- [Collect all tenants major compactions3,ff4dc6aa060a2d63,18a0c66c6ce4cff4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:19:58.335 INFO 2981023 --- [Collect all tenants major compactions1,ff4dc6aa060a2d63,234b0693ce6fcd71] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:19:58.335 INFO 2981023 --- [Collect all tenants major compactions3,ff4dc6aa060a2d63,18a0c66c6ce4cff4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:19:58.335 INFO 2981023 --- [Collect all tenants major compactions1,ff4dc6aa060a2d63,234b0693ce6fcd71] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:19:58.336 INFO 2981023 --- [Collect all tenants major compactions2,ff4dc6aa060a2d63,0bb47dd13347b2bc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:19:58.336 INFO 2981023 --- [Collect all tenants major compactions2,ff4dc6aa060a2d63,0bb47dd13347b2bc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:19:58.336 INFO 2981023 --- [Collect all tenants major compactions6,ff4dc6aa060a2d63,2af835a762c4a021] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:19:58.336 INFO 2981023 --- [Collect all tenants major compactions5,ff4dc6aa060a2d63,e0cbfcead6d0c5e7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:19:58.336 INFO 2981023 --- [Collect all tenants major compactions5,ff4dc6aa060a2d63,e0cbfcead6d0c5e7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:19:58.337 INFO 2981023 --- [Collect all tenants major compactions4,ff4dc6aa060a2d63,4f67af17125ca6d6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:19:58.337 INFO 2981023 --- [Collect all tenants major compactions4,ff4dc6aa060a2d63,4f67af17125ca6d6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:19:58.337 INFO 2981023 --- [Collect all tenants major compactions6,ff4dc6aa060a2d63,2af835a762c4a021] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:19:58.339 INFO 2981023 --- [Collect all tenants major compactions6,ff4dc6aa060a2d63,2af835a762c4a021] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:19:58.339 INFO 2981023 --- [Collect all tenants major compactions6,ff4dc6aa060a2d63,2af835a762c4a021] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:19:58.340 INFO 2981023 --- [ocp-schedules-22,ff4dc6aa060a2d63,d05d856e83827e6e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:19:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,3d12790db0d8c8e8,b3872dfc2f5ae96c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050000, interval=60 2024-09-11 18:19:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,3d12790db0d8c8e8,b3872dfc2f5ae96c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:19:59.439 INFO 2981023 --- [ocp-async-2,5b836eea96789f8d,217ba7c30808f0fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:19:59.541 INFO 2981023 --- [ocp-schedules-18,847b009754619fb2,0115be9b9329480c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:19:59.541+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:19:59.546 INFO 2981023 --- [ocp-schedules-18,847b009754619fb2,0115be9b9329480c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:19:59.704 INFO 2981023 --- [ocp-schedules-18,847b009754619fb2,0115be9b9329480c] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 43 to SqlAggregateData 2024-09-11 18:19:59.705 INFO 2981023 --- [ocp-schedules-18,847b009754619fb2,0115be9b9329480c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:20:00.169 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:00.170 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:00.586 INFO 2981023 --- [ocp-schedules-12,0687273a38644233,4649c10e82e02a97] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:20:00.587 INFO 2981023 --- [ocp-schedules-12,0687273a38644233,4649c10e82e02a97] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:20:00.615 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:20:00.615 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:20:00.615 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.615 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.618 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:20:00.618 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:20:00.619 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:20:00.619 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.621 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:20:00.622 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.622 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.624 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.624 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.625 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.626 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.627 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.627 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.634 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.634 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:20:00.635 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.636 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.638 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.639 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.641 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:20:00.645 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.645 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:20:00.650 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.651 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.652 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.653 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:20:00.657 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.658 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.658 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:20:00.659 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5891D6F1-0-0] 2024-09-11 18:20:00.659 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:20:00.659 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.660 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:20:00.662 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.663 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:20:00.663 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.664 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:20:00.664 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.665 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:20:00.666 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:20:00.666 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.667 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:20:00.667 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.668 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:20:00.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.669 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:20:00.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:20:00.670 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.671 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:20:00.671 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.672 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:20:00.672 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.673 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:20:00.674 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.674 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:20:00.675 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.676 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:20:00.676 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.677 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:20:00.677 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:20:00.677 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.678 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:20:00.679 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:20:00.687 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:00.688 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.689 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.691 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:00.691 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.692 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:20:00.693 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:20:00.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5891D6FB-0-0] 2024-09-11 18:20:00.693 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:20:00.693 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:20:00.700 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:20:00.701 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.702 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:20:00.702 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.703 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:20:00.703 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:20:00.704 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:20:00.704 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.704 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:20:00.705 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:20:00.705 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:20:00.705 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:20:00.706 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.707 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:20:00.708 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.709 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:20:00.710 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.710 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:15:01.0, nextRun=2024-09-11 18:20:00.0, currentDate=Wed Sep 11 18:20:00 HKT 2024 2024-09-11 18:20:00.710 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:15:01.0, nextRunTime=2024-09-11 18:20:00.0} 2024-09-11 18:20:00.710 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:20:00.711 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:20:00.711 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.712 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:20:00.713 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:20:00.713 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.713 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.715 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 18:20:00.715 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:20:00.715 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:20:00.715+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:20:00.719 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:20:00.720 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:20:00.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:20:00.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:20:00.721 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:20:00.722 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:20:00.731 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:15:01.0, nextRun=2024-09-11 18:20:00.0, currentDate=Wed Sep 11 18:20:00 HKT 2024 2024-09-11 18:20:00.731 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:15:01.0, nextRunTime=2024-09-11 18:20:00.0} 2024-09-11 18:20:00.735 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:00.751 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.752 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:20:00.758 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 18:20:00.765 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:20:00.767 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:20:00.767 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:00.768 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:20:00.772 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 18:20:00.775 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 18:20:00.775 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:20:00.775 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d417e706a66b2b2c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T18:20:00.775+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:20:00.781 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:20:00.782 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:20:00.782 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:20:00.782 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:20:00.783 INFO 2981023 --- [ocp-schedules-12,0687273a38644233,4649c10e82e02a97] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:20:01.439 INFO 2981023 --- [ocp-async-8,8734ab6239a16d4a,4a882e49f4014509] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:01.516 INFO 2981023 --- [ocp-schedules-3,804bc52528419d5b,c67960500bad5f62] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:20:01.519 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,755b33e18ef5254e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048247, operation:EXECUTE, state: RUNNING 2024-09-11 18:20:01.545 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,755b33e18ef5254e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048248, operation:EXECUTE, state: RUNNING 2024-09-11 18:20:01.575 WARN 2981023 --- [scanner-13648125-1,804bc52528419d5b,e4765e3a43795f4d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:20:01.577 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048247, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048217, task_operation=execute, latest_execution_start_time=2024-09-11T18:20:01.512+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048247}, listMap={}}, executor=10.31.35.10 2024-09-11 18:20:01.618 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:20:01.594 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048248, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048218, task_operation=execute, latest_execution_start_time=2024-09-11T18:20:01.542+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048248}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 18:20:01.665 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 18:20:01.703 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 18:20:01.712 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 18:20:01.747 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 18:20:01.750 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 18:20:01.750 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 18:20:01.781 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 18:20:01.786 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 18:20:01.791 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:20:01.794 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,60815c97a06bd56e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048248, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:20:01.862 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:20:01.890 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:20:01.909 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:20:01.919 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 18:20:01.920 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 18:20:01.925 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:20:02.025 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:20:02.073 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:20:02.094 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:20:02.110 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:20:02.142 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:20:02.156 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:20:02.177 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:20:02.184 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 18:20:02.184 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3395.459579986], b:[-5.807309276841022E12] 2024-09-11 18:20:02.189 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:20:02.247 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:20:02.286 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:20:02.298 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:20:02.304 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:20:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,f2649c0bde56a21b,588081c6e01ca08f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:02.315 INFO 2981023 --- [prometheus-minute-schedule-1,f2649c0bde56a21b,588081c6e01ca08f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:02.330 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:20:02.336 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:20:02.342 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:20:02.344 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 18:20:02.344 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:20:02.358 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:20:02.367 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:20:02.379 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,862788e01583db62] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048247, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:20:02.681 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048217, status:SUCCESS 2024-09-11 18:20:02.683 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048217, status:SUCCESS 2024-09-11 18:20:02.689 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048218, status:SUCCESS 2024-09-11 18:20:02.689 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048218, status:SUCCESS 2024-09-11 18:20:02.715 INFO 2981023 --- [ocp-schedules-7,fe3e34f7cf0efb20,3a68b5d0a7b104c0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:20:03.069 INFO 2981023 --- [ocp-schedules-9,82f212cfb69ca5ab,3ba29b16e11db77d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:20:03.070 INFO 2981023 --- [ocp-schedules-9,82f212cfb69ca5ab,3ba29b16e11db77d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:20:03.091 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:03.091 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:20:03.096 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:03.098 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:20:03.098 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050003098000} 2024-09-11 18:20:03.114 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:20:03.118 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000234, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:20:03.118 INFO 2981023 --- [ocp-schedules-9,82f212cfb69ca5ab,3ba29b16e11db77d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:20:03.170 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:03.170 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:03.439 INFO 2981023 --- [ocp-async-6,ec306624c1cc249a,31bf00f17c87c6fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:03.746 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:20:03.746 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:20:03.746 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:20:03.751 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:20:03.752 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000234, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:20:04.269 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000234 2024-09-11 18:20:04.877 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2e7aef5544503c1] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:10:05.0, nextRun=2024-09-11 18:20:04.0, currentDate=Wed Sep 11 18:20:04 HKT 2024 2024-09-11 18:20:04.877 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2e7aef5544503c1] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-09-11 18:10:05.0, nextRunTime=2024-09-11 18:20:04.0} 2024-09-11 18:20:04.881 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2e7aef5544503c1] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-09-11 18:20:04.881 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2e7aef5544503c1] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:20:04.881 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2e7aef5544503c1] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:20:04.881+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:20:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,619627297c97eee2,b157fd7062ecc729] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,619627297c97eee2,b157fd7062ecc729] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:05.439 INFO 2981023 --- [ocp-async-7,6f63b0338c636f6b,56345546a0ca3c09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:05.575 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,20749f0e6fb5342e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048249, operation:EXECUTE, state: RUNNING 2024-09-11 18:20:05.592 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048249, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048219, task_operation=execute, latest_execution_start_time=2024-09-11T18:20:05.574+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=4048249}, listMap={}}, executor=10.31.35.10 2024-09-11 18:20:05.601 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 18:20:05.601 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 18:20:05.602 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.602 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:20:05.604 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.605 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.605 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.606 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 18:20:05.617 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.617 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.618 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.618 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 18:20:05.626 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.626 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.627 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.627 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 18:20:05.635 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.636 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.636 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.636 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 18:20:05.646 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.646 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.646 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.647 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 18:20:05.650 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.651 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.651 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.651 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 18:20:05.655 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.655 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.656 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.656 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 18:20:05.659 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.659 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.660 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.660 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 18:20:05.664 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.665 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.665 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.666 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 18:20:05.669 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.670 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.670 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.671 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 18:20:05.674 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.674 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.675 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.675 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 18:20:05.681 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.681 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.681 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.682 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 18:20:05.691 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.691 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.692 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.692 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 18:20:05.695 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.696 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.696 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.696 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 18:20:05.700 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.701 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.701 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.701 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 18:20:05.705 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.705 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.705 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.706 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 18:20:05.708 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.709 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.709 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.709 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 18:20:05.712 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.712 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.713 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.713 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 18:20:05.716 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.716 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.716 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.717 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 18:20:05.721 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.721 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.721 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.722 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 18:20:05.724 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.725 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.726 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.726 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 18:20:05.729 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.729 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.730 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.730 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 18:20:05.733 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.733 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.733 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.734 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 18:20:05.736 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.737 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.737 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.737 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 18:20:05.740 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.740 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.741 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.741 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 18:20:05.744 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.745 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.745 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.745 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 18:20:05.748 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.749 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.749 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.749 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 18:20:05.752 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.752 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.753 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.753 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 18:20:05.756 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.756 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.757 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.757 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 18:20:05.760 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.760 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.760 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.761 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 18:20:05.763 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.764 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.764 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.765 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 18:20:05.767 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.768 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.768 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.768 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 18:20:05.771 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.772 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.772 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.772 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 18:20:05.775 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.775 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.776 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.776 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 18:20:05.781 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.781 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.782 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.782 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 18:20:05.786 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.786 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.787 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.787 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 18:20:05.790 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.791 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.791 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.792 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 18:20:05.795 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.795 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.796 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.796 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 18:20:05.798 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.798 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.799 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.799 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 18:20:05.801 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.802 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.802 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.802 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 18:20:05.805 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.805 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.805 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.806 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 18:20:05.808 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.808 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.809 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.809 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 18:20:05.815 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.816 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.816 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.816 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 18:20:05.831 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.832 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.832 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.832 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 18:20:05.837 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.837 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.837 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.838 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 18:20:05.842 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.842 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.843 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.843 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 18:20:05.847 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.848 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.848 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.849 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 18:20:05.851 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.852 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.852 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.852 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 18:20:05.856 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.856 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.857 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.857 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 18:20:05.859 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 18:20:05.859 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 18:20:05.864 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.865 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:20:05.866 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.867 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.867 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.868 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 18:20:05.871 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.872 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:05.872 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.873 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 18:20:05.875 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 18:20:05.884 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,f8bddf1da67854a8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048249, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:20:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:20:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:20:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:20:05.984 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:20:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:20:05.995 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:20:05.995 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:20:05.995 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:20:05.999 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:06.000 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:20:06.000 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:06.001 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:20:06.048 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:17:41, start=2024-09-11T18:15:11, end=2024-09-11T18:17:41, updates=297, granule=120000ms, delay=150000ms 2024-09-11 18:20:06.057 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:17:41, target=2024-09-11T18:17:41, granule=120000ms, delay=0ms) 2024-09-11 18:20:06.057 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:17:41, target=2024-09-11T18:10:51, granule=600000ms, delay=410000ms) 2024-09-11 18:20:06.057 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=87ms, period=120000ms 2024-09-11 18:20:06.171 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:06.171 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:20:06.351807+08:00, lastProcessTime=2024-09-11T18:19:56.351125+08:00 2024-09-11 18:20:06.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:20:06.351807+08:00 2024-09-11 18:20:06.706 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048219, status:SUCCESS 2024-09-11 18:20:06.706 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048219, status:SUCCESS 2024-09-11 18:20:07.440 INFO 2981023 --- [ocp-async-1,721b44eb4ba0be47,049ad3e147868111] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:07.790 INFO 2981023 --- [ocp-async-5,bccfde73ef016bd8,60a5ae9e5b030262] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:20:07.969 INFO 2981023 --- [ocp-schedules-9,d0e542160674d4fc,285140cb8c1116de] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:20:08.084 INFO 2981023 --- [ocp-schedules-12,8d1387119aac6011,eaddc5b7ca04f39f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:20:08.158 INFO 2981023 --- [ocp-schedules-20,b9d25412874d0519,9ae2f3351ee1f614] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:20:08.158+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:20:08.162 INFO 2981023 --- [ocp-schedules-20,b9d25412874d0519,9ae2f3351ee1f614] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:20:08.163 INFO 2981023 --- [ocp-schedules-20,b9d25412874d0519,9ae2f3351ee1f614] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:20:08.164 INFO 2981023 --- [ocp-schedules-20,b9d25412874d0519,9ae2f3351ee1f614] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:20:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,a00597fb4437d8ec,0c888595d38cba84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,a00597fb4437d8ec,0c888595d38cba84] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:09.171 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:09.172 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:09.440 INFO 2981023 --- [ocp-async-4,8ffcf921cce9a74a,b3783d51f53863f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:10.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:20:10.003 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:20:10.327 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7cb5cca6d4448bb6,c6875efe7aa45744] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=7cb5cca6d4448bb6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:20:10.327 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f073f880d5a30bcd,98592e5fbf525f74] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=f073f880d5a30bcd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:20:10.329 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,7cb5cca6d4448bb6,c6875efe7aa45744] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=7cb5cca6d4448bb6, duration=5 ms] 2024-09-11 18:20:10.338 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,9ed0207c16de9597,0eac30bbdfdf6ea9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=9ed0207c16de9597, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:20:10.341 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,9ed0207c16de9597,0eac30bbdfdf6ea9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=9ed0207c16de9597, duration=5 ms] 2024-09-11 18:20:10.345 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,f073f880d5a30bcd,98592e5fbf525f74] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=f073f880d5a30bcd, duration=22 ms] 2024-09-11 18:20:10.913 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:20:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:20:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,7c08e4640a21c785,3eb91c88a7fa23cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,7c08e4640a21c785,3eb91c88a7fa23cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:11.440 INFO 2981023 --- [ocp-async-9,3eec7481a70eae18,b6a90790441d09a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:11.774 INFO 2981023 --- [ocp-schedules-28,071d0419c9b0dfe0,9c5c534c183d56ed] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:20:11.824 WARN 2981023 --- [scanner-13648586-1,071d0419c9b0dfe0,1cb2b4c159aee6bc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:20:11.848 INFO 2981023 --- [alarm-detect-20,071d0419c9b0dfe0,a90876acd815f514] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645955, data is empty. 2024-09-11 18:20:11.852 INFO 2981023 --- [alarm-detect-20,071d0419c9b0dfe0,a90876acd815f514] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645998, data is empty. 2024-09-11 18:20:11.856 INFO 2981023 --- [alarm-detect-20,071d0419c9b0dfe0,a90876acd815f514] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645998, data is empty. 2024-09-11 18:20:11.872 INFO 2981023 --- [alarm-detect-5,071d0419c9b0dfe0,d8e6784406000daa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645955, data is empty. 2024-09-11 18:20:11.878 INFO 2981023 --- [alarm-detect-5,071d0419c9b0dfe0,d8e6784406000daa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645998, data is empty. 2024-09-11 18:20:11.882 INFO 2981023 --- [alarm-detect-20,071d0419c9b0dfe0,8983310ed84aa13c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645955, data is empty. 2024-09-11 18:20:11.885 INFO 2981023 --- [alarm-detect-5,071d0419c9b0dfe0,d8e6784406000daa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645998, data is empty. 2024-09-11 18:20:11.887 INFO 2981023 --- [alarm-detect-20,071d0419c9b0dfe0,8983310ed84aa13c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645998, data is empty. 2024-09-11 18:20:11.892 INFO 2981023 --- [alarm-detect-20,071d0419c9b0dfe0,8983310ed84aa13c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645998, data is empty. 2024-09-11 18:20:12.031 INFO 2981023 --- [alarm-detect-8,071d0419c9b0dfe0,56a5e4f4e543204b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645955, data is empty. 2024-09-11 18:20:12.036 INFO 2981023 --- [alarm-detect-8,071d0419c9b0dfe0,56a5e4f4e543204b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645998, data is empty. 2024-09-11 18:20:12.043 INFO 2981023 --- [alarm-detect-8,071d0419c9b0dfe0,56a5e4f4e543204b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645998, data is empty. 2024-09-11 18:20:12.172 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:12.172 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:12.665 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6c814e1ec52263a8,0d3f5d1b0c519792] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&version=2&ObClusterId=1718689631, client=10.31.35.5, traceId=6c814e1ec52263a8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=rtdob,1718689631,2,{"ObRegion":"rtdob","ObCluster":"rtdob","ObRegionId":1718689631,"ObClusterId":1718689631,"Type":"PRIMARY","timestamp":1726050012663339,"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:20:12.676 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6c814e1ec52263a8,0d3f5d1b0c519792] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=rtdob, obClusterId=1718689631, rootServerJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, groupName=rtdob, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:20:12.676 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6c814e1ec52263a8,0d3f5d1b0c519792] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:20:12.676 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6c814e1ec52263a8,0d3f5d1b0c519792] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=rtdob, clusterId=1718689631, affectRows=1 2024-09-11 18:20:12.677 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,6c814e1ec52263a8,0d3f5d1b0c519792] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.5, traceId=6c814e1ec52263a8, duration=13 ms] 2024-09-11 18:20:12.718 INFO 2981023 --- [ocp-schedules-24,6fcf6d220b44d7e5,0b11647a86dbe28f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:20:13.398 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049940, nodeCount=1000, affectRows=1000 2024-09-11 18:20:13.400 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049940, nodeCount=999, validCount=999 2024-09-11 18:20:13.403 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049940, nodeCount=1000, affectRows=1000 2024-09-11 18:20:13.405 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049940, nodeCount=1000, validCount=1000 2024-09-11 18:20:13.405 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726049940, nodeCount=48, affectRows=48 2024-09-11 18:20:13.405 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726049940, nodeCount=46, validCount=46 2024-09-11 18:20:13.440 INFO 2981023 --- [ocp-async-2,f1b033000db2ef73,fc5733753b590ac7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,ea91375999158672,68305655095ee7e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,b84cd4ac4c6f9d69,511a9632e15edb8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:20:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,ea91375999158672,68305655095ee7e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:14.650 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726049940, nodeCount=480, affectRows=480 2024-09-11 18:20:15.172 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:15.173 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:15.440 INFO 2981023 --- [ocp-async-8,773188172676f26b,709b3b94b5f893e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:15.571 INFO 2981023 --- [ocp-schedules-24,07b93223f0a70775,6653d370a290317a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:20:15.583 INFO 2981023 --- [ocp-schedules-24,07b93223f0a70775,6653d370a290317a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:20:15.583 INFO 2981023 --- [ocp-schedules-24,07b93223f0a70775,6653d370a290317a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:20:16.023 INFO 2981023 --- [ocp-schedules-8,91dffcae067e323e,1e72c2cd6a6b6d9a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:20:16.024 INFO 2981023 --- [ocp-schedules-25,d6a4508cd5a012c8,30013b2dd056f563] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:20:16.024 INFO 2981023 --- [host-check-clockdiff1,91dffcae067e323e,d5471a7d3bb5bd66] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:20:16.024 INFO 2981023 --- [host-check-clockdiff2,91dffcae067e323e,ff618ec3645ffcae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:20:16.026 INFO 2981023 --- [host-check-clockdiff3,91dffcae067e323e,a3752efbbac21201] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:20:16.026 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:20:16.026 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:20:16.027 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:20:16.027 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:20:16.040 INFO 2981023 --- [host-check-clockdiff4,91dffcae067e323e,4b18832859fcaf83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:20:16.049 INFO 2981023 --- [host-check-clockdiff3,91dffcae067e323e,a3752efbbac21201] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050016 0 0 , error= 2024-09-11 18:20:16.050 INFO 2981023 --- [host-check-clockdiff1,91dffcae067e323e,d5471a7d3bb5bd66] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050016 0 0 , error= 2024-09-11 18:20:16.055 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:20:16.055 INFO 2981023 --- [ocp-schedules-28,4669d8b35ae903f0,566a095d026db4e3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:20:16.055 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:20:16.056 INFO 2981023 --- [host-check-agent-status1,4669d8b35ae903f0,63eb059d0a16f515] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:20:16.056 INFO 2981023 --- [host-check-agent-status2,4669d8b35ae903f0,cf055d9b9d893e99] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:20:16.056 INFO 2981023 --- [host-check-agent-status3,4669d8b35ae903f0,c70ea7cefcfb081b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:20:16.057 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:20:16.059 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:20:16.059 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:20:16.060 INFO 2981023 --- [host-check-agent-status4,4669d8b35ae903f0,0e6b01b75d4e7d8f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:20:16.061 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:20:16.062 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:20:16.064 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:20:16.065 INFO 2981023 --- [host-check-agent-status1,4669d8b35ae903f0,63eb059d0a16f515] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:20:16.065 INFO 2981023 --- [host-check-agent-status3,4669d8b35ae903f0,c70ea7cefcfb081b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:20:16.066 INFO 2981023 --- [host-check-agent-status3,4669d8b35ae903f0,c70ea7cefcfb081b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:20:16.067 INFO 2981023 --- [host-check-agent-status1,4669d8b35ae903f0,63eb059d0a16f515] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:20:16.068 INFO 2981023 --- [host-check-agent-status4,4669d8b35ae903f0,0e6b01b75d4e7d8f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:20:16.069 INFO 2981023 --- [host-check-agent-status4,4669d8b35ae903f0,0e6b01b75d4e7d8f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:20:16.098 INFO 2981023 --- [host-check-agent-status2,4669d8b35ae903f0,cf055d9b9d893e99] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:20:16.182 INFO 2981023 --- [host-check-agent-status2,4669d8b35ae903f0,cf055d9b9d893e99] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:20:16.192 INFO 2981023 --- [ocp-schedules-20,a8cace74bd9d3685,e68a66c7a2c254fd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:20:16.193 INFO 2981023 --- [ocp-schedules-23,2c4c89136517cfad,7f4ec3d10ecbe8fd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:20:16.204 INFO 2981023 --- [ocp-schedules-27,f7a30e0bf454ac25,dd4567c22021efb8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:20:16.206 INFO 2981023 --- [ocp-schedules-9,8f478ab44006924f,793e36a7039ae599] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:20:16.263 INFO 2981023 --- [ocp-schedules-20,a8cace74bd9d3685,e68a66c7a2c254fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:20:16.264 INFO 2981023 --- [ocp-schedules-23,2c4c89136517cfad,7f4ec3d10ecbe8fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:20:16.265 INFO 2981023 --- [ocp-schedules-27,f7a30e0bf454ac25,dd4567c22021efb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:20:16.267 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050003098000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:20:16.280 INFO 2981023 --- [ocp-schedules-23,2c4c89136517cfad,7f4ec3d10ecbe8fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:20:16.296 INFO 2981023 --- [ocp-schedules-6,4210ca374babe763,0b6f1417a737a368] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:20:16.317 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:20:16.319 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:20:16.324090297+08:00 2024-09-11 18:20:16.319 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:20:16.319+08:00 2024-09-11 18:20:16.321 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:20:16.319+08:00 and status AVAILABLE 2024-09-11 18:20:16.327 INFO 2981023 --- [host-check-agent-status1,4669d8b35ae903f0,63eb059d0a16f515] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:20:16.331 INFO 2981023 --- [host-check-agent-status1,4669d8b35ae903f0,63eb059d0a16f515] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:20:16.331 INFO 2981023 --- [host-check-agent-status1,4669d8b35ae903f0,63eb059d0a16f515] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:20:16.331 INFO 2981023 --- [host-check-agent-status1,4669d8b35ae903f0,63eb059d0a16f515] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:20:16.353 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:20:16.354 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:20:16.355 INFO 2981023 --- [host-check-agent-status3,4669d8b35ae903f0,c70ea7cefcfb081b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:20:16.356 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:20:16.355439689+08:00 2024-09-11 18:20:16.357 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:20:16.356+08:00 2024-09-11 18:20:16.358 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:20:16.360 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:20:16.356+08:00 and status AVAILABLE 2024-09-11 18:20:16.363 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:20:16.361875252+08:00 2024-09-11 18:20:16.364 INFO 2981023 --- [host-check-agent-status3,4669d8b35ae903f0,c70ea7cefcfb081b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:20:16.364 INFO 2981023 --- [host-check-agent-status3,4669d8b35ae903f0,c70ea7cefcfb081b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:20:16.364 INFO 2981023 --- [host-check-agent-status3,4669d8b35ae903f0,c70ea7cefcfb081b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:20:16.364 INFO 2981023 --- [host-check-agent-status4,4669d8b35ae903f0,0e6b01b75d4e7d8f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:20:16.365 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.366 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:20:16.367 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:20:16.367 INFO 2981023 --- [host-check-agent-status4,4669d8b35ae903f0,0e6b01b75d4e7d8f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:20:16.367 INFO 2981023 --- [host-check-agent-status4,4669d8b35ae903f0,0e6b01b75d4e7d8f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:20:16.367 INFO 2981023 --- [host-check-agent-status4,4669d8b35ae903f0,0e6b01b75d4e7d8f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:20:16.374 INFO 2981023 --- [host-check-agent-status2,4669d8b35ae903f0,cf055d9b9d893e99] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:20:16.375 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:20:16.373777+08:00, lastProcessTime=2024-09-11T18:20:06.351807+08:00 2024-09-11 18:20:16.376 INFO 2981023 --- [host-check-agent-status2,4669d8b35ae903f0,cf055d9b9d893e99] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:20:16.376 INFO 2981023 --- [host-check-agent-status2,4669d8b35ae903f0,cf055d9b9d893e99] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:20:16.376 INFO 2981023 --- [host-check-agent-status2,4669d8b35ae903f0,cf055d9b9d893e99] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:20:16.377 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:20:16.363+08:00 2024-09-11 18:20:16.379 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:20:16.363+08:00 and status AVAILABLE 2024-09-11 18:20:16.379 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:20:16.380 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:20:16.381 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:20:16.379852857+08:00 2024-09-11 18:20:16.382 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:20:16.381+08:00 2024-09-11 18:20:16.384 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.385 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:20:16.381+08:00 and status AVAILABLE 2024-09-11 18:20:16.385 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:20:16.386 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:20:16.386 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:20:16.387 INFO 2981023 --- [ocp-schedules-20,a8cace74bd9d3685,e68a66c7a2c254fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:20:16.388 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:20:16.390 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:20:16.390 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.391 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:16.393 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:20:16.394 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:20:16.394 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:20:16.396 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:20:16.398 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:20:16.406 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.407 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:16.408 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:20:16.408 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.409 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:16.409 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:20:16.426 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:20:16.428 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:20:16.430 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:20:16.433 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:16.433 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.434 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:20:16.450 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:20:16.456 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:20:16.460 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:20:16.470 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:20:16.470 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:20:16.472 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:20:16.474 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:20:16.501 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:20:16.501 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050003098000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:20:16.503 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:20:16.503 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:16.504 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.504 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:16.505 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:20:16.506 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.508 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:20:16.509 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:20:16.509 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:20:16.512 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:20:16.516 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:20:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:20:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:20:16.530 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:20:16.534 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:20:16.537 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:20:16.542 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:20:16.550 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:20:16.564 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:20:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:20:16.568 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:20:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.576 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:20:16.576 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:20:16.579 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:20:16.581 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:20:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:20:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:20:16.590 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:20:16.591 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:20:16.591 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:20:16.599 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:16.600 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.601 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:20:16.601 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:20:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:20:16.610 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:20:16.611 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:20:16.615 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:20:16.622 INFO 2981023 --- [host-check-host-status4,d6a4508cd5a012c8,c58dbd2a4e76f2a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:20:16.630 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:20:16.642 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:20:16.648 INFO 2981023 --- [host-check-host-status2,d6a4508cd5a012c8,95c400c0da8ebee6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:20:16.649 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:16.652 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.653 INFO 2981023 --- [host-check-host-status3,d6a4508cd5a012c8,24596df40f01d978] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:20:16.653 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:16.653 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:20:16.654 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:20:16.654 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.657 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:20:16.658 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:20:16.669 INFO 2981023 --- [host-check-host-status1,d6a4508cd5a012c8,30dfb5c343cf8d59] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:20:16.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:20:16.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:20:16.687 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:20:16.695 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:20:16.698 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:20:16.701 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:20:16.701 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:20:16.705 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:20:16.706 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:20:16.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:16.708 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.708 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:20:16.373777+08:00 2024-09-11 18:20:16.709 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:20:16.715 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:20:16.715 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:20:16.719 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:20:16.801 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:20:16.807 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:16.808 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:16.808 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:20:16.814 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:20:16.814 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:20:16.814 INFO 2981023 --- [ocp-schedules-27,f7a30e0bf454ac25,dd4567c22021efb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:20:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,f34a9e78051ad3e5,c8ca906b93ea407f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,f34a9e78051ad3e5,c8ca906b93ea407f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:17.324 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726049940, nodeCount=480, validCount=480 2024-09-11 18:20:17.440 INFO 2981023 --- [ocp-async-4,6c7617ef8d744277,d0112077218a5703] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:17.790 INFO 2981023 --- [ocp-async-10,3f76cd7d3cf4f77c,c004f72170c4afa5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:20:17.839 INFO 2981023 --- [host-check-clockdiff2,91dffcae067e323e,ff618ec3645ffcae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050017 5 5 , error= 2024-09-11 18:20:18.173 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:18.173 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:19.440 INFO 2981023 --- [ocp-async-3,0b0bca06ded5d16e,eae98fbbd50c39f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,01a81ffa039c1f41,0c73965b4af962d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,01a81ffa039c1f41,0c73965b4af962d1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:21.174 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:21.174 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:21.440 INFO 2981023 --- [ocp-async-10,849374a1422f2cb9,7360419ca4143756] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:22.035 INFO 2981023 --- [host-check-clockdiff4,91dffcae067e323e,4b18832859fcaf83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050022 0 0 , error= 2024-09-11 18:20:22.096 INFO 2981023 --- [ocp-schedules-17,eeb1f3710b373c9d,a243da162514d33c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:20:22.195 WARN 2981023 --- [scanner-13650360-1,eeb1f3710b373c9d,b23f0fe1eb2378a4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:20:22.224 INFO 2981023 --- [alarm-detect-5,eeb1f3710b373c9d,e585bfa3efd4c03d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726049985 ~ 1726050012 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050000, value=2.258)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.258, values={1726050000=2.258}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:20:22.224 INFO 2981023 --- [alarm-detect-5,eeb1f3710b373c9d,e585bfa3efd4c03d] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Pending 2024-09-11 18:20:22.721 INFO 2981023 --- [ocp-schedules-27,cde4a2a4514e959f,eca71960a39370f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:20:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,c4adaf997d01f70f,14723f32bb904cff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,c4adaf997d01f70f,14723f32bb904cff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:23.441 INFO 2981023 --- [ocp-async-5,14afade4bacbe090,26bb9b0ea04ee506] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:24.174 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:24.175 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:24.468 INFO 2981023 --- [ocp-schedules-30,71e357c03355af4c,a9144b1bb2a8f1d2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:20:25.441 INFO 2981023 --- [ocp-async-9,fe99e5d81f83dc65,26c1ebd959b8763c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,68ba71dc6445a5a7,6160efcc49ee5d2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,68ba71dc6445a5a7,6160efcc49ee5d2b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:20:26.337413+08:00, lastProcessTime=2024-09-11T18:20:16.373777+08:00 2024-09-11 18:20:26.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:20:26.337413+08:00 2024-09-11 18:20:27.114 INFO 2981023 --- [ocp-schedules-27,2b5f9273ba5f1d52,8dc21057715d2945] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:20:27.114 INFO 2981023 --- [ocp-schedules-27,2b5f9273ba5f1d52,8dc21057715d2945] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:20:27.114 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:20:27.114 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:20:27.114 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:20:27.118 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:20:27.119 INFO 2981023 --- [ocp-schedules-27,2b5f9273ba5f1d52,8dc21057715d2945] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:20:27.119 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:20:27.119 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:20:27.119 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:20:27.120 INFO 2981023 --- [ocp-schedules-27,2b5f9273ba5f1d52,8dc21057715d2945] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:20:27.120 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:20:27.121 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:20:27.137 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:20:27.137 INFO 2981023 --- [ocp-schedules-27,2b5f9273ba5f1d52,8dc21057715d2945] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:20:27.138 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:20:27.140 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:20:27.142 INFO 2981023 --- [ocp-schedules-27,2b5f9273ba5f1d52,8dc21057715d2945] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:20:27.175 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:27.175 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:27.441 INFO 2981023 --- [ocp-async-3,2a4b5a193b2a6474,7cbaf3f4894424f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:27.791 INFO 2981023 --- [ocp-async-4,351c4d192fe6e5ce,503f53e32458e2e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:20:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,8fb49225ef285367,4d763b79da370d59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,8fb49225ef285367,4d763b79da370d59] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:29.441 INFO 2981023 --- [ocp-async-7,e92a88551d31c749,7fa9ab8ae47ab205] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:30.175 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:30.177 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:31.441 INFO 2981023 --- [ocp-async-4,35492fc56a1c4fd1,206ee3aaa7b57469] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:32.289 INFO 2981023 --- [ocp-schedules-26,40c19df9b5950501,db3687d1032e1f86] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:20:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,4d88de29fe6146eb,3e1724cbd7c70b4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:32.320 INFO 2981023 --- [prometheus-minute-schedule-1,4d88de29fe6146eb,3e1724cbd7c70b4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:32.345 WARN 2981023 --- [scanner-13650553-1,40c19df9b5950501,3769482435858ce9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:20:32.381 INFO 2981023 --- [alarm-detect-14,40c19df9b5950501,632b53adf8ff8ea3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645998, data is empty. 2024-09-11 18:20:32.385 INFO 2981023 --- [alarm-detect-14,40c19df9b5950501,632b53adf8ff8ea3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645998, data is empty. 2024-09-11 18:20:32.388 INFO 2981023 --- [alarm-detect-14,40c19df9b5950501,632b53adf8ff8ea3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645998, data is empty. 2024-09-11 18:20:32.406 INFO 2981023 --- [alarm-detect-15,40c19df9b5950501,892664b18fa09caf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645998, data is empty. 2024-09-11 18:20:32.407 INFO 2981023 --- [alarm-detect-8,40c19df9b5950501,c4a48861b055278f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645998, data is empty. 2024-09-11 18:20:32.410 INFO 2981023 --- [alarm-detect-8,40c19df9b5950501,c4a48861b055278f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645998, data is empty. 2024-09-11 18:20:32.411 INFO 2981023 --- [alarm-detect-15,40c19df9b5950501,892664b18fa09caf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645998, data is empty. 2024-09-11 18:20:32.413 INFO 2981023 --- [alarm-detect-8,40c19df9b5950501,c4a48861b055278f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645998, data is empty. 2024-09-11 18:20:32.415 INFO 2981023 --- [alarm-detect-15,40c19df9b5950501,892664b18fa09caf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645998, data is empty. 2024-09-11 18:20:32.476 INFO 2981023 --- [alarm-detect-9,40c19df9b5950501,25e42835083be882] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050000 ~ 1726050022 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050015, value=2.047)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.047, values={1726050015=2.047}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:20:32.569 INFO 2981023 --- [alarm-detect-2,40c19df9b5950501,ade4ec86b8caea2a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645998, data is empty. 2024-09-11 18:20:32.575 INFO 2981023 --- [alarm-detect-2,40c19df9b5950501,ade4ec86b8caea2a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645998, data is empty. 2024-09-11 18:20:32.580 INFO 2981023 --- [alarm-detect-2,40c19df9b5950501,ade4ec86b8caea2a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645998, data is empty. 2024-09-11 18:20:32.724 INFO 2981023 --- [ocp-schedules-22,e40d65431b3383c7,9f66f585d06176bb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:20:33.177 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:33.178 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:33.441 INFO 2981023 --- [ocp-async-6,23eb1e05265b5dd0,a13008b8bbc23491] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,d611a8d4223dc853,c9aac4f0d1aa821e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:35.317 INFO 2981023 --- [prometheus-minute-schedule-1,d611a8d4223dc853,c9aac4f0d1aa821e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:35.441 INFO 2981023 --- [ocp-async-8,7814488bf94ca7bd,d938a8ac0e67f2da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:35.586 INFO 2981023 --- [ocp-schedules-7,d0fd78c38f07077a,c58c86931e1d4fda] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:20:35.594 INFO 2981023 --- [ocp-schedules-7,d0fd78c38f07077a,c58c86931e1d4fda] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:20:35.594 INFO 2981023 --- [ocp-schedules-7,d0fd78c38f07077a,c58c86931e1d4fda] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:20:36.178 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:36.179 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:36.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:20:36.352419+08:00, lastProcessTime=2024-09-11T18:20:26.337413+08:00 2024-09-11 18:20:36.488 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:20:36.352419+08:00 2024-09-11 18:20:37.314 INFO 2981023 --- [prometheus-cache-monitor1,ab15eb518f0339af,70bcd759b699269f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:20:37.314 INFO 2981023 --- [prometheus-cache-monitor1,ab15eb518f0339af,70bcd759b699269f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:20:37.441 INFO 2981023 --- [ocp-async-1,1bee184eee8e8795,372c92af577c0a33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:37.791 INFO 2981023 --- [ocp-async-10,8d19a9ccffcf420d,e67fb975372114ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:20:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,34f0080eac0e83aa,49a2cd7e399e0fb6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,34f0080eac0e83aa,49a2cd7e399e0fb6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:39.179 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:39.180 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:39.442 INFO 2981023 --- [ocp-async-3,22d9373cef5b5bbb,2eb0ee4651f8b5e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:40.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:20:40.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:20:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,aea61863b91b5ab9,06ac09626a258bb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,aea61863b91b5ab9,06ac09626a258bb0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:41.442 INFO 2981023 --- [ocp-async-10,c5a1203a1b34dc70,f46f40c93a95e050] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:42.180 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:42.181 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:42.631 INFO 2981023 --- [ocp-schedules-27,d832a8d4f7308cf5,d686ff267b20eb9b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:20:42.684 WARN 2981023 --- [scanner-13650891-1,d832a8d4f7308cf5,3c895585d4fae22b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:20:42.726 INFO 2981023 --- [ocp-schedules-26,b8bdb47f90f390c8,56a3099508bbdd18] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:20:43.442 INFO 2981023 --- [ocp-async-9,5b1f787725ce5d51,cc94fc6926636be1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,ab2a535569b0c9b4,7425a5d65e6b197a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,ab2a535569b0c9b4,7425a5d65e6b197a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:45.101 INFO 2981023 --- [prometheus-second-schedule-1,007861674301d2ba,5041dacfc859fc91] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:20:45.181 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:45.181 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:45.442 INFO 2981023 --- [ocp-async-2,9820dca0f6f37702,dc076132af2fe2a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:46.018 INFO 2981023 --- [ocp-schedules-23,c45be6fb498103c2,0ae373865efda41d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:20:46.019 INFO 2981023 --- [host-check-clockdiff1,c45be6fb498103c2,b77253fdd267e40b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:20:46.019 INFO 2981023 --- [host-check-clockdiff2,c45be6fb498103c2,bfde82c1adfba696] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:20:46.019 INFO 2981023 --- [host-check-clockdiff3,c45be6fb498103c2,839dd7499fae16c7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:20:46.021 INFO 2981023 --- [host-check-clockdiff4,c45be6fb498103c2,1ca6dc97e7a1afb2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:20:46.022 INFO 2981023 --- [ocp-schedules-30,8ece7fb9f1af48fd,45350f524b389562] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:20:46.027 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:20:46.027 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:20:46.028 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:20:46.028 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:20:46.032 INFO 2981023 --- [ocp-schedules-13,9063f22641454735,6521fde05a20d536] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:20:46.032 INFO 2981023 --- [host-check-agent-status1,9063f22641454735,0c2639b1df0e7cdb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:20:46.033 INFO 2981023 --- [host-check-agent-status2,9063f22641454735,fdd9c2d6d00ab11b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:20:46.033 INFO 2981023 --- [host-check-agent-status3,9063f22641454735,8b3e68523339a9d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:20:46.044 INFO 2981023 --- [host-check-agent-status4,9063f22641454735,3b97ffccaeafac7b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:20:46.047 INFO 2981023 --- [host-check-clockdiff3,c45be6fb498103c2,839dd7499fae16c7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050046 0 0 , error= 2024-09-11 18:20:46.047 INFO 2981023 --- [host-check-clockdiff1,c45be6fb498103c2,b77253fdd267e40b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050046 0 0 , error= 2024-09-11 18:20:46.067 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:20:46.067 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:20:46.067 INFO 2981023 --- [host-check-agent-status1,9063f22641454735,0c2639b1df0e7cdb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:20:46.068 INFO 2981023 --- [host-check-agent-status2,9063f22641454735,fdd9c2d6d00ab11b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:20:46.071 INFO 2981023 --- [host-check-agent-status2,9063f22641454735,fdd9c2d6d00ab11b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:20:46.071 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:20:46.072 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:20:46.072 INFO 2981023 --- [host-check-agent-status4,9063f22641454735,3b97ffccaeafac7b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:20:46.073 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:20:46.073 INFO 2981023 --- [host-check-agent-status4,9063f22641454735,3b97ffccaeafac7b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:20:46.074 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:20:46.074 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:20:46.081 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:20:46.087 INFO 2981023 --- [host-check-agent-status1,9063f22641454735,0c2639b1df0e7cdb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:20:46.165 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:20:46.192 INFO 2981023 --- [ocp-schedules-22,dc9b03b9ff01d4ce,653586440863ef4b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:20:46.192 INFO 2981023 --- [ocp-schedules-5,917837a382b24399,4072e00afb57daf6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:20:46.195 INFO 2981023 --- [ocp-schedules-14,5175f199c33a14c9,7fb1ef9b341564fd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:20:46.195 INFO 2981023 --- [ocp-schedules-27,4764cde38b87f1cd,f54e8cc6c4d5cdfe] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:20:46.261 INFO 2981023 --- [ocp-schedules-14,5175f199c33a14c9,7fb1ef9b341564fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:20:46.269 INFO 2981023 --- [host-check-agent-status3,9063f22641454735,8b3e68523339a9d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:20:46.269 INFO 2981023 --- [ocp-schedules-8,5f756b261f8d5237,01fea053e5677a45] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:20:46.273 INFO 2981023 --- [ocp-schedules-22,dc9b03b9ff01d4ce,653586440863ef4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:20:46.273 INFO 2981023 --- [host-check-agent-status3,9063f22641454735,8b3e68523339a9d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:20:46.294 INFO 2981023 --- [ocp-schedules-5,917837a382b24399,4072e00afb57daf6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:20:46.307 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050003098000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:20:46.307 INFO 2981023 --- [ocp-schedules-22,dc9b03b9ff01d4ce,653586440863ef4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:20:46.369 INFO 2981023 --- [ocp-schedules-8,5f756b261f8d5237,01fea053e5677a45] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:20:46.379 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:20:46.393 INFO 2981023 --- [host-check-agent-status2,9063f22641454735,fdd9c2d6d00ab11b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:20:46.397 INFO 2981023 --- [host-check-agent-status2,9063f22641454735,fdd9c2d6d00ab11b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:20:46.397 INFO 2981023 --- [host-check-agent-status2,9063f22641454735,fdd9c2d6d00ab11b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:20:46.397 INFO 2981023 --- [host-check-agent-status2,9063f22641454735,fdd9c2d6d00ab11b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:20:46.398 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:20:46.405 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:20:46.404903024+08:00 2024-09-11 18:20:46.455 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:20:46.405+08:00 2024-09-11 18:20:46.459 INFO 2981023 --- [ocp-schedules-9,5130faf943c53945,a7be025f589d498b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:20:46.460 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:20:46.405+08:00 and status AVAILABLE 2024-09-11 18:20:46.466 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:20:46.464713+08:00, lastProcessTime=2024-09-11T18:20:36.352419+08:00 2024-09-11 18:20:46.470 INFO 2981023 --- [host-check-agent-status3,9063f22641454735,8b3e68523339a9d3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:20:46.474 INFO 2981023 --- [host-check-agent-status3,9063f22641454735,8b3e68523339a9d3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:20:46.474 INFO 2981023 --- [host-check-agent-status3,9063f22641454735,8b3e68523339a9d3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:20:46.474 INFO 2981023 --- [host-check-agent-status3,9063f22641454735,8b3e68523339a9d3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:20:46.476 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:20:46.480 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:20:46.481 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:20:46.480967387+08:00 2024-09-11 18:20:46.484 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:20:46.481+08:00 2024-09-11 18:20:46.488 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:20:46.490 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:20:46.481+08:00 and status AVAILABLE 2024-09-11 18:20:46.490 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:20:46.489983291+08:00 2024-09-11 18:20:46.491 INFO 2981023 --- [host-check-agent-status4,9063f22641454735,3b97ffccaeafac7b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:20:46.494 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:20:46.490+08:00 2024-09-11 18:20:46.495 INFO 2981023 --- [host-check-agent-status4,9063f22641454735,3b97ffccaeafac7b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:20:46.495 INFO 2981023 --- [host-check-agent-status4,9063f22641454735,3b97ffccaeafac7b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:20:46.495 INFO 2981023 --- [host-check-agent-status4,9063f22641454735,3b97ffccaeafac7b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:20:46.495 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.498 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:20:46.500 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:20:46.490+08:00 and status AVAILABLE 2024-09-11 18:20:46.501 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:20:46.502 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:20:46.507 INFO 2981023 --- [host-check-agent-status1,9063f22641454735,0c2639b1df0e7cdb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:20:46.515 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:20:46.515 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:20:46.516 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:20:46.517 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:20:46.518 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:20:46.523296916+08:00 2024-09-11 18:20:46.519 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:20:46.518+08:00 2024-09-11 18:20:46.521 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:20:46.522 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:20:46.518+08:00 and status AVAILABLE 2024-09-11 18:20:46.523 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:20:46.532 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.534 INFO 2981023 --- [host-check-agent-status1,9063f22641454735,0c2639b1df0e7cdb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:20:46.534 INFO 2981023 --- [host-check-agent-status1,9063f22641454735,0c2639b1df0e7cdb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:20:46.534 INFO 2981023 --- [host-check-agent-status1,9063f22641454735,0c2639b1df0e7cdb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:20:46.537 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:20:46.542 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:20:46.542 INFO 2981023 --- [ocp-schedules-5,917837a382b24399,4072e00afb57daf6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:20:46.558 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:20:46.558 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:20:46.558 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.559 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:46.559 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:20:46.561 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.561 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:46.570 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:20:46.572 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:20:46.572 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:20:46.575 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:20:46.592 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.592 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.593 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:46.596 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:20:46.598 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:20:46.602 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:20:46.602 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:20:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.606 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.607 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:20:46.615 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:20:46.617 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:20:46.619 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:20:46.619 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050003098000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:20:46.625 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:20:46.628 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:20:46.630 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.631 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:20:46.640 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:20:46.641 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.642 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.642 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:46.643 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:20:46.643 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.645 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:20:46.646 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:20:46.646 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:20:46.662 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:20:46.664 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:20:46.664 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:20:46.677 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:20:46.685 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:20:46.687 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:20:46.696 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:20:46.696 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.697 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:20:46.697 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.698 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:20:46.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:20:46.702 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:20:46.706 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:20:46.707 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:20:46.708 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:20:46.709 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:20:46.709 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:20:46.710 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:20:46.719 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:20:46.724 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:20:46.727 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.729 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.729 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:20:46.730 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:20:46.730 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.734 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:20:46.735 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:20:46.752 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:20:46.764 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:20:46.772 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:20:46.778 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:20:46.779 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:20:46.782 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.789 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:20:46.797 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.799 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:20:46.801 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:20:46.806 INFO 2981023 --- [host-check-host-status4,8ece7fb9f1af48fd,9c240c91686b7236] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:20:46.807 INFO 2981023 --- [host-check-host-status3,8ece7fb9f1af48fd,1ea48ce28c2b9f85] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:20:46.808 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:20:46.811 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.819 INFO 2981023 --- [host-check-host-status2,8ece7fb9f1af48fd,cf7d860484d8f2a6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:20:46.826 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.837 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:20:46.843 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.843 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.844 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.844 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:20:46.846 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.846 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.847 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:20:46.464713+08:00 2024-09-11 18:20:46.850 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:20:46.850 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:20:46.854 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:20:46.860 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:20:46.860 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.860 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:20:46.861 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:20:46.862 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:20:46.866 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.866 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:20:46.871 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:20:46.871 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:20:46.872 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:20:46.873 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:20:46.880 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:20:46.880 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.885 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.887 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:20:46.900 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.904 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:20:46.905 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:20:46.908 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:20:46.911 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:20:46.911 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:20:46.915 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:20:46.926 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:20:46.928 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:46.945 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.951 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:20:46.956 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:20:46.961 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:46.969 INFO 2981023 --- [host-check-clockdiff2,c45be6fb498103c2,bfde82c1adfba696] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050046 5 5 , error= 2024-09-11 18:20:46.980 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:20:47.009 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:20:47.023 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:47.024 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:47.024 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:20:47.025 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:20:47.025 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:47.049 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:20:47.062 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:20:47.072 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:20:47.076 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:20:47.083 INFO 2981023 --- [ocp-schedules-4,dd5f6c5b061442c6,62691322a44805a5] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:20:47.103 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:20:47.105 WARN 2981023 --- [ocp-schedules-4,dd5f6c5b061442c6,62691322a44805a5] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:20:47.110 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:20:47.113 INFO 2981023 --- [ocp-schedules-2,59e2c18ae557ac4a,b8a6c1c2134547c4] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:20:47.268 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:20:47.274 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:20:47.279 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:47.283 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:20:47.286 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:20:47.292 INFO 2981023 --- [host-check-host-status1,8ece7fb9f1af48fd,74ad2892e298b870] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:20:47.295 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:20:47.295 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:20:47.295 INFO 2981023 --- [ocp-schedules-14,5175f199c33a14c9,7fb1ef9b341564fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:20:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,7bcb6a4f74512a4d,5f97d006cc197b03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,7bcb6a4f74512a4d,5f97d006cc197b03] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:47.442 INFO 2981023 --- [ocp-async-10,478acc4f2f18a831,676390687f7ea701] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:47.791 INFO 2981023 --- [ocp-async-3,5aa4f97e8ca33650,93512c38b1281975] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:20:48.181 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:48.182 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:49.442 INFO 2981023 --- [ocp-async-4,32d4176a1b06afc9,4c58ec84cf910388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:20:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:20:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,a400ce7a2ed70767,41b0ef97eaeecf20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,a400ce7a2ed70767,41b0ef97eaeecf20] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:50.347 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.352 INFO 2981023 --- [ocp-schedules-29,ae7d2966cb1335b2,92bb45bf4b7c81df] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:20:50.355 INFO 2981023 --- [ocp-schedules-28,c1e3963d3fcc28b3,95c32091579e053f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:20:51.182 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:51.182 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:51.442 INFO 2981023 --- [ocp-async-3,6537048bcab2eb71,6ca7abeb42afd8f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:51.632 INFO 2981023 --- [ocp-schedules-25,d19f993c85a902c9,24f42b63204a2a75] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:20:51.633 INFO 2981023 --- [ocp-schedules-25,d19f993c85a902c9,24f42b63204a2a75] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:20:51.633 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:20:51.633 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:20:51.633 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:20:51.634 INFO 2981023 --- [ocp-schedules-25,d19f993c85a902c9,24f42b63204a2a75] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:20:52.002 INFO 2981023 --- [host-check-clockdiff4,c45be6fb498103c2,1ca6dc97e7a1afb2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050051 0 0 , error= 2024-09-11 18:20:52.737 INFO 2981023 --- [ocp-schedules-2,01453dc4f8bfee0f,4b1cdfbb618a5127] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:20:52.852 INFO 2981023 --- [ocp-schedules-10,89f42040a24437c1,80e314352c02b392] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:20:52.864 WARN 2981023 --- [scanner-13651076-1,89f42040a24437c1,783a2b7993cbe2ff] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:20:52.918 INFO 2981023 --- [alarm-detect-9,89f42040a24437c1,9302abdc5cb7c656] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645998, data is empty. 2024-09-11 18:20:52.923 INFO 2981023 --- [alarm-detect-9,89f42040a24437c1,9302abdc5cb7c656] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645998, data is empty. 2024-09-11 18:20:52.930 INFO 2981023 --- [alarm-detect-9,89f42040a24437c1,9302abdc5cb7c656] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645998, data is empty. 2024-09-11 18:20:52.951 INFO 2981023 --- [alarm-detect-16,89f42040a24437c1,97de3fc3971fe733] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645998, data is empty. 2024-09-11 18:20:52.951 INFO 2981023 --- [alarm-detect-14,89f42040a24437c1,8f8669c755bb85cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645998, data is empty. 2024-09-11 18:20:52.956 INFO 2981023 --- [alarm-detect-16,89f42040a24437c1,97de3fc3971fe733] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645998, data is empty. 2024-09-11 18:20:52.957 INFO 2981023 --- [alarm-detect-14,89f42040a24437c1,8f8669c755bb85cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645998, data is empty. 2024-09-11 18:20:52.961 INFO 2981023 --- [alarm-detect-16,89f42040a24437c1,97de3fc3971fe733] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645998, data is empty. 2024-09-11 18:20:52.965 INFO 2981023 --- [alarm-detect-14,89f42040a24437c1,8f8669c755bb85cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645998, data is empty. 2024-09-11 18:20:53.008 INFO 2981023 --- [alarm-detect-11,89f42040a24437c1,808a95c0a9feeb2b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050015 ~ 1726050042 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050030, value=3.896)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=3.896, values={1726050030=3.896}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:20:53.008 INFO 2981023 --- [alarm-detect-11,89f42040a24437c1,808a95c0a9feeb2b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050015 ~ 1726050042 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050030, value=3.896)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=3.896, values={1726050030=3.896}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:20:53.008 INFO 2981023 --- [alarm-detect-11,89f42040a24437c1,808a95c0a9feeb2b] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-2, currentState=Pending 2024-09-11 18:20:53.079 INFO 2981023 --- [alarm-detect-10,89f42040a24437c1,1def319bf272c9b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645998, data is empty. 2024-09-11 18:20:53.084 INFO 2981023 --- [alarm-detect-10,89f42040a24437c1,1def319bf272c9b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645998, data is empty. 2024-09-11 18:20:53.087 INFO 2981023 --- [alarm-detect-10,89f42040a24437c1,1def319bf272c9b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645998, data is empty. 2024-09-11 18:20:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,b43e54216de96e69,4156ce5d5971a800] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,b43e54216de96e69,4156ce5d5971a800] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:53.350 INFO 2981023 --- [ocp-schedules-17,db8bc1b30ad66269,638ce013889db249] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:20:53.350 INFO 2981023 --- [ocp-schedules-17,db8bc1b30ad66269,638ce013889db249] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:20:53.442 INFO 2981023 --- [ocp-async-7,ce0ddcf9bc47f926,81ce0f3fe7870aaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:54.183 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:54.183 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:54.470 INFO 2981023 --- [ocp-schedules-25,efd0074784a7eb83,827d2a64cbf498f0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:20:55.082 INFO 2981023 --- [ocp-schedules-9,c990eee875b6d13c,1a54b46fcad2f411] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:20:55.099 WARN 2981023 --- [ocp-schedules-9,c990eee875b6d13c,1a54b46fcad2f411] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:20:55.443 INFO 2981023 --- [ocp-async-9,6f2211ea8e17bf58,35c4a730f6debfce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:55.595 INFO 2981023 --- [ocp-schedules-13,30d6cfe72dc88ca0,9023b9db5d591938] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:20:55.601 INFO 2981023 --- [ocp-schedules-13,30d6cfe72dc88ca0,9023b9db5d591938] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:20:55.601 INFO 2981023 --- [ocp-schedules-13,30d6cfe72dc88ca0,9023b9db5d591938] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:20:55.896 INFO 2981023 --- [ocp-schedules-26,95a35ba375bc248d,b9a0252d267134e8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:20:55.896 INFO 2981023 --- [ocp-schedules-26,95a35ba375bc248d,b9a0252d267134e8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:20:55.896 INFO 2981023 --- [ocp-schedules-26,95a35ba375bc248d,b9a0252d267134e8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:20:55.896 INFO 2981023 --- [ocp-schedules-26,95a35ba375bc248d,b9a0252d267134e8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:20:55.897 INFO 2981023 --- [ocp-schedules-26,95a35ba375bc248d,b9a0252d267134e8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:20:55.897 INFO 2981023 --- [ocp-schedules-26,95a35ba375bc248d,b9a0252d267134e8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:20:56.102 INFO 2981023 --- [ocp-schedules-24,57c94b39b592b8a2,00d3a9ed0d8cb993] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:20:56.108 INFO 2981023 --- [ocp-schedules-24,57c94b39b592b8a2,00d3a9ed0d8cb993] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:20:56.108 INFO 2981023 --- [ocp-schedules-24,57c94b39b592b8a2,00d3a9ed0d8cb993] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:20:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,4ad2c4da478cb7b9,f6d173e6a603c789] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,4ad2c4da478cb7b9,f6d173e6a603c789] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:56.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:20:56.337425+08:00, lastProcessTime=2024-09-11T18:20:46.464713+08:00 2024-09-11 18:20:56.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:20:56.337425+08:00 2024-09-11 18:20:57.183 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:20:57.184 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:20:57.443 INFO 2981023 --- [ocp-async-8,56d0607309ae437d,103ea676766901e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:57.791 INFO 2981023 --- [ocp-async-1,ea6a5f6018974b3e,1f88e83cbae1bc87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:20:58.340 INFO 2981023 --- [ocp-schedules-16,520f6c59feffd340,021b85ddf88bfaff] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:20:58.340 INFO 2981023 --- [ocp-schedules-16,520f6c59feffd340,021b85ddf88bfaff] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:20:58.356 INFO 2981023 --- [Collect all tenants major compactions5,520f6c59feffd340,9c26f71ca2074b10] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:20:58.356 INFO 2981023 --- [Collect all tenants major compactions4,520f6c59feffd340,8b22006dfe01fd84] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:20:58.357 INFO 2981023 --- [Collect all tenants major compactions6,520f6c59feffd340,ff327bc138d372d3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:20:58.357 INFO 2981023 --- [Collect all tenants major compactions2,520f6c59feffd340,4241e46d84332630] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:20:58.359 INFO 2981023 --- [Collect all tenants major compactions1,520f6c59feffd340,af2b1a578e001f33] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:20:58.361 INFO 2981023 --- [Collect all tenants major compactions3,520f6c59feffd340,a7c88cc142f13508] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:20:58.369 INFO 2981023 --- [Collect all tenants major compactions2,520f6c59feffd340,4241e46d84332630] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:58.371 INFO 2981023 --- [Collect all tenants major compactions1,520f6c59feffd340,af2b1a578e001f33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:58.371 INFO 2981023 --- [Collect all tenants major compactions2,520f6c59feffd340,4241e46d84332630] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:20:58.371 INFO 2981023 --- [Collect all tenants major compactions6,520f6c59feffd340,ff327bc138d372d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:58.372 INFO 2981023 --- [Collect all tenants major compactions3,520f6c59feffd340,a7c88cc142f13508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:58.372 INFO 2981023 --- [Collect all tenants major compactions1,520f6c59feffd340,af2b1a578e001f33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:20:58.373 INFO 2981023 --- [Collect all tenants major compactions6,520f6c59feffd340,ff327bc138d372d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:20:58.374 INFO 2981023 --- [Collect all tenants major compactions2,520f6c59feffd340,4241e46d84332630] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:20:58.374 INFO 2981023 --- [Collect all tenants major compactions2,520f6c59feffd340,4241e46d84332630] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:20:58.374 INFO 2981023 --- [Collect all tenants major compactions3,520f6c59feffd340,a7c88cc142f13508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:20:58.375 INFO 2981023 --- [Collect all tenants major compactions6,520f6c59feffd340,ff327bc138d372d3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:20:58.375 INFO 2981023 --- [Collect all tenants major compactions6,520f6c59feffd340,ff327bc138d372d3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:20:58.375 INFO 2981023 --- [Collect all tenants major compactions1,520f6c59feffd340,af2b1a578e001f33] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:20:58.375 INFO 2981023 --- [Collect all tenants major compactions1,520f6c59feffd340,af2b1a578e001f33] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:20:58.376 INFO 2981023 --- [Collect all tenants major compactions3,520f6c59feffd340,a7c88cc142f13508] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:20:58.376 INFO 2981023 --- [Collect all tenants major compactions3,520f6c59feffd340,a7c88cc142f13508] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:20:58.387 INFO 2981023 --- [Collect all tenants major compactions4,520f6c59feffd340,8b22006dfe01fd84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:58.389 INFO 2981023 --- [Collect all tenants major compactions5,520f6c59feffd340,9c26f71ca2074b10] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:20:58.390 INFO 2981023 --- [Collect all tenants major compactions4,520f6c59feffd340,8b22006dfe01fd84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:20:58.395 INFO 2981023 --- [Collect all tenants major compactions5,520f6c59feffd340,9c26f71ca2074b10] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:20:58.395 INFO 2981023 --- [Collect all tenants major compactions4,520f6c59feffd340,8b22006dfe01fd84] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:20:58.395 INFO 2981023 --- [Collect all tenants major compactions4,520f6c59feffd340,8b22006dfe01fd84] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:20:58.400 INFO 2981023 --- [Collect all tenants major compactions5,520f6c59feffd340,9c26f71ca2074b10] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:20:58.400 INFO 2981023 --- [Collect all tenants major compactions5,520f6c59feffd340,9c26f71ca2074b10] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:20:58.400 INFO 2981023 --- [ocp-schedules-16,520f6c59feffd340,021b85ddf88bfaff] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:20:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,4577fb2faff004f8,12e1db814ab0bdf8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050060, interval=60 2024-09-11 18:20:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,4577fb2faff004f8,12e1db814ab0bdf8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:20:59.443 INFO 2981023 --- [ocp-async-7,5ae2f89db7b19bfd,8b6dae72e314f678] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:20:59.719 INFO 2981023 --- [ocp-schedules-29,1f99712e8c1251c0,9444a360014c4243] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:20:59.719+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:20:59.721 INFO 2981023 --- [ocp-schedules-29,1f99712e8c1251c0,9444a360014c4243] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:20:59.773 INFO 2981023 --- [ocp-schedules-29,1f99712e8c1251c0,9444a360014c4243] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 51 to SqlAggregateData 2024-09-11 18:20:59.774 INFO 2981023 --- [ocp-schedules-29,1f99712e8c1251c0,9444a360014c4243] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:21:00.184 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:00.184 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:00.783 INFO 2981023 --- [ocp-schedules-1,d9b1c590ded78a23,0b1d8e82b2c5d506] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:21:00.784 INFO 2981023 --- [ocp-schedules-1,d9b1c590ded78a23,0b1d8e82b2c5d506] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:21:00.811 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:21:00.811 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.812 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:21:00.814 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:21:00.814 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:21:00.815 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.817 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:21:00.818 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.818 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.821 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.821 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.823 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.823 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.827 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.828 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.828 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:21:00.834 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.834 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.838 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.838 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:21:00.846 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.846 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:21:00.850 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.851 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.853 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.854 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:21:00.860 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:21:00.864 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.864 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:21:00.865 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.865 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:21:00.866 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.867 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:21:00.867 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.867 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:21:00.867 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:21:00.868 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.868 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:21:00.869 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.869 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:21:00.870 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.870 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:21:00.871 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.871 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:21:00.871 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:21:00.872 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.872 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:21:00.873 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.874 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:21:00.874 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.875 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:21:00.875 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.876 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:21:00.876 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:21:00.878 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:21:00.886 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:00.887 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.888 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.890 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:00.894 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.894 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:21:00.905 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:21:00.908 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:21:00.908 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.909 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.911 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:21:00.911 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:21:00.911 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:21:00.912 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:21:00.912 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.912 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:21:00.918 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:21:00.920 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:21:00.920 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.927 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:21:00.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:21:00.929 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:21:00.929 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:21:00.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.948 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.953 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:00.953 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:21:00.959 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5A01D700-0-0] 2024-09-11 18:21:00.959 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:21:00.966 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:00.977 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:21:01.052 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.053 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:21:01.071 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.073 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:21:01.090 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:21:01.098 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5A21D6D9-0-0] 2024-09-11 18:21:01.098 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:21:01.108 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.112 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:21:01.122 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.131 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:21:01.132 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.132 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:21:01.133 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.134 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:21:01.141 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.145 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:21:01.157 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.158 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:21:01.160 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:21:01.162 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:21:01.174 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:01.187 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.190 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:21:01.211 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:21:01.213 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:21:01.213 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:01.214 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:21:01.222 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:21:01.224 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:21:01.224 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:21:01.224 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:21:01.235 INFO 2981023 --- [ocp-schedules-1,d9b1c590ded78a23,0b1d8e82b2c5d506] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:21:01.443 INFO 2981023 --- [ocp-async-9,49ee2405e393d420,b21d20e8b5efe723] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,50c203361ad65a9e,35f45ca102d952b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,50c203361ad65a9e,35f45ca102d952b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:02.740 INFO 2981023 --- [ocp-schedules-30,f1d288668369cf3f,670d4470789bf811] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:21:03.123 INFO 2981023 --- [ocp-schedules-20,c8dcb49efcc52548,7e162f60f1082208] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:21:03.182 INFO 2981023 --- [alarm-detect-8,c8dcb49efcc52548,266f5f12b8af53b1] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050030 ~ 1726050053 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050045, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050045, value=97.065)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=97.065, values={1726050045=97.065}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050045=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:03.182 INFO 2981023 --- [alarm-detect-8,c8dcb49efcc52548,266f5f12b8af53b1] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=os_tsar_cpu_util_hwm, target=alarm_template_id=0:host=10.31.35.5-2, currentState=Pending 2024-09-11 18:21:03.185 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:03.188 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:03.209 INFO 2981023 --- [alarm-detect-2,c8dcb49efcc52548,90b14eb6486c3672] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050030 ~ 1726050053 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050045, value=97.065)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=97.065, values={1726050045=97.065}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:03.210 INFO 2981023 --- [alarm-detect-2,c8dcb49efcc52548,90b14eb6486c3672] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_cpu_percent_over_threshold, target=alarm_template_id=0:host=10.31.35.5-2, currentState=Pending 2024-09-11 18:21:03.212 INFO 2981023 --- [alarm-detect-10,c8dcb49efcc52548,005e26dd3e0da2de] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050030 ~ 1726050053 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050045, value=4.938)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=4.938, values={1726050045=4.938}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:03.212 INFO 2981023 --- [alarm-detect-10,c8dcb49efcc52548,005e26dd3e0da2de] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050030 ~ 1726050053 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050045, value=4.938)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=4.938, values={1726050045=4.938}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:21:03.213 WARN 2981023 --- [scanner-13651418-1,c8dcb49efcc52548,cc4e9aba837f6266] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:21:03.318 INFO 2981023 --- [alarm-detect-17,c8dcb49efcc52548,aee69d24330c6aeb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050030 ~ 1726050053 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050045, value=99.523)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=99.523, values={1726050045=99.523}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:03.318 INFO 2981023 --- [alarm-detect-17,c8dcb49efcc52548,aee69d24330c6aeb] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_cpu_usage_over_threshold, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:tenant_name=linkreit:host=10.31.35.5-2, currentState=Pending 2024-09-11 18:21:03.443 INFO 2981023 --- [ocp-async-3,8398ce6fd0898f6d,c038161069a26995] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,3338fb29df43bb81,1764e957759db9f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,3338fb29df43bb81,1764e957759db9f8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:05.443 INFO 2981023 --- [ocp-async-10,50550bb1256ae4da,c17fc069b7e3fa42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:06.188 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:06.190 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:06.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:21:06.353536+08:00, lastProcessTime=2024-09-11T18:20:56.337425+08:00 2024-09-11 18:21:06.497 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:21:06.353536+08:00 2024-09-11 18:21:07.443 INFO 2981023 --- [ocp-async-1,2c5060c77305906b,176a1df20cccddc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:07.791 INFO 2981023 --- [ocp-async-3,1810575b30bc2e70,57254dcd8bc8db8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:21:08.104 INFO 2981023 --- [ocp-schedules-2,68933a983c00284e,7f1df0d0a164ff30] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:21:08.178 INFO 2981023 --- [ocp-schedules-13,5012ca4a6e730cd6,46af7fbdad496a5b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:21:08.178+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:21:08.179 INFO 2981023 --- [ocp-schedules-13,5012ca4a6e730cd6,46af7fbdad496a5b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:21:08.182 INFO 2981023 --- [ocp-schedules-13,5012ca4a6e730cd6,46af7fbdad496a5b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:21:08.182 INFO 2981023 --- [ocp-schedules-13,5012ca4a6e730cd6,46af7fbdad496a5b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:21:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,9544d62f6291493f,6667112a015a1971] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:08.318 INFO 2981023 --- [prometheus-minute-schedule-1,9544d62f6291493f,6667112a015a1971] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:08.466 INFO 2981023 --- [ocp-schedules-25,1143f86d26cd16e5,4287274df3f47a83] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:21:09.190 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:09.192 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:09.443 INFO 2981023 --- [ocp-async-2,ff2f3b69c32a9350,3ae41dc2c8793245] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:10.003 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:21:10.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:21:10.394 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c53096964458acff,5129e992898a1b7c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=c53096964458acff, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:21:10.394 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,106b44771bfaaccd,674024974007e105] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=106b44771bfaaccd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:21:10.395 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,106b44771bfaaccd,674024974007e105] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=106b44771bfaaccd, duration=3 ms] 2024-09-11 18:21:10.398 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,c53096964458acff,5129e992898a1b7c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=c53096964458acff, duration=8 ms] 2024-09-11 18:21:10.406 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,d89b7ffd8fc491da,69bad3644eb5c049] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=d89b7ffd8fc491da, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:21:10.406 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,d89b7ffd8fc491da,69bad3644eb5c049] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=d89b7ffd8fc491da, duration=2 ms] 2024-09-11 18:21:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:21:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:21:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,3ee56e5c20165504,d8048016a6253c2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,3ee56e5c20165504,d8048016a6253c2c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:11.407 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:17:11.008, end=2024-09-11T18:19:11.404} 2024-09-11 18:21:11.413 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, 2024-09-11 18:21:11.414 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:21:11.417 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:11.008, end=2024-09-11T18:19:11.404},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:21:11.417 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:21:11.417 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:21:11.441 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:21:11.444 INFO 2981023 --- [ocp-async-3,87422d3c60e50e72,4b535668e26848cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:11.481 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671481000, endTimeUs=1726049951404000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=2381E12B5B29FAF856BA02292CF5632F, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T10:21:11.481Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000029, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000010, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 18:21:11.481 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671481000, endTimeUs=1726049951404000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=E9E2014C8CE705871C555597A6A32456, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T10:21:11.481Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000029, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000010, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 18:21:11.487 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:21:11.487 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.491 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 3 2024-09-11 18:21:11.491 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.499 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.499 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.503 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.503 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=2381E12B5B29FAF856BA02292CF5632F, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.504 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.504 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.506 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.506 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.514 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.514 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.516 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:21:11.516 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.481, end=2024-09-11T18:19:11.404}, clusterId=1000006, tenantId=1000006, sqlId=E9E2014C8CE705871C555597A6A32456, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.518 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.518 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=2381E12B5B29FAF856BA02292CF5632F, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.519 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.528 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.481, end=2024-09-11T18:19:11.404}, clusterId=1000006, tenantId=1000006, sqlId=2381E12B5B29FAF856BA02292CF5632F, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.529 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:21:11.530 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:11.008, end=2024-09-11T18:19:11.404},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:21:11.530 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:21:11.530 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:21:11.531 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:21:11.533 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:11.008, end=2024-09-11T18:19:11.404},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:21:11.533 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:21:11.533 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:21:11.552 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:21:11.574 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671574000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.574Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.577 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:21:11.578 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.584 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.584 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.586 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.586 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.595 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.595 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.596 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:21:11.596 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.574, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.597 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:21:11.603 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:11.008, end=2024-09-11T18:19:11.404},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:21:11.604 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:21:11.604 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:21:11.665 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671664000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.664Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.666 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671666000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.666Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.667 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671667000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=9614ED28A0A32BEDB356D27DF4FFA3D8, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.667Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.670 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671670000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.670Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.675 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671675000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.675Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.678 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671678000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.678Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.681 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671681000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.681Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.681 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671681000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.681Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.719 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:21:11.719 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.721 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 18 2024-09-11 18:21:11.721 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.727 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:21:11.727 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.728 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:21:11.728 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.732 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 69 2024-09-11 18:21:11.732 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.733 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:21:11.733 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.741 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:21:11.741 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.744 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.744 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.745 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.745 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9614ED28A0A32BEDB356D27DF4FFA3D8, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.747 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:21:11.747 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.751 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.751 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.755 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.755 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.756 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.756 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.756 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.756 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.759 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.759 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.759 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.759 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.763 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.763 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.764 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.764 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.764 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.764 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.766 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.766 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.769 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.769 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.772 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.772 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.773 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.773 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.775 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.775 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.775 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.775 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.776 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.776 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9614ED28A0A32BEDB356D27DF4FFA3D8, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.777 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.778 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.778 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.778 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.779 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.782 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.782 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.783 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.783 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.784 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.784 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.788 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.791 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.791 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.793 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.664, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.793 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.793 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.794 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.794 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.798 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.667, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=9614ED28A0A32BEDB356D27DF4FFA3D8, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.799 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.800 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.666, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.800 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.802 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.67, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.802 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.675, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.807 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.678, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.808 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.681, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.808 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.681, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.839 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671839000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.839Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.844 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671844000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.844Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.846 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671846000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.846Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.856 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:21:11.856 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.857 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671857000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.857Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.858 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671858000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.858Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.859 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:21:11.859 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.860 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:21:11.860 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.864 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671864000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.864Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.869 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:21:11.869 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.870 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963671870000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:21:11.870Z, endTime=2024-09-11T10:19:11.404Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:21:11.875 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:21:11.875 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.881 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.881 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.881 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.881 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.887 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.887 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.888 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:21:11.888 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.889 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.889 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.889 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.889 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:21:11.889 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:21:11.889 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.890 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.890 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.891 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.891 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.898 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.898 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.902 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.902 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.903 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.903 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.906 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.906 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.906 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.906 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.907 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.907 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.907 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.907 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.908 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.909 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.909 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.910 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.910 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.910 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.916 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:21:11.916 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:21:11.917 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.917 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.846, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.920 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.920 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.922 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.922 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.844, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.923 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.923 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.923 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.923 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.924 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.839, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.926 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.927 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.930 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:21:11.930 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726049951404000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:21:11.930 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.858, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.933 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:21:11.934 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.87, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.934 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.857, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.937 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:21:11.864, end=2024-09-11T18:19:11.404}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:21:11.937 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:21:11.938 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:11.008, end=2024-09-11T18:19:11.404},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:21:11.938 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:21:11.938 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:21:11.940 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:21:11.941 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:11.008, end=2024-09-11T18:19:11.404},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:21:11.941 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:21:11.941 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:21:11.943 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:21:12.192 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:12.192 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:12.351 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050000, nodeCount=480, affectRows=480 2024-09-11 18:21:12.743 INFO 2981023 --- [ocp-schedules-26,e50fb35807bfd9eb,0e8b7b62f0ecd920] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:21:13.332 INFO 2981023 --- [ocp-schedules-15,37d2ac12fb3ce319,f12edb60f0d195d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:21:13.376 INFO 2981023 --- [alarm-detect-8,37d2ac12fb3ce319,bd41a396be7dc7b4] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050045 ~ 1726050063 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050060, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050060, value=97.592)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=97.592, values={1726050060=97.592}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050060=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:13.391 WARN 2981023 --- [scanner-13653153-1,37d2ac12fb3ce319,fa2019a477bdba71] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:21:13.401 INFO 2981023 --- [alarm-detect-7,37d2ac12fb3ce319,dd55f59515774177] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8645998, data is empty. 2024-09-11 18:21:13.408 INFO 2981023 --- [alarm-detect-7,37d2ac12fb3ce319,dd55f59515774177] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646049, data is empty. 2024-09-11 18:21:13.414 INFO 2981023 --- [alarm-detect-7,37d2ac12fb3ce319,dd55f59515774177] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646049, data is empty. 2024-09-11 18:21:13.440 INFO 2981023 --- [alarm-detect-10,37d2ac12fb3ce319,381e58e440878d7a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8645998, data is empty. 2024-09-11 18:21:13.444 INFO 2981023 --- [ocp-async-7,f1e0a9a8df6be38a,69de1d4b497b051b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:13.447 INFO 2981023 --- [alarm-detect-8,37d2ac12fb3ce319,54f28edf2553ac34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8645998, data is empty. 2024-09-11 18:21:13.461 INFO 2981023 --- [alarm-detect-8,37d2ac12fb3ce319,54f28edf2553ac34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646049, data is empty. 2024-09-11 18:21:13.465 INFO 2981023 --- [alarm-detect-8,37d2ac12fb3ce319,54f28edf2553ac34] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646049, data is empty. 2024-09-11 18:21:13.470 INFO 2981023 --- [alarm-detect-10,37d2ac12fb3ce319,381e58e440878d7a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646049, data is empty. 2024-09-11 18:21:13.485 INFO 2981023 --- [alarm-detect-10,37d2ac12fb3ce319,381e58e440878d7a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646049, data is empty. 2024-09-11 18:21:13.486 INFO 2981023 --- [alarm-detect-11,37d2ac12fb3ce319,fef0a1d595385694] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050045 ~ 1726050063 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050060, value=97.592)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=97.592, values={1726050060=97.592}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:13.607 INFO 2981023 --- [alarm-detect-17,37d2ac12fb3ce319,e61d4fd503211482] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8645998, data is empty. 2024-09-11 18:21:13.614 INFO 2981023 --- [alarm-detect-17,37d2ac12fb3ce319,e61d4fd503211482] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646049, data is empty. 2024-09-11 18:21:13.621 INFO 2981023 --- [alarm-detect-17,37d2ac12fb3ce319,e61d4fd503211482] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646049, data is empty. 2024-09-11 18:21:13.785 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050000, nodeCount=1000, affectRows=1000 2024-09-11 18:21:13.787 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050000, nodeCount=999, validCount=999 2024-09-11 18:21:13.797 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050000, nodeCount=48, affectRows=48 2024-09-11 18:21:13.797 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050000, nodeCount=46, validCount=46 2024-09-11 18:21:13.798 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050000, nodeCount=1000, affectRows=1000 2024-09-11 18:21:13.800 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050000, nodeCount=1000, validCount=1000 2024-09-11 18:21:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,560b314b0e9edbe3,cc148112c8e6c7a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,fabf0f90c76b1be7,9268fcd2885bc604] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:21:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,560b314b0e9edbe3,cc148112c8e6c7a9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:14.896 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050000, nodeCount=480, validCount=480 2024-09-11 18:21:15.193 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:15.193 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:15.444 INFO 2981023 --- [ocp-async-9,454f9270eacfe8a0,1e7acf02fe91d54e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:15.604 INFO 2981023 --- [ocp-schedules-23,eb19c0a869bd78cb,bbf7b6d1d3191bc1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:21:15.613 INFO 2981023 --- [ocp-schedules-23,eb19c0a869bd78cb,bbf7b6d1d3191bc1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:21:15.613 INFO 2981023 --- [ocp-schedules-23,eb19c0a869bd78cb,bbf7b6d1d3191bc1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:21:16.018 INFO 2981023 --- [ocp-schedules-15,c5659c657f7e0077,c94d28d3a95c4070] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:21:16.018 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:21:16.018 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:21:16.019 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:21:16.019 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:21:16.025 INFO 2981023 --- [ocp-schedules-12,61c586cb9adcdf50,9dca3957853f8084] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:21:16.026 INFO 2981023 --- [ocp-schedules-20,0038489b6a145357,e6305ae756be0e58] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:21:16.040 INFO 2981023 --- [host-check-clockdiff1,61c586cb9adcdf50,a11dd37d57d7fa99] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:21:16.040 INFO 2981023 --- [host-check-clockdiff2,61c586cb9adcdf50,8701a80d0c5608a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:21:16.041 INFO 2981023 --- [host-check-clockdiff4,61c586cb9adcdf50,911997a1f80e91fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:21:16.042 INFO 2981023 --- [host-check-clockdiff3,61c586cb9adcdf50,0eb90a50e21433ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:21:16.046 INFO 2981023 --- [host-check-agent-status1,0038489b6a145357,55c405e2fd9f8b70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:21:16.046 INFO 2981023 --- [host-check-agent-status2,0038489b6a145357,f9015b6591920ad0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:21:16.047 INFO 2981023 --- [host-check-agent-status3,0038489b6a145357,8bbce32ef6a043a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:21:16.047 INFO 2981023 --- [host-check-agent-status4,0038489b6a145357,bb3bc9c3887cb2ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:21:16.075 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:21:16.075 INFO 2981023 --- [host-check-agent-status2,0038489b6a145357,f9015b6591920ad0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:21:16.080 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:21:16.080 INFO 2981023 --- [host-check-agent-status4,0038489b6a145357,bb3bc9c3887cb2ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:21:16.083 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:21:16.083 INFO 2981023 --- [host-check-agent-status4,0038489b6a145357,bb3bc9c3887cb2ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:21:16.084 INFO 2981023 --- [host-check-agent-status2,0038489b6a145357,f9015b6591920ad0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:21:16.094 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:21:16.095 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:21:16.096 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:21:16.096 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:21:16.099 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:21:16.100 INFO 2981023 --- [host-check-clockdiff2,61c586cb9adcdf50,8701a80d0c5608a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050076 5 5 , error= 2024-09-11 18:21:16.100 INFO 2981023 --- [host-check-clockdiff3,61c586cb9adcdf50,0eb90a50e21433ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050076 0 0 , error= 2024-09-11 18:21:16.131 INFO 2981023 --- [host-check-agent-status3,0038489b6a145357,8bbce32ef6a043a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:21:16.166 INFO 2981023 --- [host-check-agent-status1,0038489b6a145357,55c405e2fd9f8b70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:21:16.168 INFO 2981023 --- [host-check-agent-status3,0038489b6a145357,8bbce32ef6a043a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:21:16.190 INFO 2981023 --- [ocp-schedules-1,6bceac23745943f8,1ba15c6c21dc8fbd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:21:16.190 INFO 2981023 --- [ocp-schedules-24,1844555ce6107981,dafde065baac6954] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:21:16.190 INFO 2981023 --- [ocp-schedules-26,61da8ae385a2e38b,c28d575d5929f728] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:21:16.190 INFO 2981023 --- [ocp-schedules-13,b18a7cf07a3df6ac,9643af427a3ed4aa] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:21:16.208 INFO 2981023 --- [host-check-agent-status1,0038489b6a145357,55c405e2fd9f8b70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:21:16.264 INFO 2981023 --- [ocp-schedules-24,1844555ce6107981,dafde065baac6954] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:21:16.265 INFO 2981023 --- [ocp-schedules-26,61da8ae385a2e38b,c28d575d5929f728] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:21:16.280 INFO 2981023 --- [ocp-schedules-1,6bceac23745943f8,1ba15c6c21dc8fbd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:21:16.303 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050003098000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:21:16.319 INFO 2981023 --- [ocp-schedules-24,1844555ce6107981,dafde065baac6954] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:21:16.365 INFO 2981023 --- [ocp-schedules-21,70aedce35fe11d93,f9e77a3855fe8070] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:21:16.388 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:21:16.391 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:21:16.391357630+08:00 2024-09-11 18:21:16.393 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:21:16.391+08:00 2024-09-11 18:21:16.395 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:21:16.391+08:00 and status AVAILABLE 2024-09-11 18:21:16.395 INFO 2981023 --- [host-check-agent-status2,0038489b6a145357,f9015b6591920ad0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:21:16.398 INFO 2981023 --- [host-check-agent-status3,0038489b6a145357,8bbce32ef6a043a8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:21:16.400 INFO 2981023 --- [host-check-agent-status2,0038489b6a145357,f9015b6591920ad0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:21:16.401 INFO 2981023 --- [host-check-agent-status2,0038489b6a145357,f9015b6591920ad0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:21:16.401 INFO 2981023 --- [host-check-agent-status2,0038489b6a145357,f9015b6591920ad0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:21:16.402 INFO 2981023 --- [host-check-agent-status4,0038489b6a145357,bb3bc9c3887cb2ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:21:16.404 INFO 2981023 --- [host-check-agent-status3,0038489b6a145357,8bbce32ef6a043a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:21:16.404 INFO 2981023 --- [host-check-agent-status3,0038489b6a145357,8bbce32ef6a043a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:21:16.404 INFO 2981023 --- [host-check-agent-status3,0038489b6a145357,8bbce32ef6a043a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:21:16.405 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:21:16.405 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:21:16.410881687+08:00 2024-09-11 18:21:16.407 INFO 2981023 --- [host-check-agent-status4,0038489b6a145357,bb3bc9c3887cb2ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:21:16.407 INFO 2981023 --- [host-check-agent-status4,0038489b6a145357,bb3bc9c3887cb2ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:21:16.407 INFO 2981023 --- [host-check-agent-status4,0038489b6a145357,bb3bc9c3887cb2ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:21:16.455 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:21:16.405+08:00 2024-09-11 18:21:16.456 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:21:16.456 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:21:16.456662+08:00, lastProcessTime=2024-09-11T18:21:06.353536+08:00 2024-09-11 18:21:16.458 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:21:16.459 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:21:16.405+08:00 and status AVAILABLE 2024-09-11 18:21:16.459 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:21:16.459317557+08:00 2024-09-11 18:21:16.461 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:21:16.459+08:00 2024-09-11 18:21:16.468 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:21:16.468 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:21:16.459+08:00 and status AVAILABLE 2024-09-11 18:21:16.469 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:21:16.468843351+08:00 2024-09-11 18:21:16.469 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:21:16.469+08:00 2024-09-11 18:21:16.473 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:21:16.469+08:00 and status AVAILABLE 2024-09-11 18:21:16.476 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:21:16.479 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.479 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:21:16.481 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:21:16.481 INFO 2981023 --- [host-check-agent-status1,0038489b6a145357,55c405e2fd9f8b70] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:21:16.481 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.481 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.487 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:16.488 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:21:16.489 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:21:16.489 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.490 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:21:16.490 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:16.498 INFO 2981023 --- [host-check-agent-status1,0038489b6a145357,55c405e2fd9f8b70] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:21:16.498 INFO 2981023 --- [host-check-agent-status1,0038489b6a145357,55c405e2fd9f8b70] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:21:16.498 INFO 2981023 --- [host-check-agent-status1,0038489b6a145357,55c405e2fd9f8b70] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:21:16.498 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:21:16.498 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:21:16.498 INFO 2981023 --- [ocp-schedules-1,6bceac23745943f8,1ba15c6c21dc8fbd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:21:16.499 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:21:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.501 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:16.509 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:21:16.509 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:21:16.510 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:21:16.510 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:21:16.510 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:21:16.512 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:21:16.520 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:21:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:16.569 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.569 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:21:16.571 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:21:16.572 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:21:16.574 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:21:16.578 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:21:16.578 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050003098000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:21:16.593 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:21:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:16.604 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:21:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.607 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:21:16.608 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:21:16.612 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:21:16.618 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:21:16.619 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:21:16.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:21:16.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:21:16.626 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:21:16.641 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:21:16.646 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:21:16.656 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:16.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:21:16.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:21:16.662 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:21:16.662 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:21:16.665 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:21:16.666 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:21:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:21:16.674 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:16.679 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:21:16.683 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:21:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.686 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:21:16.693 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:21:16.695 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:21:16.702 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:21:16.719 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:21:16.720 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:21:16.721 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:21:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:16.732 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:21:16.732 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.732 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:16.733 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:21:16.733 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.737 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:21:16.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:21:16.741 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:21:16.753 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:21:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:21:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:21:16.758 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:21:16.769 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:21:16.773 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:21:16.775 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:21:16.780 INFO 2981023 --- [host-check-host-status4,c5659c657f7e0077,c859d5439abb2740] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:21:16.783 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:21:16.783 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:16.784 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.784 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:21:16.792 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:16.793 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:16.793 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:21:16.794 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:21:16.794 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:21:16.794 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:21:16.796 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:21:16.797 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:21:16.798 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:21:16.801 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:21:16.804 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:21:16.809 INFO 2981023 --- [host-check-host-status2,c5659c657f7e0077,4c7aebe3b2418457] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:21:16.813 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:21:16.456662+08:00 2024-09-11 18:21:16.824 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:21:16.826 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:21:16.832 INFO 2981023 --- [host-check-host-status3,c5659c657f7e0077,5bbdf08d4371b7e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:21:17.020 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:21:17.026 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:21:17.042 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:21:17.114 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:21:17.121 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:17.133 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:17.140 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:21:17.149 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:21:17.149 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:21:17.150 INFO 2981023 --- [ocp-schedules-26,61da8ae385a2e38b,c28d575d5929f728] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:21:17.256 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:21:17.262 INFO 2981023 --- [host-check-host-status1,c5659c657f7e0077,8b313d3a5033245a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:21:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,7fa47ff29a4f948b,343e56b26debc35b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:17.315 INFO 2981023 --- [prometheus-minute-schedule-1,7fa47ff29a4f948b,343e56b26debc35b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:17.444 INFO 2981023 --- [ocp-async-2,57bf1c11b88dfc63,a9127e978cf33c31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:17.791 INFO 2981023 --- [ocp-async-7,edb70559e82f154d,48a30840e830e961] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:21:18.193 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:18.194 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:18.586 INFO 2981023 --- [host-check-clockdiff1,61c586cb9adcdf50,a11dd37d57d7fa99] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050078 0 0 , error= 2024-09-11 18:21:19.444 INFO 2981023 --- [ocp-async-1,18f85db84dde4ca1,25f7fe2ad48504f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,440b988f81caa01a,cc05ef4cc3864712] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,440b988f81caa01a,cc05ef4cc3864712] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:21.194 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:21.195 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:21.444 INFO 2981023 --- [ocp-async-7,dc45699117179ce0,5de094d10f7549b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:22.043 INFO 2981023 --- [host-check-clockdiff4,61c586cb9adcdf50,911997a1f80e91fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050082 0 0 , error= 2024-09-11 18:21:22.745 INFO 2981023 --- [ocp-schedules-16,977b2bb4db1e52a3,2bcb110107757455] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:21:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,640572c3921be771,19e7ae5c5e7bc334] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,640572c3921be771,19e7ae5c5e7bc334] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:23.444 INFO 2981023 --- [ocp-async-10,e096a7dbc95700c6,a3435d2e44df7658] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:23.667 INFO 2981023 --- [ocp-schedules-10,7b5aa00461f7497e,5014e2281f42a374] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:21:23.710 WARN 2981023 --- [scanner-13653436-1,7b5aa00461f7497e,446d0522ed7888d0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:21:23.806 INFO 2981023 --- [alarm-detect-19,7b5aa00461f7497e,8ccce3b7dafa6a42] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050045 ~ 1726050073 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050060, value=5.773)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=5.773, values={1726050060=5.773}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:23.806 INFO 2981023 --- [alarm-detect-19,7b5aa00461f7497e,8ccce3b7dafa6a42] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050045 ~ 1726050073 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050060, value=5.773)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=5.773, values={1726050060=5.773}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:21:23.807 INFO 2981023 --- [alarm-detect-19,7b5aa00461f7497e,8ccce3b7dafa6a42] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-3, currentState=Firing 2024-09-11 18:21:23.807 INFO 2981023 --- [alarm-detect-19,7b5aa00461f7497e,8ccce3b7dafa6a42] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:21:23.849 INFO 2981023 --- [alarm-detect-19,7b5aa00461f7497e,8ccce3b7dafa6a42] c.o.o.alarm.service.OcpAlarmServiceImpl : delete alarm event 6007109 2024-09-11 18:21:23.852 INFO 2981023 --- [alarm-detect-19,7b5aa00461f7497e,8ccce3b7dafa6a42] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007110 2024-09-11 18:21:23.875 INFO 2981023 --- [alarm-detect-20,7b5aa00461f7497e,6738557fb876973b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050045 ~ 1726050073 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050060, value=105.357)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.357, values={1726050060=105.357}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:24.195 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:24.196 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:24.466 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,423ff7156b5197d8,43f23ecfdc00245d] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&version=2&ObClusterId=1719994443, client=10.31.35.6, traceId=423ff7156b5197d8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkrtd,1719994443,2,{"ObRegion":"linkrtd","ObCluster":"linkrtd","ObRegionId":1719994443,"ObClusterId":1719994443,"Type":"PRIMARY","timestamp":1726050084469890,"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:21:24.474 INFO 2981023 --- [ocp-schedules-19,1df08016474f1af3,e4d376ca09ee03fd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:21:24.475 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,423ff7156b5197d8,43f23ecfdc00245d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkrtd, obClusterId=1719994443, rootServerJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050084466000}, groupName=linkrtd, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:21:24.475 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,423ff7156b5197d8,43f23ecfdc00245d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:21:24.475 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,423ff7156b5197d8,43f23ecfdc00245d] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkrtd, clusterId=1719994443, affectRows=1 2024-09-11 18:21:24.476 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,423ff7156b5197d8,43f23ecfdc00245d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.6, traceId=423ff7156b5197d8, duration=11 ms] 2024-09-11 18:21:25.445 INFO 2981023 --- [ocp-async-3,55f926519e1df51a,638e671ba947fe18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,df3e5d12fbaaa150,54143b77fe3bc198] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,df3e5d12fbaaa150,54143b77fe3bc198] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:21:26.352147+08:00, lastProcessTime=2024-09-11T18:21:16.456662+08:00 2024-09-11 18:21:26.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:21:26.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:21:26.352147+08:00 2024-09-11 18:21:27.145 INFO 2981023 --- [ocp-schedules-25,3cb6d67f1a6116a8,ac78100d8b775583] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:21:27.145 INFO 2981023 --- [ocp-schedules-25,3cb6d67f1a6116a8,ac78100d8b775583] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:21:27.145 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:21:27.146 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:21:27.146 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:21:27.167 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:21:27.168 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:21:27.171 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:21:27.172 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:21:27.172 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:21:27.173 INFO 2981023 --- [ocp-schedules-25,3cb6d67f1a6116a8,ac78100d8b775583] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:21:27.173 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:21:27.176 INFO 2981023 --- [ocp-schedules-25,3cb6d67f1a6116a8,ac78100d8b775583] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:21:27.196 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:27.198 INFO 2981023 --- [ocp-schedules-25,3cb6d67f1a6116a8,ac78100d8b775583] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:21:27.200 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:21:27.201 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:27.201 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:21:27.203 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:21:27.204 INFO 2981023 --- [ocp-schedules-25,3cb6d67f1a6116a8,ac78100d8b775583] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:21:27.445 INFO 2981023 --- [ocp-async-1,2385e8a0614d7979,2445895be178b837] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:27.791 INFO 2981023 --- [ocp-async-5,ba2c536dc75e0d20,1e5818f9989d6d4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:21:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,4ca08abbfa0a2f3e,22e4c9dbd7d93d71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,4ca08abbfa0a2f3e,22e4c9dbd7d93d71] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:29.445 INFO 2981023 --- [ocp-async-2,743b69c1f110fac2,17cd5b8296cf11be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:30.201 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:30.201 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:31.445 INFO 2981023 --- [ocp-async-5,2a6322278274503e,22410580d6877742] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,1f923575a5559cac,a39c287221cd4a8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,1f923575a5559cac,a39c287221cd4a8a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:32.747 INFO 2981023 --- [ocp-schedules-4,ada3205506b222cd,ace394fdd9186eea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:21:33.202 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:33.202 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:33.445 INFO 2981023 --- [ocp-async-8,fd89ed796f9aaa43,60a8414cfc107468] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:33.902 INFO 2981023 --- [ocp-schedules-30,881eab8ad8471d33,5deb379b31ec0542] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:21:33.918 WARN 2981023 --- [scanner-13653633-1,881eab8ad8471d33,5900c3c2300f0e21] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:21:33.965 INFO 2981023 --- [alarm-detect-4,881eab8ad8471d33,dfc54f3694a5aa12] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050060 ~ 1726050083 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050075, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050075, value=98.508)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.508, values={1726050075=98.508}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050075=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:33.966 INFO 2981023 --- [alarm-detect-4,881eab8ad8471d33,dfc54f3694a5aa12] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=os_tsar_cpu_util_hwm, target=alarm_template_id=0:host=10.31.35.5-2, currentState=Firing 2024-09-11 18:21:33.966 INFO 2981023 --- [alarm-detect-4,881eab8ad8471d33,dfc54f3694a5aa12] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=os_tsar_cpu_util_hwm, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:21:33.977 INFO 2981023 --- [alarm-detect-1,881eab8ad8471d33,24e57a4988bb369f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646049, data is empty. 2024-09-11 18:21:33.984 INFO 2981023 --- [alarm-detect-1,881eab8ad8471d33,24e57a4988bb369f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646049, data is empty. 2024-09-11 18:21:33.987 INFO 2981023 --- [alarm-detect-1,881eab8ad8471d33,24e57a4988bb369f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646049, data is empty. 2024-09-11 18:21:33.987 INFO 2981023 --- [alarm-detect-4,881eab8ad8471d33,dfc54f3694a5aa12] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007111 2024-09-11 18:21:34.008 INFO 2981023 --- [alarm-detect-14,881eab8ad8471d33,86fdbbcec488b99a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646049, data is empty. 2024-09-11 18:21:34.009 INFO 2981023 --- [alarm-detect-2,881eab8ad8471d33,8be2c8ea756ec99c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646049, data is empty. 2024-09-11 18:21:34.016 INFO 2981023 --- [alarm-detect-2,881eab8ad8471d33,8be2c8ea756ec99c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646049, data is empty. 2024-09-11 18:21:34.016 INFO 2981023 --- [alarm-detect-14,881eab8ad8471d33,86fdbbcec488b99a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646049, data is empty. 2024-09-11 18:21:34.021 INFO 2981023 --- [alarm-detect-16,881eab8ad8471d33,da140e88624ceca4] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050060 ~ 1726050084 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050075, value=98.508)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.508, values={1726050075=98.508}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:34.021 INFO 2981023 --- [alarm-detect-16,881eab8ad8471d33,da140e88624ceca4] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=os_tsar_cpu_util_full, target=alarm_template_id=0:host=10.31.35.5-2, currentState=Pending 2024-09-11 18:21:34.021 INFO 2981023 --- [alarm-detect-2,881eab8ad8471d33,8be2c8ea756ec99c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646049, data is empty. 2024-09-11 18:21:34.024 INFO 2981023 --- [alarm-detect-14,881eab8ad8471d33,86fdbbcec488b99a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646049, data is empty. 2024-09-11 18:21:34.058 INFO 2981023 --- [alarm-detect-2,881eab8ad8471d33,954b35488655fc89] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050060 ~ 1726050084 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050075, value=5.596)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=5.596, values={1726050075=5.596}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:34.058 INFO 2981023 --- [alarm-detect-2,881eab8ad8471d33,954b35488655fc89] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050060 ~ 1726050084 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050075, value=5.596)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=5.596, values={1726050075=5.596}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:21:34.066 INFO 2981023 --- [alarm-detect-18,881eab8ad8471d33,c02bac262ae8c2cd] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050060 ~ 1726050084 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050075, value=98.508)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.508, values={1726050075=98.508}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:34.136 INFO 2981023 --- [alarm-detect-10,881eab8ad8471d33,f070339d104b92aa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646049, data is empty. 2024-09-11 18:21:34.141 INFO 2981023 --- [alarm-detect-10,881eab8ad8471d33,f070339d104b92aa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646049, data is empty. 2024-09-11 18:21:34.144 INFO 2981023 --- [alarm-detect-10,881eab8ad8471d33,f070339d104b92aa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646049, data is empty. 2024-09-11 18:21:34.189 INFO 2981023 --- [alarm-detect-17,881eab8ad8471d33,693bcaf80dd88b1c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050060 ~ 1726050084 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050075, value=104.576)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.576, values={1726050075=104.576}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,6905a938a6756793,c78a2df9b15f656c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,6905a938a6756793,c78a2df9b15f656c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:35.445 INFO 2981023 --- [ocp-async-9,fc15dc3556976dde,341b9696acf115c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:35.616 INFO 2981023 --- [ocp-schedules-22,a0275199a65f0ca2,be6cbbc2f96ce8f0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:21:35.625 INFO 2981023 --- [ocp-schedules-22,a0275199a65f0ca2,be6cbbc2f96ce8f0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:21:35.625 INFO 2981023 --- [ocp-schedules-22,a0275199a65f0ca2,be6cbbc2f96ce8f0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:21:36.202 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:36.203 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:21:36.337926+08:00, lastProcessTime=2024-09-11T18:21:26.352147+08:00 2024-09-11 18:21:36.477 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:21:36.481 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:21:36.337926+08:00 2024-09-11 18:21:37.314 INFO 2981023 --- [prometheus-cache-monitor1,7a8b0f0e2f79e275,29fac64a221b520d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:21:37.315 INFO 2981023 --- [prometheus-cache-monitor1,7a8b0f0e2f79e275,29fac64a221b520d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:21:37.445 INFO 2981023 --- [ocp-async-6,77b6c4b94ba303d9,05a72fd6401cc43d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:37.791 INFO 2981023 --- [ocp-async-8,4e11a99d6b1aad2e,28628664104ea8be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:21:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,43f51fc5be648304,cb1b059b3e4a22a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,43f51fc5be648304,cb1b059b3e4a22a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:39.203 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:39.204 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:39.445 INFO 2981023 --- [ocp-async-1,cdfeca6b0534f85a,f3925a4dcfd0d226] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:40.004 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:21:40.004 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:21:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,83495ecdc011379b,23ee4318126dd997] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,83495ecdc011379b,23ee4318126dd997] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:41.445 INFO 2981023 --- [ocp-async-10,09591e9eda7b412d,91634209ba434aa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:42.204 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:42.204 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:42.750 INFO 2981023 --- [ocp-schedules-24,a95782ba168499a1,d1945e4dbb901aad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:21:43.446 INFO 2981023 --- [ocp-async-3,689f88cc9edf6a07,072c36ae011cf345] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:43.792 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:21:43.792 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:21:43.794 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:21:43.794 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:21:43.794 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:21:43.795 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:21:43.797 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:21:43.797 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:21:43.797 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:21:43.797 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:21:43.819 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:21:43.823 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:21:43.823 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:21:43.823 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:21:43.823 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:21:43.823 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:21:43.827 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:21:44.063 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:21:44.197 INFO 2981023 --- [ocp-schedules-3,8b9a53e7f2419379,0d44970e29c4bd77] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:21:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,8f54401b973d327e,7a73e0f1305154d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,8f54401b973d327e,7a73e0f1305154d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:44.337 WARN 2981023 --- [scanner-13653999-1,8b9a53e7f2419379,9f6572e226d79fbd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:21:44.363 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.56569468S 2024-09-11 18:21:44.363 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:21:44.424 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:21:44.424 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:21:44.459 INFO 2981023 --- [alarm-detect-13,8b9a53e7f2419379,d5f14950a9ed7e70] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050075 ~ 1726050094 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050090, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050090, value=97.498)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=97.498, values={1726050090=97.498}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050090=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:44.466 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:21:44.597 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:21:44.627 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:21:44.627 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:21:44.627 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:21:44.627 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:21:44.627 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:21:44.634 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:21:44.719 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] o.s.boot.SpringApplication : Started application in 0.924 seconds (JVM running for 85904.592) 2024-09-11 18:21:44.719 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.925047373S 2024-09-11 18:21:44.925 INFO 2981023 --- [ocp-schedules-16,2b3411fec2e8d481,f496e9c3cc641fd6] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:21:45.107 INFO 2981023 --- [prometheus-second-schedule-1,effa260d84042891,5ef0cd57d5dacec7] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:21:45.205 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:45.205 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:45.446 INFO 2981023 --- [ocp-async-4,e3f0aacdf4c6a0e8,730d35f3f9e62efa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:46.021 INFO 2981023 --- [ocp-schedules-3,0f541b360c82f5cd,6956c2144632041a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:21:46.021 INFO 2981023 --- [ocp-schedules-17,0b5fd55554758f15,297d480d147d690a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:21:46.021 INFO 2981023 --- [host-check-agent-status1,0f541b360c82f5cd,7969ac366f668179] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:21:46.022 INFO 2981023 --- [host-check-agent-status2,0f541b360c82f5cd,f519da266c18f2c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:21:46.022 INFO 2981023 --- [host-check-agent-status3,0f541b360c82f5cd,ef21e0bafbf4cb42] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:21:46.022 INFO 2981023 --- [host-check-clockdiff1,0b5fd55554758f15,3662c89a0d32e666] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:21:46.022 INFO 2981023 --- [host-check-clockdiff2,0b5fd55554758f15,0b5d044ea62d5fb5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:21:46.022 INFO 2981023 --- [host-check-clockdiff3,0b5fd55554758f15,0dc045a639ac4a8c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:21:46.024 INFO 2981023 --- [host-check-clockdiff4,0b5fd55554758f15,8d63989197002dd7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:21:46.028 INFO 2981023 --- [host-check-agent-status4,0f541b360c82f5cd,fcaf8ac8c17b8e12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:21:46.028 INFO 2981023 --- [ocp-schedules-23,c33c3cc2aa155398,62a27149a124facf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:21:46.029 INFO 2981023 --- [host-check-agent-status3,0f541b360c82f5cd,ef21e0bafbf4cb42] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:21:46.030 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:21:46.030 INFO 2981023 --- [host-check-agent-status3,0f541b360c82f5cd,ef21e0bafbf4cb42] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:21:46.030 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:21:46.030 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:21:46.034 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:21:46.034 INFO 2981023 --- [host-check-agent-status2,0f541b360c82f5cd,f519da266c18f2c4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:21:46.035 INFO 2981023 --- [host-check-agent-status1,0f541b360c82f5cd,7969ac366f668179] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:21:46.036 INFO 2981023 --- [host-check-agent-status1,0f541b360c82f5cd,7969ac366f668179] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:21:46.037 INFO 2981023 --- [host-check-agent-status2,0f541b360c82f5cd,f519da266c18f2c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:21:46.039 INFO 2981023 --- [host-check-agent-status4,0f541b360c82f5cd,fcaf8ac8c17b8e12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:21:46.041 INFO 2981023 --- [host-check-agent-status4,0f541b360c82f5cd,fcaf8ac8c17b8e12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:21:46.041 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:21:46.041 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:21:46.042 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:21:46.043 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:21:46.046 INFO 2981023 --- [host-check-clockdiff3,0b5fd55554758f15,0dc045a639ac4a8c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050106 0 0 , error= 2024-09-11 18:21:46.046 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:21:46.048 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:21:46.048 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:21:46.049 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:21:46.071 INFO 2981023 --- [host-check-agent-status1,0f541b360c82f5cd,7969ac366f668179] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:21:46.071 INFO 2981023 --- [host-check-agent-status2,0f541b360c82f5cd,f519da266c18f2c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:21:46.074 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:21:46.075 INFO 2981023 --- [host-check-agent-status2,0f541b360c82f5cd,f519da266c18f2c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:21:46.075 INFO 2981023 --- [host-check-agent-status2,0f541b360c82f5cd,f519da266c18f2c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:21:46.075 INFO 2981023 --- [host-check-agent-status2,0f541b360c82f5cd,f519da266c18f2c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:21:46.076 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:21:46.076 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:21:46.081657588+08:00 2024-09-11 18:21:46.077 INFO 2981023 --- [host-check-agent-status3,0f541b360c82f5cd,ef21e0bafbf4cb42] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:21:46.077 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:21:46.077042250+08:00 2024-09-11 18:21:46.078 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:21:46.078 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:21:46.076+08:00 2024-09-11 18:21:46.078 INFO 2981023 --- [host-check-agent-status1,0f541b360c82f5cd,7969ac366f668179] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:21:46.078 INFO 2981023 --- [host-check-agent-status1,0f541b360c82f5cd,7969ac366f668179] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:21:46.078 INFO 2981023 --- [host-check-agent-status1,0f541b360c82f5cd,7969ac366f668179] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:21:46.078 INFO 2981023 --- [host-check-agent-status4,0f541b360c82f5cd,fcaf8ac8c17b8e12] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:21:46.079 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:21:46.077+08:00 2024-09-11 18:21:46.080 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:21:46.079989474+08:00 2024-09-11 18:21:46.081 INFO 2981023 --- [host-check-agent-status3,0f541b360c82f5cd,ef21e0bafbf4cb42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:21:46.081 INFO 2981023 --- [host-check-agent-status3,0f541b360c82f5cd,ef21e0bafbf4cb42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:21:46.081 INFO 2981023 --- [host-check-agent-status3,0f541b360c82f5cd,ef21e0bafbf4cb42] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:21:46.082 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:21:46.080+08:00 2024-09-11 18:21:46.084 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:21:46.084 INFO 2981023 --- [host-check-agent-status4,0f541b360c82f5cd,fcaf8ac8c17b8e12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:21:46.084 INFO 2981023 --- [host-check-agent-status4,0f541b360c82f5cd,fcaf8ac8c17b8e12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:21:46.084 INFO 2981023 --- [host-check-agent-status4,0f541b360c82f5cd,fcaf8ac8c17b8e12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:21:46.085 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:21:46.084437518+08:00 2024-09-11 18:21:46.085 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:21:46.076+08:00 and status AVAILABLE 2024-09-11 18:21:46.085 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:21:46.077+08:00 and status AVAILABLE 2024-09-11 18:21:46.091 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:21:46.085+08:00 2024-09-11 18:21:46.093 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:21:46.085+08:00 and status AVAILABLE 2024-09-11 18:21:46.107 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:21:46.080+08:00 and status AVAILABLE 2024-09-11 18:21:46.111 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:21:46.134 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:21:46.135 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:21:46.136 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:21:46.139 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:21:46.141 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:21:46.144 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:21:46.146 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:21:46.155 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:21:46.160 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:21:46.163 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:21:46.163 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:21:46.167 INFO 2981023 --- [ocp-schedules-1,ff80064e3fb85d80,ec56229e7d2ff6fd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:21:46.170 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:21:46.177 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:21:46.183 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:21:46.187 INFO 2981023 --- [ocp-schedules-10,330f9fb1f98eaf8d,668698241a5d079b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:21:46.188 INFO 2981023 --- [ocp-schedules-10,330f9fb1f98eaf8d,668698241a5d079b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:21:46.189 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050084466000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:21:46.193 INFO 2981023 --- [ocp-schedules-24,cee88bd3ff6059d6,4d17ef258075fb6b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:21:46.193 INFO 2981023 --- [ocp-schedules-28,e89a4c96135211ba,0eb97d07de6b9798] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:21:46.194 INFO 2981023 --- [ocp-schedules-30,0bbf624c60487e1b,1faaae644b4bcb26] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:21:46.194 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:21:46.198 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:21:46.199 INFO 2981023 --- [ocp-schedules-1,ff80064e3fb85d80,ec56229e7d2ff6fd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:21:46.201 INFO 2981023 --- [ocp-schedules-28,e89a4c96135211ba,0eb97d07de6b9798] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:21:46.212 INFO 2981023 --- [ocp-schedules-24,cee88bd3ff6059d6,4d17ef258075fb6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:21:46.236 INFO 2981023 --- [ocp-schedules-19,9b776e547837e5f9,1164d4950fa1454a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 18:21:46.238 INFO 2981023 --- [ocp-schedules-19,9b776e547837e5f9,1164d4950fa1454a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:21:46.245 INFO 2981023 --- [ocp-schedules-11,35e00735e20a3bc1,e8007e0ac31cfb8c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:21:46.254 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:21:46.254 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:21:46.256 INFO 2981023 --- [ocp-schedules-24,cee88bd3ff6059d6,4d17ef258075fb6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:21:46.271 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.271 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:21:46.272 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:21:46.273 INFO 2981023 --- [ocp-schedules-11,d47209fb3b198725,87ecc59f9fe2d498] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 18:21:46.273 INFO 2981023 --- [ocp-schedules-11,d47209fb3b198725,87ecc59f9fe2d498] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 18:21:46.282 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:21:46.283 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:21:46.288 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.288 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.288 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.289 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:46.293 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.294 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:21:46.296 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.297 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.297 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.297 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:46.297 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:21:46.298 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:21:46.300 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:46.303 INFO 2981023 --- [ocp-schedules-10,330f9fb1f98eaf8d,668698241a5d079b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:21:46.304 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:21:46.308 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,c66bdde302e267fe,5e4521e29eb2bc54] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=c66bdde302e267fe, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 18:21:46.308 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,c66bdde302e267fe,5e4521e29eb2bc54] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=c66bdde302e267fe, duration=1 ms] 2024-09-11 18:21:46.310 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.311 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.311 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:21:46.314 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:21:46.314 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:21:46.319 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:21:46.321 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:21:46.321 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.324 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 18:21:46.324 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:21:46.325 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:21:46.326 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.333 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,5b0762ea0d35bc66,f4f5557c849a33c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=5b0762ea0d35bc66, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 18:21:46.333 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,5b0762ea0d35bc66,f4f5557c849a33c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=5b0762ea0d35bc66, duration=1 ms] 2024-09-11 18:21:46.335 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.347 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:21:46.346714+08:00, lastProcessTime=2024-09-11T18:21:36.337926+08:00 2024-09-11 18:21:46.348 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:21:46.349 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:21:46.352 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:21:46.355 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:21:46.356 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.364 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 18:21:46.372 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:21:46.375 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:21:46.376 INFO 2981023 --- [ocp-schedules-19,9b776e547837e5f9,1164d4950fa1454a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 18:21:46.384 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:21:46.387 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:21:46.388 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:21:46.392 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:21:46.396 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.399 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.399 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:21:46.400 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:21:46.406 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:21:46.406 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050084466000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:21:46.411 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:21:46.431 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.432 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:46.433 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:21:46.433 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.436 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:21:46.436 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:21:46.443 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:21:46.459 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:21:46.459 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:21:46.464 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:21:46.464 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:21:46.468 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:21:46.473 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.473 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:21:46.474 INFO 2981023 --- [host-check-host-status4,c33c3cc2aa155398,e0d30c4307341b64] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:21:46.474 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:21:46.475 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.494 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:21:46.496 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:21:46.501 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:21:46.520 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.522 INFO 2981023 --- [pool-10-thread-9,962e1803a154afcd,4ac6cecafff34aff] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 18:21:46.523 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.524 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:21:46.524 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:21:46.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:21:46.532 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:21:46.537 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:21:46.538 INFO 2981023 --- [pool-10-thread-12,962e1803a154afcd,875dcf2c7ba03af9] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 18:21:46.542 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:21:46.545 INFO 2981023 --- [pool-10-thread-11,962e1803a154afcd,699b6101c9232a4b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 18:21:46.560 INFO 2981023 --- [host-check-host-status2,c33c3cc2aa155398,bcca4116646e7cbe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:21:46.561 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 18:21:46.561 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 18:21:46.562 INFO 2981023 --- [ForkJoinPool-6-worker-1,962e1803a154afcd,c9dad5bf3d0e4d3a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 18:21:46.562 INFO 2981023 --- [ForkJoinPool-6-worker-1,962e1803a154afcd,c9dad5bf3d0e4d3a] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 18:21:46.562 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:21:46.563 INFO 2981023 --- [pool-10-thread-12,962e1803a154afcd,875dcf2c7ba03af9] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 18:21:46.563 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 18:21:46.563 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 18:21:46.563 INFO 2981023 --- [pool-10-thread-12,962e1803a154afcd,875dcf2c7ba03af9] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 18:21:46.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:21:46.576 INFO 2981023 --- [pool-10-thread-10,962e1803a154afcd,0ff14d5a376ddf93] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 18:21:46.578 INFO 2981023 --- [host-check-host-status3,c33c3cc2aa155398,3f827d8fdec81b0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:21:46.583 INFO 2981023 --- [pool-10-thread-10,962e1803a154afcd,0ff14d5a376ddf93] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 18:21:46.588 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.610 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:21:46.623 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.623 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.624 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:21:46.624 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:21:46.634 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:21:46.641 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.642 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:21:46.643 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:21:46.644 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.650 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:21:46.677 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.678 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:21:46.684 WARN 2981023 --- [ForkJoinPool-6-worker-1,962e1803a154afcd,c9dad5bf3d0e4d3a] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 18:21:46.686 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.688 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:21:46.707 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.708 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:21:46.710 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.710 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:21:46.715 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 18:21:46.718 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:21:46.736 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:21:46.757 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:21:46.762 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:21:46.762 INFO 2981023 --- [ForkJoinPool-6-worker-5,962e1803a154afcd,d5bc93194e8917e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.763 INFO 2981023 --- [ForkJoinPool-6-worker-5,962e1803a154afcd,d5bc93194e8917e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:21:46.772 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.779 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.788 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.788 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:21:46.789 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.791 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.792 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 18:21:46.792 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:21:46.793 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:21:46.793 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.795 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:21:46.796 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:21:46.812 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.818 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.818 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:21:46.819 INFO 2981023 --- [ForkJoinPool-6-worker-5,962e1803a154afcd,d5bc93194e8917e6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:21:46.820 INFO 2981023 --- [ForkJoinPool-6-worker-5,962e1803a154afcd,d5bc93194e8917e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.820 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:21:46.821 INFO 2981023 --- [ForkJoinPool-6-worker-5,962e1803a154afcd,d5bc93194e8917e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:21:46.822 INFO 2981023 --- [ForkJoinPool-6-worker-5,962e1803a154afcd,d5bc93194e8917e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.823 INFO 2981023 --- [ForkJoinPool-6-worker-5,962e1803a154afcd,d5bc93194e8917e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:21:46.825 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.825 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:21:46.826 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:21:46.826 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.827 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.828 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:21:46.829 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.829 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:21:46.830 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 18:21:46.831 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:21:46.831 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:21:46.832 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:21:46.832 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:21:46.837 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:21:46.846 INFO 2981023 --- [ocp-schedules-13,f35b196e874e627c,a46ce36802457408] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:21:46.846 INFO 2981023 --- [ocp-schedules-13,f35b196e874e627c,a46ce36802457408] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:21:46.848 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:21:46.849 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.850 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:21:46, 2024-09-11 18:21:46] 2024-09-11 18:21:46.851 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 18:21:46.851 INFO 2981023 --- [ForkJoinPool-6-worker-10,962e1803a154afcd,2345ae8afe0b0ee1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 18:21:46.873 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:21:46.874 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:21:46.875 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:21:46.877 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.880 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.881 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:21:46.346714+08:00 2024-09-11 18:21:46.882 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.883 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.883 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:21:46.884 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:21:46.885 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.887 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:21:46.887 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.888 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:21:46.891 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:21:46, 2024-09-11 18:21:46] 2024-09-11 18:21:46.892 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:21:46.893 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.893 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:21:46.896 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 18:21:46.900 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:21:46.906 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:21:46.907 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:21:46.914 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:46.925 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.926 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:21:46.929 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:21:46.933 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.934 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:21:46.935 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:21:46, 2024-09-11 18:21:46] 2024-09-11 18:21:46.935 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.939 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.945 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 18:21:46.945 INFO 2981023 --- [ForkJoinPool-6-worker-8,962e1803a154afcd,3401acbf16a98a67] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 18:21:46.950 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:21:46.951 INFO 2981023 --- [pool-10-thread-11,962e1803a154afcd,699b6101c9232a4b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 18:21:46.972 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:21:46.973 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.974 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:21:46.975 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:46.975 INFO 2981023 --- [ForkJoinPool-6-worker-3,962e1803a154afcd,03b7a985bc8ac429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:21:46.978 INFO 2981023 --- [pool-10-thread-9,962e1803a154afcd,4ac6cecafff34aff] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 18:21:46.985 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:21:46.999 INFO 2981023 --- [pool-10-thread-10,962e1803a154afcd,0ff14d5a376ddf93] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 18:21:47.004 INFO 2981023 --- [pool-10-thread-10,962e1803a154afcd,0ff14d5a376ddf93] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 18:21:47.022 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:21:47.028 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:21:47.030 INFO 2981023 --- [ocp-schedules-27,74132970d72a66a5,9b5de2212c3de0ea] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:21:47.174 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:21:47.178 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:21:47.181 INFO 2981023 --- [host-check-host-status1,c33c3cc2aa155398,942784d25a1d95fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:21:47.186 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:21:47.189 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:47.191 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:21:47.207 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:21:47.207 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:21:47.207 INFO 2981023 --- [ocp-schedules-28,e89a4c96135211ba,0eb97d07de6b9798] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:21:47.242 INFO 2981023 --- [ocp-schedules-14,177c7993295023ad,5030d7c7d11d186a] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:21:47.263 WARN 2981023 --- [ocp-schedules-14,177c7993295023ad,5030d7c7d11d186a] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:21:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,c4c8fc6bb8ac8ec1,bd2635f01c111f54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,c4c8fc6bb8ac8ec1,bd2635f01c111f54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:47.446 INFO 2981023 --- [ocp-async-10,30859e66caa8db54,e62b4a14f8b498fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:47.791 INFO 2981023 --- [ocp-async-7,9ab56db96a4fe7e8,1e5c5b76319eb856] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:21:47.810 INFO 2981023 --- [host-check-clockdiff2,0b5fd55554758f15,0b5d044ea62d5fb5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050107 6 6 , error= 2024-09-11 18:21:48.205 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:48.206 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:49.446 INFO 2981023 --- [ocp-async-5,a21074e0ca1deb92,1efc193e49da734c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:21:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:21:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,22a5f061eebcac9c,5e532dec1dd86862] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,22a5f061eebcac9c,5e532dec1dd86862] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:50.358 INFO 2981023 --- [ocp-schedules-1,9178a01498f2c6f6,9204efeb94cbd378] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:21:50.841 INFO 2981023 --- [host-check-clockdiff1,0b5fd55554758f15,3662c89a0d32e666] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050110 0 0 , error= 2024-09-11 18:21:51.206 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:51.207 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:51.446 INFO 2981023 --- [ocp-async-4,7b67cdb58c8ef9f2,5545fe0db29e978a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:51.634 INFO 2981023 --- [ocp-schedules-18,6e1a845fa021e178,83be61d79a1f3aff] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:21:51.635 INFO 2981023 --- [ocp-schedules-18,6e1a845fa021e178,83be61d79a1f3aff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:21:51.635 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:21:51.635 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:21:51.635 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:21:51.635 INFO 2981023 --- [ocp-schedules-18,6e1a845fa021e178,83be61d79a1f3aff] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:21:51.988 INFO 2981023 --- [host-check-clockdiff4,0b5fd55554758f15,8d63989197002dd7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050111 0 0 , error= 2024-09-11 18:21:52.752 INFO 2981023 --- [ocp-schedules-6,887419b0d86fd042,a76b026651f00a79] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:21:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,0f407f77a876f27f,bf5b4891a7a02759] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,0f407f77a876f27f,bf5b4891a7a02759] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:53.353 INFO 2981023 --- [ocp-schedules-15,e0ce09689a969567,cda30d202627f9b4] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:21:53.353 INFO 2981023 --- [ocp-schedules-15,e0ce09689a969567,cda30d202627f9b4] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:21:53.446 INFO 2981023 --- [ocp-async-2,4cf6cf7dc11e3c93,03e6ec13ece1436f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:54.207 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:54.209 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:54.477 INFO 2981023 --- [ocp-schedules-24,cc3d65be7b53afc5,22bc2696951a4aee] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:21:54.746 INFO 2981023 --- [ocp-schedules-18,3c6fc5e221bd7701,4b1103b418bb1159] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:21:54.770 WARN 2981023 --- [scanner-13654182-1,3c6fc5e221bd7701,2ca05008a4e4b666] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:21:54.809 INFO 2981023 --- [alarm-detect-6,3c6fc5e221bd7701,f61c8fa53bc510ca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646049, data is empty. 2024-09-11 18:21:54.815 INFO 2981023 --- [alarm-detect-6,3c6fc5e221bd7701,f61c8fa53bc510ca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646049, data is empty. 2024-09-11 18:21:54.818 INFO 2981023 --- [alarm-detect-6,3c6fc5e221bd7701,f61c8fa53bc510ca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646049, data is empty. 2024-09-11 18:21:54.843 INFO 2981023 --- [alarm-detect-3,3c6fc5e221bd7701,054a584797051d67] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646049, data is empty. 2024-09-11 18:21:54.846 INFO 2981023 --- [alarm-detect-14,3c6fc5e221bd7701,acf6d512fe509154] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646049, data is empty. 2024-09-11 18:21:54.848 INFO 2981023 --- [alarm-detect-3,3c6fc5e221bd7701,054a584797051d67] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646049, data is empty. 2024-09-11 18:21:54.850 INFO 2981023 --- [alarm-detect-14,3c6fc5e221bd7701,acf6d512fe509154] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646049, data is empty. 2024-09-11 18:21:54.853 INFO 2981023 --- [alarm-detect-3,3c6fc5e221bd7701,054a584797051d67] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646049, data is empty. 2024-09-11 18:21:54.854 INFO 2981023 --- [alarm-detect-14,3c6fc5e221bd7701,acf6d512fe509154] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646049, data is empty. 2024-09-11 18:21:54.884 INFO 2981023 --- [alarm-detect-12,3c6fc5e221bd7701,b21d9cfcd96ce8d4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050075 ~ 1726050104 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050090, value=7.256)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.256, values={1726050090=7.256}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:54.884 INFO 2981023 --- [alarm-detect-12,3c6fc5e221bd7701,b21d9cfcd96ce8d4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050075 ~ 1726050104 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050090, value=7.256)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.256, values={1726050090=7.256}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:21:54.884 INFO 2981023 --- [alarm-detect-12,3c6fc5e221bd7701,b21d9cfcd96ce8d4] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.31.35.5-2, currentState=Firing 2024-09-11 18:21:54.884 INFO 2981023 --- [alarm-detect-12,3c6fc5e221bd7701,b21d9cfcd96ce8d4] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:21:54.933 INFO 2981023 --- [alarm-detect-12,3c6fc5e221bd7701,b21d9cfcd96ce8d4] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm ob_host_load1_per_cpu_over_threshold changed, level 3 -> 2, status Active -> Active 2024-09-11 18:21:54.968 INFO 2981023 --- [alarm-detect-1,3c6fc5e221bd7701,88820c01147a05f8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646049, data is empty. 2024-09-11 18:21:54.974 INFO 2981023 --- [alarm-detect-1,3c6fc5e221bd7701,88820c01147a05f8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646049, data is empty. 2024-09-11 18:21:54.980 INFO 2981023 --- [alarm-detect-1,3c6fc5e221bd7701,88820c01147a05f8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646049, data is empty. 2024-09-11 18:21:54.992 INFO 2981023 --- [alarm-detect-19,3c6fc5e221bd7701,b5b06078815f6a24] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050075 ~ 1726050104 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050090, value=100.309)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=100.309, values={1726050090=100.309}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:21:55.101 INFO 2981023 --- [ocp-schedules-16,77d45167e1145e9a,583a97b637d1bf21] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:21:55.101 INFO 2981023 --- [ocp-schedules-16,77d45167e1145e9a,583a97b637d1bf21] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:21:55.105 INFO 2981023 --- [ocp-schedules-16,77d45167e1145e9a,583a97b637d1bf21] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:21:55.105 INFO 2981023 --- [ocp-schedules-16,77d45167e1145e9a,583a97b637d1bf21] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:21:55.105 INFO 2981023 --- [ocp-schedules-16,77d45167e1145e9a,583a97b637d1bf21] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:21:55.115 INFO 2981023 --- [ocp-schedules-16,77d45167e1145e9a,583a97b637d1bf21] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:21:55.115 INFO 2981023 --- [ocp-schedules-16,77d45167e1145e9a,583a97b637d1bf21] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:21:55.115 INFO 2981023 --- [ocp-schedules-16,77d45167e1145e9a,583a97b637d1bf21] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:21:55.126 INFO 2981023 --- [ocp-schedules-25,248a7e0a4095f280,b3ff91325e30b032] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:21:55.131 INFO 2981023 --- [ocp-schedules-16,77d45167e1145e9a,583a97b637d1bf21] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:21:55.145 WARN 2981023 --- [ocp-schedules-25,248a7e0a4095f280,b3ff91325e30b032] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:21:55.446 INFO 2981023 --- [ocp-async-3,52a4ac8843a020c5,77e1512c60a13575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:55.628 INFO 2981023 --- [ocp-schedules-20,1e287a10e5036484,d34b84a7776d668b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:21:55.636 INFO 2981023 --- [ocp-schedules-20,1e287a10e5036484,d34b84a7776d668b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:21:55.636 INFO 2981023 --- [ocp-schedules-20,1e287a10e5036484,d34b84a7776d668b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:21:55.897 INFO 2981023 --- [ocp-schedules-23,476c34540110e115,4fc4a3b123a2042f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:21:55.897 INFO 2981023 --- [ocp-schedules-23,476c34540110e115,4fc4a3b123a2042f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:21:55.900 INFO 2981023 --- [ocp-schedules-23,476c34540110e115,4fc4a3b123a2042f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:21:55.900 INFO 2981023 --- [ocp-schedules-23,476c34540110e115,4fc4a3b123a2042f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:21:55.902 INFO 2981023 --- [ocp-schedules-23,476c34540110e115,4fc4a3b123a2042f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:21:55.902 INFO 2981023 --- [ocp-schedules-23,476c34540110e115,4fc4a3b123a2042f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:21:56.109 INFO 2981023 --- [ocp-schedules-9,b8dab509283f3da2,b890359908d573c4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:21:56.115 INFO 2981023 --- [ocp-schedules-9,b8dab509283f3da2,b890359908d573c4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:21:56.115 INFO 2981023 --- [ocp-schedules-9,b8dab509283f3da2,b890359908d573c4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:21:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,672a5d73f6427f1c,33665ff92103c922] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,672a5d73f6427f1c,33665ff92103c922] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:56.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:21:56.353728+08:00, lastProcessTime=2024-09-11T18:21:46.346714+08:00 2024-09-11 18:21:56.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:21:56.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:21:56.353728+08:00 2024-09-11 18:21:57.209 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:21:57.210 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:21:57.446 INFO 2981023 --- [ocp-async-5,652b3ae9b54d3ad0,6c9a1ab2e0749aed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:57.791 INFO 2981023 --- [ocp-async-6,819f9132773ff0de,071646f62e60a054] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:21:58.401 INFO 2981023 --- [ocp-schedules-14,7a281dfb4e1aa803,8766f5fa997d3104] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:21:58.401 INFO 2981023 --- [ocp-schedules-14,7a281dfb4e1aa803,8766f5fa997d3104] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:21:58.433 INFO 2981023 --- [Collect all tenants major compactions2,7a281dfb4e1aa803,c5835994a7ce37ca] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:21:58.434 INFO 2981023 --- [Collect all tenants major compactions3,7a281dfb4e1aa803,7c82b303f3e2dbaf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:21:58.436 INFO 2981023 --- [Collect all tenants major compactions5,7a281dfb4e1aa803,01b46a6c421bfd48] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:21:58.441 INFO 2981023 --- [Collect all tenants major compactions6,7a281dfb4e1aa803,36539b63fd973788] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:21:58.441 INFO 2981023 --- [Collect all tenants major compactions4,7a281dfb4e1aa803,02790b975cdc31a4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:21:58.441 INFO 2981023 --- [Collect all tenants major compactions1,7a281dfb4e1aa803,089b7cdd38304129] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:21:58.461 INFO 2981023 --- [Collect all tenants major compactions3,7a281dfb4e1aa803,7c82b303f3e2dbaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:58.463 INFO 2981023 --- [Collect all tenants major compactions3,7a281dfb4e1aa803,7c82b303f3e2dbaf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:21:58.463 INFO 2981023 --- [Collect all tenants major compactions1,7a281dfb4e1aa803,089b7cdd38304129] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:58.464 INFO 2981023 --- [Collect all tenants major compactions1,7a281dfb4e1aa803,089b7cdd38304129] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:21:58.464 INFO 2981023 --- [Collect all tenants major compactions2,7a281dfb4e1aa803,c5835994a7ce37ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:58.466 INFO 2981023 --- [Collect all tenants major compactions6,7a281dfb4e1aa803,36539b63fd973788] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:58.466 INFO 2981023 --- [Collect all tenants major compactions3,7a281dfb4e1aa803,7c82b303f3e2dbaf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:21:58.467 INFO 2981023 --- [Collect all tenants major compactions2,7a281dfb4e1aa803,c5835994a7ce37ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:21:58.467 INFO 2981023 --- [Collect all tenants major compactions6,7a281dfb4e1aa803,36539b63fd973788] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:21:58.467 INFO 2981023 --- [Collect all tenants major compactions1,7a281dfb4e1aa803,089b7cdd38304129] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:21:58.469 INFO 2981023 --- [Collect all tenants major compactions2,7a281dfb4e1aa803,c5835994a7ce37ca] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:21:58.470 INFO 2981023 --- [Collect all tenants major compactions1,7a281dfb4e1aa803,089b7cdd38304129] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:21:58.470 INFO 2981023 --- [Collect all tenants major compactions4,7a281dfb4e1aa803,02790b975cdc31a4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:58.471 INFO 2981023 --- [Collect all tenants major compactions6,7a281dfb4e1aa803,36539b63fd973788] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:21:58.471 INFO 2981023 --- [Collect all tenants major compactions6,7a281dfb4e1aa803,36539b63fd973788] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:21:58.472 INFO 2981023 --- [Collect all tenants major compactions3,7a281dfb4e1aa803,7c82b303f3e2dbaf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:21:58.473 INFO 2981023 --- [Collect all tenants major compactions2,7a281dfb4e1aa803,c5835994a7ce37ca] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:21:58.473 INFO 2981023 --- [Collect all tenants major compactions4,7a281dfb4e1aa803,02790b975cdc31a4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:21:58.478 INFO 2981023 --- [Collect all tenants major compactions4,7a281dfb4e1aa803,02790b975cdc31a4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:21:58.478 INFO 2981023 --- [Collect all tenants major compactions4,7a281dfb4e1aa803,02790b975cdc31a4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:21:58.485 INFO 2981023 --- [Collect all tenants major compactions5,7a281dfb4e1aa803,01b46a6c421bfd48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:21:58.486 INFO 2981023 --- [Collect all tenants major compactions5,7a281dfb4e1aa803,01b46a6c421bfd48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:21:58.491 INFO 2981023 --- [Collect all tenants major compactions5,7a281dfb4e1aa803,01b46a6c421bfd48] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:21:58.491 INFO 2981023 --- [Collect all tenants major compactions5,7a281dfb4e1aa803,01b46a6c421bfd48] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:21:58.491 INFO 2981023 --- [ocp-schedules-14,7a281dfb4e1aa803,8766f5fa997d3104] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:21:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,e85c9c76c7c36f81,e06354478c01f519] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050120, interval=60 2024-09-11 18:21:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,e85c9c76c7c36f81,e06354478c01f519] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:21:59.446 INFO 2981023 --- [ocp-async-10,8fb31ef28a93906d,d843d140f8c5db2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:21:59.788 INFO 2981023 --- [ocp-schedules-22,ce11071eba724683,9ca6ffcf81e47b74] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:21:59.788+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:21:59.794 INFO 2981023 --- [ocp-schedules-22,ce11071eba724683,9ca6ffcf81e47b74] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:21:59.978 INFO 2981023 --- [ocp-schedules-22,ce11071eba724683,9ca6ffcf81e47b74] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 58 to SqlAggregateData 2024-09-11 18:21:59.979 INFO 2981023 --- [ocp-schedules-22,ce11071eba724683,9ca6ffcf81e47b74] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:22:00.210 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:00.211 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:01.238 INFO 2981023 --- [ocp-schedules-14,7f8fe51d21a4a394,5f7be73654a3f9b9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:22:01.239 INFO 2981023 --- [ocp-schedules-14,7f8fe51d21a4a394,5f7be73654a3f9b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:01.340 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:22:01.340 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.345 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:22:01.354 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.354 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.363 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.364 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.364 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:22:01.365 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.372 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:22:01.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.387 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.387 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:22:01.390 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.407 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.408 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:22:01.415 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.415 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:22:01.422 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.422 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.447 INFO 2981023 --- [ocp-async-6,f029bb42f601db67,e7ec6b65ab685657] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:01.449 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.453 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.453 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:22:01.462 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.475 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.475 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:22:01.476 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.477 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:22:01.477 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.478 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:22:01.478 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.479 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:22:01.479 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.479 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:22:01.480 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.480 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:22:01.480 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.480 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:22:01.481 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:22:01.481 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.482 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:22:01.482 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.483 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:22:01.483 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:22:01.483 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.484 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:22:01.484 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:22:01.486 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.487 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.488 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5C11D6EE-0-0] 2024-09-11 18:22:01.488 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:22:01.491 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:22:01.498 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.503 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.503 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.505 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:01.507 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:22:01.518 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.518 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.518 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.518 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:22:01.519 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.524 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:22:01.524 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:22:01.527 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:22:01.527 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.528 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:22:01.530 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.531 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:22:01.531 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.532 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:22:01.532 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.533 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:22:01.533 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:22:01.534 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:22:01.535 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:22:01.536 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:22:01.536 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.537 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.541 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.542 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:22:01.543 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:22:01.543 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:22:01.543 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5C21D6DB-0-0] 2024-09-11 18:22:01.543 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:22:01.544 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:22:01.544 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:22:01.544 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:22:01.549 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:01.550 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.551 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.551 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.552 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:22:01.552 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.553 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:22:01.553 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.554 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:22:01.554 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.555 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:22:01.555 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.556 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:22:01.556 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.557 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:22:01.557 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:22:01.559 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:22:01.561 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:22:01.563 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:22:01.563 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:22:01.563 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:22:01.569 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:01.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:22:01.584 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:22:01.585 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:22:01.585 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:01.586 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:22:01.594 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:22:01.595 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:22:01.595 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:22:01.595 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:22:01.595 INFO 2981023 --- [ocp-schedules-14,7f8fe51d21a4a394,5f7be73654a3f9b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:22:02.281 INFO 2981023 --- [ocp-schedules-12,b082a20c27fd193b,1c79c23cbaae8bce] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:22:02.281 INFO 2981023 --- [ocp-schedules-12,b082a20c27fd193b,1c79c23cbaae8bce] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:22:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,ab33418f1a72ae8f,07cc3c7a522d872d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,ab33418f1a72ae8f,07cc3c7a522d872d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:02.755 INFO 2981023 --- [ocp-schedules-1,d0f12851f97189ae,f2351f9c67ba3d08] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:22:03.118 INFO 2981023 --- [ocp-schedules-19,2f9519d39de99876,d941e1b6fc8664a5] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:22:03.119 INFO 2981023 --- [ocp-schedules-19,2f9519d39de99876,d941e1b6fc8664a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:03.161 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:03.161 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:22:03.167 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:03.169 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:22:03.170 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050123170000} 2024-09-11 18:22:03.175 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:22:03.178 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000235, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:22:03.179 INFO 2981023 --- [ocp-schedules-19,2f9519d39de99876,d941e1b6fc8664a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:22:03.211 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:03.211 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:03.447 INFO 2981023 --- [ocp-async-8,9a9562ab766e2a2b,e6e5fe5f27ce0e7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:03.871 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:22:03.871 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:22:03.871 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:22:03.876 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:22:03.879 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000235, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:22:04.302 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000235 2024-09-11 18:22:05.023 INFO 2981023 --- [ocp-schedules-4,999ee7dbbb2a52fd,37973e3e13251d02] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:22:05.040 WARN 2981023 --- [scanner-13654487-1,999ee7dbbb2a52fd,6e1ef0e251f5884f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:22:05.150 INFO 2981023 --- [alarm-detect-5,999ee7dbbb2a52fd,eaaa79f4ac1935d8] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050090 ~ 1726050115 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050105, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050105, value=98.15)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.15, values={1726050105=98.15}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050105=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:05.152 INFO 2981023 --- [alarm-detect-8,999ee7dbbb2a52fd,f1b2e645cd7f0bbc] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050090 ~ 1726050115 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050105, value=98.15)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.15, values={1726050105=98.15}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:05.152 INFO 2981023 --- [alarm-detect-8,999ee7dbbb2a52fd,f1b2e645cd7f0bbc] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=os_tsar_cpu_util_full, target=alarm_template_id=0:host=10.31.35.5-2, currentState=Firing 2024-09-11 18:22:05.152 INFO 2981023 --- [alarm-detect-8,999ee7dbbb2a52fd,f1b2e645cd7f0bbc] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=os_tsar_cpu_util_full, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:22:05.171 INFO 2981023 --- [alarm-detect-3,999ee7dbbb2a52fd,d671f89677eaaf8c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050090 ~ 1726050115 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050105, value=7.547)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.547, values={1726050105=7.547}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:05.171 INFO 2981023 --- [alarm-detect-3,999ee7dbbb2a52fd,d671f89677eaaf8c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050090 ~ 1726050115 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050105, value=7.547)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.547, values={1726050105=7.547}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:22:05.187 INFO 2981023 --- [alarm-detect-8,999ee7dbbb2a52fd,f1b2e645cd7f0bbc] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007112 2024-09-11 18:22:05.210 INFO 2981023 --- [alarm-detect-7,999ee7dbbb2a52fd,ec0300b4b8fa2e9f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050090 ~ 1726050115 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050105, value=98.15)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.15, values={1726050105=98.15}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:05.210 INFO 2981023 --- [alarm-detect-7,999ee7dbbb2a52fd,ec0300b4b8fa2e9f] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_host_cpu_percent_over_threshold, target=alarm_template_id=0:host=10.31.35.5-2, currentState=Firing 2024-09-11 18:22:05.210 INFO 2981023 --- [alarm-detect-7,999ee7dbbb2a52fd,ec0300b4b8fa2e9f] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_cpu_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:22:05.229 INFO 2981023 --- [alarm-detect-7,999ee7dbbb2a52fd,ec0300b4b8fa2e9f] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007113 2024-09-11 18:22:05.267 INFO 2981023 --- [alarm-detect-2,999ee7dbbb2a52fd,4d8e27c1dc133b71] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050090 ~ 1726050115 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050105, value=101.228)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=101.228, values={1726050105=101.228}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:05.267 INFO 2981023 --- [alarm-detect-2,999ee7dbbb2a52fd,4d8e27c1dc133b71] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_tenant_cpu_usage_over_threshold, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:tenant_name=linkreit:host=10.31.35.5-2, currentState=Firing 2024-09-11 18:22:05.267 INFO 2981023 --- [alarm-detect-2,999ee7dbbb2a52fd,4d8e27c1dc133b71] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_cpu_usage_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=rtdob-1718689631:tenant_name=linkreit:host=10.31.35.5-2] 2024-09-11 18:22:05.287 INFO 2981023 --- [alarm-detect-2,999ee7dbbb2a52fd,4d8e27c1dc133b71] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007114 2024-09-11 18:22:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,caafea68b1413964,f2c91277be9ae6d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,caafea68b1413964,f2c91277be9ae6d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:05.447 INFO 2981023 --- [ocp-async-9,2509d50c894e9203,80da4cdc11dd6991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:22:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:22:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:22:05.992 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:22:05.997 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:22:06.002 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:22:06.002 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:22:06.002 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:22:06.007 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T18:19:41, target=2024-09-11T18:17:41, granule=120000ms, delay=120000ms) 2024-09-11 18:22:06.007 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:17:41, target=2024-09-11T18:10:51, granule=600000ms, delay=410000ms) 2024-09-11 18:22:06.007 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=37ms, period=120000ms 2024-09-11 18:22:06.212 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:06.213 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:22:06.353070+08:00, lastProcessTime=2024-09-11T18:21:56.353728+08:00 2024-09-11 18:22:06.479 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=3, resultsCount=3 2024-09-11 18:22:06.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:22:06.353070+08:00 2024-09-11 18:22:07.447 INFO 2981023 --- [ocp-async-10,e8c6d312ff22d602,c730e8bbb4bd1956] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:07.792 INFO 2981023 --- [ocp-async-5,c3ef44653c01e9f3,dc0ebefb3e45ed3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:22:07.807 INFO 2981023 --- [ocp-async-5,c3ef44653c01e9f3,dc0ebefb3e45ed3a] c.o.o.a.s.OcpAlarmSnapshotServiceImpl : alarm id 6007114, alarm type ob_tenant_cpu_usage_over_threshold (collectGroupTypes [tenant_high_cpu]) snapshot count 0 2024-09-11 18:22:07.807 INFO 2981023 --- [ocp-async-5,c3ef44653c01e9f3,dc0ebefb3e45ed3a] c.o.o.a.s.AlarmEventSnapshotObserver : try to create alarm snapshot task ob_tenant_cpu_usage_over_threshold 6007114 2024-09-11 18:22:07.808 INFO 2981023 --- [ocp-async-5,c3ef44653c01e9f3,dc0ebefb3e45ed3a] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Collect alarm snapshot, nodeCount=1, taskCount=1 2024-09-11 18:22:07.808 INFO 2981023 --- [ocp-async-5,c3ef44653c01e9f3,dc0ebefb3e45ed3a] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:22:07.808 INFO 2981023 --- [ocp-async-5,c3ef44653c01e9f3,dc0ebefb3e45ed3a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Collect alarm snapshot', taskDefinitionId=-1, state=RUNNING, type=SYS_ONCE, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={ob_tenant_id=1004, tenant_name=linkreit, host_ip=10.31.35.5, prohibit_rollback=false, ob_cluster_name=rtdob, svr_ip=10.31.35.5, no_alarm=true, ob_cluster_id=1718689631, ob_tenant=linkreit, alarm_active_at=2024-09-11T10:21:03Z, alarmId=6007114, triggerEventName=ob_tenant_cpu_usage_over_threshold, ob_cluster=rtdob, ocp_ip=10.31.35.10}, listMap={host_ids=[2], collectSnapshotRequests=[tenant_high_cpu]}}, createTime=null, startTime=2024-09-11T18:22:07.808+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Collect alarm snapshot, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CollectAlarmSnapshotTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='c3ef44653c01e9f3', operation=EXECUTE} 2024-09-11 18:22:07.877 INFO 2981023 --- [subtask-coordinator3,ddd72be6e49ae268,d122f7d22d7b708f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048250, operation:EXECUTE, state: RUNNING 2024-09-11 18:22:07.897 INFO 2981023 --- [subtask-executor30,c3ef44653c01e9f3,56829a4f48696d9e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048250, context=Context{parallelIdx=0, stringMap={ob_tenant_id=1004, tenant_name=linkreit, host_ip=10.31.35.5, subtask_splitter=host_ids, ob_cluster_name=rtdob, svr_ip=10.31.35.5, task_instance_id=4048220, no_alarm=true, task_operation=execute, ob_cluster_id=1718689631, ob_tenant=linkreit, alarm_active_at=2024-09-11T10:21:03Z, alarmId=6007114, triggerEventName=ob_tenant_cpu_usage_over_threshold, ob_cluster=rtdob, ocp_ip=10.31.35.10, latest_execution_start_time=2024-09-11T18:22:07.869+08:00, sub_task_instance_name=Collect alarm snapshot, sub_task_instance_id=4048250}, listMap={host_ids=[2], collectSnapshotRequests=[tenant_high_cpu]}}, executor=10.31.35.10 2024-09-11 18:22:07.901 INFO 2981023 --- [subtask-executor30,c3ef44653c01e9f3,56829a4f48696d9e] c.o.o.c.agent.HostAgentServiceImpl : Collect alarm snapshot: hostId=2 2024-09-11 18:22:07.921 INFO 2981023 --- [subtask-executor30,c3ef44653c01e9f3,56829a4f48696d9e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:07.929 INFO 2981023 --- [subtask-executor30,c3ef44653c01e9f3,56829a4f48696d9e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:07.992 INFO 2981023 --- [subtask-executor30,c3ef44653c01e9f3,56829a4f48696d9e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/snapshot/collect, request body:CollectSnapshotRequest(collectExecutorGroupType=tenant_high_cpu, collectArgs={subtask_splitter=host_ids, ob_tenant_id=1004, tenant_name=linkreit, host_ip=10.31.35.5, ob_cluster_name=rtdob, svr_ip=10.31.35.5, no_alarm=true, ob_cluster_id=1718689631, ob_tenant=linkreit, alarm_active_at=2024-09-11T10:21:03Z, alarmId=6007114, triggerEventName=ob_tenant_cpu_usage_over_threshold, ob_cluster=rtdob, ocp_ip=10.31.35.10, task_instance_id=4048220, latest_execution_start_time=2024-09-11T18:22:07.869+08:00, task_operation=execute, sub_task_instance_id=4048250, sub_task_instance_name=Collect alarm snapshot}), params:null 2024-09-11 18:22:08.015 ERROR 2981023 --- [subtask-executor30,c3ef44653c01e9f3,56829a4f48696d9e] c.o.o.s.t.b.h.CollectAlarmSnapshotTask : Create collection failed. org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'com.oceanbase.ocp.collector.base.CollectorService' available at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:351) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:342) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1186) at com.oceanbase.ocp.core.util.BeanUtils.getBean(BeanUtils.java:33) at com.oceanbase.ocp.service.task.business.host.CollectAlarmSnapshotTask.triggerCollection(CollectAlarmSnapshotTask.java:52) at com.oceanbase.ocp.service.task.business.host.CollectAlarmSnapshotTask.run(CollectAlarmSnapshotTask.java:42) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:200) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 18:22:08.024 INFO 2981023 --- [subtask-executor30,c3ef44653c01e9f3,56829a4f48696d9e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048250, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:22:08.118 INFO 2981023 --- [ocp-schedules-16,9f92148be35bd5bc,dec36472715bcc62] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:22:08.195 INFO 2981023 --- [ocp-schedules-14,4735345c819e8dc4,7f15c3a1f4d2143d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:22:08.195+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:22:08.196 INFO 2981023 --- [ocp-schedules-14,4735345c819e8dc4,7f15c3a1f4d2143d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:22:08.198 INFO 2981023 --- [ocp-schedules-14,4735345c819e8dc4,7f15c3a1f4d2143d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:22:08.198 INFO 2981023 --- [ocp-schedules-14,4735345c819e8dc4,7f15c3a1f4d2143d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:22:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,51cefb00dd3ff2bd,5c68e3c6ed9c037b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,51cefb00dd3ff2bd,5c68e3c6ed9c037b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:08.883 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048220, status:SUCCESS 2024-09-11 18:22:08.885 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048220, status:SUCCESS 2024-09-11 18:22:08.905 INFO 2981023 --- [ocp-schedules-9,c828a912ff702cd9,56b96a67697d946c] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:22:09.213 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:09.214 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:09.447 INFO 2981023 --- [ocp-async-4,6206561083cf1024,c7b224b93499e158] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:10.003 INFO 2981023 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:22:10.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:22:10.332 INFO 2981023 --- [ocp-schedules-12,11ef9b63276d1f2c,dbeb95f9e66c8933] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-09-11 18:22:10.332 INFO 2981023 --- [ocp-schedules-12,11ef9b63276d1f2c,dbeb95f9e66c8933] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-09-11 18:22:10.332 INFO 2981023 --- [ocp-schedules-12,11ef9b63276d1f2c,dbeb95f9e66c8933] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=11798 2024-09-11 18:22:10.380 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,5f1fae5eef20e9c5,787ee277a7c82118] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=5f1fae5eef20e9c5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:22:10.381 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,5f1fae5eef20e9c5,787ee277a7c82118] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=5f1fae5eef20e9c5, duration=5 ms] 2024-09-11 18:22:10.391 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5c12ce33a1323db7,f290ab55d9dbc2e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=5c12ce33a1323db7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:22:10.392 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5c12ce33a1323db7,f290ab55d9dbc2e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=5c12ce33a1323db7, duration=3 ms] 2024-09-11 18:22:10.393 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ea081dc26632a24f,3dd47f52b7fef7da] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=ea081dc26632a24f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:22:10.399 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,ea081dc26632a24f,3dd47f52b7fef7da] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=ea081dc26632a24f, duration=10 ms] 2024-09-11 18:22:10.569 INFO 2981023 --- [ocp-schedules-12,11ef9b63276d1f2c,dbeb95f9e66c8933] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=11798 2024-09-11 18:22:10.919 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:22:10.921 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:22:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,1d825ff0795d7a64,4bf4a6b086652528] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,1d825ff0795d7a64,4bf4a6b086652528] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:11.448 INFO 2981023 --- [ocp-async-5,11271d0a91797342,ca6e794eb02bf49c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:12.214 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:12.215 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:12.757 INFO 2981023 --- [ocp-schedules-8,f62297100ee58cff,d8dbebf533678778] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:22:13.447 INFO 2981023 --- [ocp-async-7,957bfaecb546b7ac,d2e516e1b2142f50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:13.533 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 18:22:13.544 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:13.544 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:13.556 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:22:13.557 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:22:13.568 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:13.569 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:13.580 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:22:13.581 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:22:13.592 INFO 2981023 --- [ocp-schedules-10,c792c79b8b639387,1065d80d8c0ad7b9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2024-09-11 18:22:13.970 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050060, nodeCount=1000, affectRows=1000 2024-09-11 18:22:13.972 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050060, nodeCount=1000, validCount=1000 2024-09-11 18:22:13.973 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050060, nodeCount=1000, affectRows=1000 2024-09-11 18:22:13.975 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050060, nodeCount=999, validCount=999 2024-09-11 18:22:13.983 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050060, nodeCount=48, affectRows=48 2024-09-11 18:22:13.983 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050060, nodeCount=46, validCount=46 2024-09-11 18:22:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,c3299c15ba99fb89,6a5786b7bd6d0d4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,5196b4ce3b69f73e,3cd352e4ade93c6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:22:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,c3299c15ba99fb89,6a5786b7bd6d0d4a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:15.007 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050060, nodeCount=480, affectRows=480 2024-09-11 18:22:15.215 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:15.215 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:15.295 INFO 2981023 --- [ocp-schedules-30,1a76b057202fd8b9,42d344243f8f1b7a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:22:15.348 WARN 2981023 --- [scanner-13656247-1,1a76b057202fd8b9,80290da2fa12abea] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:22:15.373 INFO 2981023 --- [alarm-detect-20,1a76b057202fd8b9,c0e51bc48a277881] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646049, data is empty. 2024-09-11 18:22:15.380 INFO 2981023 --- [alarm-detect-20,1a76b057202fd8b9,c0e51bc48a277881] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646107, data is empty. 2024-09-11 18:22:15.394 INFO 2981023 --- [alarm-detect-20,1a76b057202fd8b9,c0e51bc48a277881] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646107, data is empty. 2024-09-11 18:22:15.398 INFO 2981023 --- [alarm-detect-7,1a76b057202fd8b9,2dbb5fab63661f24] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646049, data is empty. 2024-09-11 18:22:15.400 INFO 2981023 --- [alarm-detect-9,1a76b057202fd8b9,0ec5eec79d5db226] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646049, data is empty. 2024-09-11 18:22:15.401 INFO 2981023 --- [alarm-detect-1,1a76b057202fd8b9,69f00db27314f723] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050105 ~ 1726050125 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050120, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050120, value=98.573)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.573, values={1726050120=98.573}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050120=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:15.405 INFO 2981023 --- [alarm-detect-7,1a76b057202fd8b9,2dbb5fab63661f24] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646107, data is empty. 2024-09-11 18:22:15.408 INFO 2981023 --- [alarm-detect-9,1a76b057202fd8b9,0ec5eec79d5db226] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646107, data is empty. 2024-09-11 18:22:15.412 INFO 2981023 --- [alarm-detect-7,1a76b057202fd8b9,2dbb5fab63661f24] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646107, data is empty. 2024-09-11 18:22:15.414 INFO 2981023 --- [alarm-detect-9,1a76b057202fd8b9,0ec5eec79d5db226] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646107, data is empty. 2024-09-11 18:22:15.415 INFO 2981023 --- [alarm-detect-11,1a76b057202fd8b9,8295eea7787a2a73] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050105 ~ 1726050125 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050120, value=98.573)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.573, values={1726050120=98.573}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:15.426 INFO 2981023 --- [alarm-detect-11,1a76b057202fd8b9,503c551538322f1e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050105 ~ 1726050125 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050120, value=7.728)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.728, values={1726050120=7.728}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:15.426 INFO 2981023 --- [alarm-detect-11,1a76b057202fd8b9,503c551538322f1e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050105 ~ 1726050125 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050120, value=7.728)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.728, values={1726050120=7.728}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:22:15.427 INFO 2981023 --- [alarm-detect-9,1a76b057202fd8b9,5123b33e78f0049f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050105 ~ 1726050125 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050120, value=98.573)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.573, values={1726050120=98.573}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:15.448 INFO 2981023 --- [ocp-async-6,ac8524d3c445216d,7e3214f04e8e22b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:15.518 INFO 2981023 --- [alarm-detect-1,1a76b057202fd8b9,67096edf4805dbf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646049, data is empty. 2024-09-11 18:22:15.526 INFO 2981023 --- [alarm-detect-1,1a76b057202fd8b9,67096edf4805dbf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646107, data is empty. 2024-09-11 18:22:15.530 INFO 2981023 --- [alarm-detect-1,1a76b057202fd8b9,67096edf4805dbf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646107, data is empty. 2024-09-11 18:22:15.552 INFO 2981023 --- [alarm-detect-12,1a76b057202fd8b9,720c78f942bb52a6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050105 ~ 1726050125 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050120, value=105.538)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.538, values={1726050120=105.538}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:15.639 INFO 2981023 --- [ocp-schedules-3,dd61babb3a33ed5c,7774679d6a03d719] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:22:15.647 INFO 2981023 --- [ocp-schedules-3,dd61babb3a33ed5c,7774679d6a03d719] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:22:15.647 INFO 2981023 --- [ocp-schedules-3,dd61babb3a33ed5c,7774679d6a03d719] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:22:16.013 INFO 2981023 --- [ocp-schedules-27,b4aa5775a9afaaf4,848c9f174be1220d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:22:16.014 INFO 2981023 --- [host-check-clockdiff1,b4aa5775a9afaaf4,bb81581bc12d8a2b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:22:16.014 INFO 2981023 --- [host-check-clockdiff2,b4aa5775a9afaaf4,856b4f20db3e79b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:22:16.016 INFO 2981023 --- [host-check-clockdiff3,b4aa5775a9afaaf4,d79564d294eb2d59] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:22:16.016 INFO 2981023 --- [host-check-clockdiff4,b4aa5775a9afaaf4,e81df77ee85f7894] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:22:16.023 INFO 2981023 --- [ocp-schedules-23,fc6074ea8df21d0f,2a3aba8e93d8aa75] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:22:16.028 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:22:16.031 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:22:16.031 INFO 2981023 --- [host-check-clockdiff3,b4aa5775a9afaaf4,d79564d294eb2d59] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050136 0 0 , error= 2024-09-11 18:22:16.031 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:22:16.031 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:22:16.033 INFO 2981023 --- [ocp-schedules-18,c6bae843d07157c5,87b940aacc48a86e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:22:16.033 INFO 2981023 --- [host-check-agent-status1,c6bae843d07157c5,60086b64598e3579] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:22:16.033 INFO 2981023 --- [host-check-agent-status2,c6bae843d07157c5,e0b63779cf33275f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:22:16.033 INFO 2981023 --- [host-check-agent-status4,c6bae843d07157c5,01ea51b504874eb7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:22:16.035 INFO 2981023 --- [host-check-agent-status3,c6bae843d07157c5,056de2d463f50d9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:22:16.038 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:16.040 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:16.040 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:16.041 INFO 2981023 --- [host-check-agent-status1,c6bae843d07157c5,60086b64598e3579] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:16.041 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:22:16.041 INFO 2981023 --- [host-check-agent-status2,c6bae843d07157c5,e0b63779cf33275f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:22:16.042 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:22:16.042 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:16.043 INFO 2981023 --- [host-check-agent-status2,c6bae843d07157c5,e0b63779cf33275f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:22:16.043 INFO 2981023 --- [host-check-agent-status4,c6bae843d07157c5,01ea51b504874eb7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:22:16.043 INFO 2981023 --- [host-check-agent-status1,c6bae843d07157c5,60086b64598e3579] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:16.043 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:22:16.044 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:22:16.044 INFO 2981023 --- [host-check-agent-status4,c6bae843d07157c5,01ea51b504874eb7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:22:16.046 INFO 2981023 --- [host-check-agent-status3,c6bae843d07157c5,056de2d463f50d9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:16.049 INFO 2981023 --- [host-check-agent-status3,c6bae843d07157c5,056de2d463f50d9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:16.074 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:22:16.074 INFO 2981023 --- [host-check-agent-status1,c6bae843d07157c5,60086b64598e3579] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:22:16.076 INFO 2981023 --- [host-check-agent-status2,c6bae843d07157c5,e0b63779cf33275f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:22:16.077 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:22:16.078 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:22:16.078 INFO 2981023 --- [host-check-agent-status4,c6bae843d07157c5,01ea51b504874eb7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:22:16.079 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:22:16.084311018+08:00 2024-09-11 18:22:16.079 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:22:16.079461625+08:00 2024-09-11 18:22:16.080 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:22:16.079+08:00 2024-09-11 18:22:16.080 INFO 2981023 --- [host-check-agent-status2,c6bae843d07157c5,e0b63779cf33275f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:22:16.080 INFO 2981023 --- [host-check-agent-status2,c6bae843d07157c5,e0b63779cf33275f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:22:16.080 INFO 2981023 --- [host-check-agent-status2,c6bae843d07157c5,e0b63779cf33275f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:22:16.080 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:22:16.079+08:00 2024-09-11 18:22:16.081 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:22:16.079+08:00 and status AVAILABLE 2024-09-11 18:22:16.082 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:22:16.082095690+08:00 2024-09-11 18:22:16.082 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:22:16.079+08:00 and status AVAILABLE 2024-09-11 18:22:16.085 INFO 2981023 --- [host-check-agent-status4,c6bae843d07157c5,01ea51b504874eb7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:22:16.085 INFO 2981023 --- [host-check-agent-status4,c6bae843d07157c5,01ea51b504874eb7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:22:16.085 INFO 2981023 --- [host-check-agent-status4,c6bae843d07157c5,01ea51b504874eb7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:22:16.085 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:22:16.085 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:22:16.082+08:00 2024-09-11 18:22:16.086 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:22:16.085992493+08:00 2024-09-11 18:22:16.087 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:22:16.082+08:00 and status AVAILABLE 2024-09-11 18:22:16.087 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:22:16.086+08:00 2024-09-11 18:22:16.089 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:22:16.086+08:00 and status AVAILABLE 2024-09-11 18:22:16.089 INFO 2981023 --- [host-check-agent-status3,c6bae843d07157c5,056de2d463f50d9f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:22:16.091 INFO 2981023 --- [host-check-agent-status3,c6bae843d07157c5,056de2d463f50d9f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:22:16.091 INFO 2981023 --- [host-check-agent-status3,c6bae843d07157c5,056de2d463f50d9f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:22:16.091 INFO 2981023 --- [host-check-agent-status3,c6bae843d07157c5,056de2d463f50d9f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:22:16.094 INFO 2981023 --- [host-check-agent-status1,c6bae843d07157c5,60086b64598e3579] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:22:16.094 INFO 2981023 --- [host-check-agent-status1,c6bae843d07157c5,60086b64598e3579] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:22:16.094 INFO 2981023 --- [host-check-agent-status1,c6bae843d07157c5,60086b64598e3579] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:22:16.096 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:22:16.133 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:22:16.134 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:22:16.134 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:22:16.143 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:16.144 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:16.144 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:16.145 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:16.145 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:22:16.146 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:22:16.147 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:22:16.148 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:22:16.179 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:22:16.187 INFO 2981023 --- [ocp-schedules-9,d847df1662357a66,72570a50c60ea721] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:22:16.188 INFO 2981023 --- [ocp-schedules-2,30a60fabd34de26c,d487077848e319e3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:22:16.189 INFO 2981023 --- [ocp-schedules-2,30a60fabd34de26c,d487077848e319e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:16.189 INFO 2981023 --- [ocp-schedules-20,07ccbe4b3de37be3,4d83165a56d264bf] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:22:16.190 INFO 2981023 --- [ocp-schedules-16,1acaffa5e93f1e45,d6d773a6de9fab22] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:22:16.190 INFO 2981023 --- [ocp-schedules-9,d847df1662357a66,72570a50c60ea721] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:16.191 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:22:16.196 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:22:16.200 INFO 2981023 --- [ocp-schedules-20,07ccbe4b3de37be3,4d83165a56d264bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:16.212 INFO 2981023 --- [ocp-schedules-5,c59d6798d9682f9d,b917a2b150c09d5a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:22:16.215 INFO 2981023 --- [ocp-schedules-2,30a60fabd34de26c,d487077848e319e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:22:16.225 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:22:16.236 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:22:16.246 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:22:16.257 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:22:16.257 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:22:16.276 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:22:16.276 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.277 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:22:16.278 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:22:16.280 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050123170000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:22:16.286 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.286 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:16.288 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.289 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:16.293 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:22:16.295 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.297 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:22:16.297 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.304 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:22:16.306 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:22:16.308 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:22:16.308 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:16.312 INFO 2981023 --- [ocp-schedules-9,d847df1662357a66,72570a50c60ea721] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:22:16.335 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:22:16.337 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:16.338 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.338 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:22:16.343 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:22:16.344 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:22:16.343470+08:00, lastProcessTime=2024-09-11T18:22:06.353070+08:00 2024-09-11 18:22:16.348 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:22:16.348 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050123170000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:22:16.352 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:22:16.355 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:22:16.357 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:22:16.366 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:22:16.372 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:22:16.377 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:22:16.384 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:22:16.387 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:16.387 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:22:16.388 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.388 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:22:16.389 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:16.389 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:22:16.390 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.392 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:22:16.393 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:22:16.396 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:22:16.405 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:22:16.405 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:22:16.408 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:22:16.410 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:22:16.412 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:22:16.423 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:22:16.423 INFO 2981023 --- [host-check-host-status2,fc6074ea8df21d0f,8da3ff4c1a34d145] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:22:16.426 INFO 2981023 --- [host-check-host-status4,fc6074ea8df21d0f,d2bf3aaed519c9c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:22:16.435 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:16.436 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.437 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:22:16.438 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:22:16.441 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:22:16.445 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:22:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:16.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:16.457 INFO 2981023 --- [host-check-host-status3,fc6074ea8df21d0f,60c31288cc2fac77] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:22:16.472 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:22:16.475 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:16.480 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.481 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:22:16.520 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:16.522 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.523 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:16.523 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:22:16.524 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.526 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:22:16.528 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:22:16.543 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:22:16.543 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:22:16.560 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:22:16.564 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:22:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:22:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:22:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:22:16.579 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.585 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:22:16.586 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:22:16.587 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:22:16.343470+08:00 2024-09-11 18:22:16.589 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:22:16.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:16.592 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:16.604 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:22:16.612 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:22:16.618 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:22:16.631 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:22:16.835 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:22:16.841 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:16.846 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:16.851 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:22:16.860 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:22:16.861 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:22:16.861 INFO 2981023 --- [ocp-schedules-20,07ccbe4b3de37be3,4d83165a56d264bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:22:16.961 INFO 2981023 --- [host-check-clockdiff2,b4aa5775a9afaaf4,856b4f20db3e79b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050136 5 5 , error= 2024-09-11 18:22:16.963 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:22:16.970 INFO 2981023 --- [host-check-host-status1,fc6074ea8df21d0f,b02d14b1a19d6f3d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:22:17.132 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050060, nodeCount=480, validCount=480 2024-09-11 18:22:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,ebcba3ed7fd27b30,2de9c88c06315eb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,ebcba3ed7fd27b30,2de9c88c06315eb2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:17.448 INFO 2981023 --- [ocp-async-2,8714cc64868e36a7,b871212bf3611b3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:17.792 INFO 2981023 --- [ocp-async-3,41227efa0693cc9f,b244a5b58da83dc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:22:18.215 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:18.216 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:19.448 INFO 2981023 --- [ocp-async-10,ebc37f1fa97fe782,592eb87ba8f93725] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,5cabcc64df7753b0,3a98255bd66f8538] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,5cabcc64df7753b0,3a98255bd66f8538] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:21.081 INFO 2981023 --- [host-check-clockdiff1,b4aa5775a9afaaf4,bb81581bc12d8a2b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050141 0 0 , error= 2024-09-11 18:22:21.216 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:21.217 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:21.448 INFO 2981023 --- [ocp-async-3,c8bdcd8d1dedb203,4ed1b9cfc0b698ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:22.002 INFO 2981023 --- [host-check-clockdiff4,b4aa5775a9afaaf4,e81df77ee85f7894] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050141 0 0 , error= 2024-09-11 18:22:22.760 INFO 2981023 --- [ocp-schedules-7,fe4fe084dbb4ce0f,61e1e042e5445c0d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:22:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,8b12285b94afff97,ada6fb2b2af37767] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,8b12285b94afff97,ada6fb2b2af37767] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:23.448 INFO 2981023 --- [ocp-async-9,2353b742a734a041,d1a8ad41faad813c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:24.218 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:24.218 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:24.480 INFO 2981023 --- [ocp-schedules-8,6ee49312e91eb5fc,0df934f30cf2a664] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:22:25.448 INFO 2981023 --- [ocp-async-5,c90413ebd800b27b,f98ae55ae168f981] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:25.569 INFO 2981023 --- [ocp-schedules-25,bfb5c90aa17438f8,c4b602741e6895cd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:22:25.613 WARN 2981023 --- [scanner-13656533-1,bfb5c90aa17438f8,1796204d596d8dcf] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:22:25.620 INFO 2981023 --- [alarm-detect-8,bfb5c90aa17438f8,49d7e89db11db890] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050120 ~ 1726050135 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050135, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050135, value=98.929)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.929, values={1726050135=98.929}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050135=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:25.646 INFO 2981023 --- [alarm-detect-7,bfb5c90aa17438f8,5267ad2fb6e58952] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050120 ~ 1726050135 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050135, value=98.929)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.929, values={1726050135=98.929}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:25.676 INFO 2981023 --- [alarm-detect-13,bfb5c90aa17438f8,c637e493ab6ccc3b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050120 ~ 1726050135 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050135, value=98.929)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.929, values={1726050135=98.929}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,23848208f9e435e6,6f838ced31b5dc94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,23848208f9e435e6,6f838ced31b5dc94] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:22:26.351968+08:00, lastProcessTime=2024-09-11T18:22:16.343470+08:00 2024-09-11 18:22:26.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:22:26.351968+08:00 2024-09-11 18:22:27.208 INFO 2981023 --- [ocp-schedules-28,0a3fb7239389b1ef,2be4a31e56d846ba] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:22:27.208 INFO 2981023 --- [ocp-schedules-28,0a3fb7239389b1ef,2be4a31e56d846ba] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:22:27.208 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:22:27.208 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:22:27.208 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:22:27.212 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:22:27.214 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:27.214 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:22:27.215 INFO 2981023 --- [ocp-schedules-28,0a3fb7239389b1ef,2be4a31e56d846ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:27.217 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:27.217 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:22:27.218 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:22:27.218 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:27.218 INFO 2981023 --- [ocp-schedules-28,0a3fb7239389b1ef,2be4a31e56d846ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:27.221 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:27.246 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:22:27.248 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:22:27.248 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:22:27.249 INFO 2981023 --- [ocp-schedules-28,0a3fb7239389b1ef,2be4a31e56d846ba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:22:27.254 INFO 2981023 --- [ocp-schedules-28,0a3fb7239389b1ef,2be4a31e56d846ba] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:22:27.448 INFO 2981023 --- [ocp-async-8,b25f6992cce8b54c,4e01c515421e6244] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:27.792 INFO 2981023 --- [ocp-async-9,0bd944a7b619604c,92168c32a2215cec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:22:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,e4656ffbecaffd9d,1a66683218e2b218] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,e4656ffbecaffd9d,1a66683218e2b218] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:29.449 INFO 2981023 --- [ocp-async-2,103e3ffc28fdc998,33c5577d79bef41d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:30.221 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:30.222 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:31.449 INFO 2981023 --- [ocp-async-9,18f0533ac216e85a,5b693fdfd9186306] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,90c50d5677711243,e77f5d10c1b1e1fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,90c50d5677711243,e77f5d10c1b1e1fc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:32.763 INFO 2981023 --- [ocp-schedules-10,875b5fd2c64bccee,96aae34454d73083] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:22:33.222 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:33.223 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:33.449 INFO 2981023 --- [ocp-async-4,27302d27ecf283ab,d4009658bf2796ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,efb237bd5b3fcaf7,a246d779309027ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,efb237bd5b3fcaf7,a246d779309027ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:35.449 INFO 2981023 --- [ocp-async-6,ac7f676b40f451a1,62c3c99968249d1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:35.650 INFO 2981023 --- [ocp-schedules-10,2c3b8c597598eb0d,e414a3af20520d09] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:22:35.659 INFO 2981023 --- [ocp-schedules-10,2c3b8c597598eb0d,e414a3af20520d09] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:22:35.659 INFO 2981023 --- [ocp-schedules-10,2c3b8c597598eb0d,e414a3af20520d09] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:22:35.764 INFO 2981023 --- [ocp-schedules-14,87e87fdc21c7b3e5,f545dcb97ae5fb0e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:22:35.818 WARN 2981023 --- [scanner-13656775-1,87e87fdc21c7b3e5,7c00385d8274628a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:22:35.841 INFO 2981023 --- [alarm-detect-6,87e87fdc21c7b3e5,16ea8035b8d0ea80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646107, data is empty. 2024-09-11 18:22:35.844 INFO 2981023 --- [alarm-detect-6,87e87fdc21c7b3e5,16ea8035b8d0ea80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646107, data is empty. 2024-09-11 18:22:35.847 INFO 2981023 --- [alarm-detect-6,87e87fdc21c7b3e5,16ea8035b8d0ea80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646107, data is empty. 2024-09-11 18:22:35.870 INFO 2981023 --- [alarm-detect-8,87e87fdc21c7b3e5,733c0b2a02a3d6f5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646107, data is empty. 2024-09-11 18:22:35.871 INFO 2981023 --- [alarm-detect-15,87e87fdc21c7b3e5,c0de42da2779360c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646107, data is empty. 2024-09-11 18:22:35.874 INFO 2981023 --- [alarm-detect-8,87e87fdc21c7b3e5,733c0b2a02a3d6f5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646107, data is empty. 2024-09-11 18:22:35.875 INFO 2981023 --- [alarm-detect-15,87e87fdc21c7b3e5,c0de42da2779360c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646107, data is empty. 2024-09-11 18:22:35.877 INFO 2981023 --- [alarm-detect-8,87e87fdc21c7b3e5,733c0b2a02a3d6f5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646107, data is empty. 2024-09-11 18:22:35.879 INFO 2981023 --- [alarm-detect-15,87e87fdc21c7b3e5,c0de42da2779360c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646107, data is empty. 2024-09-11 18:22:35.902 INFO 2981023 --- [alarm-detect-14,87e87fdc21c7b3e5,02fee7a2e52dc932] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050120 ~ 1726050145 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050135, value=7.432)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.432, values={1726050135=7.432}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:35.903 INFO 2981023 --- [alarm-detect-14,87e87fdc21c7b3e5,02fee7a2e52dc932] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050120 ~ 1726050145 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050135, value=7.432)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.432, values={1726050135=7.432}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:22:35.986 INFO 2981023 --- [alarm-detect-1,87e87fdc21c7b3e5,484ccfdf04074f92] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646107, data is empty. 2024-09-11 18:22:35.988 INFO 2981023 --- [alarm-detect-1,87e87fdc21c7b3e5,484ccfdf04074f92] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646107, data is empty. 2024-09-11 18:22:35.992 INFO 2981023 --- [alarm-detect-1,87e87fdc21c7b3e5,484ccfdf04074f92] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646107, data is empty. 2024-09-11 18:22:36.015 INFO 2981023 --- [alarm-detect-17,87e87fdc21c7b3e5,badb6459c3a61825] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050120 ~ 1726050145 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050135, value=104.357)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.357, values={1726050135=104.357}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:36.223 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:36.224 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:22:36.336569+08:00, lastProcessTime=2024-09-11T18:22:26.351968+08:00 2024-09-11 18:22:36.470 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:22:36.336569+08:00 2024-09-11 18:22:37.314 INFO 2981023 --- [prometheus-cache-monitor1,7279c3953c880207,91b5599d056dae4f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8192,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:22:37.315 INFO 2981023 --- [prometheus-cache-monitor1,7279c3953c880207,91b5599d056dae4f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:22:37.449 INFO 2981023 --- [ocp-async-7,fd369196b270ec86,1cef68131a9c4454] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:37.792 INFO 2981023 --- [ocp-async-4,8571c2d8b3b2260b,3c60f86c540da5c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:22:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,ab60055609034f94,2fdcd539b0d39162] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,ab60055609034f94,2fdcd539b0d39162] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:39.224 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:39.224 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:39.449 INFO 2981023 --- [ocp-async-8,34d893f546a20648,d9a0285678a30816] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:40.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:22:40.004 INFO 2981023 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:22:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,85fd7f9fa354be85,f825fedad9ed6343] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,85fd7f9fa354be85,f825fedad9ed6343] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:41.449 INFO 2981023 --- [ocp-async-1,4aa79974eec210a9,8af4ecd81ee01bdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:42.225 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:42.225 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:42.766 INFO 2981023 --- [ocp-schedules-12,b2d747bdcf9d547b,fa67cb513dca918f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:22:43.449 INFO 2981023 --- [ocp-async-5,d345e788cb97dcda,4f0998782c0129ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,c0028cf3a839c5b7,d6feef0ab5b0a506] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,c0028cf3a839c5b7,d6feef0ab5b0a506] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:45.101 INFO 2981023 --- [prometheus-second-schedule-1,a4991ba704b0d4a2,556222f885179d0b] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:22:45.225 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:45.226 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:45.449 INFO 2981023 --- [ocp-async-10,710ce6573a13c254,30174740d557af3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:46.020 INFO 2981023 --- [ocp-schedules-19,6c5f4680c07d74e5,43b6b32786709c81] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:22:46.021 INFO 2981023 --- [ocp-schedules-26,989081451354a6dd,8f7f98c9f36cdabf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:22:46.024 INFO 2981023 --- [host-check-clockdiff2,6c5f4680c07d74e5,fc26e7fba3c85900] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:22:46.024 INFO 2981023 --- [host-check-clockdiff1,6c5f4680c07d74e5,12be1d8942470bed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:22:46.027 INFO 2981023 --- [host-check-clockdiff3,6c5f4680c07d74e5,2dcd67e03202ea06] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:22:46.028 INFO 2981023 --- [host-check-clockdiff4,6c5f4680c07d74e5,8d4e0fd0e5757bc2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:22:46.030 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:22:46.030 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:22:46.031 INFO 2981023 --- [ocp-schedules-12,66720e7a55598e8d,a38d7dbe7c6f373e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:22:46.031 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:22:46.031 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:22:46.040 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:22:46.040 INFO 2981023 --- [host-check-agent-status1,66720e7a55598e8d,fd7caa398b16e866] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:22:46.041 INFO 2981023 --- [host-check-agent-status2,66720e7a55598e8d,ac36ba56956ce54a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:22:46.041 INFO 2981023 --- [host-check-agent-status3,66720e7a55598e8d,3ed939d6a636b110] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:22:46.041 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:22:46.041 INFO 2981023 --- [host-check-agent-status4,66720e7a55598e8d,548a07164bcbaa4e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:22:46.042 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:22:46.044 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:22:46.047 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:46.047 INFO 2981023 --- [host-check-clockdiff2,6c5f4680c07d74e5,fc26e7fba3c85900] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050166 5 5 , error= 2024-09-11 18:22:46.047 INFO 2981023 --- [ocp-schedules-20,30a081bf1426ad48,c5eb7229f5ebbcce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:22:46.052 INFO 2981023 --- [host-check-agent-status1,66720e7a55598e8d,fd7caa398b16e866] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:46.053 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:46.054 INFO 2981023 --- [host-check-agent-status2,66720e7a55598e8d,ac36ba56956ce54a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:22:46.055 INFO 2981023 --- [host-check-agent-status4,66720e7a55598e8d,548a07164bcbaa4e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:22:46.056 INFO 2981023 --- [host-check-agent-status2,66720e7a55598e8d,ac36ba56956ce54a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:22:46.056 INFO 2981023 --- [host-check-agent-status1,66720e7a55598e8d,fd7caa398b16e866] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:46.058 INFO 2981023 --- [host-check-agent-status4,66720e7a55598e8d,548a07164bcbaa4e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:22:46.070 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:46.071 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:46.072 INFO 2981023 --- [host-check-agent-status3,66720e7a55598e8d,3ed939d6a636b110] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:46.073 INFO 2981023 --- [host-check-agent-status3,66720e7a55598e8d,3ed939d6a636b110] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:46.115 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:22:46.118 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:22:46.117440104+08:00 2024-09-11 18:22:46.118 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:22:46.118+08:00 2024-09-11 18:22:46.121 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:22:46.118+08:00 and status AVAILABLE 2024-09-11 18:22:46.124 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:22:46.128 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:22:46.130 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:22:46.132 INFO 2981023 --- [host-check-agent-status4,66720e7a55598e8d,548a07164bcbaa4e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:22:46.133 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:22:46.132532449+08:00 2024-09-11 18:22:46.134 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:22:46.133+08:00 2024-09-11 18:22:46.137 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:22:46.133+08:00 and status AVAILABLE 2024-09-11 18:22:46.137 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:22:46.137 INFO 2981023 --- [host-check-agent-status4,66720e7a55598e8d,548a07164bcbaa4e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:22:46.137 INFO 2981023 --- [host-check-agent-status4,66720e7a55598e8d,548a07164bcbaa4e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:22:46.137 INFO 2981023 --- [host-check-agent-status4,66720e7a55598e8d,548a07164bcbaa4e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:22:46.139 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:22:46.139221074+08:00 2024-09-11 18:22:46.145 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:22:46.139+08:00 2024-09-11 18:22:46.146 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:22:46.147 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:22:46.139+08:00 and status AVAILABLE 2024-09-11 18:22:46.154 INFO 2981023 --- [host-check-agent-status3,66720e7a55598e8d,3ed939d6a636b110] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:22:46.155 INFO 2981023 --- [host-check-agent-status2,66720e7a55598e8d,ac36ba56956ce54a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:22:46.158 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:22:46.161 INFO 2981023 --- [host-check-agent-status3,66720e7a55598e8d,3ed939d6a636b110] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:22:46.161 INFO 2981023 --- [host-check-agent-status3,66720e7a55598e8d,3ed939d6a636b110] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:22:46.161 INFO 2981023 --- [host-check-agent-status3,66720e7a55598e8d,3ed939d6a636b110] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:22:46.164 INFO 2981023 --- [host-check-agent-status2,66720e7a55598e8d,ac36ba56956ce54a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:22:46.164 INFO 2981023 --- [host-check-agent-status2,66720e7a55598e8d,ac36ba56956ce54a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:22:46.164 INFO 2981023 --- [host-check-agent-status2,66720e7a55598e8d,ac36ba56956ce54a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:22:46.165 INFO 2981023 --- [ocp-schedules-23,264d9e3cc7c6295d,f912d2de6eebc078] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:22:46.177 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:22:46.191 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:46.191 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:46.192 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:46.192 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:46.193 INFO 2981023 --- [ocp-schedules-17,992ff73f963d8cc5,6a40a3a02b64a5e9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:22:46.193 INFO 2981023 --- [ocp-schedules-15,53a65f3c06258dea,905fe9ace29985ae] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:22:46.193 INFO 2981023 --- [ocp-schedules-24,5a19c02254bc9740,02a0d875299e12b5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:22:46.193 INFO 2981023 --- [ocp-schedules-29,6c683e5288e0cca3,e975de19d74d13df] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:22:46.198 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:22:46.201 INFO 2981023 --- [ocp-schedules-24,5a19c02254bc9740,02a0d875299e12b5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:46.201 INFO 2981023 --- [ocp-schedules-17,992ff73f963d8cc5,6a40a3a02b64a5e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:46.201 INFO 2981023 --- [ocp-schedules-15,53a65f3c06258dea,905fe9ace29985ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:46.202 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:22:46.216 INFO 2981023 --- [ocp-schedules-23,264d9e3cc7c6295d,f912d2de6eebc078] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:22:46.225 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:22:46.231 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:22:46.238 INFO 2981023 --- [ocp-schedules-27,d00b696f393fd59f,2fe657c8d3e0a17e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:22:46.240 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:22:46.264 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:22:46.276 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:22:46.285 INFO 2981023 --- [host-check-agent-status1,66720e7a55598e8d,fd7caa398b16e866] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:22:46.285 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:22:46.296 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:22:46.300 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.301 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:22:46.306 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:22:46.307 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:22:46.311 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:22:46.312 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.316 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:22:46.327 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:22:46.329 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:22:46.334 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:22:46.340 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:22:46.339665+08:00, lastProcessTime=2024-09-11T18:22:36.336569+08:00 2024-09-11 18:22:46.367 INFO 2981023 --- [host-check-clockdiff3,6c5f4680c07d74e5,2dcd67e03202ea06] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050166 0 0 , error= 2024-09-11 18:22:46.372 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:22:46.372 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:22:46.376348680+08:00 2024-09-11 18:22:46.375 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:22:46.372+08:00 2024-09-11 18:22:46.376 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:22:46.372+08:00 and status AVAILABLE 2024-09-11 18:22:46.379 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:22:46.379 WARN 2981023 --- [scanner-13657113-1,30a081bf1426ad48,5814225c0c7e0237] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:22:46.382 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.385 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:22:46.385 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:22:46.393 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.397 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:22:46.398 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:22:46.399 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:22:46.401 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:22:46.407 INFO 2981023 --- [host-check-agent-status1,66720e7a55598e8d,fd7caa398b16e866] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:22:46.407 INFO 2981023 --- [host-check-agent-status1,66720e7a55598e8d,fd7caa398b16e866] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:22:46.407 INFO 2981023 --- [host-check-agent-status1,66720e7a55598e8d,fd7caa398b16e866] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:22:46.415 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:22:46.416 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.417 INFO 2981023 --- [host-check-host-status4,989081451354a6dd,9263092d13adfe8a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:22:46.422 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:22:46.427 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:22:46.433 INFO 2981023 --- [alarm-detect-12,30a081bf1426ad48,ac04cce9535f92b1] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050135 ~ 1726050156 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050150, value=98.543)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.543, values={1726050150=98.543}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:46.437 INFO 2981023 --- [alarm-detect-2,30a081bf1426ad48,a06ef2718af0ca65] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050135 ~ 1726050156 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050150, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050150, value=98.543)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.543, values={1726050150=98.543}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050150=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:46.444 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:22:46.453 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.453 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:22:46.454 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:22:46.455 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:22:46.464 INFO 2981023 --- [alarm-detect-15,30a081bf1426ad48,5e05144ec71f13b2] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050135 ~ 1726050156 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050150, value=7.73)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.73, values={1726050150=7.73}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:46.464 INFO 2981023 --- [alarm-detect-15,30a081bf1426ad48,5e05144ec71f13b2] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050135 ~ 1726050156 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050150, value=7.73)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.73, values={1726050150=7.73}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:22:46.466 INFO 2981023 --- [alarm-detect-1,30a081bf1426ad48,7c42527691579687] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050135 ~ 1726050156 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050150, value=98.543)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.543, values={1726050150=98.543}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:46.466 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:22:46.472 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:22:46.478 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050123170000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:22:46.487 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:22:46.491 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:22:46.504 INFO 2981023 --- [ocp-schedules-15,53a65f3c06258dea,905fe9ace29985ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:22:46.517 INFO 2981023 --- [host-check-host-status3,989081451354a6dd,e40b2778e92254bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:22:46.533 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.537 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.538 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:46.551 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:22:46.555 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.560 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:22:46.564 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:22:46.565 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.567 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:22:46.571 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.571 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:22:46.572 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:22:46.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.572 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.573 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:46.576 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:22:46.577 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:22:46.577 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.583 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:22:46.586 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.586 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:46.602 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:22:46.621 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:22:46.624 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:22:46.625 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:22:46.625 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.627 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.627 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:22:46.650 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:22:46.650 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050123170000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:22:46.655 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:22:46.658 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.659 INFO 2981023 --- [ocp-schedules-17,992ff73f963d8cc5,6a40a3a02b64a5e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:22:46.661 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.662 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:22:46.663 INFO 2981023 --- [alarm-detect-14,30a081bf1426ad48,d80649c1b7bedec4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050135 ~ 1726050156 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050150, value=103.833)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=103.833, values={1726050150=103.833}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:46.664 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:22:46.664 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.672 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.673 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.673 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:46.674 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:22:46.674 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:22:46.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:22:46.683 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:22:46.688 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:22:46.693 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:22:46.693 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:22:46.711 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:22:46.716 INFO 2981023 --- [host-check-host-status2,989081451354a6dd,7866c128f2c4d095] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:22:46.724 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:22:46.745 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.746 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.746 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:22:46.747 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.747 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:22:46.750 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:22:46.753 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:22:46.754 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.754 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.754 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:22:46.754 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.756 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:22:46.339665+08:00 2024-09-11 18:22:46.758 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:22:46.758 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:22:46.765 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:22:46.771 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:22:46.771 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.787 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.794 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.799 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:22:46.803 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:22:46.810 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.818 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:22:46.821 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:22:46.824 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:22:46.828 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:22:46.828 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:22:46.830 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:22:46.832 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:22:46.839 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:22:46.847 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.847 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.847 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:22:46.849 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:22:46.853 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:22:46.854 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:46.854 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:46.855 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:22:46.857 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:22:46.861 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:22:46.861 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:22:46.861 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:22:46.864 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:22:46.864 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:22:46.866 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:22:46.867 INFO 2981023 --- [ocp-schedules-13,7982242f30c07e22,d6b48b924621dcbe] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:22:46.867 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:22:46.880 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:22:47.032 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:22:47.036 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:22:47.040 INFO 2981023 --- [host-check-host-status1,989081451354a6dd,396adee29850e195] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:22:47.044 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:47.052 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:47.056 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:22:47.065 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:22:47.065 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:22:47.065 INFO 2981023 --- [ocp-schedules-24,5a19c02254bc9740,02a0d875299e12b5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:22:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,64d6af36f35fc68c,be85baa24ea2c528] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,64d6af36f35fc68c,be85baa24ea2c528] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:47.408 INFO 2981023 --- [ocp-schedules-23,35c9818e067cac0c,9ff75dae32bf2494] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:22:47.426 WARN 2981023 --- [ocp-schedules-23,35c9818e067cac0c,9ff75dae32bf2494] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:22:47.449 INFO 2981023 --- [ocp-async-6,13cc5b09e141ec75,ddd7662874fd5a4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:47.484 INFO 2981023 --- [ocp-schedules-22,98e81bb853252e5c,bc8c4eb4da3b13ce] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-09-11 18:22:47.484 INFO 2981023 --- [ocp-schedules-22,98e81bb853252e5c,bc8c4eb4da3b13ce] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-09-11 18:22:47.792 INFO 2981023 --- [ocp-async-10,e84f8d802f7954c2,7cbf8fa416f734a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:22:48.226 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:48.227 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:49.450 INFO 2981023 --- [ocp-async-5,e1626872d6588ecc,746b5c62685758cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:49.628 INFO 2981023 --- [ocp-schedules-16,4c1490605d325304,fa1c9493fc68b63e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-09-11 18:22:49.629 INFO 2981023 --- [ocp-schedules-16,4c1490605d325304,fa1c9493fc68b63e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:49.666 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:49.667 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:49.669 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 02:03:06.281634, 2024-09-11 18:22:49.649, 1000] 2024-09-11 18:22:49.669 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 18:00:06.461339, 2024-09-11 18:22:49.634, 1000] 2024-09-11 18:22:49.670 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000006, startTime 2024-09-11 18:00:06.461339, endTime 2024-09-11 18:22:49.634 2024-09-11 18:22:49.670 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-09-11 02:03:06.281634, endTime 2024-09-11 18:22:49.649 2024-09-11 18:22:49.673 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:49.680 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 10:40:18.658505, 2024-09-11 18:22:49.649, 1000] 2024-09-11 18:22:49.684 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2024-09-11 10:40:18.658505, endTime 2024-09-11 18:22:49.649 2024-09-11 18:22:49.687 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:49.690 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:49.711 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 18:12:34.59165, 2024-09-11 18:22:49.677, 1000] 2024-09-11 18:22:49.711 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000006, startTime 2024-09-11 18:12:34.59165, endTime 2024-09-11 18:22:49.677 2024-09-11 18:22:49.717 INFO 2981023 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:49.717 INFO 2981023 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:22:49.717 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:49.718 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:49.735 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 08:00:10.321716, 2024-09-11 18:22:49.711, 1000] 2024-09-11 18:22:49.736 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-09-11 08:00:10.321716, endTime 2024-09-11 18:22:49.711 2024-09-11 18:22:49.737 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 18:01:30.332457, 2024-09-11 18:22:49.711, 1000] 2024-09-11 18:22:49.745 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2024-09-11 18:01:30.332457, endTime 2024-09-11 18:22:49.711 2024-09-11 18:22:49.745 INFO 2981023 --- [ocp-schedules-16,4c1490605d325304,fa1c9493fc68b63e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, tryCount=3, successCount=3 2024-09-11 18:22:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:22:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:22:50.168 INFO 2981023 --- [host-check-clockdiff1,6c5f4680c07d74e5,12be1d8942470bed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050170 0 0 , error= 2024-09-11 18:22:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,0a71740bb5fd69e4,be2fc9007af01297] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,0a71740bb5fd69e4,be2fc9007af01297] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:50.361 INFO 2981023 --- [ocp-schedules-27,4ca392d3e7d4165b,71e8b0b59fc04503] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:22:51.227 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:51.229 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:51.450 INFO 2981023 --- [ocp-async-10,3b29c897bde5cecb,77d3c7c707e36352] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:51.635 INFO 2981023 --- [ocp-schedules-30,54c21737d0ccd3db,92d7d78fa50d653e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:22:51.636 INFO 2981023 --- [ocp-schedules-30,54c21737d0ccd3db,92d7d78fa50d653e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:22:51.637 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:22:51.637 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:22:51.637 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:22:51.637 INFO 2981023 --- [ocp-schedules-30,54c21737d0ccd3db,92d7d78fa50d653e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:22:52.015 INFO 2981023 --- [host-check-clockdiff4,6c5f4680c07d74e5,8d4e0fd0e5757bc2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050172 0 0 , error= 2024-09-11 18:22:52.768 INFO 2981023 --- [ocp-schedules-17,b6114593c54e6238,787edb6952736d8f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:22:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,b07093e0b1b2dd32,f5378624c719c0a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,b07093e0b1b2dd32,f5378624c719c0a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:53.360 INFO 2981023 --- [ocp-schedules-20,128703f7eac93d55,96712e4e079a76b6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:22:53.360 INFO 2981023 --- [ocp-schedules-20,128703f7eac93d55,96712e4e079a76b6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:22:53.450 INFO 2981023 --- [ocp-async-1,0b31d45d7532c634,97e94df915101a98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:54.229 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:54.231 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:54.483 INFO 2981023 --- [ocp-schedules-17,2f66ec2bd3c85a3d,a3f24e29fe97d09a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:22:55.174 INFO 2981023 --- [ocp-schedules-29,0d7214d53ea3ae20,2450ed848ed9af0b] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:22:55.198 WARN 2981023 --- [ocp-schedules-29,0d7214d53ea3ae20,2450ed848ed9af0b] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:22:55.450 INFO 2981023 --- [ocp-async-7,74cef060c33679b2,2e66b4d57d72dac5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:55.662 INFO 2981023 --- [ocp-schedules-16,d179ec95feb4b556,75a700c03acef878] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:22:55.671 INFO 2981023 --- [ocp-schedules-16,d179ec95feb4b556,75a700c03acef878] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:22:55.671 INFO 2981023 --- [ocp-schedules-16,d179ec95feb4b556,75a700c03acef878] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:22:55.904 INFO 2981023 --- [ocp-schedules-21,b7b26084c12cd158,f6361bcdeaafc41e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:22:55.904 INFO 2981023 --- [ocp-schedules-21,b7b26084c12cd158,f6361bcdeaafc41e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:22:55.905 INFO 2981023 --- [ocp-schedules-21,b7b26084c12cd158,f6361bcdeaafc41e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:22:55.905 INFO 2981023 --- [ocp-schedules-21,b7b26084c12cd158,f6361bcdeaafc41e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:22:55.905 INFO 2981023 --- [ocp-schedules-21,b7b26084c12cd158,f6361bcdeaafc41e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:22:55.905 INFO 2981023 --- [ocp-schedules-21,b7b26084c12cd158,f6361bcdeaafc41e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:22:56.115 INFO 2981023 --- [ocp-schedules-17,5454b7eb881ff2d1,68a3f5c10c584eba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:22:56.122 INFO 2981023 --- [ocp-schedules-17,5454b7eb881ff2d1,68a3f5c10c584eba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:22:56.122 INFO 2981023 --- [ocp-schedules-17,5454b7eb881ff2d1,68a3f5c10c584eba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:22:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,a05b8da70e74604e,91ab52b10ef6aa7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,a05b8da70e74604e,91ab52b10ef6aa7d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:22:56.353614+08:00, lastProcessTime=2024-09-11T18:22:46.339665+08:00 2024-09-11 18:22:56.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:22:56.353614+08:00 2024-09-11 18:22:56.746 INFO 2981023 --- [ocp-schedules-24,f5fcbe2474507abc,d5104ec1960dc732] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:22:56.795 INFO 2981023 --- [alarm-detect-5,f5fcbe2474507abc,ee65d2607521c0a0] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050150 ~ 1726050166 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050165, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050165, value=98.384)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.384, values={1726050165=98.384}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050165=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:56.801 WARN 2981023 --- [scanner-13657336-1,f5fcbe2474507abc,44a06937b349e1f8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:22:56.816 INFO 2981023 --- [alarm-detect-18,f5fcbe2474507abc,f4f44822d191788e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646107, data is empty. 2024-09-11 18:22:56.824 INFO 2981023 --- [alarm-detect-18,f5fcbe2474507abc,f4f44822d191788e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646107, data is empty. 2024-09-11 18:22:56.830 INFO 2981023 --- [alarm-detect-18,f5fcbe2474507abc,f4f44822d191788e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646107, data is empty. 2024-09-11 18:22:56.843 INFO 2981023 --- [alarm-detect-16,f5fcbe2474507abc,df0725f1e5a49ed0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646107, data is empty. 2024-09-11 18:22:56.845 INFO 2981023 --- [alarm-detect-20,f5fcbe2474507abc,76296b28e7ea48f9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646107, data is empty. 2024-09-11 18:22:56.849 INFO 2981023 --- [alarm-detect-16,f5fcbe2474507abc,df0725f1e5a49ed0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646107, data is empty. 2024-09-11 18:22:56.849 INFO 2981023 --- [alarm-detect-20,f5fcbe2474507abc,76296b28e7ea48f9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646107, data is empty. 2024-09-11 18:22:56.855 INFO 2981023 --- [alarm-detect-20,f5fcbe2474507abc,76296b28e7ea48f9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646107, data is empty. 2024-09-11 18:22:56.855 INFO 2981023 --- [alarm-detect-16,f5fcbe2474507abc,df0725f1e5a49ed0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646107, data is empty. 2024-09-11 18:22:56.863 INFO 2981023 --- [alarm-detect-20,f5fcbe2474507abc,e698e5af41ad7003] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050150 ~ 1726050166 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050165, value=98.384)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.384, values={1726050165=98.384}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:56.867 INFO 2981023 --- [alarm-detect-18,f5fcbe2474507abc,8fd378ae28ae5309] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050150 ~ 1726050166 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050165, value=98.384)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.384, values={1726050165=98.384}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:22:56.982 INFO 2981023 --- [alarm-detect-17,f5fcbe2474507abc,ddfe217c2d187a0d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646107, data is empty. 2024-09-11 18:22:56.988 INFO 2981023 --- [alarm-detect-17,f5fcbe2474507abc,ddfe217c2d187a0d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646107, data is empty. 2024-09-11 18:22:56.991 INFO 2981023 --- [alarm-detect-17,f5fcbe2474507abc,ddfe217c2d187a0d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646107, data is empty. 2024-09-11 18:22:57.231 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:22:57.231 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:22:57.450 INFO 2981023 --- [ocp-async-5,f16248f1cd411908,c11ef80b2a52d3d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:57.792 INFO 2981023 --- [ocp-async-9,1924d3b0ac3cc3a9,15d55cdf0c5fe2e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:22:58.492 INFO 2981023 --- [ocp-schedules-6,17b9ce161197317e,99cde0b1f5117008] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:22:58.492 INFO 2981023 --- [ocp-schedules-6,17b9ce161197317e,99cde0b1f5117008] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:22:58.523 INFO 2981023 --- [Collect all tenants major compactions1,17b9ce161197317e,451265f23ae51a50] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:22:58.524 INFO 2981023 --- [Collect all tenants major compactions4,17b9ce161197317e,960aafa838060088] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:22:58.526 INFO 2981023 --- [Collect all tenants major compactions5,17b9ce161197317e,f3c34267ea550b0b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:22:58.526 INFO 2981023 --- [Collect all tenants major compactions6,17b9ce161197317e,9c9f0303e64bc7ee] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:22:58.529 INFO 2981023 --- [Collect all tenants major compactions3,17b9ce161197317e,a5ae4762a5122134] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:22:58.535 INFO 2981023 --- [Collect all tenants major compactions2,17b9ce161197317e,05194ad7a7d3ccb7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:22:58.546 INFO 2981023 --- [Collect all tenants major compactions1,17b9ce161197317e,451265f23ae51a50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:58.547 INFO 2981023 --- [Collect all tenants major compactions3,17b9ce161197317e,a5ae4762a5122134] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:58.548 INFO 2981023 --- [Collect all tenants major compactions3,17b9ce161197317e,a5ae4762a5122134] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:22:58.549 INFO 2981023 --- [Collect all tenants major compactions1,17b9ce161197317e,451265f23ae51a50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:22:58.551 INFO 2981023 --- [Collect all tenants major compactions6,17b9ce161197317e,9c9f0303e64bc7ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:58.551 INFO 2981023 --- [Collect all tenants major compactions2,17b9ce161197317e,05194ad7a7d3ccb7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:58.552 INFO 2981023 --- [Collect all tenants major compactions3,17b9ce161197317e,a5ae4762a5122134] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:22:58.552 INFO 2981023 --- [Collect all tenants major compactions1,17b9ce161197317e,451265f23ae51a50] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:22:58.552 INFO 2981023 --- [Collect all tenants major compactions1,17b9ce161197317e,451265f23ae51a50] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:22:58.552 INFO 2981023 --- [Collect all tenants major compactions3,17b9ce161197317e,a5ae4762a5122134] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:22:58.552 INFO 2981023 --- [Collect all tenants major compactions6,17b9ce161197317e,9c9f0303e64bc7ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:22:58.552 INFO 2981023 --- [Collect all tenants major compactions2,17b9ce161197317e,05194ad7a7d3ccb7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:22:58.554 INFO 2981023 --- [Collect all tenants major compactions2,17b9ce161197317e,05194ad7a7d3ccb7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:22:58.554 INFO 2981023 --- [Collect all tenants major compactions2,17b9ce161197317e,05194ad7a7d3ccb7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:22:58.554 INFO 2981023 --- [Collect all tenants major compactions4,17b9ce161197317e,960aafa838060088] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:58.554 INFO 2981023 --- [Collect all tenants major compactions6,17b9ce161197317e,9c9f0303e64bc7ee] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:22:58.554 INFO 2981023 --- [Collect all tenants major compactions6,17b9ce161197317e,9c9f0303e64bc7ee] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:22:58.559 INFO 2981023 --- [Collect all tenants major compactions5,17b9ce161197317e,f3c34267ea550b0b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:22:58.559 INFO 2981023 --- [Collect all tenants major compactions4,17b9ce161197317e,960aafa838060088] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:22:58.566 INFO 2981023 --- [Collect all tenants major compactions5,17b9ce161197317e,f3c34267ea550b0b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:22:58.566 INFO 2981023 --- [Collect all tenants major compactions4,17b9ce161197317e,960aafa838060088] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:22:58.566 INFO 2981023 --- [Collect all tenants major compactions4,17b9ce161197317e,960aafa838060088] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:22:58.571 INFO 2981023 --- [Collect all tenants major compactions5,17b9ce161197317e,f3c34267ea550b0b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:22:58.571 INFO 2981023 --- [Collect all tenants major compactions5,17b9ce161197317e,f3c34267ea550b0b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:22:58.571 INFO 2981023 --- [ocp-schedules-6,17b9ce161197317e,99cde0b1f5117008] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:22:59.148 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.152 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, o 2024-09-11 18:22:59.158 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.166 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.167 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.172 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.175 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.178 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.181 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.183 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.184 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.185 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.186 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.186 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.191 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.194 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.196 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.199 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.202 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.205 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.208 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.211 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.211 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.212 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.213 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.213 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:16:59.077, end=2024-09-11T18:21:59.145} 2024-09-11 18:22:59.213 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 18:22:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,8abae2aa9ba6aae3,f4177578468b42f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050180, interval=60 2024-09-11 18:22:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,8abae2aa9ba6aae3,f4177578468b42f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:22:59.450 INFO 2981023 --- [ocp-async-2,8d2145be8312df3e,acb382e7b6f910ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:22:59.993 INFO 2981023 --- [ocp-schedules-6,614fa802021c7156,09edfd0420768d4c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:22:59.993+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:22:59.996 INFO 2981023 --- [ocp-schedules-6,614fa802021c7156,09edfd0420768d4c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:23:00.160 INFO 2981023 --- [ocp-schedules-6,614fa802021c7156,09edfd0420768d4c] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 60 to SqlAggregateData 2024-09-11 18:23:00.162 INFO 2981023 --- [ocp-schedules-6,614fa802021c7156,09edfd0420768d4c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:23:00.231 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:00.232 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:01.450 INFO 2981023 --- [ocp-async-9,8dc47602896b74f2,fe6e5e839ee9f7e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:01.596 INFO 2981023 --- [ocp-schedules-6,09e3df90f6ce8c2e,ca25d65ca4a73a4a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:23:01.597 INFO 2981023 --- [ocp-schedules-6,09e3df90f6ce8c2e,ca25d65ca4a73a4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:23:01.624 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:23:01.624 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.627 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:23:01.627 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:23:01.627 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.629 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:23:01.629 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.630 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:23:01.632 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:23:01.634 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.634 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.635 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.636 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.638 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.639 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.641 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.642 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.644 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.645 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:23:01.649 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.649 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.653 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.654 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:23:01.654 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.659 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.659 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:23:01.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.662 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.663 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.666 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:23:01.675 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.675 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:23:01.676 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.676 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:23:01.677 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.677 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.677 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:23:01.678 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:23:01.678 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.678 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:23:01.678 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.678 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:23:01.679 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.679 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.679 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:23:01.680 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.680 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:23:01.680 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5DE1D6FD-0-0] 2024-09-11 18:23:01.680 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:23:01.680 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:23:01.681 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.681 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:23:01.681 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:23:01.682 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.682 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:23:01.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:23:01.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:23:01.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:23:01.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:23:01.687 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:23:01.696 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:23:01.696 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:01.696 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.697 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:23:01.697 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.698 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.707 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:01.708 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.709 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:23:01.710 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:23:01.713 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:23:01.713 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:23:01.713 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:23:01.714 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:23:01.714 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.715 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:23:01.715 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:23:01.715 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.715 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:23:01.720 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:23:01.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:23:01.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.721 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:23:01.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.722 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE5E61D6CE-0-0] 2024-09-11 18:23:01.722 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:23:01.728 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:23:01.728 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:23:01.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.730 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:23:01.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:23:01.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:23:01.730 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:23:01.731 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.731 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:23:01.746 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.747 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:23:01.749 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.756 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:23:01.777 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.778 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:23:01.780 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:23:01.782 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:23:01.792 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:01.797 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.798 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:23:01.810 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:23:01.812 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:23:01.812 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:01.816 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:23:01.853 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:23:01.855 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:23:01.855 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:23:01.855 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:23:01.855 INFO 2981023 --- [ocp-schedules-6,09e3df90f6ce8c2e,ca25d65ca4a73a4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:23:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,50d108a6dc4503eb,0b03325065621117] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,50d108a6dc4503eb,0b03325065621117] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:02.771 INFO 2981023 --- [ocp-schedules-17,f3bb0529bef26c46,cd519e5d298eb0cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:23:03.232 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:03.233 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:03.450 INFO 2981023 --- [ocp-async-8,a752992ebe8fecf8,561b0542dd6f5e30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,672949834b7be6e8,2a2e4b3ff531735a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,672949834b7be6e8,2a2e4b3ff531735a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:05.451 INFO 2981023 --- [ocp-async-6,02e2b98d1eb851de,5e007a336ffa5f58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:05.594 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2d847dcb3be16890,39246c2330ef2330] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=2d847dcb3be16890, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:23:05.594 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2d847dcb3be16890,39246c2330ef2330] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=2d847dcb3be16890, duration=4 ms] 2024-09-11 18:23:05.601 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8fba658d843f255b,e827c2af6483f3ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=8fba658d843f255b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:23:05.601 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4c87a689618d13a6,1b73179210909fcf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=4c87a689618d13a6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:23:05.602 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,4c87a689618d13a6,1b73179210909fcf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=4c87a689618d13a6, duration=3 ms] 2024-09-11 18:23:05.608 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,8fba658d843f255b,e827c2af6483f3ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=8fba658d843f255b, duration=10 ms] 2024-09-11 18:23:06.233 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:06.233 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:23:06.336140+08:00, lastProcessTime=2024-09-11T18:22:56.353614+08:00 2024-09-11 18:23:06.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:23:06.336140+08:00 2024-09-11 18:23:07.024 INFO 2981023 --- [ocp-schedules-10,13d0d3f2a3b3a3e5,6cb546fc5d961876] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:23:07.065 WARN 2981023 --- [scanner-13657635-1,13d0d3f2a3b3a3e5,ffbbae6f2dfd4b51] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:23:07.118 INFO 2981023 --- [alarm-detect-1,13d0d3f2a3b3a3e5,16493e070eb9a134] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050150 ~ 1726050177 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050165, value=7.531)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.531, values={1726050165=7.531}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:07.118 INFO 2981023 --- [alarm-detect-1,13d0d3f2a3b3a3e5,16493e070eb9a134] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050150 ~ 1726050177 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050165, value=7.531)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.531, values={1726050165=7.531}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:23:07.204 INFO 2981023 --- [alarm-detect-6,13d0d3f2a3b3a3e5,a54651164a7db7c0] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050150 ~ 1726050177 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050165, value=105.142)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.142, values={1726050165=105.142}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:07.451 INFO 2981023 --- [ocp-async-3,025a6b2a28ac3c21,bc808fbc28015e0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:07.792 INFO 2981023 --- [ocp-async-8,edc3942f590ce877,f47d6b5401bce08f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:23:08.133 INFO 2981023 --- [ocp-schedules-17,f18753bd22d1bad8,5818fa27c3fe8476] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:23:08.211 INFO 2981023 --- [ocp-schedules-11,884be697ea7278fb,dda5711adacd8b65] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:23:08.211+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:23:08.212 INFO 2981023 --- [ocp-schedules-11,884be697ea7278fb,dda5711adacd8b65] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:23:08.214 INFO 2981023 --- [ocp-schedules-11,884be697ea7278fb,dda5711adacd8b65] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:23:08.214 INFO 2981023 --- [ocp-schedules-11,884be697ea7278fb,dda5711adacd8b65] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:23:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,257ffe9d4484acaf,3d65f63f3ca5174f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,257ffe9d4484acaf,3d65f63f3ca5174f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:09.233 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:09.234 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:09.348 INFO 2981023 --- [ocp-schedules-4,1f3d37e43a1f9065,3cf8dbde02987a44] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:23:09.451 INFO 2981023 --- [ocp-async-5,b0acf550828564d9,190d1388c1c7b435] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:10.003 INFO 2981023 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:23:10.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:23:10.889 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,6c162b7a2e9d3eb4,354d55e64a78a19b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=6c162b7a2e9d3eb4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:23:10.890 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,6c162b7a2e9d3eb4,354d55e64a78a19b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=6c162b7a2e9d3eb4, duration=3 ms] 2024-09-11 18:23:10.897 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e03ef913bd19dc9c,fd3decae13d970ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=e03ef913bd19dc9c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:23:10.898 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,66b3e110c01a7301,15532767f1c886c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=66b3e110c01a7301, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:23:10.899 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,66b3e110c01a7301,15532767f1c886c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=66b3e110c01a7301, duration=3 ms] 2024-09-11 18:23:10.902 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e03ef913bd19dc9c,fd3decae13d970ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=e03ef913bd19dc9c, duration=7 ms] 2024-09-11 18:23:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:23:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:23:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,867edbefb523a99b,d478ecea939a94d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,867edbefb523a99b,d478ecea939a94d1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:11.451 INFO 2981023 --- [ocp-async-8,b571c6dd2b332ccf,bba570d5aa86c246] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:11.966 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:19:11.404, end=2024-09-11T18:21:11.963} 2024-09-11 18:23:11.969 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterI 2024-09-11 18:23:11.970 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:23:11.972 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:19:11.404, end=2024-09-11T18:21:11.963},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:23:11.972 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:23:11.972 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:23:11.983 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:23:12.006 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792006000, endTimeUs=1726050071963000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=BA36471C4B4CB3F1E4A9F3D16634B586, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T10:23:12.006Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000029, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000010, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 18:23:12.011 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 5 2024-09-11 18:23:12.011 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.019 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.019 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=BA36471C4B4CB3F1E4A9F3D16634B586, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.022 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.022 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.030 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.030 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=BA36471C4B4CB3F1E4A9F3D16634B586, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.032 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.037 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.006, end=2024-09-11T18:21:11.963}, clusterId=1000006, tenantId=1000006, sqlId=BA36471C4B4CB3F1E4A9F3D16634B586, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.037 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:23:12.038 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:19:11.404, end=2024-09-11T18:21:11.963},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:23:12.038 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:23:12.038 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:23:12.040 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:23:12.041 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:19:11.404, end=2024-09-11T18:21:11.963},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:23:12.041 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:23:12.041 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:23:12.051 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:23:12.067 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792067000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.067Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.069 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:23:12.069 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.074 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.074 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.076 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.076 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.081 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.081 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.082 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:23:12.082 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.067, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.083 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:23:12.085 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:19:11.404, end=2024-09-11T18:21:11.963},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:23:12.085 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:23:12.085 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:23:12.127 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792127000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.127Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.129 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792129000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.129Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.131 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792131000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.131Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.131 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792131000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.131Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.133 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792133000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.133Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.133 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792133000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.133Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.145 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792145000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.145Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.145 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792145000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=8A1B851989150B802AEEACA99152D145, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.145Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.177 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:23:12.177 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.177 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:23:12.177 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.184 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.184 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.186 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.186 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.186 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 69 2024-09-11 18:23:12.186 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.191 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.191 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.192 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.192 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.194 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.194 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.200 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.200 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.200 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.200 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.200 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.200 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.203 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 29 2024-09-11 18:23:12.203 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.205 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.205 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.206 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.206 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.209 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.209 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.210 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.210 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.217 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.217 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.217 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.217 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.218 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.218 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.219 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.219 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.219 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.219 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.219 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.219 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.220 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.220 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.222 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.222 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.232 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.232 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.233 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.233 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.234 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.234 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.234 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.234 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.235 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:12.236 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:12.236 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.236 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.236 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.237 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.237 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.237 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.238 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.238 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.239 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.239 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.239 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.240 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.240 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.241 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.244 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.249 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.249 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.129, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.249 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.133, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.249 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.131, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.251 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.131, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.251 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.133, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.252 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.127, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.256 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.145, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.258 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.145, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=8A1B851989150B802AEEACA99152D145, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.284 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792284000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.284Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.287 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792287000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.287Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.296 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792296000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.296Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.296 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:23:12.296 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.298 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792298000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1ED31157F7096F47941EA2233AA4101C, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.298Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.302 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.302 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.308 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.308 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.310 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792310000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.310Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.311 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792311000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.311Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.312 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.312 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.313 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792313000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.313Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.318 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.318 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.318 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963792318000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:23:12.318Z, endTime=2024-09-11T10:21:11.963Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:23:12.319 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.319 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.321 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.321 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.324 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.324 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.325 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.325 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.326 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.326 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.327 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.327 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.331 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.331 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.331 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.331 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.332 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.332 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.332 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.332 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.336 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.336 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.340 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:23:12.340 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:23:12.341 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.341 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.342 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.342 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.345 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.345 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.345 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.345 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.346 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.346 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.346 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.346 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.348 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.348 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.348 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.349 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.349 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.350 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.356 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.356 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.357 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.287, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.358 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.358 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.358 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.358 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.359 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.298, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=1ED31157F7096F47941EA2233AA4101C, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.360 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.284, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.360 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.296, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.361 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:23:12.361 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:23:12.361 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.361 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.364 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.364 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.364 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.367 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.370 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.31, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.370 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.370 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.371 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:23:12.371 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050071963000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:23:12.372 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.372 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.311, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.373 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:23:12.376 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.313, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.376 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:23:12.318, end=2024-09-11T18:21:11.963}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:23:12.377 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:23:12.377 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:19:11.404, end=2024-09-11T18:21:11.963},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:23:12.377 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:23:12.377 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:23:12.378 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:23:12.379 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:19:11.404, end=2024-09-11T18:21:11.963},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:23:12.379 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:23:12.379 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:23:12.380 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:23:12.424 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050120, nodeCount=480, affectRows=480 2024-09-11 18:23:12.458 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bd3405de25666ba7,af015c593a8e0c78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.31.33.41, traceId=bd3405de25666ba7, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2024-09-11 18:23:12.465 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a5fc608a221890ac,3329e3b60d52c36a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP&maxPoints=360, client=10.31.33.41, traceId=a5fc608a221890ac, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,] 2024-09-11 18:23:12.472 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,91508e53c665d822,f99613a9cdd832fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.31.33.41, traceId=91508e53c665d822, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2024-09-11 18:23:12.475 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bd3405de25666ba7,af015c593a8e0c78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=bd3405de25666ba7, duration=20 ms] 2024-09-11 18:23:12.478 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,a5fc608a221890ac,3329e3b60d52c36a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=a5fc608a221890ac, duration=17 ms] 2024-09-11 18:23:12.482 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,91508e53c665d822,f99613a9cdd832fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.31.33.41, traceId=91508e53c665d822, duration=21 ms] 2024-09-11 18:23:12.550 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,4478e9c7fdf185c1,93db82294434b55c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=4478e9c7fdf185c1, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 18:23:12.585 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,4478e9c7fdf185c1,93db82294434b55c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=4478e9c7fdf185c1, duration=38 ms] 2024-09-11 18:23:12.601 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4ea47e24ed73fa63,07e1e7cb7b906c52] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.31.33.41, traceId=4ea47e24ed73fa63, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2024-09-11 18:23:12.620 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,73fc5fcd39714dfa,1f24c07c03485d42] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2024-09-11T17%3A23%3A12%2B08%3A00&endTime=2024-09-11T18%3A23%3A12%2B08%3A00&maxPoints=360, client=10.31.33.41, traceId=73fc5fcd39714dfa, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2024-09-11T17:23:12+08:00,2024-09-11T18:23:12+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2024-09-11 18:23:12.636 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:12.638 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 18:23:12.643 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:12.644 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:12.646 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 18:23:12.646 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 18:23:12.669 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,73fc5fcd39714dfa,1f24c07c03485d42] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.31.33.41, traceId=73fc5fcd39714dfa, duration=51 ms] 2024-09-11 18:23:12.670 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4ea47e24ed73fa63,07e1e7cb7b906c52] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.31.33.41, traceId=4ea47e24ed73fa63, duration=72 ms] 2024-09-11 18:23:12.773 INFO 2981023 --- [ocp-schedules-25,c69b479a2ab34612,2bf719ef4ba5839b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:23:13.451 INFO 2981023 --- [ocp-async-7,88fa49dd2e73db88,e5c8d0ccb36aec62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:14.138 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050120, nodeCount=1000, affectRows=1000 2024-09-11 18:23:14.140 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050120, nodeCount=1000, affectRows=1000 2024-09-11 18:23:14.140 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050120, nodeCount=999, validCount=999 2024-09-11 18:23:14.142 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050120, nodeCount=1000, validCount=1000 2024-09-11 18:23:14.148 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050120, nodeCount=49, affectRows=49 2024-09-11 18:23:14.148 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050120, nodeCount=47, validCount=47 2024-09-11 18:23:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,ec03efba6a3efc16,3b97bd1f6c55eb4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,5f630e58393c5046,c20d18a173657f43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:23:14.318 INFO 2981023 --- [prometheus-minute-schedule-1,ec03efba6a3efc16,3b97bd1f6c55eb4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:14.530 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050120, nodeCount=480, validCount=480 2024-09-11 18:23:15.236 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:15.237 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:15.451 INFO 2981023 --- [ocp-async-4,f0537e8447438c99,254a608e16c07fe0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:15.674 INFO 2981023 --- [ocp-schedules-18,2dc56d9d6988533a,7bead876e1a92aed] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:23:15.682 INFO 2981023 --- [ocp-schedules-18,2dc56d9d6988533a,7bead876e1a92aed] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:23:15.682 INFO 2981023 --- [ocp-schedules-18,2dc56d9d6988533a,7bead876e1a92aed] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:23:16.016 INFO 2981023 --- [ocp-schedules-23,603e791d46b38c0a,9de5374a59ba9bbb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:23:16.021 INFO 2981023 --- [ocp-schedules-26,63536b8bcb85995a,141ed3a3af5cf058] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:23:16.023 INFO 2981023 --- [host-check-clockdiff1,603e791d46b38c0a,c57fa3bdb6899296] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:23:16.023 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:23:16.024 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:23:16.024 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:23:16.024 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:23:16.024 INFO 2981023 --- [host-check-clockdiff2,603e791d46b38c0a,c660cd07e5dcb871] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:23:16.025 INFO 2981023 --- [host-check-clockdiff3,603e791d46b38c0a,c81f3e673f8b3e12] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:23:16.027 INFO 2981023 --- [host-check-clockdiff4,603e791d46b38c0a,528d12f6d4538b75] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:23:16.037 INFO 2981023 --- [ocp-schedules-13,b7e64de9829ac133,d7b189adaa7afaa0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:23:16.045 INFO 2981023 --- [host-check-agent-status1,b7e64de9829ac133,c8211624962032f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:23:16.046 INFO 2981023 --- [host-check-agent-status2,b7e64de9829ac133,5d6e0ddeb86324d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:23:16.048 INFO 2981023 --- [host-check-agent-status3,b7e64de9829ac133,ec0756f9035a597f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:23:16.048 INFO 2981023 --- [host-check-agent-status4,b7e64de9829ac133,194e9ed2ace398c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:23:16.060 INFO 2981023 --- [host-check-clockdiff3,603e791d46b38c0a,c81f3e673f8b3e12] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050196 0 0 , error= 2024-09-11 18:23:16.062 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:23:16.062 INFO 2981023 --- [host-check-agent-status2,b7e64de9829ac133,5d6e0ddeb86324d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:23:16.063 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:23:16.067 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:23:16.068 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:23:16.070 INFO 2981023 --- [host-check-agent-status2,b7e64de9829ac133,5d6e0ddeb86324d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:23:16.071 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:23:16.072 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:23:16.073 INFO 2981023 --- [host-check-agent-status1,b7e64de9829ac133,c8211624962032f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:23:16.074 INFO 2981023 --- [host-check-agent-status3,b7e64de9829ac133,ec0756f9035a597f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:23:16.075 INFO 2981023 --- [host-check-agent-status4,b7e64de9829ac133,194e9ed2ace398c4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:23:16.075 INFO 2981023 --- [host-check-agent-status1,b7e64de9829ac133,c8211624962032f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:23:16.076 INFO 2981023 --- [host-check-agent-status3,b7e64de9829ac133,ec0756f9035a597f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:23:16.077 INFO 2981023 --- [host-check-agent-status4,b7e64de9829ac133,194e9ed2ace398c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:23:16.123 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:23:16.187 INFO 2981023 --- [ocp-schedules-28,e1398fc3dbf05d24,b065611c06b35be0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:23:16.187 INFO 2981023 --- [ocp-schedules-22,3364dbbaecc9ca74,2214f13058a7358b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:23:16.188 INFO 2981023 --- [ocp-schedules-30,3b6ef052e6bfd3cc,b324d1059988acae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:23:16.188 INFO 2981023 --- [ocp-schedules-9,e3063d07b89d5a9b,3b17e95dfd6835ab] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:23:16.189 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:23:16.190 INFO 2981023 --- [ocp-schedules-28,e1398fc3dbf05d24,b065611c06b35be0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:23:16.202 INFO 2981023 --- [ocp-schedules-30,3b6ef052e6bfd3cc,b324d1059988acae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:23:16.211 INFO 2981023 --- [ocp-schedules-22,3364dbbaecc9ca74,2214f13058a7358b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:23:16.220 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050123170000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:23:16.222 INFO 2981023 --- [ocp-schedules-22,3364dbbaecc9ca74,2214f13058a7358b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:23:16.268 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:23:16.271 INFO 2981023 --- [ocp-schedules-29,950421624ff945da,cb93f51ab3b892cd] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:23:16.272 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:23:16.271092253+08:00 2024-09-11 18:23:16.276 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:23:16.272+08:00 2024-09-11 18:23:16.278 INFO 2981023 --- [host-check-agent-status4,b7e64de9829ac133,194e9ed2ace398c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:23:16.279 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:23:16.272+08:00 and status AVAILABLE 2024-09-11 18:23:16.283 INFO 2981023 --- [host-check-agent-status4,b7e64de9829ac133,194e9ed2ace398c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:23:16.283 INFO 2981023 --- [host-check-agent-status4,b7e64de9829ac133,194e9ed2ace398c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:23:16.283 INFO 2981023 --- [host-check-agent-status4,b7e64de9829ac133,194e9ed2ace398c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:23:16.296 INFO 2981023 --- [host-check-agent-status3,b7e64de9829ac133,ec0756f9035a597f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:23:16.302 INFO 2981023 --- [host-check-agent-status3,b7e64de9829ac133,ec0756f9035a597f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:23:16.302 INFO 2981023 --- [host-check-agent-status3,b7e64de9829ac133,ec0756f9035a597f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:23:16.302 INFO 2981023 --- [host-check-agent-status3,b7e64de9829ac133,ec0756f9035a597f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:23:16.355 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:23:16.369 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.370 INFO 2981023 --- [host-check-agent-status2,b7e64de9829ac133,5d6e0ddeb86324d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:23:16.371 INFO 2981023 --- [host-check-agent-status1,b7e64de9829ac133,c8211624962032f1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:23:16.375 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:23:16.383 INFO 2981023 --- [host-check-agent-status2,b7e64de9829ac133,5d6e0ddeb86324d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:23:16.383 INFO 2981023 --- [host-check-agent-status2,b7e64de9829ac133,5d6e0ddeb86324d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:23:16.383 INFO 2981023 --- [host-check-agent-status2,b7e64de9829ac133,5d6e0ddeb86324d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:23:16.384 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.384 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:16.384 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:23:16.384 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:23:16.385 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:23:16.385168403+08:00 2024-09-11 18:23:16.385 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:23:16.386 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:23:16.385+08:00 2024-09-11 18:23:16.386 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:23:16.387 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:23:16.387 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:23:16.392844270+08:00 2024-09-11 18:23:16.388 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:23:16.385+08:00 and status AVAILABLE 2024-09-11 18:23:16.388 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:23:16.387+08:00 2024-09-11 18:23:16.389 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.391 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:23:16.391 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:23:16.387+08:00 and status AVAILABLE 2024-09-11 18:23:16.392 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:23:16.393 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:23:16.393 INFO 2981023 --- [host-check-agent-status1,b7e64de9829ac133,c8211624962032f1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:23:16.393 INFO 2981023 --- [host-check-agent-status1,b7e64de9829ac133,c8211624962032f1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:23:16.393 INFO 2981023 --- [host-check-agent-status1,b7e64de9829ac133,c8211624962032f1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:23:16.394 INFO 2981023 --- [ocp-schedules-28,e1398fc3dbf05d24,b065611c06b35be0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:23:16.396 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.396 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:23:16.396 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:16.396 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:23:16.395982+08:00, lastProcessTime=2024-09-11T18:23:06.336140+08:00 2024-09-11 18:23:16.400 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:23:16.401 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.403 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:23:16.404 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:23:16.405 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:23:16.400974232+08:00 2024-09-11 18:23:16.407 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:16.407 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:23:16.405+08:00 2024-09-11 18:23:16.411 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:23:16.405+08:00 and status AVAILABLE 2024-09-11 18:23:16.413 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:23:16.415 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:23:16.454 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:23:16.459 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:23:16.465 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:23:16.466 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:23:16.467 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:16.468 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.468 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:23:16.470 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:23:16.483 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:23:16.498 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:23:16.500 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:23:16.506 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:23:16.507 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:23:16.514 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:23:16.514 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050123170000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:23:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:16.517 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:23:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:16.518 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:23:16.518 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:23:16.522 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:23:16.529 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:23:16.533 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:23:16.533 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:23:16.535 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:23:16.539 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:23:16.549 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:23:16.562 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:23:16.563 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:23:16.568 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:23:16.568 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:23:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:16.577 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.577 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:23:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:23:16.585 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:23:16.585 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:23:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:23:16.590 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:23:16.592 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:23:16.592 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:23:16.606 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:16.606 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:23:16.607 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.607 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:23:16.614 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:23:16.614 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:23:16.627 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:23:16.629 INFO 2981023 --- [host-check-host-status4,63536b8bcb85995a,cebf96ecf093054c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:23:16.651 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:23:16.662 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:16.662 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:23:16.662 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.663 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:16.665 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:23:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:23:16.669 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:23:16.669 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:23:16.674 INFO 2981023 --- [host-check-host-status3,63536b8bcb85995a,20c84f4571e3c814] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:23:16.680 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:23:16.680 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:23:16.680 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:23:16.680 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:23:16.688 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:23:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:23:16.697 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:16.697 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.698 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:23:16.700 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:23:16.705 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:16.706 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:23:16.706 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:23:16.707 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:23:16.708 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:23:16.710 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:23:16.713 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:23:16.714 INFO 2981023 --- [host-check-host-status2,63536b8bcb85995a,96c5b9c71ca321ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:23:16.741 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:23:16.742 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:23:16.756 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:23:16.757 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:23:16.395982+08:00 2024-09-11 18:23:16.806 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:23:16.812 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:23:16.824 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:23:16.990 INFO 2981023 --- [host-check-clockdiff2,603e791d46b38c0a,c660cd07e5dcb871] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050196 5 5 , error= 2024-09-11 18:23:17.028 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:23:17.033 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:17.038 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:17.042 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:23:17.042 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:23:17.048 INFO 2981023 --- [host-check-host-status1,63536b8bcb85995a,b0d6048dc0988515] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:23:17.053 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:23:17.053 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:23:17.054 INFO 2981023 --- [ocp-schedules-30,3b6ef052e6bfd3cc,b324d1059988acae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:23:17.210 INFO 2981023 --- [ocp-schedules-25,d666ff86d2f93520,344d71523a7d64d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:23:17.268 WARN 2981023 --- [scanner-13659347-1,d666ff86d2f93520,f1c2312cdd9906c3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:23:17.272 INFO 2981023 --- [alarm-detect-5,d666ff86d2f93520,13ae03ca1ebd6788] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646107, data is empty. 2024-09-11 18:23:17.275 INFO 2981023 --- [alarm-detect-5,d666ff86d2f93520,13ae03ca1ebd6788] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646167, data is empty. 2024-09-11 18:23:17.275 INFO 2981023 --- [alarm-detect-16,d666ff86d2f93520,711ce224293655de] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050165 ~ 1726050187 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050180, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050180, value=98.556)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.556, values={1726050180=98.556}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050180=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:17.282 INFO 2981023 --- [alarm-detect-5,d666ff86d2f93520,13ae03ca1ebd6788] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646167, data is empty. 2024-09-11 18:23:17.306 INFO 2981023 --- [alarm-detect-19,d666ff86d2f93520,5c7ccf9e998c4c09] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646107, data is empty. 2024-09-11 18:23:17.308 INFO 2981023 --- [alarm-detect-3,d666ff86d2f93520,00dd2a06bd2d7cfd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646107, data is empty. 2024-09-11 18:23:17.309 INFO 2981023 --- [alarm-detect-19,d666ff86d2f93520,5c7ccf9e998c4c09] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646167, data is empty. 2024-09-11 18:23:17.311 INFO 2981023 --- [alarm-detect-3,d666ff86d2f93520,00dd2a06bd2d7cfd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646167, data is empty. 2024-09-11 18:23:17.311 INFO 2981023 --- [alarm-detect-19,d666ff86d2f93520,5c7ccf9e998c4c09] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646167, data is empty. 2024-09-11 18:23:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,60e1d4c8629d8126,18d3bd4309141a56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:17.314 INFO 2981023 --- [alarm-detect-3,d666ff86d2f93520,00dd2a06bd2d7cfd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646167, data is empty. 2024-09-11 18:23:17.318 INFO 2981023 --- [prometheus-minute-schedule-1,60e1d4c8629d8126,18d3bd4309141a56] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:17.341 INFO 2981023 --- [alarm-detect-4,d666ff86d2f93520,26701d60ae21bc22] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050165 ~ 1726050187 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050180, value=98.556)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.556, values={1726050180=98.556}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:17.346 INFO 2981023 --- [alarm-detect-1,d666ff86d2f93520,e7a54a4fd6d0211d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050165 ~ 1726050187 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050180, value=98.556)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.556, values={1726050180=98.556}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:17.378 INFO 2981023 --- [alarm-detect-8,d666ff86d2f93520,c68a44cb2f827d70] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050165 ~ 1726050187 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050180, value=7.38)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.38, values={1726050180=7.38}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:17.379 INFO 2981023 --- [alarm-detect-8,d666ff86d2f93520,c68a44cb2f827d70] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050165 ~ 1726050187 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050180, value=7.38)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.38, values={1726050180=7.38}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:23:17.432 INFO 2981023 --- [alarm-detect-6,d666ff86d2f93520,7f03da0ce5ffadf4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646107, data is empty. 2024-09-11 18:23:17.436 INFO 2981023 --- [alarm-detect-6,d666ff86d2f93520,7f03da0ce5ffadf4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646167, data is empty. 2024-09-11 18:23:17.441 INFO 2981023 --- [alarm-detect-6,d666ff86d2f93520,7f03da0ce5ffadf4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646167, data is empty. 2024-09-11 18:23:17.451 INFO 2981023 --- [ocp-async-1,665b87ecc2bbd0e4,dd435d7b46b038cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:17.468 INFO 2981023 --- [alarm-detect-12,d666ff86d2f93520,d0c329f25c65695f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050165 ~ 1726050187 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050180, value=106.085)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=106.085, values={1726050180=106.085}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:17.792 INFO 2981023 --- [ocp-async-7,ffa833eee36363b6,a82fe258848f14cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:23:18.237 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:18.238 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:19.201 INFO 2981023 --- [host-check-clockdiff1,603e791d46b38c0a,c57fa3bdb6899296] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050199 0 0 , error= 2024-09-11 18:23:19.451 INFO 2981023 --- [ocp-async-3,66d18f872ab861a9,cc1f95442d3dbfea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,a2a14fea27df692d,74ef2b2f5a32569e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,a2a14fea27df692d,74ef2b2f5a32569e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:21.238 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:21.240 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:21.452 INFO 2981023 --- [ocp-async-10,17f715b51c79898f,e002f571362c1019] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:22.058 INFO 2981023 --- [host-check-clockdiff4,603e791d46b38c0a,528d12f6d4538b75] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050202 0 0 , error= 2024-09-11 18:23:22.776 INFO 2981023 --- [ocp-schedules-22,2ac0da50b45bda71,8318a629d4b62a9d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:23:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,64065213d7b51ea8,f6258b06c4034b96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,64065213d7b51ea8,f6258b06c4034b96] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:23.452 INFO 2981023 --- [ocp-async-6,ce65255f80810470,d7bd4401a0877bdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:23.648 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,447088514df297a6,4323b4c8b4e5aa15] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.31.33.41, traceId=447088514df297a6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-09-11 18:23:23.650 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e8da0d982c2a5b8e,ded57318403a4ff1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.31.33.41, traceId=e8da0d982c2a5b8e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-09-11 18:23:23.650 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,e8da0d982c2a5b8e,ded57318403a4ff1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.31.33.41, traceId=e8da0d982c2a5b8e, duration=2 ms] 2024-09-11 18:23:23.653 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,09c43270a0ce6fc4,95f333bd60c69d7c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.31.33.41, traceId=09c43270a0ce6fc4, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-09-11 18:23:23.653 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,09c43270a0ce6fc4,95f333bd60c69d7c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.31.33.41, traceId=09c43270a0ce6fc4, duration=2 ms] 2024-09-11 18:23:23.656 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,447088514df297a6,4323b4c8b4e5aa15] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.31.33.41, traceId=447088514df297a6, duration=10 ms] 2024-09-11 18:23:23.658 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,14ae19eeb9efb05c,ccf7433bf8d96fff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.31.33.41, traceId=14ae19eeb9efb05c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 18:23:23.659 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,bb6dd0bfafb72c8a,af3a6e399a9bd902] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1000006&status=RUNNING%2CFAILED, client=10.31.33.41, traceId=bb6dd0bfafb72c8a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1000006,] 2024-09-11 18:23:23.669 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,14ae19eeb9efb05c,ccf7433bf8d96fff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=14ae19eeb9efb05c, duration=14 ms] 2024-09-11 18:23:23.686 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,bb6dd0bfafb72c8a,af3a6e399a9bd902] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.31.33.41, traceId=bb6dd0bfafb72c8a, duration=31 ms] 2024-09-11 18:23:23.776 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0873496464632e09,dc1ac18979bfdae4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.31.33.41, traceId=0873496464632e09, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-09-11 18:23:23.778 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0873496464632e09,dc1ac18979bfdae4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.31.33.41, traceId=0873496464632e09, duration=4 ms] 2024-09-11 18:23:23.793 INFO 2981023 --- [ocp-schedules-29,e48874060c140c2b,6d7e0f38bc8b63e6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 18:23:23.793 INFO 2981023 --- [ocp-schedules-29,e48874060c140c2b,6d7e0f38bc8b63e6] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 18:23:23.794 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,c6fde419ed17eb34,25b9e5420ea53cd2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.31.33.41, traceId=c6fde419ed17eb34, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-09-11 18:23:23.794 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,c6fde419ed17eb34,25b9e5420ea53cd2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.31.33.41, traceId=c6fde419ed17eb34, duration=11 ms] 2024-09-11 18:23:23.798 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,bae7153f86f423da,1a83853ffe1c5eec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.31.33.41, traceId=bae7153f86f423da, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-09-11 18:23:23.798 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,bae7153f86f423da,1a83853ffe1c5eec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.31.33.41, traceId=bae7153f86f423da, duration=3 ms] 2024-09-11 18:23:23.823 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,27f089e8f6c832d4,cf88cc375f1cf763] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.31.33.41, traceId=27f089e8f6c832d4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 18:23:23.850 INFO 2981023 --- [pool-618-thread-1,e48874060c140c2b,ece0a606ef86bb0d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:23:23.851 INFO 2981023 --- [pool-618-thread-2,e48874060c140c2b,32b04b4abcae44d7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:23:23.864 INFO 2981023 --- [pool-618-thread-3,e48874060c140c2b,bac053d641833653] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:23:23.907 INFO 2981023 --- [pool-618-thread-1,e48874060c140c2b,ece0a606ef86bb0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:23.907 INFO 2981023 --- [pool-618-thread-2,e48874060c140c2b,32b04b4abcae44d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:23.908 INFO 2981023 --- [pool-618-thread-3,e48874060c140c2b,bac053d641833653] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:23.910 INFO 2981023 --- [pool-618-thread-1,e48874060c140c2b,ece0a606ef86bb0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049603000000, 1726050203000000] 2024-09-11 18:23:23.910 INFO 2981023 --- [pool-618-thread-3,e48874060c140c2b,bac053d641833653] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049603000000, 1726050203000000] 2024-09-11 18:23:23.912 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:23:23.912 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, client=10.31.33.41, traceId=] 2024-09-11 18:23:23.912 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:23:23.915 INFO 2981023 --- [pool-618-thread-3,e48874060c140c2b,bac053d641833653] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:23:23.918 INFO 2981023 --- [pool-618-thread-2,e48874060c140c2b,32b04b4abcae44d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726049603000000, 1726050203000000] 2024-09-11 18:23:23.924 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1760f35a8f681c88,c1993de247006d00] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/summary?id=1000006, client=10.31.33.41, traceId=1760f35a8f681c88, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1000006,] 2024-09-11 18:23:23.930 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:23:23.930 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, client=10.31.33.41, traceId=] 2024-09-11 18:23:23.930 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:23:23.942 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8c18807d76d14a70,21b02dc2fc011d5a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/topSql/switch?id=1000006, client=10.31.33.41, traceId=8c18807d76d14a70, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1000006,] 2024-09-11 18:23:23.969 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d6a4e3dfd7281782,9471252214b7a477] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/stats?id=1000006, client=10.31.33.41, traceId=d6a4e3dfd7281782, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1000006,] 2024-09-11 18:23:23.983 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1760f35a8f681c88,c1993de247006d00] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:23.985 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1760f35a8f681c88,c1993de247006d00] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 18:23:23.988 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:23.989 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:23:23.997 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,8c18807d76d14a70,21b02dc2fc011d5a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/topSql/switch client=10.31.33.41, traceId=8c18807d76d14a70, duration=58 ms] 2024-09-11 18:23:24.001 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d6a4e3dfd7281782,9471252214b7a477] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:24.003 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d6a4e3dfd7281782,9471252214b7a477] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 18:23:24.004 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:24.006 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT ZONE, SVR_IP, SVR_PORT, NAME, VALUE, DATA_TYPE, SECTION, EDIT_LEVEL, INFO FROM GV$OB_PARAMETERS WHERE SCOPE = 'CLUSTER' AND name = ?, args: [_lcl_op_interval] 2024-09-11 18:23:24.009 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,d6a4e3dfd7281782,9471252214b7a477] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/stats client=10.31.33.41, traceId=d6a4e3dfd7281782, duration=45 ms] 2024-09-11 18:23:24.024 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:24.028 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1] 2024-09-11 18:23:24.032 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:24.033 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [resource_hard_limit] 2024-09-11 18:23:24.037 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:24.038 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2024-09-11 18:23:24.040 INFO 2981023 --- [pool-618-thread-1,e48874060c140c2b,ece0a606ef86bb0d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:23:24.051 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1760f35a8f681c88,c1993de247006d00] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/summary client=10.31.33.41, traceId=1760f35a8f681c88, duration=130 ms] 2024-09-11 18:23:24.057 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:24.058 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:24.062 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1] 2024-09-11 18:23:24.122 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,27f089e8f6c832d4,cf88cc375f1cf763] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.31.33.41, traceId=27f089e8f6c832d4, duration=317 ms] 2024-09-11 18:23:24.161 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,8047fffacd8fac95,d032bd2371130a8e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.31.33.41, traceId=8047fffacd8fac95, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-09-11 18:23:24.165 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,8047fffacd8fac95,d032bd2371130a8e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.31.33.41, traceId=8047fffacd8fac95, duration=6 ms] 2024-09-11 18:23:24.240 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:24.241 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:24.255 INFO 2981023 --- [pool-618-thread-2,e48874060c140c2b,32b04b4abcae44d7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:23:24.255 INFO 2981023 --- [ocp-schedules-29,e48874060c140c2b,6d7e0f38bc8b63e6] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 18:23:24.486 INFO 2981023 --- [ocp-schedules-28,5bef071feb85417f,d8999c3f3e89f245] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:23:24.706 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,28428117bd794b3f,4f601509f15ae01c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/eventMeta?id=1000006, client=10.31.33.41, traceId=28428117bd794b3f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1000006,] 2024-09-11 18:23:24.706 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,28428117bd794b3f,4f601509f15ae01c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/eventMeta client=10.31.33.41, traceId=28428117bd794b3f, duration=6 ms] 2024-09-11 18:23:24.706 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,64c157aab4cecc62,176ca7e2e16a3d93] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.31.33.41, traceId=64c157aab4cecc62, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 18:23:24.707 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,58ecee638f1a5a0c,d594358c30cd8955] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/serverStats?id=1000006, client=10.31.33.41, traceId=58ecee638f1a5a0c, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=1000006,] 2024-09-11 18:23:24.708 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-4,64c157aab4cecc62,176ca7e2e16a3d93] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 18:23:24.716 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bc8bf0f413ee8b91,bf0c46ceaff28bfe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/alarmSummaries?id=1000006, client=10.31.33.41, traceId=bc8bf0f413ee8b91, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=1000006,] 2024-09-11 18:23:24.721 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,308184e2c5b14951,b293b043184ee48a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?status=AVAILABLE&status=ONLINE&excludedServiceType=OB_CLUSTER&excludedServiceType=BACKUP_AGENT&excludedServiceType=RESTORE_AGENT&excludedServiceType=BACKUP_RESTORE_AGENT&excludedServiceType=ARBITRATION&excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=308184e2c5b14951, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[AVAILABLE, ONLINE],[OB_CLUSTER, BACKUP_AGENT, RESTORE_AGENT, BACKUP_RESTORE_AGENT, ARBITRATION, BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 18:23:24.730 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,a05d1cf7e3490305,ac5d15f1b9be4e87] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.31.33.41, traceId=a05d1cf7e3490305, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 18:23:24.731 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-7,308184e2c5b14951,b293b043184ee48a] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 18:23:24.740 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,bc8bf0f413ee8b91,bf0c46ceaff28bfe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/alarmSummaries client=10.31.33.41, traceId=bc8bf0f413ee8b91, duration=28 ms] 2024-09-11 18:23:24.754 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,a05d1cf7e3490305,ac5d15f1b9be4e87] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=a05d1cf7e3490305, duration=44 ms] 2024-09-11 18:23:24.758 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,64c157aab4cecc62,176ca7e2e16a3d93] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=64c157aab4cecc62, duration=55 ms] 2024-09-11 18:23:24.764 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,58ecee638f1a5a0c,d594358c30cd8955] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:24.764 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,308184e2c5b14951,b293b043184ee48a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=308184e2c5b14951, duration=51 ms] 2024-09-11 18:23:24.766 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,58ecee638f1a5a0c,d594358c30cd8955] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-09-11 18:23:24.766 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,58ecee638f1a5a0c,d594358c30cd8955] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/serverStats client=10.31.33.41, traceId=58ecee638f1a5a0c, duration=62 ms] 2024-09-11 18:23:24.865 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6007eed4ae741899,7baadf4a70d739c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top/realtime?metrics=active_session%2Cqps%2Ctps&labels=app%3AOB%2Cob_cluster_name%3Alinkrtd%2Cob_cluster_id%3A1719994443&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&queryAsync=true&endTime=2024-09-11T18%3A23%3A24%2B08%3A00&limit=1&maxPoints=360, client=10.31.33.41, traceId=6007eed4ae741899, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryRealtimeTopMetrics(OffsetDateTime,Long,String,String,Long,Long,Integer,Integer,String,Boolean,QueryRouteType), args=2024-09-11T18:23:24+08:00,60,active_session,qps,tps,app,ob_cluster_name,ob_cluster_id,0,0,360,1,app:OB,ob_cluster_name:linkrtd,ob_cluster_id:1719994443,true,NOT_ROUTED,] 2024-09-11 18:23:24.865 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,6e7c19d85dd6e64f,7af9a029c395a8aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=6e7c19d85dd6e64f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 18:23:24.870 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6007eed4ae741899,7baadf4a70d739c3] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = qps 2024-09-11 18:23:24.870 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6007eed4ae741899,7baadf4a70d739c3] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = qps, minTimestamp = 0, minTimeBuffer = 0, startTime = 1726050150 2024-09-11 18:23:24.874 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,6e7c19d85dd6e64f,7af9a029c395a8aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=6e7c19d85dd6e64f, duration=11 ms] 2024-09-11 18:23:24.877 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,03db82a027e00fed,ff6578e7536a5e37] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/topSql/status/events?null, client=10.31.33.41, traceId=03db82a027e00fed, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitchEvents(Long,OffsetDateTime,OffsetDateTime,Pageable), args=1000006,Page request [number: 0, size 5000, sort: timestamp: ASC],] 2024-09-11 18:23:24.883 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,5b9aa13ec02d8a97,16dd243d38522e39] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top/realtime?metrics=active_session%2Cqps%2Ctps&labels=app%3AOB%2Cob_cluster_name%3Alinkrtd%2Cob_cluster_id%3A1719994443&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&queryAsync=true&minStep=1&endTime=2024-09-11T18%3A23%3A24%2B08%3A00&limit=1&maxPoints=360, client=10.31.33.41, traceId=5b9aa13ec02d8a97, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryRealtimeTopMetrics(OffsetDateTime,Long,String,String,Long,Long,Integer,Integer,String,Boolean,QueryRouteType), args=2024-09-11T18:23:24+08:00,60,active_session,qps,tps,app,ob_cluster_name,ob_cluster_id,0,1,360,1,app:OB,ob_cluster_name:linkrtd,ob_cluster_id:1719994443,true,NOT_ROUTED,] 2024-09-11 18:23:24.885 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-8,5b9aa13ec02d8a97,16dd243d38522e39] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = qps 2024-09-11 18:23:24.885 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-8,5b9aa13ec02d8a97,16dd243d38522e39] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = qps, minTimestamp = 0, minTimeBuffer = 0, startTime = 1726050150 2024-09-11 18:23:24.886 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,6007eed4ae741899,7baadf4a70d739c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top/realtime client=10.31.33.41, traceId=6007eed4ae741899, duration=23 ms] 2024-09-11 18:23:24.891 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7437cc3a7c77a724,f8e7f2c7cea2c872] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=7437cc3a7c77a724, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 18:23:24.891 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,5b9aa13ec02d8a97,16dd243d38522e39] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top/realtime client=10.31.33.41, traceId=5b9aa13ec02d8a97, duration=11 ms] 2024-09-11 18:23:24.892 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,03db82a027e00fed,ff6578e7536a5e37] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/topSql/status/events client=10.31.33.41, traceId=03db82a027e00fed, duration=22 ms] 2024-09-11 18:23:24.895 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7437cc3a7c77a724,f8e7f2c7cea2c872] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 18:23:24.898 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9195602cacd84237,4a8dfcb8d8c9e661] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.31.33.41, traceId=9195602cacd84237, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 18:23:24.901 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9195602cacd84237,4a8dfcb8d8c9e661] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.31.33.41, traceId=9195602cacd84237, duration=5 ms] 2024-09-11 18:23:24.908 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,7437cc3a7c77a724,f8e7f2c7cea2c872] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=7437cc3a7c77a724, duration=21 ms] 2024-09-11 18:23:24.999 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,52b740ada629b58d,c66dfa976ac80e68] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1000006/events?startTime=2024-09-04T18%3A23%3A24%2B08%3A00&endTime=2024-09-11T18%3A23%3A24%2B08%3A00&page=1&size=7, client=10.31.33.41, traceId=52b740ada629b58d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1000006,2024-09-04T18:23:24+08:00,2024-09-11T18:23:24+08:00,Page request [number: 0, size 7, sort: timestamp: DESC],] 2024-09-11 18:23:25.004 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,7a92946f10c103e7,e261389c6d23efd4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=7a92946f10c103e7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 18:23:25.006 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,7a92946f10c103e7,e261389c6d23efd4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=7a92946f10c103e7, duration=4 ms] 2024-09-11 18:23:25.014 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,52b740ada629b58d,c66dfa976ac80e68] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1000006/events client=10.31.33.41, traceId=52b740ada629b58d, duration=18 ms] 2024-09-11 18:23:25.123 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2088c5fde8dc6f74,d743c2cd1c10be1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.31.33.41, traceId=2088c5fde8dc6f74, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2024-09-11 18:23:25.124 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-8,2088c5fde8dc6f74,d743c2cd1c10be1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.31.33.41, traceId=2088c5fde8dc6f74, duration=4 ms] 2024-09-11 18:23:25.452 INFO 2981023 --- [ocp-async-8,984625a21b30eaca,4e0aebd3efd51782] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,27215dabf9ab02bc,2f1b077ad84abed7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:26.318 INFO 2981023 --- [prometheus-minute-schedule-1,27215dabf9ab02bc,2f1b077ad84abed7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:26.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:23:26.353696+08:00, lastProcessTime=2024-09-11T18:23:16.395982+08:00 2024-09-11 18:23:26.498 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:23:26.353696+08:00 2024-09-11 18:23:27.241 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:27.241 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:27.258 INFO 2981023 --- [ocp-schedules-30,0de48f82bd0b0d5e,8d4207ea2b4ea677] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:23:27.258 INFO 2981023 --- [ocp-schedules-30,0de48f82bd0b0d5e,8d4207ea2b4ea677] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:23:27.258 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:23:27.258 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:23:27.258 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:23:27.283 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:23:27.284 INFO 2981023 --- [ocp-schedules-30,0de48f82bd0b0d5e,8d4207ea2b4ea677] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:23:27.286 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:23:27.288 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:23:27.288 INFO 2981023 --- [ocp-schedules-30,0de48f82bd0b0d5e,8d4207ea2b4ea677] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:23:27.289 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:23:27.289 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:23:27.291 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:23:27.452 INFO 2981023 --- [ocp-async-3,b6b9d995a94b8d69,7fce8d6bcb8b60d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:27.476 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:23:27.482 INFO 2981023 --- [ocp-schedules-30,0de48f82bd0b0d5e,8d4207ea2b4ea677] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:23:27.484 INFO 2981023 --- [ocp-schedules-19,e26ef410299ce582,48cfecf719224fbf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:23:27.504 WARN 2981023 --- [scanner-13659635-1,e26ef410299ce582,b17b7544cee39a26] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:23:27.528 INFO 2981023 --- [alarm-detect-19,e26ef410299ce582,bd08ef975a1364db] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050180 ~ 1726050197 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050195, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050195, value=99.189)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=99.189, values={1726050195=99.189}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050195=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:27.606 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:23:27.616 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:23:27.619 INFO 2981023 --- [ocp-schedules-30,0de48f82bd0b0d5e,8d4207ea2b4ea677] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:23:27.624 INFO 2981023 --- [alarm-detect-9,e26ef410299ce582,0d351515036c7d67] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050180 ~ 1726050197 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050195, value=99.189)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=99.189, values={1726050195=99.189}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:27.638 INFO 2981023 --- [alarm-detect-20,e26ef410299ce582,359fe13ae4f17668] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050180 ~ 1726050197 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050195, value=99.189)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=99.189, values={1726050195=99.189}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:27.793 INFO 2981023 --- [ocp-async-6,41b2f9e56240d84b,5ff197f2e35ef5cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:23:28.677 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.680 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, ob 2024-09-11 18:23:28.691 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.691 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:23:28.698 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.707 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 6 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.721 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.725 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.749 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 36 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.829 INFO 2981023 --- [sql_perf_diagnoser8,,] .s.d.i.d.PerformanceDegradationDiagnoser : Auto flushing plan cache is disabled 2024-09-11 18:23:28.847 INFO 2981023 --- [sql_perf_diagnoser8,,] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007115 2024-09-11 18:23:28.865 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 12 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.883 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 182 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.901 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.903 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.904 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.905 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.905 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.906 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.911 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.911 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:23:28.912 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.916 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 6 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.931 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.932 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.932 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:23:28.933 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.933 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.934 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.935 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.936 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.936 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.937 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.938 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:17:28.509, end=2024-09-11T18:22:28.673} 2024-09-11 18:23:28.938 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 7 dubious sqls 2024-09-11 18:23:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,6e6c6639c72d2b96,60f94f84e192dd8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:29.318 INFO 2981023 --- [prometheus-minute-schedule-1,6e6c6639c72d2b96,60f94f84e192dd8f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:29.452 INFO 2981023 --- [ocp-async-5,c23b2eb19e830851,c67d7e06db3d3b10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:30.241 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:30.243 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:31.452 INFO 2981023 --- [ocp-async-6,2416d0ea5ed112e3,2fc65be39702dc5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,cefd609dfeb50b07,ec6b741759f9f90f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,cefd609dfeb50b07,ec6b741759f9f90f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:32.778 INFO 2981023 --- [ocp-schedules-12,85c6382a0265e1be,a9407aba92f7fb71] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:23:33.243 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:33.244 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:33.452 INFO 2981023 --- [ocp-async-4,e397f7c42d7c889d,c62ab7c4440baed5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,1286b8ea91b9f14c,6ee8026d6358882d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,1286b8ea91b9f14c,6ee8026d6358882d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:35.453 INFO 2981023 --- [ocp-async-3,704aa50f15069187,0503a7959cb4a10a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:35.685 INFO 2981023 --- [ocp-schedules-9,6d94904d32f478e2,5741de547167ef45] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:23:35.693 INFO 2981023 --- [ocp-schedules-9,6d94904d32f478e2,5741de547167ef45] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:23:35.693 INFO 2981023 --- [ocp-schedules-9,6d94904d32f478e2,5741de547167ef45] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:23:36.244 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:36.245 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:23:36.352592+08:00, lastProcessTime=2024-09-11T18:23:26.353696+08:00 2024-09-11 18:23:36.473 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:23:36.478 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:23:36.352592+08:00 2024-09-11 18:23:37.314 INFO 2981023 --- [prometheus-cache-monitor1,e22dc5f9c7ad7f38,18e3793f9220e4fc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:23:37.315 INFO 2981023 --- [prometheus-cache-monitor1,e22dc5f9c7ad7f38,18e3793f9220e4fc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:23:37.453 INFO 2981023 --- [ocp-async-9,6beab950e725c647,9cc08e175841846b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:37.757 INFO 2981023 --- [ocp-schedules-24,e1c2526eaf4009c2,7b2d250502545d4f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:23:37.790 WARN 2981023 --- [scanner-13659858-1,e1c2526eaf4009c2,435cd2a8eb3e0b6b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:23:37.793 INFO 2981023 --- [ocp-async-4,39886926681b84af,c19e30ed93140a18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:23:37.841 INFO 2981023 --- [alarm-detect-16,e1c2526eaf4009c2,82feeef192e08764] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646167, data is empty. 2024-09-11 18:23:37.846 INFO 2981023 --- [alarm-detect-16,e1c2526eaf4009c2,82feeef192e08764] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646167, data is empty. 2024-09-11 18:23:37.851 INFO 2981023 --- [alarm-detect-16,e1c2526eaf4009c2,82feeef192e08764] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646167, data is empty. 2024-09-11 18:23:37.866 INFO 2981023 --- [alarm-detect-5,e1c2526eaf4009c2,f8564ac0f8b4ccdb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646167, data is empty. 2024-09-11 18:23:37.869 INFO 2981023 --- [alarm-detect-13,e1c2526eaf4009c2,eca34ab3d69f8e2c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646167, data is empty. 2024-09-11 18:23:37.872 INFO 2981023 --- [alarm-detect-5,e1c2526eaf4009c2,f8564ac0f8b4ccdb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646167, data is empty. 2024-09-11 18:23:37.875 INFO 2981023 --- [alarm-detect-5,e1c2526eaf4009c2,f8564ac0f8b4ccdb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646167, data is empty. 2024-09-11 18:23:37.882 INFO 2981023 --- [alarm-detect-13,e1c2526eaf4009c2,eca34ab3d69f8e2c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646167, data is empty. 2024-09-11 18:23:37.898 INFO 2981023 --- [alarm-detect-13,e1c2526eaf4009c2,eca34ab3d69f8e2c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646167, data is empty. 2024-09-11 18:23:37.958 INFO 2981023 --- [alarm-detect-6,e1c2526eaf4009c2,174326006ab96071] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050180 ~ 1726050207 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050195, value=7.925)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.925, values={1726050195=7.925}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:37.958 INFO 2981023 --- [alarm-detect-6,e1c2526eaf4009c2,174326006ab96071] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050180 ~ 1726050207 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050195, value=7.925)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.925, values={1726050195=7.925}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:23:37.993 INFO 2981023 --- [alarm-detect-19,e1c2526eaf4009c2,fdab5a0fa88396af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646167, data is empty. 2024-09-11 18:23:37.997 INFO 2981023 --- [alarm-detect-19,e1c2526eaf4009c2,fdab5a0fa88396af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646167, data is empty. 2024-09-11 18:23:38.001 INFO 2981023 --- [alarm-detect-19,e1c2526eaf4009c2,fdab5a0fa88396af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646167, data is empty. 2024-09-11 18:23:38.035 INFO 2981023 --- [alarm-detect-5,e1c2526eaf4009c2,4d90dbd4a20ab5d2] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050180 ~ 1726050207 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050195, value=105.314)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.314, values={1726050195=105.314}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,f57a16e8d0ff6a19,f2e5d1ef68feef58] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,f57a16e8d0ff6a19,f2e5d1ef68feef58] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:39.245 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:39.245 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:39.453 INFO 2981023 --- [ocp-async-7,68ed92a669777671,17a306086260f057] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:40.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:23:40.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:23:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,a86ae2e8c1298d01,bf30fb93777a94ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,a86ae2e8c1298d01,bf30fb93777a94ac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:41.453 INFO 2981023 --- [ocp-async-4,82d743a55ac47f77,4517a43c4c73e5aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:42.245 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:42.247 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:42.781 INFO 2981023 --- [ocp-schedules-2,30275ad2a444cc58,02f753f77a8c346b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:23:43.453 INFO 2981023 --- [ocp-async-2,cc51330d4f5190f8,244ed753256d0cec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,99f73bbf2b0ae29b,7764a2ead628da8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:44.318 INFO 2981023 --- [prometheus-minute-schedule-1,99f73bbf2b0ae29b,7764a2ead628da8a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:45.101 INFO 2981023 --- [prometheus-second-schedule-1,3066b25a68ae22b2,284e1d054202ac05] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:23:45.248 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:45.249 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:45.453 INFO 2981023 --- [ocp-async-8,991ad3db59a197f4,cb2051bd67e8f7b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:46.023 INFO 2981023 --- [ocp-schedules-11,48b97077ffc254b9,348d3237951433c2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:23:46.023 INFO 2981023 --- [ocp-schedules-21,f5c7d5ad06d12cb1,4e1fe9e2fa68593e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:23:46.023 INFO 2981023 --- [host-check-agent-status1,48b97077ffc254b9,979f38ca35d4f73e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:23:46.024 INFO 2981023 --- [ocp-schedules-10,d493a948675c4c43,ea7841283d3b6259] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:23:46.024 INFO 2981023 --- [host-check-clockdiff1,f5c7d5ad06d12cb1,ccefcf60de9d572c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:23:46.024 INFO 2981023 --- [host-check-clockdiff2,f5c7d5ad06d12cb1,61415477724e988e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:23:46.024 INFO 2981023 --- [host-check-clockdiff3,f5c7d5ad06d12cb1,a32418d0dc519ee3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:23:46.026 INFO 2981023 --- [host-check-clockdiff4,f5c7d5ad06d12cb1,d89bdfa527a0a07d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:23:46.028 INFO 2981023 --- [host-check-agent-status2,48b97077ffc254b9,49bbb221d9088ae0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:23:46.028 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:23:46.029 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:23:46.029 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:23:46.030 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:23:46.048 INFO 2981023 --- [host-check-agent-status3,48b97077ffc254b9,8171d65411b8748e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:23:46.050 INFO 2981023 --- [host-check-clockdiff3,f5c7d5ad06d12cb1,a32418d0dc519ee3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050226 0 0 , error= 2024-09-11 18:23:46.051 INFO 2981023 --- [host-check-agent-status4,48b97077ffc254b9,2f0cd5b4292467d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:23:46.067 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:23:46.072 INFO 2981023 --- [host-check-agent-status3,48b97077ffc254b9,8171d65411b8748e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:23:46.073 INFO 2981023 --- [host-check-agent-status3,48b97077ffc254b9,8171d65411b8748e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:23:46.076 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:23:46.077 INFO 2981023 --- [host-check-agent-status2,48b97077ffc254b9,49bbb221d9088ae0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:23:46.077 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:23:46.078 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:23:46.080 INFO 2981023 --- [host-check-agent-status2,48b97077ffc254b9,49bbb221d9088ae0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:23:46.170 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:23:46.176 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:23:46.186 INFO 2981023 --- [ocp-schedules-14,bb9f825014024a4e,9069ef759181608b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:23:46.186 INFO 2981023 --- [host-check-agent-status1,48b97077ffc254b9,979f38ca35d4f73e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:23:46.188 INFO 2981023 --- [ocp-schedules-20,3f5a95374e7d28f0,de79bcf7e0d495d1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:23:46.188 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:23:46.188 INFO 2981023 --- [host-check-agent-status4,48b97077ffc254b9,2f0cd5b4292467d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:23:46.195 INFO 2981023 --- [ocp-schedules-30,d6e5ac29ddd4764d,a96d9c88760640eb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:23:46.196 INFO 2981023 --- [ocp-schedules-7,f3de09043a578919,98ccc3490495ccd8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:23:46.196 INFO 2981023 --- [ocp-schedules-27,81b28dd77e5a9fb1,12e1bad27ffb2329] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:23:46.198 INFO 2981023 --- [host-check-agent-status1,48b97077ffc254b9,979f38ca35d4f73e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:23:46.199 INFO 2981023 --- [host-check-agent-status4,48b97077ffc254b9,2f0cd5b4292467d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:23:46.205 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:23:46.259 INFO 2981023 --- [ocp-schedules-7,f3de09043a578919,98ccc3490495ccd8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:23:46.276 INFO 2981023 --- [ocp-schedules-20,3f5a95374e7d28f0,de79bcf7e0d495d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:23:46.276 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:23:46.278 INFO 2981023 --- [ocp-schedules-14,bb9f825014024a4e,9069ef759181608b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:23:46.294 INFO 2981023 --- [ocp-schedules-30,d6e5ac29ddd4764d,a96d9c88760640eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:23:46.305 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050123170000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:23:46.306 INFO 2981023 --- [ocp-schedules-30,d6e5ac29ddd4764d,a96d9c88760640eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:23:46.466 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:23:46.464809+08:00, lastProcessTime=2024-09-11T18:23:36.352592+08:00 2024-09-11 18:23:46.467 INFO 2981023 --- [host-check-agent-status3,48b97077ffc254b9,8171d65411b8748e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:23:46.478 INFO 2981023 --- [host-check-agent-status3,48b97077ffc254b9,8171d65411b8748e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:23:46.478 INFO 2981023 --- [host-check-agent-status3,48b97077ffc254b9,8171d65411b8748e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:23:46.478 INFO 2981023 --- [host-check-agent-status3,48b97077ffc254b9,8171d65411b8748e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:23:46.478 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:23:46.493 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:23:46.494 INFO 2981023 --- [host-check-agent-status1,48b97077ffc254b9,979f38ca35d4f73e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:23:46.495 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:23:46.500672374+08:00 2024-09-11 18:23:46.497 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:23:46.495+08:00 2024-09-11 18:23:46.501 INFO 2981023 --- [ocp-schedules-23,2329eb514cea59a6,b851657774cc0c60] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:23:46.504 INFO 2981023 --- [host-check-agent-status1,48b97077ffc254b9,979f38ca35d4f73e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:23:46.504 INFO 2981023 --- [host-check-agent-status1,48b97077ffc254b9,979f38ca35d4f73e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:23:46.504 INFO 2981023 --- [host-check-agent-status1,48b97077ffc254b9,979f38ca35d4f73e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:23:46.557 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:23:46.495+08:00 and status AVAILABLE 2024-09-11 18:23:46.562 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:23:46.563 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:23:46.565 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:23:46.566 INFO 2981023 --- [host-check-agent-status2,48b97077ffc254b9,49bbb221d9088ae0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:23:46.566 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:23:46.564432013+08:00 2024-09-11 18:23:46.567 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:23:46.566+08:00 2024-09-11 18:23:46.571 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:23:46.571134172+08:00 2024-09-11 18:23:46.572 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:23:46.571+08:00 2024-09-11 18:23:46.573 INFO 2981023 --- [host-check-agent-status4,48b97077ffc254b9,2f0cd5b4292467d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:23:46.574 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:23:46.566+08:00 and status AVAILABLE 2024-09-11 18:23:46.574 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:23:46.571+08:00 and status AVAILABLE 2024-09-11 18:23:46.576 INFO 2981023 --- [host-check-agent-status4,48b97077ffc254b9,2f0cd5b4292467d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:23:46.576 INFO 2981023 --- [host-check-agent-status4,48b97077ffc254b9,2f0cd5b4292467d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:23:46.576 INFO 2981023 --- [host-check-agent-status4,48b97077ffc254b9,2f0cd5b4292467d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:23:46.578 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:23:46.579 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:23:46.579358359+08:00 2024-09-11 18:23:46.579 INFO 2981023 --- [host-check-agent-status2,48b97077ffc254b9,49bbb221d9088ae0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:23:46.579 INFO 2981023 --- [host-check-agent-status2,48b97077ffc254b9,49bbb221d9088ae0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:23:46.580 INFO 2981023 --- [host-check-agent-status2,48b97077ffc254b9,49bbb221d9088ae0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:23:46.586 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:23:46.579+08:00 2024-09-11 18:23:46.590 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.591 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:23:46.596 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:23:46.579+08:00 and status AVAILABLE 2024-09-11 18:23:46.596 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:23:46.597 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.597 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:23:46.598 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:23:46.602 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:23:46.606 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:23:46.607 INFO 2981023 --- [ocp-schedules-20,3f5a95374e7d28f0,de79bcf7e0d495d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:23:46.609 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:23:46.612 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:23:46.613 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:23:46.614 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:23:46.619 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.620 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.620 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:23:46.622 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.622 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:46.624 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:23:46.624 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:46.628 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:23:46.628 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.643 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:23:46.660 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:23:46.663 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:23:46.666 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:23:46.668 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:23:46.673 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:23:46.676 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:23:46.677 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.678 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:46.692 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:23:46.698 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:23:46.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.700 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.700 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:23:46.704 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.705 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:23:46.711 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.714 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:23:46.715 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:23:46.720 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:23:46.721 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:23:46.725 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.737 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:23:46.748 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:23:46.769 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:23:46.769 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050123170000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:23:46.770 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:23:46.783 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:23:46.785 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:23:46.790 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.791 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.792 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.792 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:46.794 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:23:46.794 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.796 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:23:46.798 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:23:46.805 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.809 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.809 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:23:46.811 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:23:46.815 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:23:46.815 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:23:46.816 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:23:46.820 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.825 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.825 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:23:46.825 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.827 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:23:46.827 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:23:46.827 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.835 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:23:46.837 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:23:46.840 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:23:46.845 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:23:46.847 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:23:46.847 INFO 2981023 --- [ocp-schedules-19,ef45db4525aa1ca0,d48de7c824adbc37] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:23:46.847 INFO 2981023 --- [ocp-schedules-19,ef45db4525aa1ca0,d48de7c824adbc37] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:23:46.861 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:23:46.864 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.865 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:23:46.865 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:23:46.866 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.867 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:23:46.867 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:23:46.868 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:23:46.872 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:23:46.874 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:23:46.875 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:23:46.876 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.882 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.884 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.884 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.887 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:23:46.887 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:23:46.890 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:23:46.891 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:23:46.892 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:23:46.893 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:23:46.893 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.894 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:23:46.895 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:23:46.896 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:23:46.896 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.897 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:23:46.912 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:23:46.912 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:23:46.912 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:23:46.913 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:23:46.917 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:23:46.926 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:23:46.929 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:23:46.936 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:23:46.464809+08:00 2024-09-11 18:23:46.938 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:23:46.939 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:23:46.940 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:23:46.951 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.952 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.952 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:23:46.953 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:23:46.957 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:23:46.960 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:23:46.964 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:23:46.964 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:23:46.964 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.965 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:23:46.970 INFO 2981023 --- [host-check-host-status4,d493a948675c4c43,2e363f64b17e6158] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:23:46.971 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:23:46.974 INFO 2981023 --- [host-check-host-status3,d493a948675c4c43,bb2e04b207f052d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:23:46.980 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.982 INFO 2981023 --- [host-check-clockdiff2,f5c7d5ad06d12cb1,61415477724e988e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050226 5 5 , error= 2024-09-11 18:23:46.985 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:23:46.985 INFO 2981023 --- [host-check-host-status2,d493a948675c4c43,218e64c09d19e849] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:23:46.987 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:23:46.987 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:23:46.989 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.996 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:46.996 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:46.997 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:23:46.997 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:23:47.002 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:23:47.002 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:23:47.035 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:23:47.056 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:23:47.062 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:23:47.065 INFO 2981023 --- [ocp-schedules-24,72c93d88525cf1f0,a8ca86c6007a4c8e] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:23:47.092 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:23:47.097 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:23:47.109 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:23:47.227 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:23:47.233 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:23:47.238 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:47.242 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:23:47.247 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:23:47.247 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:23:47.248 INFO 2981023 --- [ocp-schedules-7,f3de09043a578919,98ccc3490495ccd8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:23:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,ba9eefea5430606b,2dd3019281544aa3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,ba9eefea5430606b,2dd3019281544aa3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:47.323 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:23:47.330 INFO 2981023 --- [host-check-host-status1,d493a948675c4c43,428c85feb9b4b7cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:23:47.453 INFO 2981023 --- [ocp-async-4,2087128bd259192b,bed23d4429257fc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:47.573 INFO 2981023 --- [ocp-schedules-29,af3c060cb66bef7f,a45df6956ac177a7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:23:47.596 WARN 2981023 --- [ocp-schedules-29,af3c060cb66bef7f,a45df6956ac177a7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:23:47.793 INFO 2981023 --- [ocp-async-7,c256ddcff61f3256,a3dfff05e916be48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:23:48.047 INFO 2981023 --- [ocp-schedules-24,183c025a727f66ad,3c79e0f53fbb298b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:23:48.126 WARN 2981023 --- [scanner-13660210-1,183c025a727f66ad,307108750bef1621] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:23:48.133 INFO 2981023 --- [alarm-detect-18,183c025a727f66ad,e920f4b00ded36b5] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050195 ~ 1726050218 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050210, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050210, value=98.792)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.792, values={1726050210=98.792}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050210=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:48.172 INFO 2981023 --- [alarm-detect-18,183c025a727f66ad,13f3d12a2b879a64] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050195 ~ 1726050218 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050210, value=98.792)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.792, values={1726050210=98.792}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:48.192 INFO 2981023 --- [alarm-detect-4,183c025a727f66ad,e62eec47c38be8af] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050195 ~ 1726050218 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050210, value=8.313)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.313, values={1726050210=8.313}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:48.192 INFO 2981023 --- [alarm-detect-4,183c025a727f66ad,e62eec47c38be8af] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050195 ~ 1726050218 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050210, value=8.313)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.313, values={1726050210=8.313}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:23:48.196 INFO 2981023 --- [alarm-detect-6,183c025a727f66ad,6343cfc16684c78d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050195 ~ 1726050218 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050210, value=98.792)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.792, values={1726050210=98.792}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:48.249 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:48.250 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:48.272 INFO 2981023 --- [alarm-detect-3,183c025a727f66ad,7d092d45c69c8e5b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050195 ~ 1726050218 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050210, value=104.533)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.533, values={1726050210=104.533}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:48.554 INFO 2981023 --- [host-check-clockdiff1,f5c7d5ad06d12cb1,ccefcf60de9d572c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050228 0 0 , error= 2024-09-11 18:23:49.453 INFO 2981023 --- [ocp-async-6,153fbb2d6e067956,4113e16124caea27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:23:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:23:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,58b84d5eae3bf089,d327fbfc42e0f82e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:50.318 INFO 2981023 --- [prometheus-minute-schedule-1,58b84d5eae3bf089,d327fbfc42e0f82e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:50.355 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.360 INFO 2981023 --- [ocp-schedules-8,f375b6babc057b5d,729fdd2a9c7ee4ba] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:23:50.363 INFO 2981023 --- [ocp-schedules-10,9a6fa97745597238,0565c844a3a2cb07] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:23:51.250 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:51.252 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:51.454 INFO 2981023 --- [ocp-async-8,7b005f1d23c74c03,3d53d538336f7342] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:51.637 INFO 2981023 --- [ocp-schedules-4,f3124d04147f6d69,07ff238f4bd4b018] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:23:51.638 INFO 2981023 --- [ocp-schedules-4,f3124d04147f6d69,07ff238f4bd4b018] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:23:51.638 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:23:51.638 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:23:51.638 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:23:51.639 INFO 2981023 --- [ocp-schedules-4,f3124d04147f6d69,07ff238f4bd4b018] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:23:51.987 INFO 2981023 --- [host-check-clockdiff4,f5c7d5ad06d12cb1,d89bdfa527a0a07d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050231 0 0 , error= 2024-09-11 18:23:52.784 INFO 2981023 --- [ocp-schedules-6,51df3dd93e6e42af,a98c607343fa2bcf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:23:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,fed822583c337c22,f04e942f316f5bd4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,fed822583c337c22,f04e942f316f5bd4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:53.366 INFO 2981023 --- [ocp-schedules-1,b98c83ff468d0782,66eb4e8ab85dffda] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:23:53.366 INFO 2981023 --- [ocp-schedules-1,b98c83ff468d0782,66eb4e8ab85dffda] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:23:53.454 INFO 2981023 --- [ocp-async-5,12d791aa783fda60,6b43bb2cdc9889e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:54.252 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:54.252 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:54.489 INFO 2981023 --- [ocp-schedules-28,8459b47e11521472,298ce03410dfd85d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:23:54.960 INFO 2981023 --- [ocp-schedules-10,05a803ca02239c05,1d215ec403912171] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T18:23:54.960+08:00, tenantSize=6 2024-09-11 18:23:54.963 INFO 2981023 --- [ocp-schedules-10,05a803ca02239c05,1d215ec403912171] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726050234960000 2024-09-11 18:23:54.963 INFO 2981023 --- [ocp-schedules-10,05a803ca02239c05,1d215ec403912171] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726049934935000 2024-09-11 18:23:54.994 INFO 2981023 --- [ocp-schedules-10,05a803ca02239c05,1d215ec403912171] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 18:23:55.239 INFO 2981023 --- [ocp-schedules-23,96204be968d869a8,bd4a792a0da20f46] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:23:55.260 WARN 2981023 --- [ocp-schedules-23,96204be968d869a8,bd4a792a0da20f46] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:23:55.454 INFO 2981023 --- [ocp-async-9,8e0f3bba513eaccc,a4d0018425952913] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:55.696 INFO 2981023 --- [ocp-schedules-27,4ab22f3ba7aa8d8d,b76b33a43fa3f617] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:23:55.704 INFO 2981023 --- [ocp-schedules-27,4ab22f3ba7aa8d8d,b76b33a43fa3f617] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:23:55.704 INFO 2981023 --- [ocp-schedules-27,4ab22f3ba7aa8d8d,b76b33a43fa3f617] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:23:55.905 INFO 2981023 --- [ocp-schedules-19,04752b447f4ae8ca,03d093681e904394] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:23:55.905 INFO 2981023 --- [ocp-schedules-19,04752b447f4ae8ca,03d093681e904394] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:23:55.906 INFO 2981023 --- [ocp-schedules-19,04752b447f4ae8ca,03d093681e904394] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:23:55.906 INFO 2981023 --- [ocp-schedules-19,04752b447f4ae8ca,03d093681e904394] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:23:55.906 INFO 2981023 --- [ocp-schedules-19,04752b447f4ae8ca,03d093681e904394] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:23:55.906 INFO 2981023 --- [ocp-schedules-19,04752b447f4ae8ca,03d093681e904394] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:23:56.122 INFO 2981023 --- [ocp-schedules-2,ddefc3254a1055be,ec0b702af97073b8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:23:56.128 INFO 2981023 --- [ocp-schedules-2,ddefc3254a1055be,ec0b702af97073b8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:23:56.128 INFO 2981023 --- [ocp-schedules-2,ddefc3254a1055be,ec0b702af97073b8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:23:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,25cc6947c60caae1,9be3025a7d6ba105] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,25cc6947c60caae1,9be3025a7d6ba105] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:56.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:23:56.354944+08:00, lastProcessTime=2024-09-11T18:23:46.464809+08:00 2024-09-11 18:23:56.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:23:56.354944+08:00 2024-09-11 18:23:57.252 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:23:57.254 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:23:57.454 INFO 2981023 --- [ocp-async-6,8b6c78fc40eeabf2,9f3e7a89e501437f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:23:57.793 INFO 2981023 --- [ocp-async-2,b162c29b11c9f02b,a8289ac2883faa60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:23:58.281 INFO 2981023 --- [ocp-schedules-16,73ec6bed979c4bb2,96260acdd29d8874] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:23:58.326 WARN 2981023 --- [scanner-13660446-1,73ec6bed979c4bb2,5b772cdbe01f6aa9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:23:58.351 INFO 2981023 --- [alarm-detect-20,73ec6bed979c4bb2,2a720c81635c9848] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050210 ~ 1726050228 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050225, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050225, value=98.657)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.657, values={1726050225=98.657}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050225=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:58.358 INFO 2981023 --- [alarm-detect-16,73ec6bed979c4bb2,d35e0c3b27df7896] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646167, data is empty. 2024-09-11 18:23:58.364 INFO 2981023 --- [alarm-detect-16,73ec6bed979c4bb2,d35e0c3b27df7896] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646167, data is empty. 2024-09-11 18:23:58.369 INFO 2981023 --- [alarm-detect-16,73ec6bed979c4bb2,d35e0c3b27df7896] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646167, data is empty. 2024-09-11 18:23:58.389 INFO 2981023 --- [alarm-detect-10,73ec6bed979c4bb2,ab3cd97dc91fe16c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646167, data is empty. 2024-09-11 18:23:58.392 INFO 2981023 --- [alarm-detect-1,73ec6bed979c4bb2,c3394597666d6957] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646167, data is empty. 2024-09-11 18:23:58.392 INFO 2981023 --- [alarm-detect-10,73ec6bed979c4bb2,ab3cd97dc91fe16c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646167, data is empty. 2024-09-11 18:23:58.396 INFO 2981023 --- [alarm-detect-1,73ec6bed979c4bb2,c3394597666d6957] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646167, data is empty. 2024-09-11 18:23:58.398 INFO 2981023 --- [alarm-detect-10,73ec6bed979c4bb2,ab3cd97dc91fe16c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646167, data is empty. 2024-09-11 18:23:58.402 INFO 2981023 --- [alarm-detect-1,73ec6bed979c4bb2,c3394597666d6957] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646167, data is empty. 2024-09-11 18:23:58.406 INFO 2981023 --- [alarm-detect-2,73ec6bed979c4bb2,d8cd10c651a12c75] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050210 ~ 1726050228 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050225, value=98.657)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.657, values={1726050225=98.657}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:58.420 INFO 2981023 --- [alarm-detect-4,73ec6bed979c4bb2,d0d2c452f80fe9a9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050210 ~ 1726050228 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050225, value=98.657)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.657, values={1726050225=98.657}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:23:58.501 INFO 2981023 --- [alarm-detect-1,73ec6bed979c4bb2,af0ca05f8426b011] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646167, data is empty. 2024-09-11 18:23:58.504 INFO 2981023 --- [alarm-detect-1,73ec6bed979c4bb2,af0ca05f8426b011] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646167, data is empty. 2024-09-11 18:23:58.506 INFO 2981023 --- [alarm-detect-1,73ec6bed979c4bb2,af0ca05f8426b011] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646167, data is empty. 2024-09-11 18:23:58.572 INFO 2981023 --- [ocp-schedules-14,c39fb7e834ec3118,3c2d7c3ac2865d42] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:23:58.572 INFO 2981023 --- [ocp-schedules-14,c39fb7e834ec3118,3c2d7c3ac2865d42] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:23:58.587 INFO 2981023 --- [Collect all tenants major compactions2,c39fb7e834ec3118,ae2b33f539d68a0d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:23:58.587 INFO 2981023 --- [Collect all tenants major compactions1,c39fb7e834ec3118,59379247e6c6ed01] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:23:58.588 INFO 2981023 --- [Collect all tenants major compactions4,c39fb7e834ec3118,749ea828afc730ce] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:23:58.588 INFO 2981023 --- [Collect all tenants major compactions5,c39fb7e834ec3118,bc14a37fdec56229] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:23:58.595 INFO 2981023 --- [Collect all tenants major compactions6,c39fb7e834ec3118,f1a07713ec6262f3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:23:58.600 INFO 2981023 --- [Collect all tenants major compactions3,c39fb7e834ec3118,ada942590b2a3349] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:23:58.612 INFO 2981023 --- [Collect all tenants major compactions2,c39fb7e834ec3118,ae2b33f539d68a0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:58.612 INFO 2981023 --- [Collect all tenants major compactions3,c39fb7e834ec3118,ada942590b2a3349] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:58.613 INFO 2981023 --- [Collect all tenants major compactions2,c39fb7e834ec3118,ae2b33f539d68a0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:23:58.613 INFO 2981023 --- [Collect all tenants major compactions3,c39fb7e834ec3118,ada942590b2a3349] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:23:58.613 INFO 2981023 --- [Collect all tenants major compactions6,c39fb7e834ec3118,f1a07713ec6262f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:58.614 INFO 2981023 --- [Collect all tenants major compactions1,c39fb7e834ec3118,59379247e6c6ed01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:58.615 INFO 2981023 --- [Collect all tenants major compactions6,c39fb7e834ec3118,f1a07713ec6262f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:23:58.616 INFO 2981023 --- [Collect all tenants major compactions3,c39fb7e834ec3118,ada942590b2a3349] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:23:58.616 INFO 2981023 --- [Collect all tenants major compactions3,c39fb7e834ec3118,ada942590b2a3349] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:23:58.616 INFO 2981023 --- [Collect all tenants major compactions1,c39fb7e834ec3118,59379247e6c6ed01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:23:58.616 INFO 2981023 --- [Collect all tenants major compactions2,c39fb7e834ec3118,ae2b33f539d68a0d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:23:58.616 INFO 2981023 --- [Collect all tenants major compactions2,c39fb7e834ec3118,ae2b33f539d68a0d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:23:58.617 INFO 2981023 --- [Collect all tenants major compactions6,c39fb7e834ec3118,f1a07713ec6262f3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:23:58.617 INFO 2981023 --- [Collect all tenants major compactions6,c39fb7e834ec3118,f1a07713ec6262f3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:23:58.617 INFO 2981023 --- [Collect all tenants major compactions1,c39fb7e834ec3118,59379247e6c6ed01] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:23:58.618 INFO 2981023 --- [Collect all tenants major compactions1,c39fb7e834ec3118,59379247e6c6ed01] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:23:58.619 INFO 2981023 --- [Collect all tenants major compactions4,c39fb7e834ec3118,749ea828afc730ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:58.623 INFO 2981023 --- [Collect all tenants major compactions4,c39fb7e834ec3118,749ea828afc730ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:23:58.627 INFO 2981023 --- [Collect all tenants major compactions5,c39fb7e834ec3118,bc14a37fdec56229] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:23:58.629 INFO 2981023 --- [Collect all tenants major compactions5,c39fb7e834ec3118,bc14a37fdec56229] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:23:58.629 INFO 2981023 --- [Collect all tenants major compactions4,c39fb7e834ec3118,749ea828afc730ce] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:23:58.630 INFO 2981023 --- [Collect all tenants major compactions4,c39fb7e834ec3118,749ea828afc730ce] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:23:58.631 INFO 2981023 --- [Collect all tenants major compactions5,c39fb7e834ec3118,bc14a37fdec56229] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:23:58.631 INFO 2981023 --- [Collect all tenants major compactions5,c39fb7e834ec3118,bc14a37fdec56229] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:23:58.632 INFO 2981023 --- [ocp-schedules-14,c39fb7e834ec3118,3c2d7c3ac2865d42] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:23:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,a6360b1c5ea9ebde,58101c09190e5e5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050240, interval=60 2024-09-11 18:23:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,a6360b1c5ea9ebde,58101c09190e5e5c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:23:59.454 INFO 2981023 --- [ocp-async-4,21f7e6b1418eac7b,6ce0df3d3db46da0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:00.184 INFO 2981023 --- [ocp-schedules-14,ed6fdac3abb278fb,98fd60066d31256f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:24:00.184+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:24:00.187 INFO 2981023 --- [ocp-schedules-14,ed6fdac3abb278fb,98fd60066d31256f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:24:00.254 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:00.256 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:00.331 INFO 2981023 --- [ocp-schedules-14,ed6fdac3abb278fb,98fd60066d31256f] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 59 to SqlAggregateData 2024-09-11 18:24:00.333 INFO 2981023 --- [ocp-schedules-14,ed6fdac3abb278fb,98fd60066d31256f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:24:01.454 INFO 2981023 --- [ocp-async-7,631709d8ce613b56,3785c3fda84fc89a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:01.855 INFO 2981023 --- [ocp-schedules-16,45b8891a13ebf9b4,827425c16dc4fce7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:24:01.859 INFO 2981023 --- [ocp-schedules-16,45b8891a13ebf9b4,827425c16dc4fce7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:24:01.897 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:24:01.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:24:01.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.897 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.900 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:24:01.900 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:24:01.901 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:24:01.903 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.903 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.904 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.905 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.909 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.909 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.909 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:24:01.918 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.919 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.921 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:24:01.923 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.924 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:24:01.930 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.931 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:24:01.933 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.934 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:24:01.938 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.939 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.945 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.945 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:24:01.946 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.946 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:24:01.947 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.947 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:24:01.948 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:24:01.952 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.952 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:24:01.952 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:24:01.953 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.954 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:24:01.954 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.955 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:24:01.955 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.956 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:24:01.957 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.958 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:24:01.958 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.959 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:24:01.960 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.961 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:24:01.961 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.962 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:24:01.962 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.963 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:24:01.963 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:24:01.965 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:24:01.968 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.969 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.972 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:01.973 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.974 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.976 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:01.978 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.978 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:24:01.983 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:24:01.985 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.985 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:24:01.985 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.986 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:24:01.986 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6031D6D8-0-0] 2024-09-11 18:24:01.986 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:24:01.986 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:24:01.986 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.987 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.987 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:24:01.987 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:24:01.987 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:24:01.987 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:24:01.987 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:24:01.991 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:24:01.993 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:24:01.993 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:01.993 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:01.997 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:24:01.999 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:24:01.999 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:24:01.999 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:24:02.005 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.006 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:24:02.016 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.016 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:24:02.017 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:24:02.017 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6001D6EE-0-0] 2024-09-11 18:24:02.017 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:24:02.022 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.023 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:24:02.023 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.024 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:24:02.024 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.025 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:24:02.026 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.026 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:24:02.027 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.027 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:24:02.028 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.028 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:24:02.029 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:24:02.030 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:24:02.040 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:02.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:24:02.109 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:24:02.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:24:02.116 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:02.116 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:24:02.126 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:24:02.127 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:24:02.127 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:24:02.127 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:24:02.127 INFO 2981023 --- [ocp-schedules-16,45b8891a13ebf9b4,827425c16dc4fce7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:24:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,5b4b975a3b36a2ae,1a521010f48c7564] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,5b4b975a3b36a2ae,1a521010f48c7564] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:02.786 INFO 2981023 --- [ocp-schedules-21,afc47439c2650b2b,72fec209929ffacd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:24:03.179 INFO 2981023 --- [ocp-schedules-11,5e1f39cf41e454f5,9d0144d5b60c587e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:24:03.180 INFO 2981023 --- [ocp-schedules-11,5e1f39cf41e454f5,9d0144d5b60c587e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:24:03.200 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:03.200 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:24:03.205 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:03.207 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:24:03.207 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050243207000} 2024-09-11 18:24:03.213 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:24:03.217 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000236, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:24:03.217 INFO 2981023 --- [ocp-schedules-11,5e1f39cf41e454f5,9d0144d5b60c587e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:24:03.256 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:03.257 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:03.455 INFO 2981023 --- [ocp-async-3,56cae1cef7ff1138,dbafcb335511dccf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:04.079 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:24:04.079 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:24:04.079 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:24:04.083 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000236, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:24:04.086 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:24:04.343 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000236 2024-09-11 18:24:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,95da20a474ada1da,e1ec97675a7dd2f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,95da20a474ada1da,e1ec97675a7dd2f5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:05.455 INFO 2981023 --- [ocp-async-10,2bc66f7e020b2049,899e1cfee24b807d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:24:05.984 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:24:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:24:05.995 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:24:05.995 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:24:05.995 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:24:05.998 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:05.998 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:24:05.999 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:06.000 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:06.061 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:21:41, start=2024-09-11T18:17:41, end=2024-09-11T18:20:11, updates=285, granule=120000ms, delay=240000ms 2024-09-11 18:24:06.067 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:21:41, target=2024-09-11T18:20:11, granule=120000ms, delay=90000ms) 2024-09-11 18:24:06.067 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:20:11, target=2024-09-11T18:10:51, granule=600000ms, delay=560000ms) 2024-09-11 18:24:06.067 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=100ms, period=120000ms 2024-09-11 18:24:06.257 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:06.257 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:24:06.352653+08:00, lastProcessTime=2024-09-11T18:23:56.354944+08:00 2024-09-11 18:24:06.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:24:06.352653+08:00 2024-09-11 18:24:07.455 INFO 2981023 --- [ocp-async-4,44f71240c9f33c5a,26cb50be4f8f5cec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:07.793 INFO 2981023 --- [ocp-async-6,4e6a97b0f25a3132,10d608f0335f537c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:24:08.146 INFO 2981023 --- [ocp-schedules-13,31df38def4cdb6e7,5fca5370955fee58] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:24:08.228 INFO 2981023 --- [ocp-schedules-20,c07260db4120a505,1fd1990b9ef5380a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:24:08.228+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:24:08.228 INFO 2981023 --- [ocp-schedules-20,c07260db4120a505,1fd1990b9ef5380a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:24:08.231 INFO 2981023 --- [ocp-schedules-20,c07260db4120a505,1fd1990b9ef5380a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:24:08.231 INFO 2981023 --- [ocp-schedules-20,c07260db4120a505,1fd1990b9ef5380a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:24:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,7b9e471cef4f6578,17a5d566c09cd724] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,7b9e471cef4f6578,17a5d566c09cd724] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:08.557 INFO 2981023 --- [ocp-schedules-28,af3a66c7cd8c0345,c99d04ed5c26a05e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:24:08.600 WARN 2981023 --- [scanner-13660735-1,af3a66c7cd8c0345,d1708a23925598bc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:24:08.647 INFO 2981023 --- [alarm-detect-20,af3a66c7cd8c0345,38ae2eba014f9bdc] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050210 ~ 1726050238 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050225, value=7.828)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.828, values={1726050225=7.828}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:08.647 INFO 2981023 --- [alarm-detect-20,af3a66c7cd8c0345,38ae2eba014f9bdc] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050210 ~ 1726050238 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050225, value=7.828)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.828, values={1726050225=7.828}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:24:08.729 INFO 2981023 --- [alarm-detect-13,af3a66c7cd8c0345,d83e4d37a703f763] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050210 ~ 1726050238 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050225, value=101.395)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=101.395, values={1726050225=101.395}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:09.258 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:09.258 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:09.455 INFO 2981023 --- [ocp-async-8,61619746d1faed01,f4614991d946854f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:09.799 INFO 2981023 --- [ocp-schedules-18,98e4967e702c85ad,021d8b30bbc46e63] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:24:10.003 INFO 2981023 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:24:10.003 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:24:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:24:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:24:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,cb29a030e3a7fa70,23216cb0ebb8d948] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,cb29a030e3a7fa70,23216cb0ebb8d948] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:11.455 INFO 2981023 --- [ocp-async-5,060ccdb29f3ad15d,0b163e13707d859e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:12.258 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:12.260 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:12.789 INFO 2981023 --- [ocp-schedules-21,293a42a6e8a27381,d6a64fef49225ce6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:24:13.455 INFO 2981023 --- [ocp-async-9,1c0050a82316e8ff,e6502038a7ac710e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:14.239 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050180, nodeCount=1000, affectRows=1000 2024-09-11 18:24:14.240 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050180, nodeCount=1000, affectRows=1000 2024-09-11 18:24:14.242 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050180, nodeCount=999, validCount=999 2024-09-11 18:24:14.243 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050180, nodeCount=1000, validCount=1000 2024-09-11 18:24:14.249 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050180, nodeCount=49, affectRows=49 2024-09-11 18:24:14.249 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050180, nodeCount=47, validCount=47 2024-09-11 18:24:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,c558f93e4fe5dc85,47d4b2e4cddf88a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,e41f0bb73f1801f5,7e73306c100009dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:24:14.318 INFO 2981023 --- [prometheus-minute-schedule-1,c558f93e4fe5dc85,47d4b2e4cddf88a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:14.643 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050180, nodeCount=480, affectRows=480 2024-09-11 18:24:15.260 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:15.262 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:15.455 INFO 2981023 --- [ocp-async-7,66654b7ddbe29645,cea422f58cb32d4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:15.706 INFO 2981023 --- [ocp-schedules-7,607774715655db6c,a9c7ef825a041b11] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:24:15.713 INFO 2981023 --- [ocp-schedules-7,607774715655db6c,a9c7ef825a041b11] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:24:15.713 INFO 2981023 --- [ocp-schedules-7,607774715655db6c,a9c7ef825a041b11] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:24:16.015 INFO 2981023 --- [ocp-schedules-26,10d7a5c59b86dbff,45c78b846a7539bf] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:24:16.016 INFO 2981023 --- [host-check-clockdiff1,10d7a5c59b86dbff,d3fe14e901fdb872] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:24:16.016 INFO 2981023 --- [host-check-clockdiff2,10d7a5c59b86dbff,f015951a2bfc5077] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:24:16.016 INFO 2981023 --- [host-check-clockdiff3,10d7a5c59b86dbff,229bbdf9d0b1d15f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:24:16.018 INFO 2981023 --- [host-check-clockdiff4,10d7a5c59b86dbff,c1cc7b4829577127] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:24:16.024 INFO 2981023 --- [ocp-schedules-2,ad4fa9bd05e75677,268bae2a83584e25] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:24:16.028 INFO 2981023 --- [ocp-schedules-19,90d40b5475da6737,c051143c81e16746] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:24:16.029 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:24:16.030 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:24:16.030 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:24:16.030 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:24:16.031 INFO 2981023 --- [host-check-agent-status1,90d40b5475da6737,2390e92f155851a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:24:16.032 INFO 2981023 --- [host-check-agent-status2,90d40b5475da6737,5ec4f23a7f39e7a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:24:16.045 INFO 2981023 --- [host-check-agent-status3,90d40b5475da6737,a18c0d953813db2c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:24:16.046 INFO 2981023 --- [host-check-agent-status4,90d40b5475da6737,c491218cd9e72c4f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:24:16.048 INFO 2981023 --- [host-check-clockdiff3,10d7a5c59b86dbff,229bbdf9d0b1d15f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050256 0 0 , error= 2024-09-11 18:24:16.066 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:16.067 INFO 2981023 --- [host-check-agent-status1,90d40b5475da6737,2390e92f155851a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:16.068 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:24:16.072 INFO 2981023 --- [host-check-agent-status1,90d40b5475da6737,2390e92f155851a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:16.075 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:24:16.078 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:16.079 INFO 2981023 --- [host-check-agent-status3,90d40b5475da6737,a18c0d953813db2c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:16.081 INFO 2981023 --- [host-check-agent-status3,90d40b5475da6737,a18c0d953813db2c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:16.081 INFO 2981023 --- [host-check-agent-status2,90d40b5475da6737,5ec4f23a7f39e7a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:24:16.082 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:16.083 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:16.083 INFO 2981023 --- [host-check-agent-status2,90d40b5475da6737,5ec4f23a7f39e7a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:24:16.190 INFO 2981023 --- [ocp-schedules-11,02ac63f8efabe092,6b80e3982d7ba612] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:24:16.190 INFO 2981023 --- [ocp-schedules-12,3264998520007d10,c8afa2300d11882b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:24:16.194 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:24:16.197 INFO 2981023 --- [ocp-schedules-25,038ace1ad48875ab,97e1387e3afaee5c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:24:16.199 INFO 2981023 --- [ocp-schedules-21,d81307d3d635843b,1786f253eb11002e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:24:16.260 INFO 2981023 --- [host-check-agent-status4,90d40b5475da6737,c491218cd9e72c4f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:24:16.261 INFO 2981023 --- [ocp-schedules-12,3264998520007d10,c8afa2300d11882b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:24:16.262 INFO 2981023 --- [ocp-schedules-25,038ace1ad48875ab,97e1387e3afaee5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:24:16.262 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:24:16.265 INFO 2981023 --- [host-check-agent-status4,90d40b5475da6737,c491218cd9e72c4f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:24:16.269 INFO 2981023 --- [ocp-schedules-11,02ac63f8efabe092,6b80e3982d7ba612] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:24:16.290 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050243207000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:24:16.291 INFO 2981023 --- [ocp-schedules-12,3264998520007d10,c8afa2300d11882b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:24:16.385 INFO 2981023 --- [ocp-schedules-16,3e8824a5e0d81453,dd633af9f9d6ca51] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:24:16.396 INFO 2981023 --- [host-check-agent-status2,90d40b5475da6737,5ec4f23a7f39e7a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:24:16.400 INFO 2981023 --- [host-check-agent-status2,90d40b5475da6737,5ec4f23a7f39e7a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:24:16.400 INFO 2981023 --- [host-check-agent-status2,90d40b5475da6737,5ec4f23a7f39e7a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:24:16.400 INFO 2981023 --- [host-check-agent-status2,90d40b5475da6737,5ec4f23a7f39e7a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:24:16.462 INFO 2981023 --- [host-check-agent-status4,90d40b5475da6737,c491218cd9e72c4f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:24:16.463 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:24:16.464 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:24:16.465 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:24:16.469190364+08:00 2024-09-11 18:24:16.467 INFO 2981023 --- [host-check-agent-status4,90d40b5475da6737,c491218cd9e72c4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:24:16.467 INFO 2981023 --- [host-check-agent-status4,90d40b5475da6737,c491218cd9e72c4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:24:16.467 INFO 2981023 --- [host-check-agent-status4,90d40b5475da6737,c491218cd9e72c4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:24:16.467 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:24:16.466105515+08:00 2024-09-11 18:24:16.468 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:24:16.470 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:24:16.465+08:00 2024-09-11 18:24:16.470 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:24:16.470490319+08:00 2024-09-11 18:24:16.471 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:24:16.467+08:00 2024-09-11 18:24:16.473 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:24:16.465+08:00 and status AVAILABLE 2024-09-11 18:24:16.473 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:24:16.467+08:00 and status AVAILABLE 2024-09-11 18:24:16.474 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:24:16.472210+08:00, lastProcessTime=2024-09-11T18:24:06.352653+08:00 2024-09-11 18:24:16.474 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:24:16.470+08:00 2024-09-11 18:24:16.477 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:24:16.470+08:00 and status AVAILABLE 2024-09-11 18:24:16.479 INFO 2981023 --- [host-check-agent-status3,90d40b5475da6737,a18c0d953813db2c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:24:16.483 INFO 2981023 --- [host-check-agent-status3,90d40b5475da6737,a18c0d953813db2c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:24:16.483 INFO 2981023 --- [host-check-agent-status3,90d40b5475da6737,a18c0d953813db2c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:24:16.483 INFO 2981023 --- [host-check-agent-status3,90d40b5475da6737,a18c0d953813db2c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:24:16.483 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:24:16.484 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:24:16.484 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:24:16.483971426+08:00 2024-09-11 18:24:16.486 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:24:16.485+08:00 2024-09-11 18:24:16.490 INFO 2981023 --- [host-check-agent-status1,90d40b5475da6737,2390e92f155851a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:24:16.490 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:24:16.485+08:00 and status AVAILABLE 2024-09-11 18:24:16.492 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:24:16.492 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:24:16.509 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.510 INFO 2981023 --- [host-check-agent-status1,90d40b5475da6737,2390e92f155851a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:24:16.510 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:24:16.510 INFO 2981023 --- [host-check-agent-status1,90d40b5475da6737,2390e92f155851a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:24:16.510 INFO 2981023 --- [host-check-agent-status1,90d40b5475da6737,2390e92f155851a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:24:16.510 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:24:16.556 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:24:16.563 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:16.564 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:16.569 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:24:16.569 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:16.571 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:16.572 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:24:16.573 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:24:16.573 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:24:16.575 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.576 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:16.580 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:24:16.586 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.586 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:24:16.587 INFO 2981023 --- [ocp-schedules-11,02ac63f8efabe092,6b80e3982d7ba612] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:24:16.591 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:24:16.593 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:16.596 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.597 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:16.599 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:24:16.607 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:24:16.607 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:24:16.623 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:16.624 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.625 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:24:16.628 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:24:16.630 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:24:16.641 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:24:16.664 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:24:16.669 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:24:16.670 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:24:16.685 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:24:16.685 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050243207000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:24:16.696 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050180, nodeCount=480, validCount=480 2024-09-11 18:24:16.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:16.700 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.700 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:16.701 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:24:16.701 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.701 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:24:16.703 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:24:16.704 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:24:16.708 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:24:16.716 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:24:16.719 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:24:16.719 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:24:16.734 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:24:16.736 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:24:16.737 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:24:16.757 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:24:16.758 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:24:16.760 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:24:16.760 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:24:16.765 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:24:16.766 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:24:16.775 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:16.775 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:16.776 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.776 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:24:16.777 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:24:16.780 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:24:16.781 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:24:16.784 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.785 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:16.786 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:24:16.786 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:16.787 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:24:16.791 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:24:16.793 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:24:16.805 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:24:16.807 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:24:16.812 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:24:16.821 INFO 2981023 --- [host-check-host-status4,ad4fa9bd05e75677,898accf021afc783] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:24:16.822 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:16.822 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.823 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:16.823 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:24:16.823 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.826 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:24:16.827 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:24:16.828 INFO 2981023 --- [host-check-host-status2,ad4fa9bd05e75677,39cb720c9cf4bcc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:24:16.837 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:24:16.837 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:24:16.839 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:24:16.850 INFO 2981023 --- [host-check-host-status3,ad4fa9bd05e75677,0a8dd4f027cbb23c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:24:16.854 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:24:16.876 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:16.891 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.892 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:24:16.894 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:24:16.903 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:24:16.905 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:16.906 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:16.912 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:24:16.472210+08:00 2024-09-11 18:24:16.916 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:24:16.921 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:24:16.924 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:16.925 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:16.925 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:24:16.930 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:24:16.930 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:24:16.935 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:24:16.941 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:24:16.953 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:24:17.234 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:24:17.240 INFO 2981023 --- [host-check-host-status1,ad4fa9bd05e75677,b77ad705b2877bbf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:24:17.262 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:24:17.268 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:17.276 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:17.277 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:24:17.288 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:24:17.289 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:24:17.289 INFO 2981023 --- [ocp-schedules-25,038ace1ad48875ab,97e1387e3afaee5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:24:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,2ffd483d0c491c0b,3f576c2f368301cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,2ffd483d0c491c0b,3f576c2f368301cf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:17.455 INFO 2981023 --- [ocp-async-6,bd6f5a1c004f58d4,d1e7368ba56c7a9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:17.793 INFO 2981023 --- [ocp-async-9,13950c61bb37a985,245ddf1d103b5e57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:24:17.806 INFO 2981023 --- [host-check-clockdiff2,10d7a5c59b86dbff,f015951a2bfc5077] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050257 5 5 , error= 2024-09-11 18:24:18.262 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:18.263 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:18.748 INFO 2981023 --- [ocp-schedules-12,05f4d7ac3c37f464,f1a1ebadd776dcef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:24:18.780 WARN 2981023 --- [scanner-13662436-1,05f4d7ac3c37f464,301e15e3667497e2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:24:18.815 INFO 2981023 --- [alarm-detect-10,05f4d7ac3c37f464,dbc656e3da60e9fe] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050225 ~ 1726050248 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050240, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050240, value=98.82)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.82, values={1726050240=98.82}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050240=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:18.817 INFO 2981023 --- [alarm-detect-9,05f4d7ac3c37f464,0874fe67a70da0cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646167, data is empty. 2024-09-11 18:24:18.820 INFO 2981023 --- [alarm-detect-9,05f4d7ac3c37f464,0874fe67a70da0cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646226, data is empty. 2024-09-11 18:24:18.824 INFO 2981023 --- [alarm-detect-9,05f4d7ac3c37f464,0874fe67a70da0cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646226, data is empty. 2024-09-11 18:24:18.856 INFO 2981023 --- [alarm-detect-14,05f4d7ac3c37f464,30041db50b33166e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646167, data is empty. 2024-09-11 18:24:18.857 INFO 2981023 --- [alarm-detect-15,05f4d7ac3c37f464,5e249879e8574c9e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646167, data is empty. 2024-09-11 18:24:18.861 INFO 2981023 --- [alarm-detect-14,05f4d7ac3c37f464,30041db50b33166e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646226, data is empty. 2024-09-11 18:24:18.864 INFO 2981023 --- [alarm-detect-15,05f4d7ac3c37f464,5e249879e8574c9e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646226, data is empty. 2024-09-11 18:24:18.868 INFO 2981023 --- [alarm-detect-14,05f4d7ac3c37f464,30041db50b33166e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646226, data is empty. 2024-09-11 18:24:18.868 INFO 2981023 --- [alarm-detect-15,05f4d7ac3c37f464,5e249879e8574c9e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646226, data is empty. 2024-09-11 18:24:18.884 INFO 2981023 --- [alarm-detect-7,05f4d7ac3c37f464,065873b4e82616df] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050225 ~ 1726050248 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050240, value=98.82)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.82, values={1726050240=98.82}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:18.886 INFO 2981023 --- [alarm-detect-18,05f4d7ac3c37f464,9b9b0900a4f599e1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050225 ~ 1726050248 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050240, value=98.82)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.82, values={1726050240=98.82}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:18.919 INFO 2981023 --- [alarm-detect-16,05f4d7ac3c37f464,1c6919d8ec8144e0] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050225 ~ 1726050248 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050240, value=9.401)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=9.401, values={1726050240=9.401}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:18.920 INFO 2981023 --- [alarm-detect-16,05f4d7ac3c37f464,1c6919d8ec8144e0] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050225 ~ 1726050248 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050240, value=9.401)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=9.401, values={1726050240=9.401}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:24:18.970 INFO 2981023 --- [alarm-detect-8,05f4d7ac3c37f464,18b31d419917b09d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646167, data is empty. 2024-09-11 18:24:18.973 INFO 2981023 --- [alarm-detect-8,05f4d7ac3c37f464,18b31d419917b09d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646226, data is empty. 2024-09-11 18:24:18.975 INFO 2981023 --- [alarm-detect-8,05f4d7ac3c37f464,18b31d419917b09d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646226, data is empty. 2024-09-11 18:24:19.004 INFO 2981023 --- [alarm-detect-9,05f4d7ac3c37f464,69145c0a5ed72eae] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050225 ~ 1726050248 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050240, value=101.628)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=101.628, values={1726050240=101.628}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:19.456 INFO 2981023 --- [ocp-async-1,7a9c0a544451db41,cf40c5d2fd85f425] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:20.160 INFO 2981023 --- [host-check-clockdiff1,10d7a5c59b86dbff,d3fe14e901fdb872] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050260 0 0 , error= 2024-09-11 18:24:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,ff1acf265a99c7bf,a17ad68894e43b2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:20.315 INFO 2981023 --- [prometheus-minute-schedule-1,ff1acf265a99c7bf,a17ad68894e43b2b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:21.263 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:21.263 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:21.388 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:24:21.388 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000009 2024-09-11 18:24:21.388 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000010 2024-09-11 18:24:21.388 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000007 2024-09-11 18:24:21.388 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2024-09-11 18:24:21.421 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:24:21.422 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:24:21.436 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:24:21.442 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:24:21.446 WARN 2981023 --- [ForkJoinPool.commonPool-worker-2,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:24:21.455 WARN 2981023 --- [ForkJoinPool.commonPool-worker-3,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:24:21.456 INFO 2981023 --- [ocp-async-3,1466382707eee947,b7b84ef5aee1b842] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:21.466 WARN 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:24:21.467 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:24:21.467 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:24:21.467 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:24:21.468 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000007 agent sql collect property 2024-09-11 18:24:21.472 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:24:21.472 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:24:21.472 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:24:21.473 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2024-09-11 18:24:21.476 WARN 2981023 --- [ForkJoinPool.commonPool-worker-0,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:24:21.485 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:21.491 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:21.492 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:24:21.494 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:24:21.495 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:24:21.495 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:24:21.496 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:24:21.496 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000010 agent sql collect property 2024-09-11 18:24:21.497 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:24:21.497 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:24:21.497 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:24:21.498 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000009 agent sql collect property 2024-09-11 18:24:21.511 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2000009 to set sql collect property inactive 2024-09-11 18:24:21.512 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:21.512 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:24:21.514 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:24:21.514 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:21.520 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (monagent.ob.err.code.prefix=errcode=-), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.cgroup.status=active), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.run.user=admin), (ob.logcleaner.enabled=true), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.pipeline.awr.collect.disabled=true), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.ob.rpc.port=2882), (monagent.monitordb.user=root@ocp_monitor), (monagent.host.ip=10.31.35.6), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"includeKeywordsRelation":"Or"," 2024-09-11 18:24:21.520 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1718689631), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.5), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=rtdob), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"in 2024-09-11 18:24:21.536 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (monagent.ob.err.code.prefix=errcode=-), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.cgroup.status=active), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.run.user=admin), (ob.logcleaner.enabled=true), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.pipeline.awr.collect.disabled=true), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.ob.rpc.port=2882), (monagent.monitordb.user=root@ocp_monitor), (monagent.host.ip=10.31.35.7), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"includeKeywordsRelation":"Or"," 2024-09-11 18:24:21.537 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994437), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.10), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkprdocp), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"include 2024-09-11 18:24:21.539 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000007, do nothing 2024-09-11 18:24:21.543 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2024-09-11 18:24:21.552 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000009, do nothing 2024-09-11 18:24:21.553 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000010, do nothing 2024-09-11 18:24:21.554 INFO 2981023 --- [ocp-schedules-24,d538d50618fee6ae,a524e36b2dfb993e] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-09-11 18:24:21.941 INFO 2981023 --- [host-check-clockdiff4,10d7a5c59b86dbff,c1cc7b4829577127] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050261 0 0 , error= 2024-09-11 18:24:22.791 INFO 2981023 --- [ocp-schedules-10,3fd9c2d69db9c9c0,91452348345bd3cc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:24:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,98fd3148a2dd0145,aabd32efc7b7ec6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,98fd3148a2dd0145,aabd32efc7b7ec6e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:23.456 INFO 2981023 --- [ocp-async-2,6538f6c14511585a,b25435bc8ebac2ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:24.263 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:24.264 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:24.492 INFO 2981023 --- [ocp-schedules-29,c707637a98a90359,33c3e0762c3f3ad6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:24:25.456 INFO 2981023 --- [ocp-async-10,99a361b0786887b9,9fbfc5b7cd3b814a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,6492b4c11a6f9dad,bcc245b2399ecfdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,6492b4c11a6f9dad,bcc245b2399ecfdb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:26.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:24:26.354018+08:00, lastProcessTime=2024-09-11T18:24:16.472210+08:00 2024-09-11 18:24:26.495 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:24:26.354018+08:00 2024-09-11 18:24:27.264 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:27.266 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:27.456 INFO 2981023 --- [ocp-async-4,5c9cd826ca69bfce,ac1d12947b1e5d73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:27.629 INFO 2981023 --- [ocp-schedules-7,629961d51ce55128,6570e81a3226ab9d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:24:27.630 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:24:27.630 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:24:27.630 INFO 2981023 --- [ocp-schedules-7,629961d51ce55128,6570e81a3226ab9d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:24:27.630 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:24:27.634 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:27.634 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:24:27.635 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:24:27.638 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:27.638 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:24:27.639 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:24:27.642 INFO 2981023 --- [ocp-schedules-7,629961d51ce55128,6570e81a3226ab9d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:27.643 INFO 2981023 --- [ocp-schedules-7,629961d51ce55128,6570e81a3226ab9d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:27.671 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:24:27.672 INFO 2981023 --- [ocp-schedules-7,629961d51ce55128,6570e81a3226ab9d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:24:27.672 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:24:27.673 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:24:27.684 INFO 2981023 --- [ocp-schedules-7,629961d51ce55128,6570e81a3226ab9d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:24:27.793 INFO 2981023 --- [ocp-async-2,16bbb9eb4c0c90f6,29a6b1509c8d2652] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:24:28.256 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 18:24:28.256 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 18:24:28.257 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.257 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:24:28.258 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.258 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.259 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.259 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 18:24:28.270 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.270 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.271 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.271 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 18:24:28.280 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.280 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.281 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.281 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 18:24:28.290 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.290 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.291 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.291 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 18:24:28.300 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.301 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.301 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.301 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 18:24:28.305 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.305 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.306 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.306 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 18:24:28.309 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.309 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.310 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.310 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 18:24:28.313 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.313 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.314 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.314 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 18:24:28.319 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.319 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.319 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.320 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 18:24:28.324 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.325 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.325 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.326 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 18:24:28.329 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.329 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.330 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.330 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 18:24:28.336 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.336 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.336 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.337 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 18:24:28.346 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.346 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.346 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.347 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 18:24:28.349 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.350 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.350 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.350 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 18:24:28.354 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.355 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.355 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.355 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 18:24:28.359 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.359 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.359 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.360 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 18:24:28.362 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.363 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.363 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.363 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 18:24:28.366 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.367 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.367 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.367 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 18:24:28.370 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.370 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.371 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.371 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 18:24:28.374 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.374 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.375 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.375 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 18:24:28.378 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.378 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.379 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.379 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 18:24:28.382 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.382 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.383 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.383 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 18:24:28.386 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.386 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.387 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.387 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 18:24:28.390 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.390 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.390 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.391 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 18:24:28.393 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.394 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.394 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.394 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 18:24:28.397 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.397 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.398 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.398 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 18:24:28.401 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.401 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.401 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.402 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 18:24:28.404 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.405 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.405 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.405 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 18:24:28.408 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.408 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.409 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.409 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 18:24:28.412 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.412 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.413 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.413 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 18:24:28.415 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.416 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.416 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.417 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 18:24:28.419 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.420 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.420 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.420 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 18:24:28.423 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.423 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.424 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.424 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 18:24:28.426 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.427 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.427 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.427 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 18:24:28.432 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.432 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.433 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.433 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 18:24:28.436 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.437 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.437 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.437 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 18:24:28.440 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.441 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.441 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.441 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 18:24:28.445 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.445 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.445 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.446 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 18:24:28.448 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.448 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.449 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.449 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 18:24:28.451 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.452 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.452 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.452 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 18:24:28.455 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.455 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.456 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.456 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 18:24:28.459 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.459 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.460 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.460 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 18:24:28.467 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.467 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.467 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.468 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 18:24:28.483 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.484 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.484 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.484 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 18:24:28.489 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.489 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.489 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.490 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 18:24:28.494 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.494 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.495 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.495 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 18:24:28.499 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.499 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.500 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.500 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 18:24:28.502 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.503 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.503 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.503 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 18:24:28.506 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.506 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.507 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.507 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 18:24:28.509 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 18:24:28.509 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 18:24:28.510 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.511 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:24:28.512 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.513 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.513 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.514 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 18:24:28.518 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.518 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:24:28.518 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:28.519 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 18:24:28.521 INFO 2981023 --- [ocp-schedules-13,e55f8d34b4ec0356,d7705587ddeaa21b] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 18:24:29.019 INFO 2981023 --- [ocp-schedules-6,6f7224fcb19ee933,36507a2af91412b2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:24:29.068 WARN 2981023 --- [scanner-13662783-1,6f7224fcb19ee933,2d86608246cafbf1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:24:29.102 INFO 2981023 --- [alarm-detect-7,6f7224fcb19ee933,64e3aeb88c3f398e] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050240 ~ 1726050259 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050255, value=98.998)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.998, values={1726050255=98.998}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:29.109 INFO 2981023 --- [alarm-detect-14,6f7224fcb19ee933,46f7180e6b803051] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050240 ~ 1726050259 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050255, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050255, value=98.998)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.998, values={1726050255=98.998}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050255=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:29.141 INFO 2981023 --- [alarm-detect-11,6f7224fcb19ee933,c566f3930b467bdf] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050240 ~ 1726050259 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050255, value=98.998)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.998, values={1726050255=98.998}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,fbfae7b1c5769174,bc862502bd3d04e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:29.318 INFO 2981023 --- [prometheus-minute-schedule-1,fbfae7b1c5769174,bc862502bd3d04e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:29.456 INFO 2981023 --- [ocp-async-6,814d688e633083dc,7c3ae980ae4fec42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:30.266 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:30.268 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:31.456 INFO 2981023 --- [ocp-async-9,87e0ac39cb85c863,aba9bd5008cc980a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,87b9f186d498a280,5e141abca29e8973] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,87b9f186d498a280,5e141abca29e8973] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:32.794 INFO 2981023 --- [ocp-schedules-25,50c37bea671ad58c,d969a2459c2a6bb3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:24:33.268 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:33.268 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:33.456 INFO 2981023 --- [ocp-async-7,5bca9407849af52e,74d55fa43fc6a5f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,12b45c9ea7e9cace,b81c2b6b7d488bfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,12b45c9ea7e9cace,b81c2b6b7d488bfe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:35.456 INFO 2981023 --- [ocp-async-3,a5d54f88ca35fa77,0046e2b363c0cc1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:35.717 INFO 2981023 --- [ocp-schedules-4,b0a1f43639b8747f,593960fbf99e91b7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:24:35.725 INFO 2981023 --- [ocp-schedules-4,b0a1f43639b8747f,593960fbf99e91b7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:24:35.725 INFO 2981023 --- [ocp-schedules-4,b0a1f43639b8747f,593960fbf99e91b7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:24:36.268 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:36.269 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:36.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:24:36.351488+08:00, lastProcessTime=2024-09-11T18:24:26.354018+08:00 2024-09-11 18:24:36.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:24:36.351488+08:00 2024-09-11 18:24:37.314 INFO 2981023 --- [prometheus-cache-monitor1,ebc772352d239ee4,156a7fbfe0de64f9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:24:37.315 INFO 2981023 --- [prometheus-cache-monitor1,ebc772352d239ee4,156a7fbfe0de64f9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:24:37.457 INFO 2981023 --- [ocp-async-5,1890699d2f0bbe02,4ef34f65155befac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:37.793 INFO 2981023 --- [ocp-async-7,0c4c362f0dd2e015,07b4e308be00fe45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:24:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,d6eb9b58d64dfc37,c58238f98a677a79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,d6eb9b58d64dfc37,c58238f98a677a79] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:39.213 INFO 2981023 --- [ocp-schedules-22,2c05b700142f9e06,6ece5cc9e74e147b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:24:39.257 WARN 2981023 --- [scanner-13662969-1,2c05b700142f9e06,678c26e03db9aa58] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:24:39.269 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:39.270 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:39.283 INFO 2981023 --- [alarm-detect-2,2c05b700142f9e06,0d440d9499c754a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646226, data is empty. 2024-09-11 18:24:39.287 INFO 2981023 --- [alarm-detect-2,2c05b700142f9e06,0d440d9499c754a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646226, data is empty. 2024-09-11 18:24:39.298 INFO 2981023 --- [alarm-detect-2,2c05b700142f9e06,0d440d9499c754a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646226, data is empty. 2024-09-11 18:24:39.314 INFO 2981023 --- [alarm-detect-13,2c05b700142f9e06,e134d5f2c699b1a3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646226, data is empty. 2024-09-11 18:24:39.316 INFO 2981023 --- [alarm-detect-6,2c05b700142f9e06,fcfe8ba5a1ddff61] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646226, data is empty. 2024-09-11 18:24:39.318 INFO 2981023 --- [alarm-detect-13,2c05b700142f9e06,e134d5f2c699b1a3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646226, data is empty. 2024-09-11 18:24:39.322 INFO 2981023 --- [alarm-detect-6,2c05b700142f9e06,fcfe8ba5a1ddff61] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646226, data is empty. 2024-09-11 18:24:39.324 INFO 2981023 --- [alarm-detect-13,2c05b700142f9e06,e134d5f2c699b1a3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646226, data is empty. 2024-09-11 18:24:39.328 INFO 2981023 --- [alarm-detect-6,2c05b700142f9e06,fcfe8ba5a1ddff61] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646226, data is empty. 2024-09-11 18:24:39.369 INFO 2981023 --- [alarm-detect-12,2c05b700142f9e06,2c21cba09f21667e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050240 ~ 1726050269 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050255, value=9.868)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=9.868, values={1726050255=9.868}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:39.369 INFO 2981023 --- [alarm-detect-12,2c05b700142f9e06,2c21cba09f21667e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050240 ~ 1726050269 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050255, value=9.868)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=9.868, values={1726050255=9.868}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:24:39.440 INFO 2981023 --- [alarm-detect-9,2c05b700142f9e06,574cc6edfd555c16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646226, data is empty. 2024-09-11 18:24:39.446 INFO 2981023 --- [alarm-detect-9,2c05b700142f9e06,574cc6edfd555c16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646226, data is empty. 2024-09-11 18:24:39.450 INFO 2981023 --- [alarm-detect-9,2c05b700142f9e06,574cc6edfd555c16] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646226, data is empty. 2024-09-11 18:24:39.457 INFO 2981023 --- [ocp-async-4,65b57c266107626f,f412fa3ae662299c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:39.521 INFO 2981023 --- [alarm-detect-3,2c05b700142f9e06,2f7cb4f1cf236d17] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050240 ~ 1726050269 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050255, value=104.047)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.047, values={1726050255=104.047}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:40.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:24:40.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:24:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,d0159a1f76e0e5c0,f937c856ff01ad6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,d0159a1f76e0e5c0,f937c856ff01ad6e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:41.457 INFO 2981023 --- [ocp-async-2,e8a069726802165a,7b70c31a5cb64d61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:42.270 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:42.272 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:42.797 INFO 2981023 --- [ocp-schedules-24,c37b6ca6fe0219d1,8963ff773c2f0cc7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:24:43.457 INFO 2981023 --- [ocp-async-10,d2ec94aecf9600d3,4d8ed5907f55d3ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,0acb2134088e739e,d092a23111a89c03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:44.318 INFO 2981023 --- [prometheus-minute-schedule-1,0acb2134088e739e,d092a23111a89c03] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:44.925 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:24:44.925 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:24:44.928 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:24:44.928 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:24:44.928 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:24:44.928 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:24:44.930 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:24:44.930 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:24:44.930 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:24:44.930 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:24:44.952 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:24:44.955 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:24:44.955 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:24:44.955 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:24:44.955 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:24:44.955 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:24:44.959 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:24:45.101 INFO 2981023 --- [prometheus-second-schedule-1,d6487168069dc934,3bdf43023d84cce4] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:24:45.223 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:24:45.272 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:45.274 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:45.457 INFO 2981023 --- [ocp-async-9,0f6319622b1d39f3,981380fb898cc56c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:45.478 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.547766485S 2024-09-11 18:24:45.478 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:24:45.525 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:24:45.525 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:24:45.540 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:24:45.577 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:24:45.577 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:24:45.577 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:24:45.577 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:24:45.577 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:24:45.577 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:24:45.582 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:24:45.661 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] o.s.boot.SpringApplication : Started application in 0.733 seconds (JVM running for 86085.533) 2024-09-11 18:24:45.661 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.733228463S 2024-09-11 18:24:45.859 INFO 2981023 --- [ocp-schedules-22,62392f308e52ea39,946a60d871737def] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:24:46.020 INFO 2981023 --- [ocp-schedules-4,f838481a5230ec4d,f52ffed8ebb22b31] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:24:46.027 INFO 2981023 --- [ocp-schedules-28,c33e3aa523cbd77b,ba41a28dfebddfc9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:24:46.028 INFO 2981023 --- [ocp-schedules-30,68fc16fd288af7d9,cdf086220b979da0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:24:46.029 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:24:46.029 INFO 2981023 --- [host-check-clockdiff2,f838481a5230ec4d,4e5bb70d9efd7601] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:24:46.029 INFO 2981023 --- [host-check-agent-status1,c33e3aa523cbd77b,7a8c2bb6ccdfa19e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:24:46.029 INFO 2981023 --- [host-check-agent-status2,c33e3aa523cbd77b,ef5d5ee317a045ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:24:46.029 INFO 2981023 --- [host-check-agent-status3,c33e3aa523cbd77b,6f035656b560498d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:24:46.030 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:24:46.030 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:24:46.031 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:24:46.032 INFO 2981023 --- [host-check-clockdiff1,f838481a5230ec4d,0104bf44640710dd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:24:46.033 INFO 2981023 --- [host-check-clockdiff3,f838481a5230ec4d,751f3cf7a42ecbf2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:24:46.036 INFO 2981023 --- [host-check-clockdiff4,f838481a5230ec4d,a337b02caae7c511] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:24:46.036 INFO 2981023 --- [host-check-agent-status4,c33e3aa523cbd77b,b6bf94e3385f3ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:24:46.069 INFO 2981023 --- [host-check-clockdiff3,f838481a5230ec4d,751f3cf7a42ecbf2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050286 0 0 , error= 2024-09-11 18:24:46.075 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:46.076 INFO 2981023 --- [host-check-agent-status1,c33e3aa523cbd77b,7a8c2bb6ccdfa19e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:46.076 INFO 2981023 --- [host-check-agent-status3,c33e3aa523cbd77b,6f035656b560498d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:46.079 INFO 2981023 --- [host-check-agent-status4,c33e3aa523cbd77b,b6bf94e3385f3ad7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:24:46.080 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:46.080 INFO 2981023 --- [host-check-agent-status3,c33e3aa523cbd77b,6f035656b560498d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:46.080 INFO 2981023 --- [host-check-agent-status2,c33e3aa523cbd77b,ef5d5ee317a045ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:24:46.080 INFO 2981023 --- [host-check-agent-status1,c33e3aa523cbd77b,7a8c2bb6ccdfa19e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:46.081 INFO 2981023 --- [host-check-agent-status2,c33e3aa523cbd77b,ef5d5ee317a045ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:24:46.086 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:46.086 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:24:46.087 INFO 2981023 --- [host-check-agent-status4,c33e3aa523cbd77b,b6bf94e3385f3ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:24:46.088 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:46.088 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:24:46.109 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:24:46.137 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:24:46.170 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:24:46.190 INFO 2981023 --- [ocp-schedules-27,bb8eca062ce7fec9,2d30377b1dfb8c02] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:24:46.193 INFO 2981023 --- [ocp-schedules-6,af1e8f373abbbe17,2f9f964037bf4314] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:24:46.198 INFO 2981023 --- [ocp-schedules-1,dfd155c03dc14b96,b0d99eca5cfb61a5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:24:46.198 INFO 2981023 --- [ocp-schedules-13,fbd9f4f27302bd1c,18f4373fed809437] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:24:46.270 INFO 2981023 --- [ocp-schedules-8,456b35deba47219f,d4f5b2c51167e8d2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:24:46.286 INFO 2981023 --- [ocp-schedules-1,dfd155c03dc14b96,b0d99eca5cfb61a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:24:46.358 INFO 2981023 --- [ocp-schedules-27,bb8eca062ce7fec9,2d30377b1dfb8c02] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:24:46.371 INFO 2981023 --- [ocp-schedules-6,af1e8f373abbbe17,2f9f964037bf4314] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:24:46.372 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050243207000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:24:46.374 INFO 2981023 --- [ocp-schedules-6,af1e8f373abbbe17,2f9f964037bf4314] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:24:46.388 INFO 2981023 --- [ocp-schedules-8,456b35deba47219f,d4f5b2c51167e8d2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:24:46.462 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:24:46.476 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:24:46.480 INFO 2981023 --- [host-check-agent-status3,c33e3aa523cbd77b,6f035656b560498d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:24:46.482 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:24:46.484005822+08:00 2024-09-11 18:24:46.489 INFO 2981023 --- [host-check-agent-status3,c33e3aa523cbd77b,6f035656b560498d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:24:46.489 INFO 2981023 --- [host-check-agent-status3,c33e3aa523cbd77b,6f035656b560498d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:24:46.489 INFO 2981023 --- [host-check-agent-status3,c33e3aa523cbd77b,6f035656b560498d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:24:46.490 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:24:46.482+08:00 2024-09-11 18:24:46.490 INFO 2981023 --- [ocp-schedules-9,431df308de9ddc8e,b97f8cbb9a30f3e7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:24:46.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:24:46.490153+08:00, lastProcessTime=2024-09-11T18:24:36.351488+08:00 2024-09-11 18:24:46.496 INFO 2981023 --- [host-check-agent-status4,c33e3aa523cbd77b,b6bf94e3385f3ad7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:24:46.499 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:24:46.482+08:00 and status AVAILABLE 2024-09-11 18:24:46.501 INFO 2981023 --- [host-check-agent-status4,c33e3aa523cbd77b,b6bf94e3385f3ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:24:46.501 INFO 2981023 --- [host-check-agent-status4,c33e3aa523cbd77b,b6bf94e3385f3ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:24:46.501 INFO 2981023 --- [host-check-agent-status4,c33e3aa523cbd77b,b6bf94e3385f3ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:24:46.556 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:24:46.560 INFO 2981023 --- [host-check-agent-status1,c33e3aa523cbd77b,7a8c2bb6ccdfa19e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:24:46.561 INFO 2981023 --- [host-check-agent-status2,c33e3aa523cbd77b,ef5d5ee317a045ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:24:46.563 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:24:46.564 INFO 2981023 --- [host-check-agent-status2,c33e3aa523cbd77b,ef5d5ee317a045ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:24:46.564 INFO 2981023 --- [host-check-agent-status2,c33e3aa523cbd77b,ef5d5ee317a045ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:24:46.564 INFO 2981023 --- [host-check-agent-status2,c33e3aa523cbd77b,ef5d5ee317a045ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:24:46.565 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:24:46.565 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:24:46.565560453+08:00 2024-09-11 18:24:46.566 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:24:46.565+08:00 2024-09-11 18:24:46.566 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:24:46.565705538+08:00 2024-09-11 18:24:46.567 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:24:46.567 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:24:46.566+08:00 2024-09-11 18:24:46.568 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:24:46.565+08:00 and status AVAILABLE 2024-09-11 18:24:46.569 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:24:46.568255928+08:00 2024-09-11 18:24:46.570 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:24:46.566+08:00 and status AVAILABLE 2024-09-11 18:24:46.571 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:24:46.569+08:00 2024-09-11 18:24:46.573 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:24:46.569+08:00 and status AVAILABLE 2024-09-11 18:24:46.574 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:24:46.577 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:24:46.583 INFO 2981023 --- [host-check-agent-status1,c33e3aa523cbd77b,7a8c2bb6ccdfa19e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:24:46.583 INFO 2981023 --- [host-check-agent-status1,c33e3aa523cbd77b,7a8c2bb6ccdfa19e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:24:46.583 INFO 2981023 --- [host-check-agent-status1,c33e3aa523cbd77b,7a8c2bb6ccdfa19e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:24:46.583 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.584 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:24:46.584 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:24:46.587 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.589 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:24:46.593 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:24:46.596 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:24:46.597 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:24:46.598 INFO 2981023 --- [ocp-schedules-27,bb8eca062ce7fec9,2d30377b1dfb8c02] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:24:46.599 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:24:46.603 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:46.604 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:24:46.604 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:46.605 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:24:46.606 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:46.609 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:46.613 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.613 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:46.668 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.668 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.672 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:24:46.672 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:46.674 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:24:46.677 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:24:46.677 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.677 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:24:46.678 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.678 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:46.681 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:24:46.689 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:24:46.698 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:24:46.701 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:46.705 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.705 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:24:46.706 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:24:46.710 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:24:46.714 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:24:46.724 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:24:46.726 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:24:46.733 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:24:46.777 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:24:46.777 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050243207000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:24:46.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:46.784 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:24:46.784 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.785 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:46.785 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:24:46.785 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:24:46.786 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:46.789 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:24:46.789 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.791 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:24:46.792 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:24:46.804 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:24:46.804 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:24:46.814 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:24:46.817 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.819 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:24:46.822 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:24:46.823 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:24:46.826 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:24:46.829 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:24:46.830 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:24:46.835 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.835 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:24:46.855 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:46.856 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:24:46.859 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:24:46.860 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.860 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:24:46.861 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:24:46.861 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:24:46.862 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:24:46.867 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:24:46.870 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:24:46.871 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:24:46.873 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:46.873 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.874 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:24:46.876 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:24:46.876 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:24:46.881 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:24:46.885 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:24:46.889 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:24:46.893 INFO 2981023 --- [host-check-host-status4,68fc16fd288af7d9,909d076780bb9207] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:24:46.898 INFO 2981023 --- [host-check-host-status2,68fc16fd288af7d9,14c67040b3f2fc35] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:24:46.907 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:24:46.908 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:46.912 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:24:46.919 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:46.923 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.924 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:24:46.925 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:24:46.931 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:24:46.932 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:46.934 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.934 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.936 INFO 2981023 --- [host-check-host-status3,68fc16fd288af7d9,ed7c5afe1a2850fb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:24:46.937 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:24:46.937 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:24:46.940 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:24:46.943 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.947 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:24:46.949 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:24:46.951 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:24:46.952 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.952 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:24:46.490153+08:00 2024-09-11 18:24:46.953 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:24:46.957 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:24:46.958 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:24:46.958 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:24:46.967 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:24:46.975 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:46.978 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:46.978 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:24:46.979 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:24:46.981 INFO 2981023 --- [host-check-clockdiff1,f838481a5230ec4d,0104bf44640710dd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050286 0 0 , error= 2024-09-11 18:24:46.984 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:24:46.988 INFO 2981023 --- [host-check-clockdiff2,f838481a5230ec4d,4e5bb70d9efd7601] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050286 5 5 , error= 2024-09-11 18:24:46.991 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:24:46.991 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:24:46.992 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:24:46.992 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:24:46.998 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:47.002 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:47.002 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:24:47.009 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:24:47.010 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:24:47.010 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:24:47.010 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:47.021 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:47.025 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:24:47.030 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:24:47.030 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:47.045 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:24:47.062 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:24:47.074 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:47.086 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:47.091 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:24:47.096 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:24:47.096 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:47.145 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:24:47.154 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:24:47.161 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:24:47.168 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:24:47.175 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:24:47.176 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:24:47.180 INFO 2981023 --- [ocp-schedules-23,b244c7df1fe9c835,67c445b34bc7aa72] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:24:47.252 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:24:47.257 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:24:47.265 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:47.269 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:24:47.279 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:24:47.279 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:24:47.279 INFO 2981023 --- [ocp-schedules-1,dfd155c03dc14b96,b0d99eca5cfb61a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:24:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,0957228e53932e9a,666f68892b5ab5a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,0957228e53932e9a,666f68892b5ab5a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:47.450 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:24:47.456 INFO 2981023 --- [host-check-host-status1,68fc16fd288af7d9,ee93eaf5cf93e2ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:24:47.457 INFO 2981023 --- [ocp-async-3,a4f67c35631313a0,fc6239ced63129a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:47.743 INFO 2981023 --- [ocp-schedules-15,67b714730f1e09d9,107c9456590eb561] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:24:47.760 WARN 2981023 --- [ocp-schedules-15,67b714730f1e09d9,107c9456590eb561] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:24:47.794 INFO 2981023 --- [ocp-async-4,b65ba7d3a95e1c28,2d4c1cc731504ae5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:24:48.274 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:48.274 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:49.457 INFO 2981023 --- [ocp-async-10,7036d1b00b8435f6,478bcf0e4dbe9a2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:49.528 INFO 2981023 --- [ocp-schedules-17,df5d83ff49afd4fc,06635e34465c6f30] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:24:49.573 WARN 2981023 --- [scanner-13663306-1,df5d83ff49afd4fc,8ddb5bce17b91912] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:24:49.576 INFO 2981023 --- [alarm-detect-10,df5d83ff49afd4fc,e139f10a2aec4cd3] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050255 ~ 1726050279 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050270, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050270, value=98.582)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.582, values={1726050270=98.582}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050270=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:49.576 INFO 2981023 --- [alarm-detect-10,df5d83ff49afd4fc,e139f10a2aec4cd3] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=os_tsar_cpu_util_hwm, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:24:49.620 INFO 2981023 --- [alarm-detect-13,df5d83ff49afd4fc,6c938d55c54ba472] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050255 ~ 1726050279 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050270, value=98.582)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.582, values={1726050270=98.582}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:49.634 INFO 2981023 --- [alarm-detect-13,df5d83ff49afd4fc,e11800d6a2ae8703] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050255 ~ 1726050279 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050270, value=9.297)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=9.297, values={1726050270=9.297}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:49.635 INFO 2981023 --- [alarm-detect-13,df5d83ff49afd4fc,e11800d6a2ae8703] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050255 ~ 1726050279 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050270, value=9.297)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=9.297, values={1726050270=9.297}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:24:49.637 INFO 2981023 --- [alarm-detect-16,df5d83ff49afd4fc,07ac0f8d39e173cb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050255 ~ 1726050279 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050270, value=98.582)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.582, values={1726050270=98.582}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:49.723 INFO 2981023 --- [alarm-detect-19,df5d83ff49afd4fc,a1aeb8c9779a3006] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050255 ~ 1726050279 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050270, value=103.819)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=103.819, values={1726050270=103.819}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:24:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:24:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,c75f1aa2ccb27532,878f4ec97e88f4a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,c75f1aa2ccb27532,878f4ec97e88f4a7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:50.366 INFO 2981023 --- [ocp-schedules-1,aff3becab4bf857e,930e8d3eea55686e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:24:51.274 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:51.275 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:51.457 INFO 2981023 --- [ocp-async-8,ac9429caf8636b6d,67f19d5ae6946ddc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:51.639 INFO 2981023 --- [ocp-schedules-24,454c8cd6773ac1e8,d01325cbb055393b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:24:51.640 INFO 2981023 --- [ocp-schedules-24,454c8cd6773ac1e8,d01325cbb055393b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:24:51.640 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:24:51.640 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:24:51.640 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:24:51.640 INFO 2981023 --- [ocp-schedules-24,454c8cd6773ac1e8,d01325cbb055393b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:24:52.002 INFO 2981023 --- [host-check-clockdiff4,f838481a5230ec4d,a337b02caae7c511] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050291 0 0 , error= 2024-09-11 18:24:52.800 INFO 2981023 --- [ocp-schedules-15,851281251031da11,dd8e40c3f3acb051] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:24:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,329da6b8fb49cea9,7efde4ee45a9afea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,329da6b8fb49cea9,7efde4ee45a9afea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:53.372 INFO 2981023 --- [ocp-schedules-1,685ab72ab6a5828d,c34841b03b2bdf65] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:24:53.372 INFO 2981023 --- [ocp-schedules-1,685ab72ab6a5828d,c34841b03b2bdf65] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:24:53.458 INFO 2981023 --- [ocp-async-2,99c51e8a811b2fac,be1aeb02e15265d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:54.275 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:54.276 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:54.495 INFO 2981023 --- [ocp-schedules-12,a84c4f4fcd17b0e7,4639a1e8b490b133] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:24:55.131 INFO 2981023 --- [ocp-schedules-13,13ccd551fec72fdf,dcbfac26376320c9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:24:55.131 INFO 2981023 --- [ocp-schedules-13,13ccd551fec72fdf,dcbfac26376320c9] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:24:55.135 INFO 2981023 --- [ocp-schedules-13,13ccd551fec72fdf,dcbfac26376320c9] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:24:55.135 INFO 2981023 --- [ocp-schedules-13,13ccd551fec72fdf,dcbfac26376320c9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:24:55.135 INFO 2981023 --- [ocp-schedules-13,13ccd551fec72fdf,dcbfac26376320c9] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:24:55.138 INFO 2981023 --- [ocp-schedules-13,13ccd551fec72fdf,dcbfac26376320c9] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:24:55.138 INFO 2981023 --- [ocp-schedules-13,13ccd551fec72fdf,dcbfac26376320c9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:24:55.138 INFO 2981023 --- [ocp-schedules-13,13ccd551fec72fdf,dcbfac26376320c9] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:24:55.149 INFO 2981023 --- [ocp-schedules-13,13ccd551fec72fdf,dcbfac26376320c9] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:24:55.301 INFO 2981023 --- [ocp-schedules-26,fb8646a863b33b12,e6caecc3cea14bb1] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:24:55.319 WARN 2981023 --- [ocp-schedules-26,fb8646a863b33b12,e6caecc3cea14bb1] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:24:55.458 INFO 2981023 --- [ocp-async-5,a7404eb68a3534f5,fd4434cd1f62c838] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:55.728 INFO 2981023 --- [ocp-schedules-28,bd2b081e97eeb7ef,62970c4a23542da1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:24:55.736 INFO 2981023 --- [ocp-schedules-28,bd2b081e97eeb7ef,62970c4a23542da1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:24:55.736 INFO 2981023 --- [ocp-schedules-28,bd2b081e97eeb7ef,62970c4a23542da1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:24:55.907 INFO 2981023 --- [ocp-schedules-21,a88b9d9900c39181,81074e7a501aecd0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:24:55.907 INFO 2981023 --- [ocp-schedules-21,a88b9d9900c39181,81074e7a501aecd0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:24:55.910 INFO 2981023 --- [ocp-schedules-21,a88b9d9900c39181,81074e7a501aecd0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:24:55.910 INFO 2981023 --- [ocp-schedules-21,a88b9d9900c39181,81074e7a501aecd0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:24:55.912 INFO 2981023 --- [ocp-schedules-21,a88b9d9900c39181,81074e7a501aecd0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:24:55.912 INFO 2981023 --- [ocp-schedules-21,a88b9d9900c39181,81074e7a501aecd0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:24:56.129 INFO 2981023 --- [ocp-schedules-2,5f6e98f738440e47,07d017a4c552182b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:24:56.135 INFO 2981023 --- [ocp-schedules-2,5f6e98f738440e47,07d017a4c552182b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:24:56.135 INFO 2981023 --- [ocp-schedules-2,5f6e98f738440e47,07d017a4c552182b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:24:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,1b832cbcca9dbba1,816b819bde1d1592] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,1b832cbcca9dbba1,816b819bde1d1592] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:24:56.352039+08:00, lastProcessTime=2024-09-11T18:24:46.490153+08:00 2024-09-11 18:24:56.482 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:24:56.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:24:56.352039+08:00 2024-09-11 18:24:57.276 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:24:57.277 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:24:57.458 INFO 2981023 --- [ocp-async-7,54f0b75c88b8f2bb,007ea56dd02cb553] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:57.794 INFO 2981023 --- [ocp-async-9,1c196ec98b26f547,9ed0b6339fdbc609] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:24:58.632 INFO 2981023 --- [ocp-schedules-19,4c7dcd6750bb1aae,eef2b92ff70b4dcf] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:24:58.632 INFO 2981023 --- [ocp-schedules-19,4c7dcd6750bb1aae,eef2b92ff70b4dcf] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:24:58.664 INFO 2981023 --- [Collect all tenants major compactions4,4c7dcd6750bb1aae,a61c922e0ca8d89f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:24:58.664 INFO 2981023 --- [Collect all tenants major compactions2,4c7dcd6750bb1aae,0b34ac15574efb3a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:24:58.666 INFO 2981023 --- [Collect all tenants major compactions1,4c7dcd6750bb1aae,eb0c5afa9027ff3d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:24:58.667 INFO 2981023 --- [Collect all tenants major compactions3,4c7dcd6750bb1aae,b6258970efdcf844] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:24:58.670 INFO 2981023 --- [Collect all tenants major compactions6,4c7dcd6750bb1aae,ac18f846d18f66bf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:24:58.670 INFO 2981023 --- [Collect all tenants major compactions5,4c7dcd6750bb1aae,12fedccbed175ae6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:24:58.689 INFO 2981023 --- [Collect all tenants major compactions2,4c7dcd6750bb1aae,0b34ac15574efb3a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:58.690 INFO 2981023 --- [Collect all tenants major compactions2,4c7dcd6750bb1aae,0b34ac15574efb3a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:24:58.690 INFO 2981023 --- [Collect all tenants major compactions3,4c7dcd6750bb1aae,b6258970efdcf844] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:58.691 INFO 2981023 --- [Collect all tenants major compactions5,4c7dcd6750bb1aae,12fedccbed175ae6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:58.691 INFO 2981023 --- [Collect all tenants major compactions4,4c7dcd6750bb1aae,a61c922e0ca8d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:58.692 INFO 2981023 --- [Collect all tenants major compactions3,4c7dcd6750bb1aae,b6258970efdcf844] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:24:58.693 INFO 2981023 --- [Collect all tenants major compactions6,4c7dcd6750bb1aae,ac18f846d18f66bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:58.694 INFO 2981023 --- [Collect all tenants major compactions6,4c7dcd6750bb1aae,ac18f846d18f66bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:24:58.694 INFO 2981023 --- [Collect all tenants major compactions5,4c7dcd6750bb1aae,12fedccbed175ae6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:24:58.695 INFO 2981023 --- [Collect all tenants major compactions2,4c7dcd6750bb1aae,0b34ac15574efb3a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:24:58.695 INFO 2981023 --- [Collect all tenants major compactions1,4c7dcd6750bb1aae,eb0c5afa9027ff3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:24:58.695 INFO 2981023 --- [Collect all tenants major compactions3,4c7dcd6750bb1aae,b6258970efdcf844] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:24:58.696 INFO 2981023 --- [Collect all tenants major compactions1,4c7dcd6750bb1aae,eb0c5afa9027ff3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:24:58.697 INFO 2981023 --- [Collect all tenants major compactions6,4c7dcd6750bb1aae,ac18f846d18f66bf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:24:58.698 INFO 2981023 --- [Collect all tenants major compactions1,4c7dcd6750bb1aae,eb0c5afa9027ff3d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:24:58.699 INFO 2981023 --- [Collect all tenants major compactions2,4c7dcd6750bb1aae,0b34ac15574efb3a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:24:58.700 INFO 2981023 --- [Collect all tenants major compactions3,4c7dcd6750bb1aae,b6258970efdcf844] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:24:58.700 INFO 2981023 --- [Collect all tenants major compactions4,4c7dcd6750bb1aae,a61c922e0ca8d89f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:24:58.703 INFO 2981023 --- [Collect all tenants major compactions1,4c7dcd6750bb1aae,eb0c5afa9027ff3d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:24:58.704 INFO 2981023 --- [Collect all tenants major compactions5,4c7dcd6750bb1aae,12fedccbed175ae6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:24:58.704 INFO 2981023 --- [Collect all tenants major compactions5,4c7dcd6750bb1aae,12fedccbed175ae6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:24:58.705 INFO 2981023 --- [Collect all tenants major compactions6,4c7dcd6750bb1aae,ac18f846d18f66bf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:24:58.705 INFO 2981023 --- [Collect all tenants major compactions4,4c7dcd6750bb1aae,a61c922e0ca8d89f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:24:58.705 INFO 2981023 --- [Collect all tenants major compactions4,4c7dcd6750bb1aae,a61c922e0ca8d89f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:24:58.707 INFO 2981023 --- [ocp-schedules-19,4c7dcd6750bb1aae,eef2b92ff70b4dcf] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:24:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,209eb550c47876ef,473cb3f42ecc9eb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050300, interval=60 2024-09-11 18:24:59.318 INFO 2981023 --- [prometheus-minute-schedule-1,209eb550c47876ef,473cb3f42ecc9eb2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:24:59.458 INFO 2981023 --- [ocp-async-2,413e1b80b8457600,a7c525cd9b537081] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:24:59.736 INFO 2981023 --- [ocp-schedules-25,ce0358326143580c,a491aec04853c36d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:24:59.777 WARN 2981023 --- [scanner-13663544-1,ce0358326143580c,ba9a791187f33435] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:24:59.808 INFO 2981023 --- [alarm-detect-16,ce0358326143580c,be409981640d768f] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050270 ~ 1726050289 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050285, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050285, value=98.8)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.8, values={1726050285=98.8}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050285=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:59.815 INFO 2981023 --- [alarm-detect-1,ce0358326143580c,1405b38a0bcb4772] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646226, data is empty. 2024-09-11 18:24:59.826 INFO 2981023 --- [alarm-detect-1,ce0358326143580c,1405b38a0bcb4772] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646226, data is empty. 2024-09-11 18:24:59.831 INFO 2981023 --- [alarm-detect-1,ce0358326143580c,1405b38a0bcb4772] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646226, data is empty. 2024-09-11 18:24:59.853 INFO 2981023 --- [alarm-detect-14,ce0358326143580c,bd0e42e553bc9209] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646226, data is empty. 2024-09-11 18:24:59.856 INFO 2981023 --- [alarm-detect-14,ce0358326143580c,bd0e42e553bc9209] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646226, data is empty. 2024-09-11 18:24:59.857 INFO 2981023 --- [alarm-detect-4,ce0358326143580c,1583daf4a2e60283] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646226, data is empty. 2024-09-11 18:24:59.861 INFO 2981023 --- [alarm-detect-4,ce0358326143580c,1583daf4a2e60283] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646226, data is empty. 2024-09-11 18:24:59.863 INFO 2981023 --- [alarm-detect-14,ce0358326143580c,bd0e42e553bc9209] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646226, data is empty. 2024-09-11 18:24:59.866 INFO 2981023 --- [alarm-detect-4,ce0358326143580c,1583daf4a2e60283] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646226, data is empty. 2024-09-11 18:24:59.872 INFO 2981023 --- [alarm-detect-1,ce0358326143580c,6f060031787463cd] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050270 ~ 1726050289 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050285, value=98.8)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.8, values={1726050285=98.8}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:59.881 INFO 2981023 --- [alarm-detect-15,ce0358326143580c,04901768b1bbc038] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050270 ~ 1726050289 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050285, value=98.8)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.8, values={1726050285=98.8}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:24:59.963 INFO 2981023 --- [alarm-detect-17,ce0358326143580c,c5e63b3eef844fb9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646226, data is empty. 2024-09-11 18:24:59.967 INFO 2981023 --- [alarm-detect-17,ce0358326143580c,c5e63b3eef844fb9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646226, data is empty. 2024-09-11 18:24:59.973 INFO 2981023 --- [alarm-detect-17,ce0358326143580c,c5e63b3eef844fb9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646226, data is empty. 2024-09-11 18:25:00.278 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:00.279 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:00.354 INFO 2981023 --- [ocp-schedules-28,104a21e6e555fe90,af88052ced9ba6d9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:25:00.354+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:25:00.356 INFO 2981023 --- [ocp-schedules-28,104a21e6e555fe90,af88052ced9ba6d9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:25:00.473 INFO 2981023 --- [ocp-schedules-28,104a21e6e555fe90,af88052ced9ba6d9] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 54 to SqlAggregateData 2024-09-11 18:25:00.475 INFO 2981023 --- [ocp-schedules-28,104a21e6e555fe90,af88052ced9ba6d9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:25:00.963 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:20:01.0, nextRun=2024-09-11 18:25:00.0, currentDate=Wed Sep 11 18:25:00 HKT 2024 2024-09-11 18:25:00.963 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:20:01.0, nextRunTime=2024-09-11 18:25:00.0} 2024-09-11 18:25:00.970 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 18:25:00.970 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:25:00.970 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:25:00.970+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:25:00.988 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:20:01.0, nextRun=2024-09-11 18:25:00.0, currentDate=Wed Sep 11 18:25:00 HKT 2024 2024-09-11 18:25:00.988 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:20:01.0, nextRunTime=2024-09-11 18:25:00.0} 2024-09-11 18:25:01.002 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 18:25:01.012 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 18:25:01.013 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 18:25:01.013 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:25:01.013 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,7cc0acb0651ebac0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T18:25:01.013+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:25:01.458 INFO 2981023 --- [ocp-async-5,4c9e844d5f7cad91,d41319a23f8dd793] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:01.619 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,d7be50af6efc9b57] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048251, operation:EXECUTE, state: RUNNING 2024-09-11 18:25:01.631 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,d7be50af6efc9b57] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048252, operation:EXECUTE, state: RUNNING 2024-09-11 18:25:01.645 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048251, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048221, task_operation=execute, latest_execution_start_time=2024-09-11T18:25:01.611+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048251}, listMap={}}, executor=10.31.35.10 2024-09-11 18:25:01.659 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048252, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048222, task_operation=execute, latest_execution_start_time=2024-09-11T18:25:01.629+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048252}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 18:25:01.661 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 18:25:01.662 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:25:01.688 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 18:25:01.690 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 18:25:01.694 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:25:01.703 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 18:25:01.704 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 18:25:01.704 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 18:25:01.727 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 18:25:01.731 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 18:25:01.742 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,e0c55afc6a310730] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048252, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:25:01.761 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:25:01.778 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:25:01.796 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:25:01.803 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 18:25:01.803 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 18:25:01.807 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:25:01.906 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:25:01.969 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:25:01.992 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:25:02.009 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:25:02.044 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:25:02.058 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:25:02.070 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:25:02.076 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 18:25:02.076 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3395.459579986], b:[-5.807309276841022E12] 2024-09-11 18:25:02.080 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:25:02.128 INFO 2981023 --- [ocp-schedules-7,4caed26d61cc87dc,8a00f86ec25130cb] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:25:02.129 INFO 2981023 --- [ocp-schedules-7,4caed26d61cc87dc,8a00f86ec25130cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:25:02.158 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:25:02.177 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:25:02.178 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.179 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:25:02.179 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.179 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:25:02.180 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:25:02.183 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.183 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:25:02.185 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:25:02.192 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.192 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.193 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.193 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.198 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.201 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.201 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.201 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.203 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:25:02.208 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.208 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:25:02.209 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.210 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.216 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.216 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:25:02.223 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.223 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:25:02.226 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.226 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.230 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:25:02.230 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.231 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:25:02.231 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.240 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.240 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:25:02.241 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.241 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:25:02.242 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.243 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.243 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:25:02.243 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:25:02.244 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.245 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:25:02.245 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:25:02.249 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.250 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:25:02.250 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.251 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:25:02.252 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.253 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:25:02.254 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:25:02.254 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.254 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:25:02.255 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.255 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:25:02.256 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.256 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:25:02.257 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.257 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:25:02.258 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.258 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:25:02.258 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:25:02.259 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:25:02.260 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:25:02.266 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:25:02.270 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:02.272 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.272 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.274 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:02.275 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:25:02.280 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.281 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 18:25:02.282 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:25:02.284 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.284 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:25:02.285 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:25:02.286 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.287 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:25:02.289 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:25:02.289 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:25:02.289 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:25:02.291 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:25:02.291 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6271D6E1-0-0] 2024-09-11 18:25:02.292 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:25:02.295 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:25:02.295 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.296 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:25:02.300 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:25:02.302 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:25:02.302 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.302 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:25:02.305 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:25:02.308 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:25:02.311 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:25:02.311 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:25:02.311 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:25:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,79787d3410b20f24,b71034a14719f0d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,79787d3410b20f24,b71034a14719f0d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:02.327 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.332 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:25:02.336 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:25:02.347 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,0f82251fe77a9d43] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048251, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:25:02.366 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.366 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:25:02.367 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:25:02.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6271D6EC-0-0] 2024-09-11 18:25:02.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:25:02.372 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.373 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:25:02.373 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.374 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:25:02.374 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.375 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:25:02.376 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.376 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:25:02.376 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.377 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:25:02.377 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.378 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:25:02.378 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:25:02.380 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:25:02.389 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:02.446 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048221, status:SUCCESS 2024-09-11 18:25:02.448 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048221, status:SUCCESS 2024-09-11 18:25:02.454 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048222, status:SUCCESS 2024-09-11 18:25:02.454 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048222, status:SUCCESS 2024-09-11 18:25:02.477 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.478 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:25:02.487 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:25:02.488 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:25:02.495 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:02.495 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:25:02.505 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:25:02.507 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:25:02.507 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:25:02.507 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:25:02.507 INFO 2981023 --- [ocp-schedules-7,4caed26d61cc87dc,8a00f86ec25130cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:25:02.802 INFO 2981023 --- [ocp-schedules-16,e0a6efe45adee502,91b8cc8d5cd7b2ca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:25:03.279 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:03.280 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:03.458 INFO 2981023 --- [ocp-async-8,c8a44c2fa4ccc84e,cabdbfbc01ffbd43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,7a8358e562401be9,967b7ac873b2a204] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,7a8358e562401be9,967b7ac873b2a204] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:05.458 INFO 2981023 --- [ocp-async-6,37749a678e941792,2ac84dae9823da5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:06.280 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:06.282 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:25:06.352326+08:00, lastProcessTime=2024-09-11T18:24:56.352039+08:00 2024-09-11 18:25:06.496 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:25:06.352326+08:00 2024-09-11 18:25:07.458 INFO 2981023 --- [ocp-async-9,eccdd4fd5c9c1283,549edb39f3f19ad7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:07.794 INFO 2981023 --- [ocp-async-8,4b700681d7a27adc,2bade406c76b0b9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:25:08.171 INFO 2981023 --- [ocp-schedules-13,c746d266f9c92624,8682812639481755] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:25:08.245 INFO 2981023 --- [ocp-schedules-27,f12264da278b8a85,6b757e2baf85296e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:25:08.245+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:25:08.246 INFO 2981023 --- [ocp-schedules-27,f12264da278b8a85,6b757e2baf85296e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:25:08.248 INFO 2981023 --- [ocp-schedules-27,f12264da278b8a85,6b757e2baf85296e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:25:08.248 INFO 2981023 --- [ocp-schedules-27,f12264da278b8a85,6b757e2baf85296e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:25:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,ca1434b5a03adc99,cf4d57b5a2e923eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,ca1434b5a03adc99,cf4d57b5a2e923eb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:09.282 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:09.282 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:09.459 INFO 2981023 --- [ocp-async-7,c3d0a7f7bae5f26f,d318f4ec2e7faf86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:10.003 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:25:10.003 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:25:10.020 INFO 2981023 --- [ocp-schedules-9,f21039f72167485d,5fbbbfd3bacc1bce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:25:10.041 WARN 2981023 --- [scanner-13664004-1,f21039f72167485d,a1021b7f108df455] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:25:10.069 INFO 2981023 --- [alarm-detect-15,f21039f72167485d,e6bb2328f7ccb357] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050285 ~ 1726050300 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050300, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050300, value=98.13)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.13, values={1726050300=98.13}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050300=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:10.134 INFO 2981023 --- [alarm-detect-1,f21039f72167485d,f3cf5e000bfa9ee0] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050285 ~ 1726050300 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050300, value=98.13)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.13, values={1726050300=98.13}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:10.134 INFO 2981023 --- [alarm-detect-1,f21039f72167485d,f3cf5e000bfa9ee0] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=os_tsar_cpu_util_full, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:25:10.139 INFO 2981023 --- [alarm-detect-13,f21039f72167485d,d63e24af0ec84d72] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050270 ~ 1726050300 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050285, value=7.858)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.858, values={1726050285=7.858}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:10.139 INFO 2981023 --- [alarm-detect-13,f21039f72167485d,d63e24af0ec84d72] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050270 ~ 1726050300 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050285, value=7.858)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.858, values={1726050285=7.858}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:25:10.140 INFO 2981023 --- [alarm-detect-13,f21039f72167485d,d63e24af0ec84d72] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:25:10.148 INFO 2981023 --- [alarm-detect-10,f21039f72167485d,4ca5ea25a6e2f0d7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050285 ~ 1726050300 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050300, value=98.13)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.13, values={1726050300=98.13}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:10.148 INFO 2981023 --- [alarm-detect-10,f21039f72167485d,4ca5ea25a6e2f0d7] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_cpu_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:25:10.273 INFO 2981023 --- [ocp-schedules-11,746d784c316a8794,9e5cfc2e40c784fe] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:25:10.330 INFO 2981023 --- [alarm-detect-15,f21039f72167485d,b22587e5042582a6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050270 ~ 1726050300 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050285, value=105.152)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.152, values={1726050285=105.152}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:10.330 INFO 2981023 --- [alarm-detect-15,f21039f72167485d,b22587e5042582a6] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_cpu_usage_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=rtdob-1718689631:tenant_name=linkreit:host=10.31.35.5-2] 2024-09-11 18:25:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:25:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:25:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,8f5fd73809bacdc8,e70652716e955712] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:11.319 INFO 2981023 --- [prometheus-minute-schedule-1,8f5fd73809bacdc8,e70652716e955712] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:11.459 INFO 2981023 --- [ocp-async-3,2ac2e0bf4438ead6,29b41b1a4b260338] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:11.726 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050240, nodeCount=480, affectRows=480 2024-09-11 18:25:12.282 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:12.283 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:12.405 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:21:11.963, end=2024-09-11T18:23:12.399} 2024-09-11 18:25:12.410 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obC 2024-09-11 18:25:12.410 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:25:12.411 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:11.963, end=2024-09-11T18:23:12.399},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:25:12.411 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:25:12.411 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:25:12.412 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:25:12.415 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:11.963, end=2024-09-11T18:23:12.399},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:25:12.415 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:25:12.415 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:25:12.472 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912472000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.472Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.473 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912473000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.473Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.486 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912486000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.486Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.492 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912492000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.492Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.497 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912497000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=8A1B851989150B802AEEACA99152D145, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.497Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.499 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912499000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.499Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.500 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912500000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.500Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.500 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912500000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.500Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.501 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.501 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.509 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.509 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.513 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:25:12.513 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.513 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.513 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.516 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 69 2024-09-11 18:25:12.516 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.516 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 29 2024-09-11 18:25:12.516 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.525 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.525 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.526 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.526 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.526 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.526 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.526 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.526 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.531 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.531 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.533 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.533 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.535 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.535 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.536 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.536 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.537 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:25:12.537 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.537 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.537 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.538 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.538 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.539 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.539 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.540 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.540 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.540 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.540 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.542 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.542 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.545 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.545 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.551 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.551 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.552 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.552 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.552 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.552 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.554 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.557 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.557 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.557 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.558 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.558 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.558 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.558 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.559 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.559 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.559 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.560 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.566 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.567 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.472, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.567 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.567 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.569 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.473, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.571 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.492, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.572 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.572 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.572 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.499, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.573 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.574 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.486, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.574 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.574 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.575 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.577 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.582 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.497, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=8A1B851989150B802AEEACA99152D145, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.584 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.5, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.588 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.5, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.608 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912608000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.608Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.611 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912611000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.611Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.613 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912613000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.613Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.615 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912615000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.615Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.615 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912615000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.615Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.622 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.622 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.624 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912624000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1ED31157F7096F47941EA2233AA4101C, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.624Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.626 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.626 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.632 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912631000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.631Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.632 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.632 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.636 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.636 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.637 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.637 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.639 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912639000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.639Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.642 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:25:12.642 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.649 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.649 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.650 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.650 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.650 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.650 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.654 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.654 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.658 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.658 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.662 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.662 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.662 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.662 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.666 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:25:12.666 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.670 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.670 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.672 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.673 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.674 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.679 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.679 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.679 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.681 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.682 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.682 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.682 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.682 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.683 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.683 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.684 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.684 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.684 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.684 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.691 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.691 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.692 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.692 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.694 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.694 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.696 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.608, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.696 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.696 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.611, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.696 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.696 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.615, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.697 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.697 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.700 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.700 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.702 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.708 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.639, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.709 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.709 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.710 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.710 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.711 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.711 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.711 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.711 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.712 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.712 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.713 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.715 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.717 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.615, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.717 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.613, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.719 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.631, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.721 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.624, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=6, sqlId=1ED31157F7096F47941EA2233AA4101C, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.722 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:25:12.726 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:11.963, end=2024-09-11T18:23:12.399},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:25:12.726 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:25:12.726 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:25:12.737 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:25:12.754 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912754000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=AD63995D4A6E8C3787D81BDA88E06774, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.754Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.755 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725963912755000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:25:12.755Z, endTime=2024-09-11T10:23:12.399Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:25:12.757 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:25:12.757 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.766 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.766 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.768 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.768 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.771 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:25:12.771 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:25:12.774 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.774 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.775 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:25:12.775 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.755, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.777 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.777 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.779 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:25:12.779 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:25:12.785 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:25:12.785 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050192399000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 18:25:12.786 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:25:12.790 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:25:12.754, end=2024-09-11T18:23:12.399}, clusterId=2, tenantId=4, sqlId=AD63995D4A6E8C3787D81BDA88E06774, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:25:12.791 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:25:12.791 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:11.963, end=2024-09-11T18:23:12.399},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:25:12.791 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:25:12.791 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:25:12.792 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:25:12.793 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:11.963, end=2024-09-11T18:23:12.399},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:25:12.793 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:25:12.793 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:25:12.794 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:25:12.796 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 109 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:11.963, end=2024-09-11T18:23:12.399},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:25:12.806 INFO 2981023 --- [ocp-schedules-1,98329e4b9b6554ea,08c25e9c8f86e517] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:25:12.856 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:25:12.856 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:25:12.868 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:25:12.868 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:25:13.459 INFO 2981023 --- [ocp-async-10,a73433c07d4cdbdd,eb2b09369cf75aab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:14.191 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050240, nodeCount=480, validCount=480 2024-09-11 18:25:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,f2ada6ae698ae6b8,5b82e6265e70decb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,5ce4308f3c7b80cd,20890ddcd5cb6aca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:25:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,f2ada6ae698ae6b8,5b82e6265e70decb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:14.420 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050240, nodeCount=1000, affectRows=1000 2024-09-11 18:25:14.422 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050240, nodeCount=999, validCount=999 2024-09-11 18:25:14.427 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050240, nodeCount=1000, affectRows=1000 2024-09-11 18:25:14.429 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050240, nodeCount=1000, validCount=1000 2024-09-11 18:25:14.430 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050240, nodeCount=48, affectRows=48 2024-09-11 18:25:14.430 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050240, nodeCount=46, validCount=46 2024-09-11 18:25:15.283 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:15.284 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:15.459 INFO 2981023 --- [ocp-async-4,5ef34ba6baaab4ee,94324a345bbd4ea4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:15.739 INFO 2981023 --- [ocp-schedules-10,1adeda81231ac26b,8a0f814b00bf8bf6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:25:15.748 INFO 2981023 --- [ocp-schedules-10,1adeda81231ac26b,8a0f814b00bf8bf6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:25:15.748 INFO 2981023 --- [ocp-schedules-10,1adeda81231ac26b,8a0f814b00bf8bf6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:25:16.023 INFO 2981023 --- [ocp-schedules-24,278a2886da70d86a,6cb90db3b197cd18] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:25:16.023 INFO 2981023 --- [ocp-schedules-2,cfbe1cc19ae502ee,a3e0f260b1194ce9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:25:16.023 INFO 2981023 --- [host-check-agent-status1,278a2886da70d86a,68507e65f5331b51] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:25:16.023 INFO 2981023 --- [ocp-schedules-30,46518200454a1fec,60b6732670051202] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:25:16.024 INFO 2981023 --- [host-check-agent-status2,278a2886da70d86a,30a1f33fb30ca7b5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:25:16.024 INFO 2981023 --- [host-check-agent-status3,278a2886da70d86a,2042c596aa724bb6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:25:16.024 INFO 2981023 --- [host-check-agent-status4,278a2886da70d86a,320e58e82fdef9f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:25:16.024 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:25:16.025 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:25:16.025 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:25:16.026 INFO 2981023 --- [host-check-clockdiff1,cfbe1cc19ae502ee,8dc8b85c2a61377c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:25:16.026 INFO 2981023 --- [host-check-clockdiff2,cfbe1cc19ae502ee,c436c642037cebca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:25:16.029 INFO 2981023 --- [host-check-clockdiff3,cfbe1cc19ae502ee,0ef6d2fd68a95fcc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:25:16.030 INFO 2981023 --- [host-check-clockdiff4,cfbe1cc19ae502ee,a4f5bebc08d8736c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:25:16.031 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:25:16.049 INFO 2981023 --- [host-check-clockdiff2,cfbe1cc19ae502ee,c436c642037cebca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050316 5 5 , error= 2024-09-11 18:25:16.050 INFO 2981023 --- [host-check-clockdiff3,cfbe1cc19ae502ee,0ef6d2fd68a95fcc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050316 0 0 , error= 2024-09-11 18:25:16.066 INFO 2981023 --- [host-check-agent-status3,278a2886da70d86a,2042c596aa724bb6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:25:16.067 INFO 2981023 --- [host-check-agent-status2,278a2886da70d86a,30a1f33fb30ca7b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:25:16.070 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:25:16.071 INFO 2981023 --- [host-check-agent-status3,278a2886da70d86a,2042c596aa724bb6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:25:16.071 INFO 2981023 --- [host-check-agent-status1,278a2886da70d86a,68507e65f5331b51] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:25:16.072 INFO 2981023 --- [host-check-agent-status2,278a2886da70d86a,30a1f33fb30ca7b5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:25:16.072 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:25:16.073 INFO 2981023 --- [host-check-agent-status1,278a2886da70d86a,68507e65f5331b51] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:25:16.078 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:25:16.079 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:25:16.095 INFO 2981023 --- [host-check-agent-status4,278a2886da70d86a,320e58e82fdef9f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:25:16.112 INFO 2981023 --- [host-check-agent-status4,278a2886da70d86a,320e58e82fdef9f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:25:16.113 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:25:16.190 INFO 2981023 --- [ocp-schedules-26,f57c326a7e55cc05,de5d9c222a56d7d8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:25:16.191 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:25:16.192 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:25:16.197 INFO 2981023 --- [ocp-schedules-3,31c0c9d985ddd4dc,931ee7bed9971fc2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:25:16.197 INFO 2981023 --- [ocp-schedules-21,f91fe918261840dd,8a164b0d8d4f0b06] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:25:16.197 INFO 2981023 --- [ocp-schedules-5,a2fa6b3311414804,1d2bce3626707332] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:25:16.213 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:25:16.214 INFO 2981023 --- [ocp-schedules-21,f91fe918261840dd,8a164b0d8d4f0b06] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:25:16.217 INFO 2981023 --- [ocp-schedules-26,f57c326a7e55cc05,de5d9c222a56d7d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:25:16.217 INFO 2981023 --- [ocp-schedules-3,31c0c9d985ddd4dc,931ee7bed9971fc2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:25:16.272 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050243207000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:25:16.272 INFO 2981023 --- [ocp-schedules-3,31c0c9d985ddd4dc,931ee7bed9971fc2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:25:16.361 INFO 2981023 --- [host-check-agent-status1,278a2886da70d86a,68507e65f5331b51] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:25:16.368 INFO 2981023 --- [host-check-agent-status3,278a2886da70d86a,2042c596aa724bb6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:25:16.372 INFO 2981023 --- [ocp-schedules-15,acf81228bf3c57d1,99a5a0c08392e716] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:25:16.375 INFO 2981023 --- [host-check-agent-status3,278a2886da70d86a,2042c596aa724bb6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:25:16.375 INFO 2981023 --- [host-check-agent-status3,278a2886da70d86a,2042c596aa724bb6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:25:16.375 INFO 2981023 --- [host-check-agent-status3,278a2886da70d86a,2042c596aa724bb6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:25:16.376 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:25:16.376 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:25:16.376707991+08:00 2024-09-11 18:25:16.381 INFO 2981023 --- [host-check-agent-status1,278a2886da70d86a,68507e65f5331b51] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:25:16.381 INFO 2981023 --- [host-check-agent-status1,278a2886da70d86a,68507e65f5331b51] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:25:16.381 INFO 2981023 --- [host-check-agent-status1,278a2886da70d86a,68507e65f5331b51] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:25:16.382 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:25:16.376+08:00 2024-09-11 18:25:16.384 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:25:16.376+08:00 and status AVAILABLE 2024-09-11 18:25:16.393 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:25:16.391410+08:00, lastProcessTime=2024-09-11T18:25:06.352326+08:00 2024-09-11 18:25:16.398 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.399 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:25:16.400 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:25:16.403 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.403 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.406 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:25:16.406 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:25:16.407 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:16.409 INFO 2981023 --- [host-check-agent-status4,278a2886da70d86a,320e58e82fdef9f9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:25:16.410 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:25:16.409257049+08:00 2024-09-11 18:25:16.411 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:25:16.411 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:25:16.410+08:00 2024-09-11 18:25:16.413 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:25:16.414 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:25:16.410+08:00 and status AVAILABLE 2024-09-11 18:25:16.417 INFO 2981023 --- [host-check-agent-status4,278a2886da70d86a,320e58e82fdef9f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:25:16.417 INFO 2981023 --- [host-check-agent-status4,278a2886da70d86a,320e58e82fdef9f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:25:16.417 INFO 2981023 --- [host-check-agent-status4,278a2886da70d86a,320e58e82fdef9f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:25:16.419 INFO 2981023 --- [ocp-schedules-26,f57c326a7e55cc05,de5d9c222a56d7d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:25:16.425 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.425 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:16.427 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:25:16.455 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:25:16.457 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:25:16.457 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:25:16.462874006+08:00 2024-09-11 18:25:16.458 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:25:16.457+08:00 2024-09-11 18:25:16.460 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:25:16.457+08:00 and status AVAILABLE 2024-09-11 18:25:16.462 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:25:16.462040276+08:00 2024-09-11 18:25:16.463 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:25:16.462+08:00 2024-09-11 18:25:16.464 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.465 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:25:16.462+08:00 and status AVAILABLE 2024-09-11 18:25:16.466 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:16.470 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:25:16.471 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:25:16.473 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:25:16.479 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:25:16.479 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:25:16.481 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:25:16.482 INFO 2981023 --- [host-check-agent-status2,278a2886da70d86a,30a1f33fb30ca7b5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:25:16.484 INFO 2981023 --- [host-check-agent-status2,278a2886da70d86a,30a1f33fb30ca7b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:25:16.484 INFO 2981023 --- [host-check-agent-status2,278a2886da70d86a,30a1f33fb30ca7b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:25:16.484 INFO 2981023 --- [host-check-agent-status2,278a2886da70d86a,30a1f33fb30ca7b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:25:16.489 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:25:16.489 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:25:16.490 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:25:16.491 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:25:16.493 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:25:16.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:16.512 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:25:16.514 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.514 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:25:16.526 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:25:16.534 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:25:16.554 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:25:16.563 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:25:16.563 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:25:16.564 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:25:16.564 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050243207000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:25:16.565 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:25:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:16.575 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:25:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:25:16.578 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:25:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:25:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:25:16.593 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:25:16.597 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:25:16.603 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:25:16.607 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:25:16.624 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:25:16.632 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:25:16.653 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:25:16.663 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:25:16.666 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:16.666 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:25:16.666 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:25:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:25:16.674 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:25:16.676 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:25:16.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:25:16.681 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:25:16.681 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:16.684 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:25:16.686 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.690 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:25:16.694 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:25:16.696 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:25:16.706 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:25:16.710 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:25:16.714 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:25:16.722 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:25:16.727 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:25:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:16.735 INFO 2981023 --- [host-check-host-status3,46518200454a1fec,d97e98dfec634e3e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:25:16.737 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:25:16.740 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.743 INFO 2981023 --- [host-check-host-status4,46518200454a1fec,cc90b9264d1ec629] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:25:16.743 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:25:16.744 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:16.750 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:25:16.753 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.761 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:25:16.761 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:25:16.765 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=4, resultsCount=4 2024-09-11 18:25:16.770 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:25:16.771 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:25:16.391410+08:00 2024-09-11 18:25:16.772 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:25:16.780 INFO 2981023 --- [host-check-host-status2,46518200454a1fec,faa7b23d9c324a3c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:25:16.780 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:25:16.781 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:25:16.781 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:25:16.787 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:16.787 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.788 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:25:16.792 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:25:16.794 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:25:16.794 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:25:16.798 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:16.800 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:16.802 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:25:16.803 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:25:16.806 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:25:16.808 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:25:16.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:25:16.820 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:25:16.976 INFO 2981023 --- [host-check-clockdiff1,cfbe1cc19ae502ee,8dc8b85c2a61377c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050316 0 0 , error= 2024-09-11 18:25:16.991 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:25:16.997 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:25:17.013 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:25:17.121 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:25:17.127 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:17.135 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:17.141 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:25:17.148 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:25:17.148 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:25:17.148 INFO 2981023 --- [ocp-schedules-21,f91fe918261840dd,8a164b0d8d4f0b06] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:25:17.221 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:25:17.227 INFO 2981023 --- [host-check-host-status1,46518200454a1fec,fd4186a168354056] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:25:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,392676b96df22567,da6e2e35705a7175] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,392676b96df22567,da6e2e35705a7175] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:17.459 INFO 2981023 --- [ocp-async-1,500bad2b756eacba,4ed1fd6c7bc61510] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:17.794 INFO 2981023 --- [ocp-async-10,9c1da36d77016e38,d56ab54b3e06788f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:25:17.806 INFO 2981023 --- [ocp-async-10,9c1da36d77016e38,d56ab54b3e06788f] c.o.o.a.s.OcpAlarmSnapshotServiceImpl : alarm type ob_tenant_cpu_usage_over_threshold 's snapshots already exist. 2024-09-11 18:25:18.284 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:18.285 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:19.459 INFO 2981023 --- [ocp-async-9,93efc92871fea144,348217827962b7b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,b17d4b928f17b1ff,d1d8e3699b9affd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,b17d4b928f17b1ff,d1d8e3699b9affd0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:20.351 INFO 2981023 --- [ocp-schedules-6,e674b0279e4f038d,fbd89d1338895ec1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:25:20.408 WARN 2981023 --- [scanner-13665705-1,e674b0279e4f038d,dd11baec4fab233f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:25:20.427 INFO 2981023 --- [alarm-detect-15,e674b0279e4f038d,2038a5aa4360f48e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646226, data is empty. 2024-09-11 18:25:20.433 INFO 2981023 --- [alarm-detect-15,e674b0279e4f038d,2038a5aa4360f48e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646280, data is empty. 2024-09-11 18:25:20.437 INFO 2981023 --- [alarm-detect-15,e674b0279e4f038d,2038a5aa4360f48e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646280, data is empty. 2024-09-11 18:25:20.458 INFO 2981023 --- [alarm-detect-20,e674b0279e4f038d,af52076530008398] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646226, data is empty. 2024-09-11 18:25:20.458 INFO 2981023 --- [alarm-detect-1,e674b0279e4f038d,92386224dcb964d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646226, data is empty. 2024-09-11 18:25:20.463 INFO 2981023 --- [alarm-detect-1,e674b0279e4f038d,92386224dcb964d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646280, data is empty. 2024-09-11 18:25:20.463 INFO 2981023 --- [alarm-detect-20,e674b0279e4f038d,af52076530008398] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646280, data is empty. 2024-09-11 18:25:20.471 INFO 2981023 --- [alarm-detect-1,e674b0279e4f038d,92386224dcb964d2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646280, data is empty. 2024-09-11 18:25:20.471 INFO 2981023 --- [alarm-detect-20,e674b0279e4f038d,af52076530008398] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646280, data is empty. 2024-09-11 18:25:20.532 INFO 2981023 --- [alarm-detect-20,e674b0279e4f038d,b4db45307721cbb9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050285 ~ 1726050310 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050300, value=7.571)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.571, values={1726050300=7.571}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:20.532 INFO 2981023 --- [alarm-detect-20,e674b0279e4f038d,b4db45307721cbb9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050285 ~ 1726050310 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050300, value=7.571)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.571, values={1726050300=7.571}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:25:20.585 INFO 2981023 --- [alarm-detect-8,e674b0279e4f038d,0c8e36184d5324cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646226, data is empty. 2024-09-11 18:25:20.594 INFO 2981023 --- [alarm-detect-8,e674b0279e4f038d,0c8e36184d5324cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646280, data is empty. 2024-09-11 18:25:20.597 INFO 2981023 --- [alarm-detect-8,e674b0279e4f038d,0c8e36184d5324cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646280, data is empty. 2024-09-11 18:25:20.618 INFO 2981023 --- [alarm-detect-9,e674b0279e4f038d,44024505da55e0d2] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050285 ~ 1726050310 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050300, value=105.238)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.238, values={1726050300=105.238}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:21.285 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:21.287 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:21.459 INFO 2981023 --- [ocp-async-8,af549b0446da3c3e,68e716191b3b54e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:21.991 INFO 2981023 --- [host-check-clockdiff4,cfbe1cc19ae502ee,a4f5bebc08d8736c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050321 0 0 , error= 2024-09-11 18:25:22.343 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_available, cached timestamp: 1726043116 2024-09-11 18:25:22.343 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_available 2024-09-11 18:25:22.809 INFO 2981023 --- [ocp-schedules-29,bc3b06c98959b59f,b71cf78b60a92092] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:25:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,fe54e9118a206c35,5970ffb59294c15a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,fe54e9118a206c35,5970ffb59294c15a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:23.459 INFO 2981023 --- [ocp-async-7,8505cbeba3c1a717,a6e73fec04776ed2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:24.287 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:24.288 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:24.498 INFO 2981023 --- [ocp-schedules-10,c0735f9fead44dd0,091f76a12ecb141b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:25:25.459 INFO 2981023 --- [ocp-async-3,9092b1ed6941376e,5b7cb5cd8c1167f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,828011504c047a79,09907c78e1d87c56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,828011504c047a79,09907c78e1d87c56] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:26.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:25:26.354040+08:00, lastProcessTime=2024-09-11T18:25:16.391410+08:00 2024-09-11 18:25:26.497 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:25:26.354040+08:00 2024-09-11 18:25:27.288 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:27.290 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:27.460 INFO 2981023 --- [ocp-async-9,0c039d8891356ba0,2a4c2c18e16db676] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:27.696 INFO 2981023 --- [ocp-schedules-30,ee4687494985f691,adfef926976dd9ca] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:25:27.696 INFO 2981023 --- [ocp-schedules-30,ee4687494985f691,adfef926976dd9ca] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:25:27.700 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:25:27.700 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:25:27.700 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:25:27.723 INFO 2981023 --- [ocp-schedules-30,ee4687494985f691,adfef926976dd9ca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:25:27.726 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:25:27.729 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:25:27.730 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:25:27.730 INFO 2981023 --- [ocp-schedules-30,ee4687494985f691,adfef926976dd9ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:25:27.730 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:25:27.732 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:25:27.733 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:25:27.794 INFO 2981023 --- [ocp-async-7,61bcb61ef809a426,b5c1bb19a5195a54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:25:27.975 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:25:27.977 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:25:27.977 INFO 2981023 --- [ocp-schedules-30,ee4687494985f691,adfef926976dd9ca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:25:27.978 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:25:27.982 INFO 2981023 --- [ocp-schedules-30,ee4687494985f691,adfef926976dd9ca] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:25:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,85d22f5f8b7754f9,784ab1a2888c7e22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,85d22f5f8b7754f9,784ab1a2888c7e22] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:29.460 INFO 2981023 --- [ocp-async-6,d6d1c72214b86d4e,5a0973e5fe540fb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:30.290 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:30.291 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:30.627 INFO 2981023 --- [ocp-schedules-28,0796c2b87b91f470,0f374c93c5bd3820] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:25:30.674 WARN 2981023 --- [scanner-13666057-1,0796c2b87b91f470,5e1e98933d18e025] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:25:30.682 INFO 2981023 --- [alarm-detect-18,0796c2b87b91f470,57d85fa39b7bcd44] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050300 ~ 1726050320 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050315, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050315, value=98.77)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.77, values={1726050315=98.77}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050315=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:30.711 INFO 2981023 --- [alarm-detect-19,0796c2b87b91f470,d504f59a9123f13c] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050300 ~ 1726050320 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050315, value=98.77)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.77, values={1726050315=98.77}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:30.756 INFO 2981023 --- [alarm-detect-17,0796c2b87b91f470,2f4de3cfb80a0f82] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050300 ~ 1726050320 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050315, value=98.77)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.77, values={1726050315=98.77}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:30.766 INFO 2981023 --- [alarm-detect-9,0796c2b87b91f470,a03ab449589cd5c7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050300 ~ 1726050320 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050315, value=8.119)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.119, values={1726050315=8.119}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:30.766 INFO 2981023 --- [alarm-detect-9,0796c2b87b91f470,a03ab449589cd5c7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050300 ~ 1726050320 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050315, value=8.119)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.119, values={1726050315=8.119}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:25:30.817 INFO 2981023 --- [alarm-detect-18,0796c2b87b91f470,ca9b651157d16bff] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050300 ~ 1726050320 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050315, value=104.104)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.104, values={1726050315=104.104}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:31.460 INFO 2981023 --- [ocp-async-5,74df396e6b054655,264154f88de8b2d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,29484ff83470df14,95a073ebfa62ff34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:32.317 INFO 2981023 --- [prometheus-minute-schedule-1,29484ff83470df14,95a073ebfa62ff34] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:32.811 INFO 2981023 --- [ocp-schedules-4,a603598499ab8fb5,ab676d249511548e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:25:33.291 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:33.292 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:33.460 INFO 2981023 --- [ocp-async-4,69d40b941349ffb0,4c2e5be93ccb59eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,825b69bb6521394e,4ab5e92c53eed8f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,825b69bb6521394e,4ab5e92c53eed8f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:35.460 INFO 2981023 --- [ocp-async-8,31468bdfa90464b8,24521b4679559bf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:35.750 INFO 2981023 --- [ocp-schedules-16,d62e388f4224e2f1,f0f526eda8cafe8d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:25:35.761 INFO 2981023 --- [ocp-schedules-16,d62e388f4224e2f1,f0f526eda8cafe8d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:25:35.761 INFO 2981023 --- [ocp-schedules-16,d62e388f4224e2f1,f0f526eda8cafe8d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:25:36.292 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:36.293 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:25:36.352283+08:00, lastProcessTime=2024-09-11T18:25:26.354040+08:00 2024-09-11 18:25:36.477 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:25:36.352283+08:00 2024-09-11 18:25:37.314 INFO 2981023 --- [prometheus-cache-monitor1,5b842ec2f91b9e22,898aa16562bebb54] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:25:37.315 INFO 2981023 --- [prometheus-cache-monitor1,5b842ec2f91b9e22,898aa16562bebb54] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:25:37.460 INFO 2981023 --- [ocp-async-2,6c67b21b60d12aea,7db89d4dd86f2809] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:37.794 INFO 2981023 --- [ocp-async-4,8a4558932b95934a,8ae5af9344257c8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:25:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,19a7aa8b7eb49343,dbe825d998842e28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,19a7aa8b7eb49343,dbe825d998842e28] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:39.293 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:39.294 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:39.460 INFO 2981023 --- [ocp-async-10,95358b3a9c9a3255,d53c06a38f02fd1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:40.003 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:25:40.003 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:25:40.844 INFO 2981023 --- [ocp-schedules-25,9f9bc1fc337e9b39,175a610cd8ac0d1a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:25:40.887 INFO 2981023 --- [alarm-detect-19,9f9bc1fc337e9b39,283311a227b922ae] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050315 ~ 1726050330 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050330, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050330, value=98.409)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.409, values={1726050330=98.409}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050330=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:40.889 WARN 2981023 --- [scanner-13666252-1,9f9bc1fc337e9b39,603f1b0614cb839d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:25:40.913 INFO 2981023 --- [alarm-detect-17,9f9bc1fc337e9b39,9b7ef20b345d3d33] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646280, data is empty. 2024-09-11 18:25:40.918 INFO 2981023 --- [alarm-detect-17,9f9bc1fc337e9b39,9b7ef20b345d3d33] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646280, data is empty. 2024-09-11 18:25:40.927 INFO 2981023 --- [alarm-detect-17,9f9bc1fc337e9b39,9b7ef20b345d3d33] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646280, data is empty. 2024-09-11 18:25:40.941 INFO 2981023 --- [alarm-detect-16,9f9bc1fc337e9b39,c54b57b09f604137] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646280, data is empty. 2024-09-11 18:25:40.941 INFO 2981023 --- [alarm-detect-5,9f9bc1fc337e9b39,698eebcf43fcb439] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050315 ~ 1726050330 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050330, value=98.409)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.409, values={1726050330=98.409}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:40.945 INFO 2981023 --- [alarm-detect-12,9f9bc1fc337e9b39,1e713525237866ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646280, data is empty. 2024-09-11 18:25:40.948 INFO 2981023 --- [alarm-detect-16,9f9bc1fc337e9b39,c54b57b09f604137] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646280, data is empty. 2024-09-11 18:25:40.950 INFO 2981023 --- [alarm-detect-12,9f9bc1fc337e9b39,1e713525237866ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646280, data is empty. 2024-09-11 18:25:40.957 INFO 2981023 --- [alarm-detect-16,9f9bc1fc337e9b39,c54b57b09f604137] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646280, data is empty. 2024-09-11 18:25:40.957 INFO 2981023 --- [alarm-detect-12,9f9bc1fc337e9b39,1e713525237866ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646280, data is empty. 2024-09-11 18:25:41.026 INFO 2981023 --- [alarm-detect-18,9f9bc1fc337e9b39,f962cdaa95cc3119] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050315 ~ 1726050330 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050330, value=98.409)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.409, values={1726050330=98.409}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:41.096 INFO 2981023 --- [alarm-detect-17,9f9bc1fc337e9b39,6f1b1f00adb9a2bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646280, data is empty. 2024-09-11 18:25:41.100 INFO 2981023 --- [alarm-detect-17,9f9bc1fc337e9b39,6f1b1f00adb9a2bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646280, data is empty. 2024-09-11 18:25:41.104 INFO 2981023 --- [alarm-detect-17,9f9bc1fc337e9b39,6f1b1f00adb9a2bb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646280, data is empty. 2024-09-11 18:25:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,9d9c34336c4344cb,f4a4707c2a0f0cef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,9d9c34336c4344cb,f4a4707c2a0f0cef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:41.460 INFO 2981023 --- [ocp-async-1,14881471f3bd7cb7,55b65c14e1f337dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:42.294 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:42.295 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:42.814 INFO 2981023 --- [ocp-schedules-18,ab0c4220138e129a,d373fa06ba87b130] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:25:43.461 INFO 2981023 --- [ocp-async-7,f550ccee5edead67,8219e2482d370e7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,841c20e0236679c3,c6fdc2206c62bfff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,841c20e0236679c3,c6fdc2206c62bfff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:45.101 INFO 2981023 --- [prometheus-second-schedule-1,34be21a384530f51,2dccbb0a1e62cff4] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:25:45.295 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:45.298 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:45.461 INFO 2981023 --- [ocp-async-3,ab10fc1b97c9bfb1,a1e32c7fdcd152f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:46.018 INFO 2981023 --- [ocp-schedules-19,4225db5975d9317d,85f7c76577a9050e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:25:46.018 INFO 2981023 --- [ocp-schedules-3,ec874b21c3609735,8ebe12779f286012] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:25:46.018 INFO 2981023 --- [host-check-clockdiff1,4225db5975d9317d,a2969d2f7ef05971] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:25:46.018 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:25:46.020 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:25:46.020 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:25:46.036 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:25:46.036 INFO 2981023 --- [host-check-clockdiff2,4225db5975d9317d,da33daa15fb76649] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:25:46.036 INFO 2981023 --- [host-check-clockdiff3,4225db5975d9317d,0984af4c7f68b4ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:25:46.038 INFO 2981023 --- [host-check-clockdiff4,4225db5975d9317d,37f7728ab21e6829] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:25:46.041 INFO 2981023 --- [ocp-schedules-29,fbc298fc599bc3e4,03467e74686a5ca6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:25:46.044 INFO 2981023 --- [host-check-clockdiff2,4225db5975d9317d,da33daa15fb76649] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050346 5 5 , error= 2024-09-11 18:25:46.044 INFO 2981023 --- [host-check-agent-status1,fbc298fc599bc3e4,f4ca5b5a80c20a12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:25:46.044 INFO 2981023 --- [host-check-agent-status2,fbc298fc599bc3e4,64fabf7f788f2ded] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:25:46.044 INFO 2981023 --- [host-check-agent-status3,fbc298fc599bc3e4,43a193c5c4761147] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:25:46.045 INFO 2981023 --- [host-check-agent-status4,fbc298fc599bc3e4,b993b4bde2d4bed3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:25:46.046 INFO 2981023 --- [host-check-clockdiff3,4225db5975d9317d,0984af4c7f68b4ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050346 0 0 , error= 2024-09-11 18:25:46.057 INFO 2981023 --- [host-check-agent-status3,fbc298fc599bc3e4,43a193c5c4761147] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:25:46.060 INFO 2981023 --- [host-check-agent-status3,fbc298fc599bc3e4,43a193c5c4761147] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:25:46.065 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:25:46.066 INFO 2981023 --- [host-check-agent-status2,fbc298fc599bc3e4,64fabf7f788f2ded] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:25:46.066 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:25:46.067 INFO 2981023 --- [host-check-agent-status2,fbc298fc599bc3e4,64fabf7f788f2ded] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:25:46.070 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:25:46.072 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:25:46.075 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:25:46.096 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:25:46.178 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:25:46.192 INFO 2981023 --- [ocp-schedules-9,55442f87c02d697b,71bbe56b82ab5918] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:25:46.194 INFO 2981023 --- [ocp-schedules-27,51be96e8392fa057,bcdd735daba3dbad] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:25:46.194 INFO 2981023 --- [ocp-schedules-13,27e0609c3490bd95,319ecebc447f4cb7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:25:46.194 INFO 2981023 --- [ocp-schedules-20,561b28641d45d355,9fc51bebdac503d1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:25:46.198 INFO 2981023 --- [ocp-schedules-12,8fdaf779d1a25197,441b62d48ecd815c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:25:46.199 INFO 2981023 --- [host-check-agent-status4,fbc298fc599bc3e4,b993b4bde2d4bed3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:25:46.200 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:25:46.201 INFO 2981023 --- [ocp-schedules-9,55442f87c02d697b,71bbe56b82ab5918] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:25:46.202 INFO 2981023 --- [ocp-schedules-27,51be96e8392fa057,bcdd735daba3dbad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:25:46.205 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050243207000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:25:46.211 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:25:46.212 INFO 2981023 --- [host-check-agent-status4,fbc298fc599bc3e4,b993b4bde2d4bed3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:25:46.226 INFO 2981023 --- [ocp-schedules-27,51be96e8392fa057,bcdd735daba3dbad] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:25:46.260 INFO 2981023 --- [host-check-agent-status1,fbc298fc599bc3e4,f4ca5b5a80c20a12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:25:46.267 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:25:46.269 INFO 2981023 --- [host-check-agent-status1,fbc298fc599bc3e4,f4ca5b5a80c20a12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:25:46.276 INFO 2981023 --- [ocp-schedules-13,27e0609c3490bd95,319ecebc447f4cb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:25:46.287 INFO 2981023 --- [ocp-schedules-5,f68ce58ed4b9f876,4fb2dab1c604999a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:25:46.288 INFO 2981023 --- [ocp-schedules-12,8fdaf779d1a25197,441b62d48ecd815c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:25:46.327 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:25:46.329 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:25:46.334599179+08:00 2024-09-11 18:25:46.353 INFO 2981023 --- [host-check-agent-status2,fbc298fc599bc3e4,64fabf7f788f2ded] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:25:46.356 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:25:46.329+08:00 2024-09-11 18:25:46.356 INFO 2981023 --- [host-check-agent-status3,fbc298fc599bc3e4,43a193c5c4761147] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:25:46.357 INFO 2981023 --- [host-check-agent-status2,fbc298fc599bc3e4,64fabf7f788f2ded] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:25:46.357 INFO 2981023 --- [host-check-agent-status2,fbc298fc599bc3e4,64fabf7f788f2ded] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:25:46.357 INFO 2981023 --- [host-check-agent-status2,fbc298fc599bc3e4,64fabf7f788f2ded] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:25:46.365 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:25:46.366 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:25:46.368 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:25:46.367774674+08:00 2024-09-11 18:25:46.369 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:25:46.329+08:00 and status AVAILABLE 2024-09-11 18:25:46.370 INFO 2981023 --- [host-check-agent-status3,fbc298fc599bc3e4,43a193c5c4761147] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:25:46.370 INFO 2981023 --- [host-check-agent-status3,fbc298fc599bc3e4,43a193c5c4761147] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:25:46.370 INFO 2981023 --- [host-check-agent-status3,fbc298fc599bc3e4,43a193c5c4761147] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:25:46.370 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:25:46.368+08:00 2024-09-11 18:25:46.373 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:25:46.368+08:00 and status AVAILABLE 2024-09-11 18:25:46.374 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:25:46.375 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.375 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:25:46.375131017+08:00 2024-09-11 18:25:46.377 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:25:46.376953861+08:00 2024-09-11 18:25:46.378 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:25:46.375+08:00 2024-09-11 18:25:46.380 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:25:46.375+08:00 and status AVAILABLE 2024-09-11 18:25:46.380 INFO 2981023 --- [host-check-agent-status1,fbc298fc599bc3e4,f4ca5b5a80c20a12] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:25:46.383 INFO 2981023 --- [host-check-agent-status4,fbc298fc599bc3e4,b993b4bde2d4bed3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:25:46.386 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.386 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.387 INFO 2981023 --- [host-check-agent-status4,fbc298fc599bc3e4,b993b4bde2d4bed3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:25:46.387 INFO 2981023 --- [host-check-agent-status4,fbc298fc599bc3e4,b993b4bde2d4bed3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:25:46.387 INFO 2981023 --- [host-check-agent-status4,fbc298fc599bc3e4,b993b4bde2d4bed3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:25:46.388 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:25:46.377+08:00 2024-09-11 18:25:46.390 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:25:46.377+08:00 and status AVAILABLE 2024-09-11 18:25:46.392 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:25:46.392 INFO 2981023 --- [host-check-agent-status1,fbc298fc599bc3e4,f4ca5b5a80c20a12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:25:46.392 INFO 2981023 --- [host-check-agent-status1,fbc298fc599bc3e4,f4ca5b5a80c20a12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:25:46.392 INFO 2981023 --- [host-check-agent-status1,fbc298fc599bc3e4,f4ca5b5a80c20a12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:25:46.393 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:25:46.393 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.394 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:25:46.394 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:25:46.398 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.399 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:25:46.398805+08:00, lastProcessTime=2024-09-11T18:25:36.352283+08:00 2024-09-11 18:25:46.400 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:25:46.400 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:46.403 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.403 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:25:46.403 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:46.410 INFO 2981023 --- [ocp-schedules-9,55442f87c02d697b,71bbe56b82ab5918] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:25:46.412 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:25:46.415 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:25:46.422 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:25:46.422 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.436 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:25:46.455 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:25:46.455 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:25:46.456 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:25:46.463 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:25:46.465 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:25:46.466 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.467 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:46.470 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:25:46.473 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:25:46.476 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:25:46.480 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:25:46.483 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:25:46.498 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.498 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:25:46.499 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.499 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:25:46.504 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:25:46.507 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:25:46.513 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.516 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:25:46.527 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:25:46.529 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.530 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:25:46.530 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:25:46.531 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.531 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:25:46.548 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:25:46.557 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:25:46.564 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:25:46.567 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:25:46.585 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:25:46.589 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:25:46.592 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.592 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:25:46.592 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050243207000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:25:46.592 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.593 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:46.593 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:25:46.593 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.596 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:25:46.596 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:25:46.598 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:25:46.610 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:25:46.610 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:25:46.611 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:25:46.619 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:25:46.623 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:25:46.629 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.632 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.632 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:25:46.633 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:25:46.634 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.634 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:25:46.635 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:25:46.635 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.640 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:25:46.640 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:25:46.641 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:25:46.644 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:25:46.655 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:25:46.660 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.661 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.662 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:25:46.663 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:25:46.665 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.665 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.666 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:25:46.666 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:25:46.670 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:25:46.672 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:25:46.672 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:25:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:25:46.677 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:25:46.684 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:25:46.690 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:25:46.698 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.699 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.700 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:25:46.700 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.700 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.701 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:25:46.701 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.701 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:25:46.708 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:25:46.709 INFO 2981023 --- [host-check-host-status2,ec874b21c3609735,288ff4073b3f7fb0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:25:46.709 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:25:46.711 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:25:46.712 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:25:46.717 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:25:46.717 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:25:46.717 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.718 INFO 2981023 --- [host-check-host-status3,ec874b21c3609735,9a5fad75157e5694] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:25:46.727 INFO 2981023 --- [host-check-host-status4,ec874b21c3609735,5fbce4003e0f9328] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:25:46.728 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:25:46.730 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:25:46.741 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:25:46.742 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:25:46.742 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:25:46.746 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:25:46.759 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:25:46.766 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:25:46.766 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.776 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.777 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.783 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.785 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:25:46.787 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:25:46.787 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.788 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:25:46.788 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:25:46.793 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:25:46.795 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:25:46.797 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:25:46.797 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:25:46.398805+08:00 2024-09-11 18:25:46.800 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:25:46.801 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:25:46.802 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:46.802 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:46.803 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:25:46.809 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:25:46.809 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:25:46.814 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:25:46.836 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:25:46.848 INFO 2981023 --- [ocp-schedules-22,be1e0c47bb94e663,d62cd64446689e19] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:25:46.848 INFO 2981023 --- [ocp-schedules-22,be1e0c47bb94e663,d62cd64446689e19] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:25:46.859 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:25:46.865 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:25:46.867 INFO 2981023 --- [ocp-schedules-2,19a54236defedce2,fa6f166f7a1ce568] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:25:47.002 INFO 2981023 --- [host-check-clockdiff1,4225db5975d9317d,a2969d2f7ef05971] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050346 0 0 , error= 2024-09-11 18:25:47.020 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:25:47.025 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:25:47.037 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:25:47.082 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:25:47.087 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:25:47.102 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:47.106 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:25:47.116 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:25:47.116 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:25:47.116 INFO 2981023 --- [ocp-schedules-13,27e0609c3490bd95,319ecebc447f4cb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:25:47.202 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:25:47.208 INFO 2981023 --- [host-check-host-status1,ec874b21c3609735,e5609a9a4befbea8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:25:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,f24f1b7f0ab22b42,623e5743e8559cfd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:47.317 INFO 2981023 --- [prometheus-minute-schedule-1,f24f1b7f0ab22b42,623e5743e8559cfd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:47.461 INFO 2981023 --- [ocp-async-1,0332a7f1299fe61e,7f03a029ea8f3c16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:47.796 INFO 2981023 --- [ocp-async-10,03212698386ad2b2,792b012a09ed3b5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:25:47.919 INFO 2981023 --- [ocp-schedules-18,7d4f72e8ed00a6ea,2b1f2722c4e1518d] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:25:47.942 WARN 2981023 --- [ocp-schedules-18,7d4f72e8ed00a6ea,2b1f2722c4e1518d] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:25:48.298 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:48.298 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:49.461 INFO 2981023 --- [ocp-async-5,4e65c4f0bde791dc,986ca770a8fd387a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:25:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:25:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,77f0cddde5e4edc6,951ee309c557e40c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,77f0cddde5e4edc6,951ee309c557e40c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:50.369 INFO 2981023 --- [ocp-schedules-18,7b3a1113b3ec79a0,32a3b3d645a644dd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:25:51.160 INFO 2981023 --- [ocp-schedules-23,d3220c5af28f59d0,97d6853f8a59ab14] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:25:51.214 WARN 2981023 --- [scanner-13666626-1,d3220c5af28f59d0,f4f09798ad67ddc7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:25:51.258 INFO 2981023 --- [alarm-detect-19,d3220c5af28f59d0,c192f700d0b778f7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050315 ~ 1726050341 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050330, value=7.292)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.292, values={1726050330=7.292}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:51.258 INFO 2981023 --- [alarm-detect-19,d3220c5af28f59d0,c192f700d0b778f7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050315 ~ 1726050341 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050330, value=7.292)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.292, values={1726050330=7.292}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:25:51.299 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:51.303 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:51.350 INFO 2981023 --- [alarm-detect-16,d3220c5af28f59d0,87f22cce1f0992a5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050315 ~ 1726050341 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050330, value=104.476)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.476, values={1726050330=104.476}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:25:51.461 INFO 2981023 --- [ocp-async-9,3a3559018ed22d63,ebb8f4e70f194e82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:51.640 INFO 2981023 --- [ocp-schedules-6,448c131097088b21,d0e330f996cf7cf4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:25:51.642 INFO 2981023 --- [ocp-schedules-6,448c131097088b21,d0e330f996cf7cf4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:25:51.642 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:25:51.642 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:25:51.642 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:25:51.642 INFO 2981023 --- [ocp-schedules-6,448c131097088b21,d0e330f996cf7cf4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:25:52.009 INFO 2981023 --- [host-check-clockdiff4,4225db5975d9317d,37f7728ab21e6829] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050352 0 0 , error= 2024-09-11 18:25:52.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: ob_cluster_sync, cached timestamp: 1726043146 2024-09-11 18:25:52.339 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = ob_cluster_sync 2024-09-11 18:25:52.817 INFO 2981023 --- [ocp-schedules-9,b5fecf1de4f10070,618320cfd5cb0c70] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:25:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,ef82910f8e740a85,2ffd44bc8d0b010a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,ef82910f8e740a85,2ffd44bc8d0b010a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:53.375 INFO 2981023 --- [ocp-schedules-24,4c96965039255098,e477f45fbb60b883] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:25:53.375 INFO 2981023 --- [ocp-schedules-24,4c96965039255098,e477f45fbb60b883] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:25:53.461 INFO 2981023 --- [ocp-async-6,9f1f14c61af618cc,b6c2755561bff6e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:54.303 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:54.303 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:54.501 INFO 2981023 --- [ocp-schedules-22,0c9c3f28b48854fd,7c38626d5697187d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:25:55.357 INFO 2981023 --- [ocp-schedules-2,65a8368bf920b145,33780601780e7f05] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:25:55.376 WARN 2981023 --- [ocp-schedules-2,65a8368bf920b145,33780601780e7f05] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:25:55.461 INFO 2981023 --- [ocp-async-7,221617df2a6811fa,7f837997fa0ea2ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:55.764 INFO 2981023 --- [ocp-schedules-18,035b5c9c58474545,3f7813485dc1352e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:25:55.772 INFO 2981023 --- [ocp-schedules-18,035b5c9c58474545,3f7813485dc1352e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:25:55.772 INFO 2981023 --- [ocp-schedules-18,035b5c9c58474545,3f7813485dc1352e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:25:55.914 INFO 2981023 --- [ocp-schedules-30,fbf4d9ebfbb3b5f0,9bd7fda5c89e0672] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:25:55.914 INFO 2981023 --- [ocp-schedules-30,fbf4d9ebfbb3b5f0,9bd7fda5c89e0672] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:25:55.915 INFO 2981023 --- [ocp-schedules-30,fbf4d9ebfbb3b5f0,9bd7fda5c89e0672] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:25:55.915 INFO 2981023 --- [ocp-schedules-30,fbf4d9ebfbb3b5f0,9bd7fda5c89e0672] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:25:55.915 INFO 2981023 --- [ocp-schedules-30,fbf4d9ebfbb3b5f0,9bd7fda5c89e0672] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:25:55.915 INFO 2981023 --- [ocp-schedules-30,fbf4d9ebfbb3b5f0,9bd7fda5c89e0672] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:25:56.135 INFO 2981023 --- [ocp-schedules-22,fd53a314e7d8d0b6,98ea4b1b3622db6c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:25:56.141 INFO 2981023 --- [ocp-schedules-22,fd53a314e7d8d0b6,98ea4b1b3622db6c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:25:56.141 INFO 2981023 --- [ocp-schedules-22,fd53a314e7d8d0b6,98ea4b1b3622db6c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:25:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,1a855ee486d9fd9b,ff479c92379a5f84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,1a855ee486d9fd9b,ff479c92379a5f84] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:25:56.352813+08:00, lastProcessTime=2024-09-11T18:25:46.398805+08:00 2024-09-11 18:25:56.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:25:56.352813+08:00 2024-09-11 18:25:57.303 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:25:57.305 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:25:57.461 INFO 2981023 --- [ocp-async-4,8addf42262d4f6fd,73e3e5889e0b32d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:25:57.796 INFO 2981023 --- [ocp-async-2,70566072d4ce3acc,79755bd384049ef8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:25:58.707 INFO 2981023 --- [ocp-schedules-14,ad19ec8ed779f825,5a4b10b9cdb516a9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:25:58.707 INFO 2981023 --- [ocp-schedules-14,ad19ec8ed779f825,5a4b10b9cdb516a9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:25:58.742 INFO 2981023 --- [Collect all tenants major compactions2,ad19ec8ed779f825,f8b97779f559478c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:25:58.744 INFO 2981023 --- [Collect all tenants major compactions3,ad19ec8ed779f825,2988a6ebdf4f6cc5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:25:58.748 INFO 2981023 --- [Collect all tenants major compactions5,ad19ec8ed779f825,786232183fa43525] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:25:58.750 INFO 2981023 --- [Collect all tenants major compactions1,ad19ec8ed779f825,988b8ccd2fe753c4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:25:58.750 INFO 2981023 --- [Collect all tenants major compactions4,ad19ec8ed779f825,0924b39cdf5b1a9e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:25:58.750 INFO 2981023 --- [Collect all tenants major compactions6,ad19ec8ed779f825,f522820ee0489616] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:25:58.770 INFO 2981023 --- [Collect all tenants major compactions1,ad19ec8ed779f825,988b8ccd2fe753c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:58.771 INFO 2981023 --- [Collect all tenants major compactions3,ad19ec8ed779f825,2988a6ebdf4f6cc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:58.771 INFO 2981023 --- [Collect all tenants major compactions2,ad19ec8ed779f825,f8b97779f559478c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:58.771 INFO 2981023 --- [Collect all tenants major compactions6,ad19ec8ed779f825,f522820ee0489616] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:58.771 INFO 2981023 --- [Collect all tenants major compactions1,ad19ec8ed779f825,988b8ccd2fe753c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:25:58.772 INFO 2981023 --- [Collect all tenants major compactions2,ad19ec8ed779f825,f8b97779f559478c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:25:58.772 INFO 2981023 --- [Collect all tenants major compactions3,ad19ec8ed779f825,2988a6ebdf4f6cc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:25:58.773 INFO 2981023 --- [Collect all tenants major compactions6,ad19ec8ed779f825,f522820ee0489616] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:25:58.775 INFO 2981023 --- [Collect all tenants major compactions1,ad19ec8ed779f825,988b8ccd2fe753c4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:25:58.775 INFO 2981023 --- [Collect all tenants major compactions1,ad19ec8ed779f825,988b8ccd2fe753c4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:25:58.775 INFO 2981023 --- [Collect all tenants major compactions3,ad19ec8ed779f825,2988a6ebdf4f6cc5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:25:58.775 INFO 2981023 --- [Collect all tenants major compactions3,ad19ec8ed779f825,2988a6ebdf4f6cc5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:25:58.775 INFO 2981023 --- [Collect all tenants major compactions2,ad19ec8ed779f825,f8b97779f559478c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:25:58.775 INFO 2981023 --- [Collect all tenants major compactions2,ad19ec8ed779f825,f8b97779f559478c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:25:58.776 INFO 2981023 --- [Collect all tenants major compactions6,ad19ec8ed779f825,f522820ee0489616] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:25:58.776 INFO 2981023 --- [Collect all tenants major compactions6,ad19ec8ed779f825,f522820ee0489616] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:25:58.781 INFO 2981023 --- [Collect all tenants major compactions5,ad19ec8ed779f825,786232183fa43525] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:58.783 INFO 2981023 --- [Collect all tenants major compactions4,ad19ec8ed779f825,0924b39cdf5b1a9e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:25:58.784 INFO 2981023 --- [Collect all tenants major compactions5,ad19ec8ed779f825,786232183fa43525] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:25:58.794 INFO 2981023 --- [Collect all tenants major compactions4,ad19ec8ed779f825,0924b39cdf5b1a9e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:25:58.794 INFO 2981023 --- [Collect all tenants major compactions5,ad19ec8ed779f825,786232183fa43525] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:25:58.794 INFO 2981023 --- [Collect all tenants major compactions5,ad19ec8ed779f825,786232183fa43525] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:25:58.798 INFO 2981023 --- [Collect all tenants major compactions4,ad19ec8ed779f825,0924b39cdf5b1a9e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:25:58.798 INFO 2981023 --- [Collect all tenants major compactions4,ad19ec8ed779f825,0924b39cdf5b1a9e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:25:58.799 INFO 2981023 --- [ocp-schedules-14,ad19ec8ed779f825,5a4b10b9cdb516a9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:25:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,05c1605946a74149,b4d8ad9f6d40573d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050360, interval=60 2024-09-11 18:25:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,05c1605946a74149,b4d8ad9f6d40573d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:25:59.461 INFO 2981023 --- [ocp-async-6,8f31113703d828b8,2c42c0ec9dd45842] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:00.305 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:00.307 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:00.496 INFO 2981023 --- [ocp-schedules-16,554e6fd208a1b9ac,5ead2263140dea69] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:26:00.496+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:26:00.499 INFO 2981023 --- [ocp-schedules-16,554e6fd208a1b9ac,5ead2263140dea69] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:26:00.650 INFO 2981023 --- [ocp-schedules-16,554e6fd208a1b9ac,5ead2263140dea69] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 53 to SqlAggregateData 2024-09-11 18:26:00.651 INFO 2981023 --- [ocp-schedules-16,554e6fd208a1b9ac,5ead2263140dea69] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:26:01.358 INFO 2981023 --- [ocp-schedules-5,58799d8a3870f881,2d22d67feb5d32cd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:26:01.462 INFO 2981023 --- [ocp-async-7,68039aecfa934f34,fa36c3968a0af53a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:01.477 WARN 2981023 --- [scanner-13666846-1,58799d8a3870f881,e1f70542ffb994f1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:26:01.495 INFO 2981023 --- [alarm-detect-20,58799d8a3870f881,b8d8c8aa544ee08b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646280, data is empty. 2024-09-11 18:26:01.504 INFO 2981023 --- [alarm-detect-20,58799d8a3870f881,b8d8c8aa544ee08b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646333, data is empty. 2024-09-11 18:26:01.505 INFO 2981023 --- [alarm-detect-4,58799d8a3870f881,122422a299820399] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050330 ~ 1726050351 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050345, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050345, value=97.865)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=97.865, values={1726050345=97.865}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050345=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:01.509 INFO 2981023 --- [alarm-detect-20,58799d8a3870f881,b8d8c8aa544ee08b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646333, data is empty. 2024-09-11 18:26:01.526 INFO 2981023 --- [alarm-detect-4,58799d8a3870f881,d4dba5e650985e9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646280, data is empty. 2024-09-11 18:26:01.528 INFO 2981023 --- [alarm-detect-7,58799d8a3870f881,9900aabea5fe18fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646280, data is empty. 2024-09-11 18:26:01.530 INFO 2981023 --- [alarm-detect-4,58799d8a3870f881,d4dba5e650985e9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646333, data is empty. 2024-09-11 18:26:01.534 INFO 2981023 --- [alarm-detect-7,58799d8a3870f881,9900aabea5fe18fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646333, data is empty. 2024-09-11 18:26:01.536 INFO 2981023 --- [alarm-detect-4,58799d8a3870f881,d4dba5e650985e9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646333, data is empty. 2024-09-11 18:26:01.539 INFO 2981023 --- [alarm-detect-7,58799d8a3870f881,9900aabea5fe18fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646333, data is empty. 2024-09-11 18:26:01.553 INFO 2981023 --- [alarm-detect-7,58799d8a3870f881,8196ade01b8dc59a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050330 ~ 1726050351 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050345, value=8.061)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.061, values={1726050345=8.061}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:01.553 INFO 2981023 --- [alarm-detect-7,58799d8a3870f881,8196ade01b8dc59a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050330 ~ 1726050351 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050345, value=8.061)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.061, values={1726050345=8.061}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:26:01.565 INFO 2981023 --- [alarm-detect-12,58799d8a3870f881,970f869c61d6d3d7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050330 ~ 1726050351 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050345, value=97.865)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=97.865, values={1726050345=97.865}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:01.632 INFO 2981023 --- [alarm-detect-14,58799d8a3870f881,f57d335fa4c7c612] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646280, data is empty. 2024-09-11 18:26:01.635 INFO 2981023 --- [alarm-detect-14,58799d8a3870f881,f57d335fa4c7c612] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646333, data is empty. 2024-09-11 18:26:01.640 INFO 2981023 --- [alarm-detect-14,58799d8a3870f881,f57d335fa4c7c612] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646333, data is empty. 2024-09-11 18:26:01.675 INFO 2981023 --- [alarm-detect-10,58799d8a3870f881,27979ea020109342] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050330 ~ 1726050351 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050345, value=105.071)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.071, values={1726050345=105.071}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:02.303 INFO 2981023 --- [ocp-schedules-7,6a91852af070c627,340a018113c4ed82] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:26:02.303 INFO 2981023 --- [ocp-schedules-7,6a91852af070c627,340a018113c4ed82] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:26:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,a1d0cc1bf78e2823,75187017890a8f7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:02.319 INFO 2981023 --- [prometheus-minute-schedule-1,a1d0cc1bf78e2823,75187017890a8f7a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:02.508 INFO 2981023 --- [ocp-schedules-8,eefecd98c4947fad,498fbefd58a6d28b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:26:02.512 INFO 2981023 --- [ocp-schedules-8,eefecd98c4947fad,498fbefd58a6d28b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:02.545 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:26:02.546 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.547 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:26:02.548 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.548 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:26:02.551 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:26:02.552 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.553 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.555 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.556 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.557 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.558 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.561 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:26:02.561 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.563 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.563 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:26:02.567 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.568 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.568 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:26:02.571 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.571 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:26:02.575 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.575 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.576 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.576 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:26:02.579 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.582 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:26:02.587 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.588 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.613 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.614 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:26:02.614 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.615 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:26:02.615 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.616 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.616 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:26:02.616 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:26:02.616 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:26:02.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.617 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:26:02.618 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.618 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:26:02.619 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.620 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:26:02.620 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:26:02.620 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.621 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:26:02.621 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.622 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:26:02.622 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.623 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:26:02.623 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.624 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:26:02.625 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.625 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:26:02.625 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:26:02.627 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:26:02.629 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.635 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.635 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:02.637 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.637 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.639 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:02.640 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:26:02.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:26:02.645 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6431D6ED-0-0] 2024-09-11 18:26:02.645 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:26:02.646 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:26:02.649 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:26:02.650 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.650 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:26:02.653 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.654 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:26:02.655 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:26:02.656 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:26:02.658 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:26:02.658 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.658 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.659 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:26:02.659 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:26:02.659 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:26:02.664 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:26:02.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:26:02.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:26:02.665 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:26:02.674 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.678 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:26:02.689 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.694 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:26:02.699 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:26:02.703 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6431D6F5-0-0] 2024-09-11 18:26:02.703 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221563633) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:26:02.713 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.714 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:26:02.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.728 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:26:02.735 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.743 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:26:02.759 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.768 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:26:02.772 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.775 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:26:02.780 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.781 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:26:02.781 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:26:02.783 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:26:02.792 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:02.818 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.819 INFO 2981023 --- [ocp-schedules-21,8146c793a28b126c,1f4594fb5810f85a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:26:02.822 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:26:02.835 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:26:02.836 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:26:02.839 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:02.840 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:26:02.848 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:26:02.849 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:26:02.849 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:26:02.849 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:26:02.849 INFO 2981023 --- [ocp-schedules-8,eefecd98c4947fad,498fbefd58a6d28b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:26:03.217 INFO 2981023 --- [ocp-schedules-30,91897d91ca033b78,d966854088ac3951] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:26:03.218 INFO 2981023 --- [ocp-schedules-30,91897d91ca033b78,d966854088ac3951] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:03.238 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:03.239 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:26:03.243 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:03.245 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:26:03.246 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050363245000} 2024-09-11 18:26:03.251 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:26:03.255 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000237, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:26:03.255 INFO 2981023 --- [ocp-schedules-30,91897d91ca033b78,d966854088ac3951] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:26:03.307 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:03.308 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:03.462 INFO 2981023 --- [ocp-async-9,601bdf1e32205ca3,b35215e621faf384] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:03.479 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:26:03.479 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:26:03.479 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:26:03.483 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000237, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:26:03.485 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:26:04.385 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000237 2024-09-11 18:26:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,46f83e32bbe9326a,bf1520af17bb99f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,46f83e32bbe9326a,bf1520af17bb99f1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:05.462 INFO 2981023 --- [ocp-async-1,5a552dde958050b5,476ef16003b17a54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:26:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:26:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:26:05.985 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:26:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:26:05.994 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:26:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:26:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:26:05.997 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:05.997 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:26:05.998 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:05.998 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:26:06.039 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:23:41, start=2024-09-11T18:20:11, end=2024-09-11T18:22:41, updates=314, granule=120000ms, delay=210000ms 2024-09-11 18:26:06.044 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:23:41, target=2024-09-11T18:22:41, granule=120000ms, delay=60000ms) 2024-09-11 18:26:06.044 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:22:41, target=2024-09-11T18:10:51, granule=600000ms, delay=710000ms) 2024-09-11 18:26:06.044 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=74ms, period=120000ms 2024-09-11 18:26:06.308 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:06.308 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:06.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:26:06.351513+08:00, lastProcessTime=2024-09-11T18:25:56.352813+08:00 2024-09-11 18:26:06.484 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:26:06.351513+08:00 2024-09-11 18:26:07.462 INFO 2981023 --- [ocp-async-10,a68607992c863b62,542cdf11bac5d450] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:07.796 INFO 2981023 --- [ocp-async-9,4bb9a1bca6f07475,fbd5d0f249cb9fce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:26:08.187 INFO 2981023 --- [ocp-schedules-26,1289ea8e1f412118,dbce5f1f5c1a8dc9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:26:08.262 INFO 2981023 --- [ocp-schedules-24,5fee70db0cc42db8,12a6ee5b590bd04d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:26:08.262+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:26:08.263 INFO 2981023 --- [ocp-schedules-24,5fee70db0cc42db8,12a6ee5b590bd04d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:26:08.266 INFO 2981023 --- [ocp-schedules-24,5fee70db0cc42db8,12a6ee5b590bd04d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:26:08.266 INFO 2981023 --- [ocp-schedules-24,5fee70db0cc42db8,12a6ee5b590bd04d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:26:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,b14c8519269f7b4d,95fbf1c7e5595d44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,b14c8519269f7b4d,95fbf1c7e5595d44] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:09.308 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:09.309 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:09.462 INFO 2981023 --- [ocp-async-4,c3257f66d078aae1,068dd9deccbb0fdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:10.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:26:10.004 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:26:10.745 INFO 2981023 --- [ocp-schedules-7,369f126f4921fed7,3e964c5e87aebcc4] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:26:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:26:10.924 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:26:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,fa18e3da2ba08ef8,1376904fc56a75d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,fa18e3da2ba08ef8,1376904fc56a75d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:11.462 INFO 2981023 --- [ocp-async-8,51c8effb649968dd,72c68112b29ab245] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:11.685 INFO 2981023 --- [ocp-schedules-20,72c36b179e896d91,130cebcdbe8d4bf8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:26:11.744 WARN 2981023 --- [scanner-13667535-1,72c36b179e896d91,0286d625ce6dbd93] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:26:11.749 INFO 2981023 --- [alarm-detect-16,72c36b179e896d91,e2b03e808c12c015] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050345 ~ 1726050361 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050360, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050360, value=97.39)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=97.39, values={1726050360=97.39}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050360=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:11.809 INFO 2981023 --- [alarm-detect-18,72c36b179e896d91,bb6c2c2d8a33098a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050345 ~ 1726050361 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050360, value=97.39)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=97.39, values={1726050360=97.39}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:12.309 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:12.310 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:12.822 INFO 2981023 --- [ocp-schedules-9,401f0f2f02b21280,025de0ac37816fbd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:26:13.462 INFO 2981023 --- [ocp-async-6,b4f1e543dc1e8bb9,928f5c7f51d5dff2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:14.295 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050300, nodeCount=480, affectRows=480 2024-09-11 18:26:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,ca377f9cbe71a9fe,910a070dcb979461] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,ff4b7bea054948be,2bd5ce1cdf71796b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:26:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,ca377f9cbe71a9fe,910a070dcb979461] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:14.579 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050300, nodeCount=1000, affectRows=1000 2024-09-11 18:26:14.582 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050300, nodeCount=999, validCount=999 2024-09-11 18:26:14.594 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050300, nodeCount=48, affectRows=48 2024-09-11 18:26:14.594 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050300, nodeCount=46, validCount=46 2024-09-11 18:26:14.618 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050300, nodeCount=1000, affectRows=1000 2024-09-11 18:26:14.620 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050300, nodeCount=1000, validCount=1000 2024-09-11 18:26:15.311 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:15.311 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:15.462 INFO 2981023 --- [ocp-async-7,67a0279f595be71b,c354a74115fede26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:15.775 INFO 2981023 --- [ocp-schedules-5,e9d8bb42618cfbc9,ee258eb402f5068c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:26:15.783 INFO 2981023 --- [ocp-schedules-5,e9d8bb42618cfbc9,ee258eb402f5068c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:26:15.783 INFO 2981023 --- [ocp-schedules-5,e9d8bb42618cfbc9,ee258eb402f5068c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:26:16.017 INFO 2981023 --- [ocp-schedules-12,5f41fc7f852bf6af,b03e0fc3fae3398b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:26:16.017 INFO 2981023 --- [host-check-clockdiff1,5f41fc7f852bf6af,966e11367e612e2b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:26:16.017 INFO 2981023 --- [host-check-clockdiff2,5f41fc7f852bf6af,ff95560a32af3c53] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:26:16.018 INFO 2981023 --- [host-check-clockdiff3,5f41fc7f852bf6af,1e797706f4bb00a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:26:16.020 INFO 2981023 --- [host-check-clockdiff4,5f41fc7f852bf6af,6f6ceff4a4134def] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:26:16.030 INFO 2981023 --- [ocp-schedules-27,5eb1bc4da8450225,c59c22e000a75a60] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:26:16.031 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:26:16.031 INFO 2981023 --- [host-check-clockdiff2,5f41fc7f852bf6af,ff95560a32af3c53] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050376 5 5 , error= 2024-09-11 18:26:16.037 INFO 2981023 --- [host-check-clockdiff3,5f41fc7f852bf6af,1e797706f4bb00a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050376 0 0 , error= 2024-09-11 18:26:16.037 INFO 2981023 --- [ocp-schedules-8,32d264571c006250,7a586cd9f57a6a76] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:26:16.037 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:26:16.037 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:26:16.038 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:26:16.038 INFO 2981023 --- [host-check-agent-status1,32d264571c006250,8dcbae76c1febaff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:26:16.038 INFO 2981023 --- [host-check-agent-status2,32d264571c006250,68015733d4a5a7e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:26:16.049 INFO 2981023 --- [host-check-agent-status3,32d264571c006250,39663b3a26607676] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:26:16.049 INFO 2981023 --- [host-check-agent-status4,32d264571c006250,d6fa762c8211271b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:26:16.062 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:26:16.066 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:26:16.070 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:26:16.073 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:26:16.074 INFO 2981023 --- [host-check-agent-status2,32d264571c006250,68015733d4a5a7e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:26:16.074 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:26:16.074 INFO 2981023 --- [host-check-agent-status1,32d264571c006250,8dcbae76c1febaff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:26:16.074 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:26:16.075 INFO 2981023 --- [host-check-agent-status3,32d264571c006250,39663b3a26607676] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:26:16.075 INFO 2981023 --- [host-check-agent-status1,32d264571c006250,8dcbae76c1febaff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:26:16.076 INFO 2981023 --- [host-check-agent-status3,32d264571c006250,39663b3a26607676] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:26:16.077 INFO 2981023 --- [host-check-agent-status2,32d264571c006250,68015733d4a5a7e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:26:16.078 INFO 2981023 --- [host-check-agent-status4,32d264571c006250,d6fa762c8211271b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:26:16.079 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:26:16.080 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:26:16.080 INFO 2981023 --- [host-check-agent-status4,32d264571c006250,d6fa762c8211271b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:26:16.187 INFO 2981023 --- [ocp-schedules-7,bd9196151d52e441,33d0947ed4c128e0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:26:16.194 INFO 2981023 --- [ocp-schedules-10,70f3aff1bc179476,ef823e7555f8a395] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:26:16.196 INFO 2981023 --- [ocp-schedules-3,0332b5101eddb8f8,747f709e281f1465] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:26:16.196 INFO 2981023 --- [ocp-schedules-29,2b83af9f7e88f48e,3bd0d2c916886dca] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:26:16.245 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050300, nodeCount=480, validCount=480 2024-09-11 18:26:16.259 INFO 2981023 --- [ocp-schedules-7,bd9196151d52e441,33d0947ed4c128e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:16.283 INFO 2981023 --- [ocp-schedules-10,70f3aff1bc179476,ef823e7555f8a395] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:16.284 INFO 2981023 --- [ocp-schedules-3,0332b5101eddb8f8,747f709e281f1465] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:16.298 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050363245000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:26:16.313 INFO 2981023 --- [ocp-schedules-10,70f3aff1bc179476,ef823e7555f8a395] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:26:16.387 INFO 2981023 --- [ocp-schedules-30,12670d53ae3b5fda,1d1f1390063768d9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:26:16.466 INFO 2981023 --- [host-check-agent-status4,32d264571c006250,d6fa762c8211271b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:26:16.467 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:26:16.469 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:26:16.474894092+08:00 2024-09-11 18:26:16.471 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:26:16.469+08:00 2024-09-11 18:26:16.474 INFO 2981023 --- [host-check-agent-status4,32d264571c006250,d6fa762c8211271b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:26:16.474 INFO 2981023 --- [host-check-agent-status4,32d264571c006250,d6fa762c8211271b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:26:16.474 INFO 2981023 --- [host-check-agent-status4,32d264571c006250,d6fa762c8211271b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:26:16.474 INFO 2981023 --- [host-check-agent-status1,32d264571c006250,8dcbae76c1febaff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:26:16.476 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:26:16.469+08:00 and status AVAILABLE 2024-09-11 18:26:16.476 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:26:16.474719+08:00, lastProcessTime=2024-09-11T18:26:06.351513+08:00 2024-09-11 18:26:16.476 INFO 2981023 --- [host-check-agent-status2,32d264571c006250,68015733d4a5a7e7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:26:16.476 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:26:16.478 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:26:16.478732596+08:00 2024-09-11 18:26:16.479 INFO 2981023 --- [host-check-agent-status2,32d264571c006250,68015733d4a5a7e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:26:16.479 INFO 2981023 --- [host-check-agent-status2,32d264571c006250,68015733d4a5a7e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:26:16.479 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:26:16.479 INFO 2981023 --- [host-check-agent-status2,32d264571c006250,68015733d4a5a7e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:26:16.479 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:26:16.478+08:00 2024-09-11 18:26:16.480 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.481 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:26:16.480330405+08:00 2024-09-11 18:26:16.481 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:26:16.481+08:00 2024-09-11 18:26:16.481 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:26:16.482 INFO 2981023 --- [host-check-agent-status1,32d264571c006250,8dcbae76c1febaff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:26:16.482 INFO 2981023 --- [host-check-agent-status1,32d264571c006250,8dcbae76c1febaff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:26:16.482 INFO 2981023 --- [host-check-agent-status1,32d264571c006250,8dcbae76c1febaff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:26:16.482 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:26:16.478+08:00 and status AVAILABLE 2024-09-11 18:26:16.482 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:26:16.484 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:26:16.484 INFO 2981023 --- [host-check-agent-status3,32d264571c006250,39663b3a26607676] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:26:16.484 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:26:16.481+08:00 and status AVAILABLE 2024-09-11 18:26:16.486 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:26:16.485895326+08:00 2024-09-11 18:26:16.487 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:26:16.489 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:26:16.492 INFO 2981023 --- [host-check-agent-status3,32d264571c006250,39663b3a26607676] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:26:16.492 INFO 2981023 --- [host-check-agent-status3,32d264571c006250,39663b3a26607676] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:26:16.492 INFO 2981023 --- [host-check-agent-status3,32d264571c006250,39663b3a26607676] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:26:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.500 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:26:16.486+08:00 2024-09-11 18:26:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:16.502 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:26:16.503 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:26:16.486+08:00 and status AVAILABLE 2024-09-11 18:26:16.504 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.505 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:26:16.506 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:26:16.508 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.508 INFO 2981023 --- [ocp-schedules-7,bd9196151d52e441,33d0947ed4c128e0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:26:16.508 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:16.555 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:26:16.569 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:26:16.570 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:26:16.571 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:26:16.572 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:26:16.573 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:26:16.574 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:26:16.577 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:26:16.578 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:26:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.586 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:26:16.594 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:26:16.594 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.595 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:16.605 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:26:16.605 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:26:16.608 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:26:16.619 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:26:16.619 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050363245000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:26:16.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:16.621 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:26:16.621 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.624 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:26:16.625 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:26:16.626 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:26:16.634 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:26:16.636 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:26:16.663 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:26:16.663 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:26:16.667 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:26:16.668 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:26:16.668 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:26:16.681 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:26:16.693 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:26:16.695 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:16.695 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.695 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:26:16.696 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:26:16.696 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:26:16.700 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:26:16.701 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:26:16.704 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:26:16.705 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:26:16.716 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:26:16.722 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:26:16.725 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:26:16.730 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:26:16.735 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:26:16.755 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:26:16.763 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:26:16.770 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:26:16.774 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:26:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:16.782 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.785 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:26:16.788 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:26:16.792 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:26:16.793 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:26:16.795 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:26:16.802 INFO 2981023 --- [host-check-host-status4,5eb1bc4da8450225,c132736cfd055f59] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:26:16.803 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:26:16.803 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:16.804 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.804 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:26:16.811 INFO 2981023 --- [host-check-host-status3,5eb1bc4da8450225,a23a22528bf88fb9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:26:16.813 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:26:16.818 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:26:16.821 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:26:16.821 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:26:16.828 INFO 2981023 --- [host-check-host-status2,5eb1bc4da8450225,02589a11265c704d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:26:16.832 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:16.840 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.846 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:16.849 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:26:16.853 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:26:16.474719+08:00 2024-09-11 18:26:16.855 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:26:16.855 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.861 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:26:16.865 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:26:16.870 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:26:16.870 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:26:16.879 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:26:16.883 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:16.888 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:16.893 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:26:16.899 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:26:16.901 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:26:16.903 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:26:16.904 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:26:16.915 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:26:17.147 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:26:17.154 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:26:17.166 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:26:17.241 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:26:17.247 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:17.249 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:17.250 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:26:17.255 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:26:17.255 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:26:17.256 INFO 2981023 --- [ocp-schedules-3,0332b5101eddb8f8,747f709e281f1465] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:26:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,35eef6db5a5f6da6,adbe6f995fff86d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,35eef6db5a5f6da6,adbe6f995fff86d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:17.381 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:26:17.387 INFO 2981023 --- [host-check-host-status1,5eb1bc4da8450225,83e9ca40355d55ec] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:26:17.462 INFO 2981023 --- [ocp-async-4,8c35cc22004bf0b7,a0a75ab6840df39f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:17.796 INFO 2981023 --- [ocp-async-6,5301419c2752cc1b,8942b13faa9a876e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:26:18.311 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:18.312 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:18.584 INFO 2981023 --- [host-check-clockdiff1,5f41fc7f852bf6af,966e11367e612e2b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050378 0 0 , error= 2024-09-11 18:26:19.463 INFO 2981023 --- [ocp-async-10,c2af01012d1e4408,c9fa1625133e8ae5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,6b1a200d75a8d2c8,aebcdb0f278d03d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,6b1a200d75a8d2c8,aebcdb0f278d03d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:21.312 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:21.313 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:21.463 INFO 2981023 --- [ocp-async-9,520a87d5ebc2bbbe,9ce66484cf655214] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:21.913 INFO 2981023 --- [ocp-schedules-13,e0dc96df6dabdd95,efbe6495aca083f9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:26:21.958 WARN 2981023 --- [scanner-13668809-1,e0dc96df6dabdd95,26cf4f9f8c9be404] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:26:21.976 INFO 2981023 --- [host-check-clockdiff4,5f41fc7f852bf6af,6f6ceff4a4134def] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050381 0 0 , error= 2024-09-11 18:26:21.993 INFO 2981023 --- [alarm-detect-3,e0dc96df6dabdd95,7d5255ecd5d24112] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646333, data is empty. 2024-09-11 18:26:21.996 INFO 2981023 --- [alarm-detect-3,e0dc96df6dabdd95,7d5255ecd5d24112] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646333, data is empty. 2024-09-11 18:26:21.999 INFO 2981023 --- [alarm-detect-3,e0dc96df6dabdd95,7d5255ecd5d24112] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646333, data is empty. 2024-09-11 18:26:22.012 INFO 2981023 --- [alarm-detect-15,e0dc96df6dabdd95,15bc6d64fd7532fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646333, data is empty. 2024-09-11 18:26:22.014 INFO 2981023 --- [alarm-detect-2,e0dc96df6dabdd95,05e41fced4957aca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646333, data is empty. 2024-09-11 18:26:22.019 INFO 2981023 --- [alarm-detect-15,e0dc96df6dabdd95,15bc6d64fd7532fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646333, data is empty. 2024-09-11 18:26:22.021 INFO 2981023 --- [alarm-detect-2,e0dc96df6dabdd95,05e41fced4957aca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646333, data is empty. 2024-09-11 18:26:22.024 INFO 2981023 --- [alarm-detect-15,e0dc96df6dabdd95,15bc6d64fd7532fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646333, data is empty. 2024-09-11 18:26:22.025 INFO 2981023 --- [alarm-detect-2,e0dc96df6dabdd95,05e41fced4957aca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646333, data is empty. 2024-09-11 18:26:22.098 INFO 2981023 --- [alarm-detect-3,e0dc96df6dabdd95,d8cc78b512f997c5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050345 ~ 1726050372 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050360, value=8.757)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.757, values={1726050360=8.757}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:22.098 INFO 2981023 --- [alarm-detect-3,e0dc96df6dabdd95,d8cc78b512f997c5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050345 ~ 1726050372 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050360, value=8.757)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.757, values={1726050360=8.757}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:26:22.147 INFO 2981023 --- [alarm-detect-9,e0dc96df6dabdd95,e7f58c160f7c65e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646333, data is empty. 2024-09-11 18:26:22.150 INFO 2981023 --- [alarm-detect-9,e0dc96df6dabdd95,e7f58c160f7c65e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646333, data is empty. 2024-09-11 18:26:22.154 INFO 2981023 --- [alarm-detect-9,e0dc96df6dabdd95,e7f58c160f7c65e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646333, data is empty. 2024-09-11 18:26:22.170 INFO 2981023 --- [alarm-detect-7,e0dc96df6dabdd95,e9ad2a3dc40a929b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050345 ~ 1726050372 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050360, value=104.514)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.514, values={1726050360=104.514}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_agent_process_status, cached timestamp: 1726043176 2024-09-11 18:26:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_agent_process_status 2024-09-11 18:26:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_agent_process_status, cached timestamp: 1726043176 2024-09-11 18:26:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_agent_process_status 2024-09-11 18:26:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_agent_process_status, cached timestamp: 1726043176 2024-09-11 18:26:22.338 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_agent_process_status 2024-09-11 18:26:22.339 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: clock_diff_milliseconds, cached timestamp: 1726043176 2024-09-11 18:26:22.339 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = clock_diff_milliseconds 2024-09-11 18:26:22.344 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Unregister ocp-server metrics: host_available, cached timestamp: 1726043176 2024-09-11 18:26:22.344 INFO 2981023 --- [collect-inner-metric-1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Remove expired cache, metric = host_available 2024-09-11 18:26:22.824 INFO 2981023 --- [ocp-schedules-6,e2245fb59697f4b0,416afd5b9f26d0f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:26:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,e06c5b281be6450c,1c1f1dd0bc352b00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,e06c5b281be6450c,1c1f1dd0bc352b00] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:23.463 INFO 2981023 --- [ocp-async-2,9b11ef72fdf13203,296f4495a7adf425] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:24.313 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:24.313 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:24.503 INFO 2981023 --- [ocp-schedules-2,f00f02f4e8231ac5,1f5c8b0c675c80a0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:26:25.463 INFO 2981023 --- [ocp-async-8,9790e32865b4b79d,7a9f9357d4fe9c58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,1fc6c8f6e4273a16,b4cb5a47d0318180] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,1fc6c8f6e4273a16,b4cb5a47d0318180] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:26:26.337427+08:00, lastProcessTime=2024-09-11T18:26:16.474719+08:00 2024-09-11 18:26:26.466 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:26:26.337427+08:00 2024-09-11 18:26:27.313 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:27.314 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:27.463 INFO 2981023 --- [ocp-async-6,b0164c71a80e7bbc,8810f92b4b42eede] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:27.796 INFO 2981023 --- [ocp-async-1,add2e840165314f8,e8782f4a673b9362] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:26:27.985 INFO 2981023 --- [ocp-schedules-18,dafe1791fd3ac9fb,e6b9911b2321c8ef] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:26:27.985 INFO 2981023 --- [ocp-schedules-18,dafe1791fd3ac9fb,e6b9911b2321c8ef] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:26:27.985 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:26:27.985 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:26:27.985 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:26:27.989 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:26:27.989 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:26:27.990 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:26:27.990 INFO 2981023 --- [ocp-schedules-18,dafe1791fd3ac9fb,e6b9911b2321c8ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:26:27.990 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:26:27.991 INFO 2981023 --- [ocp-schedules-18,dafe1791fd3ac9fb,e6b9911b2321c8ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:26:27.991 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:26:27.991 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:26:28.009 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:26:28.009 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:26:28.010 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:26:28.011 INFO 2981023 --- [ocp-schedules-18,dafe1791fd3ac9fb,e6b9911b2321c8ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:26:28.018 INFO 2981023 --- [ocp-schedules-18,dafe1791fd3ac9fb,e6b9911b2321c8ef] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:26:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,aed692a083717d3e,74db4b9c1a2c5c76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,aed692a083717d3e,74db4b9c1a2c5c76] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:29.463 INFO 2981023 --- [ocp-async-4,9c2fa6eab9da66ac,578a9b15fe45f6ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:30.314 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:30.314 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:31.463 INFO 2981023 --- [ocp-async-5,376a4eb2e790b3b6,af23d914dd2ed3e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:32.194 INFO 2981023 --- [ocp-schedules-23,fb9148dc16258834,031c0bbc8ab57505] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:26:32.239 WARN 2981023 --- [scanner-13669128-1,fb9148dc16258834,10d09fb48f210a14] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:26:32.261 INFO 2981023 --- [alarm-detect-3,fb9148dc16258834,62f3f063105fd0fd] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050360 ~ 1726050382 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050375, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050375, value=98.645)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.645, values={1726050375=98.645}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050375=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:32.301 INFO 2981023 --- [alarm-detect-2,fb9148dc16258834,db3d4fd6c13db1d2] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050360 ~ 1726050382 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050375, value=98.645)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.645, values={1726050375=98.645}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:32.305 INFO 2981023 --- [alarm-detect-8,fb9148dc16258834,c2da028ff718cac6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050360 ~ 1726050382 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050375, value=10.192)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=10.192, values={1726050375=10.192}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:32.305 INFO 2981023 --- [alarm-detect-8,fb9148dc16258834,c2da028ff718cac6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050360 ~ 1726050382 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050375, value=10.192)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=10.192, values={1726050375=10.192}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:26:32.307 INFO 2981023 --- [alarm-detect-1,fb9148dc16258834,e991122979f96e82] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050360 ~ 1726050382 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050375, value=98.645)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.645, values={1726050375=98.645}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,b3210b6751ac48aa,3c49241a78a24301] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:32.326 INFO 2981023 --- [prometheus-minute-schedule-1,b3210b6751ac48aa,3c49241a78a24301] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:32.386 INFO 2981023 --- [alarm-detect-15,fb9148dc16258834,4de2388be137cb87] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050360 ~ 1726050382 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050375, value=100.976)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=100.976, values={1726050375=100.976}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:32.827 INFO 2981023 --- [ocp-schedules-18,fe2e8f06a6b70073,75a1db60a4acd1ca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:26:33.315 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:33.315 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:33.464 INFO 2981023 --- [ocp-async-3,b790d22df04175c4,f3d231b6d5bddc88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,8ad72874124de267,370a5f3b5cf5d81b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,8ad72874124de267,370a5f3b5cf5d81b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:35.464 INFO 2981023 --- [ocp-async-7,ed4dad7a85558cec,740eaca82b044fe3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:35.785 INFO 2981023 --- [ocp-schedules-9,da4ae867170b0fb8,b26fc8c16a45b95f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:26:35.790 INFO 2981023 --- [ocp-schedules-9,da4ae867170b0fb8,b26fc8c16a45b95f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:26:35.790 INFO 2981023 --- [ocp-schedules-9,da4ae867170b0fb8,b26fc8c16a45b95f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:26:36.315 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:36.316 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:26:36.337710+08:00, lastProcessTime=2024-09-11T18:26:26.337427+08:00 2024-09-11 18:26:36.466 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:26:36.337710+08:00 2024-09-11 18:26:37.314 INFO 2981023 --- [prometheus-cache-monitor1,1e1694fc6776ba41,3abd2d28d7b4c057] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:26:37.315 INFO 2981023 --- [prometheus-cache-monitor1,1e1694fc6776ba41,3abd2d28d7b4c057] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:26:37.464 INFO 2981023 --- [ocp-async-1,15ab96c110e57c0c,4a1555a3cfa1eae8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:37.796 INFO 2981023 --- [ocp-async-3,c9e6085f3309fa00,735c9e8e9fd78dfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:26:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,b3abf079c03fed19,9ae816380b1bc47e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,b3abf079c03fed19,9ae816380b1bc47e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:39.316 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:39.317 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:39.464 INFO 2981023 --- [ocp-async-6,aa5878080c4e742f,56c83b37c5206bba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:40.004 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:26:40.004 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:26:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,64b3b2dfacb70e44,217b1a7e99591b4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,64b3b2dfacb70e44,217b1a7e99591b4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:41.464 INFO 2981023 --- [ocp-async-2,160aaf684531c4e9,2c80ca16fa5cf926] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:42.317 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:42.319 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:42.410 INFO 2981023 --- [ocp-schedules-1,259eb309aae83c39,5dbfc63fa3054f10] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:26:42.459 INFO 2981023 --- [alarm-detect-8,259eb309aae83c39,5f47a0a8cfa3ff32] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050375 ~ 1726050392 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050390, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050390, value=98.704)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.704, values={1726050390=98.704}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050390=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:42.462 WARN 2981023 --- [scanner-13669382-1,259eb309aae83c39,36d0bb3a89bea430] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:26:42.490 INFO 2981023 --- [alarm-detect-8,259eb309aae83c39,e79fb20ca60fbff9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646333, data is empty. 2024-09-11 18:26:42.499 INFO 2981023 --- [alarm-detect-8,259eb309aae83c39,e79fb20ca60fbff9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646333, data is empty. 2024-09-11 18:26:42.509 INFO 2981023 --- [alarm-detect-8,259eb309aae83c39,e79fb20ca60fbff9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646333, data is empty. 2024-09-11 18:26:42.520 INFO 2981023 --- [alarm-detect-15,259eb309aae83c39,621824bf7e4366ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646333, data is empty. 2024-09-11 18:26:42.520 INFO 2981023 --- [alarm-detect-3,259eb309aae83c39,1b57920cd84ee9bd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646333, data is empty. 2024-09-11 18:26:42.525 INFO 2981023 --- [alarm-detect-15,259eb309aae83c39,621824bf7e4366ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646333, data is empty. 2024-09-11 18:26:42.528 INFO 2981023 --- [alarm-detect-3,259eb309aae83c39,1b57920cd84ee9bd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646333, data is empty. 2024-09-11 18:26:42.529 INFO 2981023 --- [alarm-detect-15,259eb309aae83c39,621824bf7e4366ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646333, data is empty. 2024-09-11 18:26:42.532 INFO 2981023 --- [alarm-detect-3,259eb309aae83c39,1b57920cd84ee9bd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646333, data is empty. 2024-09-11 18:26:42.544 INFO 2981023 --- [alarm-detect-10,259eb309aae83c39,3207228693b3d34f] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050375 ~ 1726050392 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050390, value=98.704)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.704, values={1726050390=98.704}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:42.553 INFO 2981023 --- [alarm-detect-7,259eb309aae83c39,a0a1ee80fd03ea77] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050375 ~ 1726050392 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050390, value=98.704)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.704, values={1726050390=98.704}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:42.703 INFO 2981023 --- [alarm-detect-18,259eb309aae83c39,fe680f2f1b5fb9e5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646333, data is empty. 2024-09-11 18:26:42.707 INFO 2981023 --- [alarm-detect-18,259eb309aae83c39,fe680f2f1b5fb9e5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646333, data is empty. 2024-09-11 18:26:42.710 INFO 2981023 --- [alarm-detect-18,259eb309aae83c39,fe680f2f1b5fb9e5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646333, data is empty. 2024-09-11 18:26:42.829 INFO 2981023 --- [ocp-schedules-22,4d92995448441135,14e16c8ebd80e707] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:26:43.464 INFO 2981023 --- [ocp-async-8,8977958ce06c6f41,f3d6fd1a713d9610] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,cd4ed838a1611dea,44b60c123d0a16b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,cd4ed838a1611dea,44b60c123d0a16b2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:45.101 INFO 2981023 --- [prometheus-second-schedule-1,e5b38172ba85ee1d,f068818110e59ba7] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:26:45.319 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:45.320 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:45.464 INFO 2981023 --- [ocp-async-10,eb23ac385ab1561a,55c723be098c6701] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:46.021 INFO 2981023 --- [ocp-schedules-27,023970a2d9532ac9,3f1ad6d82e4f63a4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:26:46.022 INFO 2981023 --- [host-check-clockdiff1,023970a2d9532ac9,265a7c3be40bb697] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:26:46.024 INFO 2981023 --- [host-check-clockdiff2,023970a2d9532ac9,515a5044129b16df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:26:46.024 INFO 2981023 --- [host-check-clockdiff3,023970a2d9532ac9,c403877e16c1e4e8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:26:46.025 INFO 2981023 --- [host-check-clockdiff4,023970a2d9532ac9,c2c4c4553908d461] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:26:46.025 INFO 2981023 --- [ocp-schedules-14,bcf308fb71bc15f6,83bcaf69b895d74a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:26:46.025 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:26:46.025 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:26:46.030 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:26:46.031 INFO 2981023 --- [ocp-schedules-21,c22c15258e31a726,2f9ef9308175d716] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:26:46.044 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:26:46.047 INFO 2981023 --- [host-check-agent-status1,c22c15258e31a726,2342c896a5965a91] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:26:46.048 INFO 2981023 --- [host-check-agent-status2,c22c15258e31a726,e36e92e003130881] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:26:46.054 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:26:46.058 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:26:46.061 INFO 2981023 --- [host-check-agent-status4,c22c15258e31a726,c9a9ca3c93572248] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:26:46.061 INFO 2981023 --- [host-check-agent-status3,c22c15258e31a726,48426a17e7f73a02] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:26:46.052 INFO 2981023 --- [host-check-clockdiff3,023970a2d9532ac9,c403877e16c1e4e8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050406 0 0 , error= 2024-09-11 18:26:46.063 INFO 2981023 --- [host-check-agent-status2,c22c15258e31a726,e36e92e003130881] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:26:46.070 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:26:46.070 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:26:46.070 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:26:46.072 INFO 2981023 --- [host-check-agent-status1,c22c15258e31a726,2342c896a5965a91] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:26:46.073 INFO 2981023 --- [host-check-agent-status1,c22c15258e31a726,2342c896a5965a91] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:26:46.076 INFO 2981023 --- [host-check-agent-status2,c22c15258e31a726,e36e92e003130881] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:26:46.092 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:26:46.094 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:26:46.099 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:26:46.157 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:26:46.177 INFO 2981023 --- [host-check-agent-status3,c22c15258e31a726,48426a17e7f73a02] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:26:46.183 INFO 2981023 --- [ocp-schedules-29,3553f0850c0a2750,9129f0fe3a9e4b6b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:26:46.187 INFO 2981023 --- [host-check-agent-status3,c22c15258e31a726,48426a17e7f73a02] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:26:46.192 INFO 2981023 --- [ocp-schedules-6,2136f05ec87eb6db,100431d941bd7da8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:26:46.192 INFO 2981023 --- [ocp-schedules-10,18110d52cff759d7,84cbe563e67a5366] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:26:46.196 INFO 2981023 --- [ocp-schedules-17,26ee31e285d8685b,93d786a6e64b1d81] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:26:46.196 INFO 2981023 --- [ocp-schedules-13,16762bea3f7b7aae,4ad83d7acfd8227e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:26:46.201 INFO 2981023 --- [ocp-schedules-6,2136f05ec87eb6db,100431d941bd7da8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:46.209 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050363245000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:26:46.255 INFO 2981023 --- [ocp-schedules-17,26ee31e285d8685b,93d786a6e64b1d81] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:46.260 INFO 2981023 --- [ocp-schedules-29,3553f0850c0a2750,9129f0fe3a9e4b6b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:26:46.269 INFO 2981023 --- [ocp-schedules-10,18110d52cff759d7,84cbe563e67a5366] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:46.272 INFO 2981023 --- [host-check-agent-status4,c22c15258e31a726,c9a9ca3c93572248] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:26:46.273 INFO 2981023 --- [ocp-schedules-18,25e685700689fa17,ad631f0d6dd966f7] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 18:26:46.273 INFO 2981023 --- [ocp-schedules-18,25e685700689fa17,ad631f0d6dd966f7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 18:26:46.275 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:26:46.277 INFO 2981023 --- [host-check-agent-status4,c22c15258e31a726,c9a9ca3c93572248] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:26:46.290 INFO 2981023 --- [ocp-schedules-10,18110d52cff759d7,84cbe563e67a5366] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:26:46.293 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:26:46.294 INFO 2981023 --- [host-check-agent-status1,c22c15258e31a726,2342c896a5965a91] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:26:46.294 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:26:46.296 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:26:46.301082611+08:00 2024-09-11 18:26:46.296 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:26:46.295590392+08:00 2024-09-11 18:26:46.311 INFO 2981023 --- [ocp-schedules-26,c2630d5341bd6e09,c9b2cb4245a1ee6d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:26:46.313 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:26:46.296+08:00 2024-09-11 18:26:46.316 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:26:46.296+08:00 2024-09-11 18:26:46.318 INFO 2981023 --- [ocp-schedules-22,62db78e4d2ff3234,0a3485a618d7f11f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 18:26:46.319 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:26:46.296+08:00 and status AVAILABLE 2024-09-11 18:26:46.320 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:26:46.296+08:00 and status AVAILABLE 2024-09-11 18:26:46.323 INFO 2981023 --- [host-check-agent-status1,c22c15258e31a726,2342c896a5965a91] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:26:46.323 INFO 2981023 --- [host-check-agent-status1,c22c15258e31a726,2342c896a5965a91] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:26:46.323 INFO 2981023 --- [host-check-agent-status1,c22c15258e31a726,2342c896a5965a91] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:26:46.333 INFO 2981023 --- [ocp-schedules-22,62db78e4d2ff3234,0a3485a618d7f11f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:46.334 INFO 2981023 --- [host-check-agent-status2,c22c15258e31a726,e36e92e003130881] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:26:46.337 INFO 2981023 --- [host-check-agent-status2,c22c15258e31a726,e36e92e003130881] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:26:46.337 INFO 2981023 --- [host-check-agent-status2,c22c15258e31a726,e36e92e003130881] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:26:46.337 INFO 2981023 --- [host-check-agent-status2,c22c15258e31a726,e36e92e003130881] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:26:46.365 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:26:46.366 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:26:46.374 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:26:46.374005797+08:00 2024-09-11 18:26:46.377 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:26:46.378 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:26:46.378 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:26:46.374+08:00 2024-09-11 18:26:46.380 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:26:46.380082813+08:00 2024-09-11 18:26:46.383 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:26:46.374+08:00 and status AVAILABLE 2024-09-11 18:26:46.384 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:26:46.380+08:00 2024-09-11 18:26:46.390 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:26:46.384013+08:00, lastProcessTime=2024-09-11T18:26:36.337710+08:00 2024-09-11 18:26:46.390 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:26:46.380+08:00 and status AVAILABLE 2024-09-11 18:26:46.398 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.398 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:26:46.399 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:26:46.456 INFO 2981023 --- [host-check-agent-status3,c22c15258e31a726,48426a17e7f73a02] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:26:46.458 INFO 2981023 --- [host-check-agent-status4,c22c15258e31a726,c9a9ca3c93572248] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:26:46.458 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:26:46.460 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.462 INFO 2981023 --- [host-check-agent-status4,c22c15258e31a726,c9a9ca3c93572248] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:26:46.462 INFO 2981023 --- [host-check-agent-status4,c22c15258e31a726,c9a9ca3c93572248] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:26:46.462 INFO 2981023 --- [host-check-agent-status4,c22c15258e31a726,c9a9ca3c93572248] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:26:46.465 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:26:46.466 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.468 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.468 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:46.473 INFO 2981023 --- [host-check-agent-status3,c22c15258e31a726,48426a17e7f73a02] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:26:46.473 INFO 2981023 --- [host-check-agent-status3,c22c15258e31a726,48426a17e7f73a02] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:26:46.473 INFO 2981023 --- [host-check-agent-status3,c22c15258e31a726,48426a17e7f73a02] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:26:46.474 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:26:46.476 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:26:46.476 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:26:46.477 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:26:46.482 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,19607404f761cb8d,5f5bd5be80f99df7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=19607404f761cb8d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 18:26:46.482 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.482 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,19607404f761cb8d,5f5bd5be80f99df7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=19607404f761cb8d, duration=1 ms] 2024-09-11 18:26:46.482 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:26:46.486 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:26:46.491 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.494 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,3c2007aee1b46ce6,cf62b510b574dcd4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=3c2007aee1b46ce6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 18:26:46.494 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,3c2007aee1b46ce6,cf62b510b574dcd4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=3c2007aee1b46ce6, duration=0 ms] 2024-09-11 18:26:46.497 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.498 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:26:46.501 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.503 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:26:46.504 INFO 2981023 --- [ocp-schedules-6,2136f05ec87eb6db,100431d941bd7da8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:26:46.504 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 18:26:46.505 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.505 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:46.510 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.511 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.513 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:26:46.514 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 18:26:46.514 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:46.514 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:26:46.516 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:26:46.524 INFO 2981023 --- [ocp-schedules-22,62db78e4d2ff3234,0a3485a618d7f11f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 18:26:46.524 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:26:46.528 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:26:46.531 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:26:46.540 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:26:46.550 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.550 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.551 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:26:46.569 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:26:46.570 INFO 2981023 --- [pool-10-thread-14,91e7eaf5980a0406,63061244f58f778b] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 18:26:46.573 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:26:46.577 INFO 2981023 --- [pool-10-thread-15,91e7eaf5980a0406,81bbffb5e0506b7d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 18:26:46.583 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 18:26:46.583 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 18:26:46.587 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.590 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:26:46.593 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:26:46.595 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:26:46.596 INFO 2981023 --- [pool-10-thread-16,91e7eaf5980a0406,cc4299e62ce90c8d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 18:26:46.596 INFO 2981023 --- [pool-10-thread-14,91e7eaf5980a0406,63061244f58f778b] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 18:26:46.597 INFO 2981023 --- [pool-10-thread-14,91e7eaf5980a0406,63061244f58f778b] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 18:26:46.600 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:26:46.603 INFO 2981023 --- [ForkJoinPool-6-worker-0,91e7eaf5980a0406,f5911be754567d51] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 18:26:46.603 INFO 2981023 --- [ForkJoinPool-6-worker-0,91e7eaf5980a0406,f5911be754567d51] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 18:26:46.604 INFO 2981023 --- [pool-10-thread-13,91e7eaf5980a0406,9b73e8070811a621] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 18:26:46.604 INFO 2981023 --- [pool-10-thread-13,91e7eaf5980a0406,9b73e8070811a621] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 18:26:46.605 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.607 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:26:46.609 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 18:26:46.610 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 18:26:46.627 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:26:46.629 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:26:46.637 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:26:46.643 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:26:46.646 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.648 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.649 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:46.650 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:26:46.650 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.655 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:26:46.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:26:46.661 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:26:46.661 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050363245000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:26:46.671 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.671 WARN 2981023 --- [ForkJoinPool-6-worker-0,91e7eaf5980a0406,f5911be754567d51] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 18:26:46.672 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:26:46.673 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.674 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:26:46.674 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 18:26:46.680 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:26:46.687 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:26:46.687 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:26:46.711 INFO 2981023 --- [ForkJoinPool-6-worker-4,91e7eaf5980a0406,e483a8cc2b818dc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.713 INFO 2981023 --- [ForkJoinPool-6-worker-4,91e7eaf5980a0406,e483a8cc2b818dc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:26:46.714 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:26:46.719 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.722 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:26:46, 2024-09-11 18:26:46] 2024-09-11 18:26:46.722 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.724 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 18:26:46.724 INFO 2981023 --- [ForkJoinPool-6-worker-9,91e7eaf5980a0406,666af4fa91a2d581] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 18:26:46.724 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:26:46.727 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.728 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:26:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.730 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:26:46.730 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.742 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.743 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:26:46.744 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:26:46.749 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:26:46.753 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.754 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:26:46.762 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:26:46.764 INFO 2981023 --- [ForkJoinPool-6-worker-4,91e7eaf5980a0406,e483a8cc2b818dc7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:26:46.764 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:26:46.764 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:26:46.766 INFO 2981023 --- [ForkJoinPool-6-worker-4,91e7eaf5980a0406,e483a8cc2b818dc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.767 INFO 2981023 --- [ForkJoinPool-6-worker-4,91e7eaf5980a0406,e483a8cc2b818dc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:26:46.768 INFO 2981023 --- [ForkJoinPool-6-worker-4,91e7eaf5980a0406,e483a8cc2b818dc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.770 INFO 2981023 --- [ForkJoinPool-6-worker-4,91e7eaf5980a0406,e483a8cc2b818dc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:26:46.783 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:26:46.789 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.793 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:26:46.793 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.794 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.794 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:26:46.794 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:26:46.797 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.798 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:26:46.799 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:26:46.799 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.799 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.801 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:26:46.802 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:26:46.802 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:26:46.804 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:26:46.805 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.806 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:26:46.808 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:26:46.808 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:26:46.809 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:26:46.814 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:26:46.817 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.817 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.818 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 18:26:46.822 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:26:46.823 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:26:46.823 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:26:46.827 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 18:26:46.829 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:26:46.829 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:26:46.836 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:26:46.850 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.857 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:26:46, 2024-09-11 18:26:46] 2024-09-11 18:26:46.872 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:26:46.873 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.875 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:26:46.885 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:26:46.888 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.890 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.891 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:26:46.897 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.900 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.900 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.901 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:26:46.902 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:26:46.902 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:26:46.903 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.906 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:26:46.906 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.906 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:26:46.906 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.906 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:26:46.909 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:26:46.909 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:26:46.909 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.909 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.909 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.913 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:26:46.913 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:26:46.920 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:26:46.921 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:26:46.922 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:26:46.923 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:26:46, 2024-09-11 18:26:46] 2024-09-11 18:26:46.924 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:26:46.925 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 18:26:46.931 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:26:46.931 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:26:46.931 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 18:26:46.931 INFO 2981023 --- [ForkJoinPool-6-worker-7,91e7eaf5980a0406,0c473e8e2438084d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 18:26:46.937 INFO 2981023 --- [pool-10-thread-15,91e7eaf5980a0406,81bbffb5e0506b7d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 18:26:46.946 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:26:46.949 INFO 2981023 --- [host-check-host-status2,bcf308fb71bc15f6,f6deb74f0e1a2e01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:26:46.952 INFO 2981023 --- [host-check-host-status4,bcf308fb71bc15f6,52326596429cac2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:26:46.955 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:26:46.965 INFO 2981023 --- [host-check-host-status3,bcf308fb71bc15f6,3884e2369e50d93a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:26:46.974 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.975 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:46.975 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:26:46.976 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:26:46.988 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:26:46.989 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:26:46.989 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:26:46.990 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:26:46.994 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:46.995 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:47.000 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:26:47.005 INFO 2981023 --- [host-check-clockdiff2,023970a2d9532ac9,515a5044129b16df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050406 5 5 , error= 2024-09-11 18:26:47.007 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:47.012 INFO 2981023 --- [ForkJoinPool-6-worker-2,91e7eaf5980a0406,d5aa90f30ae188a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:26:47.014 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:47.016 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:47.016 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:47.017 INFO 2981023 --- [pool-10-thread-16,91e7eaf5980a0406,cc4299e62ce90c8d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 18:26:47.019 INFO 2981023 --- [pool-10-thread-13,91e7eaf5980a0406,9b73e8070811a621] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 18:26:47.019 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:26:47.023 INFO 2981023 --- [pool-10-thread-13,91e7eaf5980a0406,9b73e8070811a621] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 18:26:47.024 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:26:47.024 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:47.026 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:26:47.034 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:26:47.037 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:26:47.040 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:26:47.041 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:26:47.042 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:26:47.061 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:26:47.071 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:26:46.384013+08:00 2024-09-11 18:26:47.075 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:26:47.086 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:47.096 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:47.099 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:26:47.104 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:26:47.107 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:47.141 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:26:47.147 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:26:47.154 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:26:47.159 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:26:47.190 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:26:47.196 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:26:47.199 INFO 2981023 --- [ocp-schedules-30,902ad60f33b7419e,082a9496a5b5a253] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:26:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,3cee167ef832010c,0514ed37ed05f800] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:47.317 INFO 2981023 --- [prometheus-minute-schedule-1,3cee167ef832010c,0514ed37ed05f800] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:47.377 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:26:47.383 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:26:47.384 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:47.385 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:26:47.393 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:26:47.393 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:26:47.393 INFO 2981023 --- [ocp-schedules-17,26ee31e285d8685b,93d786a6e64b1d81] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:26:47.398 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:26:47.404 INFO 2981023 --- [host-check-host-status1,bcf308fb71bc15f6,eb93d091aa4fa15b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:26:47.464 INFO 2981023 --- [ocp-async-7,151b698866e9b656,36eefab36c205b7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:47.796 INFO 2981023 --- [ocp-async-5,0fd02663c2234182,21b4afc0e37e11f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:26:48.095 INFO 2981023 --- [ocp-schedules-11,b0a2f970bdc4d2c6,a00a6e5d32bc9580] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:26:48.113 WARN 2981023 --- [ocp-schedules-11,b0a2f970bdc4d2c6,a00a6e5d32bc9580] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:26:48.320 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:48.321 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:49.464 INFO 2981023 --- [ocp-async-8,d22fe0e535e1cbc1,e1af3aee95503556] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:26:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:26:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,c50a123dd58f51d9,3985e00fb0518905] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,c50a123dd58f51d9,3985e00fb0518905] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:50.362 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:26:50.366 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.367 INFO 2981023 --- [ocp-schedules-15,4b5732e5fa427e23,b5e011a3ed2fe957] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:26:50.371 INFO 2981023 --- [ocp-schedules-8,c8678ed374fd28f7,d221d8edb9661981] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:26:51.321 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:51.323 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:51.440 INFO 2981023 --- [host-check-clockdiff1,023970a2d9532ac9,265a7c3be40bb697] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050411 0 0 , error= 2024-09-11 18:26:51.465 INFO 2981023 --- [ocp-async-6,52d57b939152d5d7,3e89a811f5147833] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:51.642 INFO 2981023 --- [ocp-schedules-21,a8e5dbe77c8e66f5,408b7176917ab803] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:26:51.644 INFO 2981023 --- [ocp-schedules-21,a8e5dbe77c8e66f5,408b7176917ab803] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:26:51.644 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:26:51.644 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:26:51.644 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:26:51.644 INFO 2981023 --- [ocp-schedules-21,a8e5dbe77c8e66f5,408b7176917ab803] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:26:52.014 INFO 2981023 --- [host-check-clockdiff4,023970a2d9532ac9,c2c4c4553908d461] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050412 0 0 , error= 2024-09-11 18:26:52.757 INFO 2981023 --- [ocp-schedules-24,8cc549013c58d3c4,ce38551a0a095402] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:26:52.809 WARN 2981023 --- [scanner-13669733-1,8cc549013c58d3c4,b58d5bd68cf91a39] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:26:52.832 INFO 2981023 --- [ocp-schedules-14,d1f9ceb298e253f0,e4005018c47b2115] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:26:52.848 INFO 2981023 --- [alarm-detect-10,8cc549013c58d3c4,ca8db442827fefc6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050375 ~ 1726050402 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050390, value=9.545)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=9.545, values={1726050390=9.545}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:52.848 INFO 2981023 --- [alarm-detect-10,8cc549013c58d3c4,ca8db442827fefc6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050375 ~ 1726050402 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050390, value=9.545)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=9.545, values={1726050390=9.545}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:26:52.950 INFO 2981023 --- [alarm-detect-15,8cc549013c58d3c4,39eb9f0fc6235bb6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050375 ~ 1726050402 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050390, value=100.966)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=100.966, values={1726050390=100.966}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:26:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,767fef6eb566262d,6dd6ea5238075f93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,767fef6eb566262d,6dd6ea5238075f93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:53.382 INFO 2981023 --- [ocp-schedules-9,8dc31c2076db3243,411b8c2bd2038f73] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:26:53.382 INFO 2981023 --- [ocp-schedules-9,8dc31c2076db3243,411b8c2bd2038f73] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:26:53.465 INFO 2981023 --- [ocp-async-2,bc7957fdba37c469,f58a8f46ae01b4d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:54.323 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:54.323 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:54.506 INFO 2981023 --- [ocp-schedules-17,f407490866437bee,c1cd940e51b4ab53] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:26:55.404 INFO 2981023 --- [ocp-schedules-29,6155c0aa85a2776d,6cfa30af5336e7a7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:26:55.423 WARN 2981023 --- [ocp-schedules-29,6155c0aa85a2776d,6cfa30af5336e7a7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:26:55.465 INFO 2981023 --- [ocp-async-1,afa055d55b65ba44,f024ddf92b9ef47a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:55.793 INFO 2981023 --- [ocp-schedules-20,b09f77235e35b5bf,4cfdc0155b4bd89c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:26:55.801 INFO 2981023 --- [ocp-schedules-20,b09f77235e35b5bf,4cfdc0155b4bd89c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:26:55.801 INFO 2981023 --- [ocp-schedules-20,b09f77235e35b5bf,4cfdc0155b4bd89c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:26:55.915 INFO 2981023 --- [ocp-schedules-30,1006d7a9a11ec076,d01cc38bae9c3172] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:26:55.915 INFO 2981023 --- [ocp-schedules-30,1006d7a9a11ec076,d01cc38bae9c3172] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:26:55.917 INFO 2981023 --- [ocp-schedules-30,1006d7a9a11ec076,d01cc38bae9c3172] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:26:55.917 INFO 2981023 --- [ocp-schedules-30,1006d7a9a11ec076,d01cc38bae9c3172] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:26:55.918 INFO 2981023 --- [ocp-schedules-30,1006d7a9a11ec076,d01cc38bae9c3172] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:26:55.918 INFO 2981023 --- [ocp-schedules-30,1006d7a9a11ec076,d01cc38bae9c3172] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:26:56.141 INFO 2981023 --- [ocp-schedules-17,52c43017a8f64017,5bd3436e69346ffb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:26:56.148 INFO 2981023 --- [ocp-schedules-17,52c43017a8f64017,5bd3436e69346ffb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:26:56.148 INFO 2981023 --- [ocp-schedules-17,52c43017a8f64017,5bd3436e69346ffb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:26:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,adffa098e5853803,9267cfec98997365] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,adffa098e5853803,9267cfec98997365] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:26:56.351631+08:00, lastProcessTime=2024-09-11T18:26:46.384013+08:00 2024-09-11 18:26:56.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:26:56.351631+08:00 2024-09-11 18:26:57.323 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:26:57.326 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:26:57.465 INFO 2981023 --- [ocp-async-3,cb71590b527afaa1,d070dcb77533b205] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:26:57.796 INFO 2981023 --- [ocp-async-4,1be31a4000ed52c8,45c2dacf5fad3217] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:26:58.799 INFO 2981023 --- [ocp-schedules-3,afe6d814016feb85,f900711a8dcfadad] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:26:58.799 INFO 2981023 --- [ocp-schedules-3,afe6d814016feb85,f900711a8dcfadad] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:26:58.832 INFO 2981023 --- [Collect all tenants major compactions2,afe6d814016feb85,44d34ca7c5092294] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:26:58.832 INFO 2981023 --- [Collect all tenants major compactions1,afe6d814016feb85,a43ac8bbb5977154] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:26:58.843 INFO 2981023 --- [Collect all tenants major compactions5,afe6d814016feb85,cac5b07b90254b62] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:26:58.843 INFO 2981023 --- [Collect all tenants major compactions4,afe6d814016feb85,e6283f76bb5ca7d6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:26:58.843 INFO 2981023 --- [Collect all tenants major compactions3,afe6d814016feb85,7ae57bee75fdadf7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:26:58.846 INFO 2981023 --- [Collect all tenants major compactions6,afe6d814016feb85,009c2df832503811] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:26:58.858 INFO 2981023 --- [Collect all tenants major compactions2,afe6d814016feb85,44d34ca7c5092294] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:58.858 INFO 2981023 --- [Collect all tenants major compactions1,afe6d814016feb85,a43ac8bbb5977154] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:58.859 INFO 2981023 --- [Collect all tenants major compactions1,afe6d814016feb85,a43ac8bbb5977154] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:26:58.860 INFO 2981023 --- [Collect all tenants major compactions6,afe6d814016feb85,009c2df832503811] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:58.861 INFO 2981023 --- [Collect all tenants major compactions2,afe6d814016feb85,44d34ca7c5092294] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:26:58.861 INFO 2981023 --- [Collect all tenants major compactions6,afe6d814016feb85,009c2df832503811] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:26:58.862 INFO 2981023 --- [Collect all tenants major compactions1,afe6d814016feb85,a43ac8bbb5977154] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:26:58.862 INFO 2981023 --- [Collect all tenants major compactions3,afe6d814016feb85,7ae57bee75fdadf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:58.862 INFO 2981023 --- [Collect all tenants major compactions1,afe6d814016feb85,a43ac8bbb5977154] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:26:58.863 INFO 2981023 --- [Collect all tenants major compactions2,afe6d814016feb85,44d34ca7c5092294] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:26:58.863 INFO 2981023 --- [Collect all tenants major compactions2,afe6d814016feb85,44d34ca7c5092294] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:26:58.863 INFO 2981023 --- [Collect all tenants major compactions3,afe6d814016feb85,7ae57bee75fdadf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:26:58.863 INFO 2981023 --- [Collect all tenants major compactions6,afe6d814016feb85,009c2df832503811] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:26:58.863 INFO 2981023 --- [Collect all tenants major compactions6,afe6d814016feb85,009c2df832503811] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:26:58.865 INFO 2981023 --- [Collect all tenants major compactions3,afe6d814016feb85,7ae57bee75fdadf7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:26:58.865 INFO 2981023 --- [Collect all tenants major compactions3,afe6d814016feb85,7ae57bee75fdadf7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:26:58.879 INFO 2981023 --- [Collect all tenants major compactions5,afe6d814016feb85,cac5b07b90254b62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:58.886 INFO 2981023 --- [Collect all tenants major compactions4,afe6d814016feb85,e6283f76bb5ca7d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:26:58.887 INFO 2981023 --- [Collect all tenants major compactions5,afe6d814016feb85,cac5b07b90254b62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:26:58.887 INFO 2981023 --- [Collect all tenants major compactions4,afe6d814016feb85,e6283f76bb5ca7d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:26:58.890 INFO 2981023 --- [Collect all tenants major compactions5,afe6d814016feb85,cac5b07b90254b62] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:26:58.890 INFO 2981023 --- [Collect all tenants major compactions5,afe6d814016feb85,cac5b07b90254b62] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:26:58.890 INFO 2981023 --- [Collect all tenants major compactions4,afe6d814016feb85,e6283f76bb5ca7d6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:26:58.890 INFO 2981023 --- [Collect all tenants major compactions4,afe6d814016feb85,e6283f76bb5ca7d6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:26:58.891 INFO 2981023 --- [ocp-schedules-3,afe6d814016feb85,f900711a8dcfadad] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:26:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,62020ae2e518bd80,65ade95ac5173491] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050420, interval=60 2024-09-11 18:26:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,62020ae2e518bd80,65ade95ac5173491] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:26:59.465 INFO 2981023 --- [ocp-async-2,de8e5631993684c9,616aff40dc6a43bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:00.326 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:00.327 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:00.673 INFO 2981023 --- [ocp-schedules-15,630e144bc84eebe8,17be98b574e29def] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:27:00.673+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:27:00.683 INFO 2981023 --- [ocp-schedules-15,630e144bc84eebe8,17be98b574e29def] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:27:00.812 INFO 2981023 --- [ocp-schedules-15,630e144bc84eebe8,17be98b574e29def] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 54 to SqlAggregateData 2024-09-11 18:27:00.813 INFO 2981023 --- [ocp-schedules-15,630e144bc84eebe8,17be98b574e29def] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:27:01.465 INFO 2981023 --- [ocp-async-1,5b9e01d08296a278,6f9b7eb94721df02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,de33984bfccdccb3,c0691df65fb61dd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:02.317 INFO 2981023 --- [prometheus-minute-schedule-1,de33984bfccdccb3,c0691df65fb61dd7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:02.836 INFO 2981023 --- [ocp-schedules-22,e0dc240a2e7ffb6b,c4294fb81a472451] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:27:02.849 INFO 2981023 --- [ocp-schedules-3,1d030011acc2ddbd,3982e0fb71378a32] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:27:02.853 INFO 2981023 --- [ocp-schedules-3,1d030011acc2ddbd,3982e0fb71378a32] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:27:02.884 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:27:02.885 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.885 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:27:02.885 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:27:02.885 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.888 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:27:02.889 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:27:02.892 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.892 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:02.893 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.893 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:02.896 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:02.902 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:27:02.904 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.904 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:27:02.906 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.906 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:02.910 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.911 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:27:02.915 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.916 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:27:02.918 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.919 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:02.921 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.922 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:27:02.927 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.927 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.928 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:27:02.929 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.929 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:27:02.930 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.930 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:27:02.931 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:27:02.934 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.934 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:27:02.935 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:27:02.935 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:27:02.936 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.937 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:27:02.937 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.938 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:27:02.938 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.939 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:27:02.939 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.940 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:27:02.941 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.941 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:27:02.942 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.942 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:27:02.943 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.943 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:27:02.943 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:27:02.945 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:27:02.948 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:02.950 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.950 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:02.955 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:02.956 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.956 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:27:02.960 INFO 2981023 --- [ocp-schedules-4,ad99eeb2e4d9bbd3,10799f73a5e3a0f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:27:02.961 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:02.962 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:27:02.963 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:27:02.970 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:27:02.971 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:02.971 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:27:02.974 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:27:02.974 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:27:02.974 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:27:03.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:27:03.020 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:27:03.021 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.021 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:03.028 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:27:03.032 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:27:03.032 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:27:03.032 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:27:03.033 WARN 2981023 --- [scanner-13669917-1,ad99eeb2e4d9bbd3,47c290af5359df06] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:27:03.042 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.055 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:03.060 INFO 2981023 --- [alarm-detect-1,ad99eeb2e4d9bbd3,0d7afcd9ae927e14] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050390 ~ 1726050412 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050405, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050405, value=97.825)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=97.825, values={1726050405=97.825}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050405=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:03.073 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.079 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:03.082 INFO 2981023 --- [alarm-detect-10,ad99eeb2e4d9bbd3,3bfa9243ff39b460] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646333, data is empty. 2024-09-11 18:27:03.087 INFO 2981023 --- [alarm-detect-10,ad99eeb2e4d9bbd3,3bfa9243ff39b460] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646387, data is empty. 2024-09-11 18:27:03.093 INFO 2981023 --- [alarm-detect-10,ad99eeb2e4d9bbd3,3bfa9243ff39b460] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646387, data is empty. 2024-09-11 18:27:03.106 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:27:03.107 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6611D6E8-0-0] 2024-09-11 18:27:03.107 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:27:03.108 INFO 2981023 --- [alarm-detect-12,ad99eeb2e4d9bbd3,493cda710ce34559] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646333, data is empty. 2024-09-11 18:27:03.109 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.111 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:27:03.112 INFO 2981023 --- [alarm-detect-5,ad99eeb2e4d9bbd3,1a1995214480bc35] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646333, data is empty. 2024-09-11 18:27:03.115 INFO 2981023 --- [alarm-detect-12,ad99eeb2e4d9bbd3,493cda710ce34559] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646387, data is empty. 2024-09-11 18:27:03.122 INFO 2981023 --- [alarm-detect-5,ad99eeb2e4d9bbd3,1a1995214480bc35] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646387, data is empty. 2024-09-11 18:27:03.127 INFO 2981023 --- [alarm-detect-12,ad99eeb2e4d9bbd3,493cda710ce34559] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646387, data is empty. 2024-09-11 18:27:03.129 INFO 2981023 --- [alarm-detect-5,ad99eeb2e4d9bbd3,1a1995214480bc35] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646387, data is empty. 2024-09-11 18:27:03.136 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.144 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:27:03.167 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.168 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:27:03.168 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:27:03.188 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6611D6F2-0-0] 2024-09-11 18:27:03.188 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:27:03.190 INFO 2981023 --- [alarm-detect-2,ad99eeb2e4d9bbd3,b56857f57324f73b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050390 ~ 1726050413 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050405, value=8.838)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.838, values={1726050405=8.838}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:03.190 INFO 2981023 --- [alarm-detect-2,ad99eeb2e4d9bbd3,b56857f57324f73b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050390 ~ 1726050413 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050405, value=8.838)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.838, values={1726050405=8.838}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:27:03.194 INFO 2981023 --- [alarm-detect-3,ad99eeb2e4d9bbd3,bb93306b46fd169d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050390 ~ 1726050413 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050405, value=97.825)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=97.825, values={1726050405=97.825}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:03.244 INFO 2981023 --- [alarm-detect-2,ad99eeb2e4d9bbd3,f88c7da3d6404e43] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646333, data is empty. 2024-09-11 18:27:03.249 INFO 2981023 --- [alarm-detect-2,ad99eeb2e4d9bbd3,f88c7da3d6404e43] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646387, data is empty. 2024-09-11 18:27:03.253 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.255 INFO 2981023 --- [alarm-detect-2,ad99eeb2e4d9bbd3,f88c7da3d6404e43] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646387, data is empty. 2024-09-11 18:27:03.271 INFO 2981023 --- [alarm-detect-1,ad99eeb2e4d9bbd3,f0633125e20af62d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050390 ~ 1726050413 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050405, value=104.852)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.852, values={1726050405=104.852}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:03.283 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:27:03.284 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.285 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:27:03.286 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.287 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:27:03.290 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.291 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:27:03.291 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.293 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:27:03.293 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.294 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:27:03.294 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:27:03.297 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:27:03.308 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:03.309 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.310 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:27:03.320 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:27:03.321 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:27:03.322 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:03.322 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:27:03.327 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:03.328 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:03.331 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:27:03.333 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:27:03.333 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:27:03.333 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:27:03.333 INFO 2981023 --- [ocp-schedules-3,1d030011acc2ddbd,3982e0fb71378a32] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:27:03.465 INFO 2981023 --- [ocp-async-6,253f13bd841cd204,75b1b75209f29376] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,1130f5d98cd26ad1,7ad550c28ee8820c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,1130f5d98cd26ad1,7ad550c28ee8820c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:05.465 INFO 2981023 --- [ocp-async-7,74934bb9d6acd4cb,f02ed603f500c769] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:06.328 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:06.330 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:06.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:27:06.351594+08:00, lastProcessTime=2024-09-11T18:26:56.351631+08:00 2024-09-11 18:27:06.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:27:06.351594+08:00 2024-09-11 18:27:07.466 INFO 2981023 --- [ocp-async-4,c85b7521d718bc18,baeb68ffea3599ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:07.797 INFO 2981023 --- [ocp-async-6,4b54ee2f44671d80,abc146da0881c0ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:27:08.209 INFO 2981023 --- [ocp-schedules-24,0c788406cc8326cb,0c892c5a32c8ec42] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:27:08.280 INFO 2981023 --- [ocp-schedules-11,80280319674fe4cd,b4dea65459cf7989] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:27:08.280+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:27:08.281 INFO 2981023 --- [ocp-schedules-11,80280319674fe4cd,b4dea65459cf7989] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:27:08.283 INFO 2981023 --- [ocp-schedules-11,80280319674fe4cd,b4dea65459cf7989] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:27:08.283 INFO 2981023 --- [ocp-schedules-11,80280319674fe4cd,b4dea65459cf7989] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:27:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,a2bd22eee10499aa,0ce23d7f3d6a75ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,a2bd22eee10499aa,0ce23d7f3d6a75ec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:09.330 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:09.331 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:09.466 INFO 2981023 --- [ocp-async-3,afca48e26b69c0c2,d4fac9bc1a0e2f2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:10.003 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:27:10.003 INFO 2981023 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:27:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:27:10.922 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:27:11.215 INFO 2981023 --- [ocp-schedules-15,4a1c4c94c5ebd9db,396ea1a6f1da441e] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:27:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,533b6c8445d07949,d6a85b1bc2690b0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:11.324 INFO 2981023 --- [prometheus-minute-schedule-1,533b6c8445d07949,d6a85b1bc2690b0a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:11.466 INFO 2981023 --- [ocp-async-10,91fa786d198c5ad1,9f6f8b84d19eb6a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:12.331 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:12.332 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:12.839 INFO 2981023 --- [ocp-schedules-17,691bea42c3486577,780088de44c8e1c5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:27:12.895 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:23:12.399, end=2024-09-11T18:25:12.892} 2024-09-11 18:27:12.901 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, 2024-09-11 18:27:12.901 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:27:12.902 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:23:12.399, end=2024-09-11T18:25:12.892},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:27:12.902 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:27:12.902 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:27:12.903 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:27:12.903 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:23:12.399, end=2024-09-11T18:25:12.892},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:27:12.903 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:27:12.903 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:27:12.904 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:27:12.906 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:23:12.399, end=2024-09-11T18:25:12.892},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:27:12.906 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:27:12.906 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:27:12.964 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964032964000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=8A1B851989150B802AEEACA99152D145, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:12.964Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:12.964 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964032964000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:12.964Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:12.967 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964032967000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:12.967Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:12.967 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964032967000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:12.967Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:12.972 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964032972000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:12.972Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:12.975 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 30 2024-09-11 18:27:12.975 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:12.977 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964032977000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:12.977Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:12.978 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964032978000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:12.978Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:12.984 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964032984000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:12.984Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:12.988 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:12.988 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:27:12.988 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:12.988 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:12.991 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 70 2024-09-11 18:27:12.991 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:12.992 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:12.992 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:12.996 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:12.996 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:12.998 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:12.998 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:12.998 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:27:12.998 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.001 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.001 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.005 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.005 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.013 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:27:13.013 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.013 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:13.013 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.016 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.016 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.017 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.017 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.018 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.018 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.018 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:13.018 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.018 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.018 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.020 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.020 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.030 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.030 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.030 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.030 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.030 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.030 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.032 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.032 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.032 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.032 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.033 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.033 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.034 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.034 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.034 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.034 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.035 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.035 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.035 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.035 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.035 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.036 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.036 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.036 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.038 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.038 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.042 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.042 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.043 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.046 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.047 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:12.972, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.048 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:12.967, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.048 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:12.967, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.049 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:12.964, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=8A1B851989150B802AEEACA99152D145, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.049 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.049 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.050 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:12.977, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.051 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:12.964, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.052 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.058 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:12.978, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.064 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.064 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.068 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.079 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:12.984, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.084 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964033084000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:13.084Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:13.087 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964033087000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:13.087Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:13.088 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964033088000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:13.088Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:13.090 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964033090000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:13.090Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:13.093 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964033093000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:13.093Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:13.094 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964033094000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:13.094Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:13.098 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:13.098 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.098 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964033098000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:13.098Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:13.103 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:13.103 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.104 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:13.104 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.109 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:13.109 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.109 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.109 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.110 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.110 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.112 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:13.112 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.116 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.116 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.117 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:13.117 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.117 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.117 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.118 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:27:13.118 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.119 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.119 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.123 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.123 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.125 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.125 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.127 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.127 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.128 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.128 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.129 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.129 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.130 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.131 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.131 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.131 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.131 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.131 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.131 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.131 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.131 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.133 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.134 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.134 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.134 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.134 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.137 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:13.084, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.137 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.137 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.139 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.139 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.140 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:13.087, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.142 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.142 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.143 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.143 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.143 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.143 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.145 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.146 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.147 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:13.09, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.148 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:13.088, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.148 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.148 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.150 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:27:13.150 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:13.093, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.150 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:13.098, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.152 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:13.094, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.153 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:27:13.154 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:23:12.399, end=2024-09-11T18:25:12.892},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:27:13.154 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:27:13.154 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:27:13.162 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:27:13.175 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964033175000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:27:13.175Z, endTime=2024-09-11T10:25:12.892Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:27:13.177 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:27:13.177 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:27:13.183 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.183 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.184 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:27:13.184 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:27:13.190 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:27:13.190 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050312892000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:27:13.192 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:27:13.192 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:27:13.175, end=2024-09-11T18:25:12.892}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:27:13.192 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:27:13.194 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:23:12.399, end=2024-09-11T18:25:12.892},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:27:13.194 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:27:13.194 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:27:13.207 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:27:13.208 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:27:13.208 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:23:12.399, end=2024-09-11T18:25:12.892},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:27:13.208 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:27:13.208 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:27:13.209 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:27:13.303 INFO 2981023 --- [ocp-schedules-25,5b72ab5d04b100a4,4a949805f6af441b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:27:13.354 INFO 2981023 --- [alarm-detect-11,5b72ab5d04b100a4,faf7948e3c3d3fe8] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050405 ~ 1726050423 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050420, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050420, value=97.398)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=97.398, values={1726050420=97.398}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050420=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:13.362 WARN 2981023 --- [scanner-13671756-1,5b72ab5d04b100a4,31735338957abdef] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:27:13.414 INFO 2981023 --- [alarm-detect-9,5b72ab5d04b100a4,aec01fd1d76df0f9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050405 ~ 1726050423 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050420, value=97.398)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=97.398, values={1726050420=97.398}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:13.466 INFO 2981023 --- [ocp-async-8,499a4b8780629ace,2bb133ffc18c73a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:13.592 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 18:27:13.611 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:13.614 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:13.670 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:27:13.671 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:27:13.682 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:13.683 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:13.693 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:27:13.693 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:27:13.704 INFO 2981023 --- [ocp-schedules-3,f5c104e531d28ed3,1b7e6ca388857332] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2024-09-11 18:27:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,5c709aadd70c894b,82b9cbd0f42b0281] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,467a6cbd9f70e042,233cab05a044dbad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:27:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,5c709aadd70c894b,82b9cbd0f42b0281] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:14.654 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050360, nodeCount=49, affectRows=49 2024-09-11 18:27:14.654 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050360, nodeCount=47, validCount=47 2024-09-11 18:27:14.716 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050360, nodeCount=1000, affectRows=1000 2024-09-11 18:27:14.719 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050360, nodeCount=999, validCount=999 2024-09-11 18:27:14.733 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050360, nodeCount=1000, affectRows=1000 2024-09-11 18:27:14.735 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050360, nodeCount=1000, validCount=1000 2024-09-11 18:27:15.332 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:15.334 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:15.466 INFO 2981023 --- [ocp-async-1,9910949f27a9bb10,3f376afac4a9c9a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:15.804 INFO 2981023 --- [ocp-schedules-4,b6d8773275c10876,5fc590af8045245a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:27:15.813 INFO 2981023 --- [ocp-schedules-4,b6d8773275c10876,5fc590af8045245a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:27:15.814 INFO 2981023 --- [ocp-schedules-4,b6d8773275c10876,5fc590af8045245a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:27:16.024 INFO 2981023 --- [ocp-schedules-8,bb5a8fb259fa1b72,6741b96fcbfe1661] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:27:16.024 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:27:16.024 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:27:16.025 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:27:16.025 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:27:16.025 INFO 2981023 --- [ocp-schedules-1,6a624c2abdb937a7,1b97f3fb8ef79cae] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:27:16.025 INFO 2981023 --- [host-check-clockdiff1,6a624c2abdb937a7,2eddb53fdc7072ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:27:16.026 INFO 2981023 --- [host-check-clockdiff2,6a624c2abdb937a7,e62a9400514ee06a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:27:16.026 INFO 2981023 --- [host-check-clockdiff4,6a624c2abdb937a7,23de4460cb85dcf1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:27:16.026 INFO 2981023 --- [host-check-clockdiff3,6a624c2abdb937a7,14b56bc4a3a659f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:27:16.050 INFO 2981023 --- [ocp-schedules-2,b6c4abd270c3aa07,0e62a13eee429a7a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:27:16.050 INFO 2981023 --- [host-check-agent-status1,b6c4abd270c3aa07,7ca529b099312b44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:27:16.050 INFO 2981023 --- [host-check-agent-status2,b6c4abd270c3aa07,5e2c227b4018a43b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:27:16.050 INFO 2981023 --- [host-check-agent-status3,b6c4abd270c3aa07,9b0d699cd9f878ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:27:16.052 INFO 2981023 --- [host-check-clockdiff3,6a624c2abdb937a7,14b56bc4a3a659f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050436 0 0 , error= 2024-09-11 18:27:16.052 INFO 2981023 --- [host-check-agent-status4,b6c4abd270c3aa07,6e56fd97f399c115] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:27:16.052 INFO 2981023 --- [host-check-clockdiff2,6a624c2abdb937a7,e62a9400514ee06a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050436 5 5 , error= 2024-09-11 18:27:16.062 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:27:16.063 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:27:16.064 INFO 2981023 --- [host-check-agent-status3,b6c4abd270c3aa07,9b0d699cd9f878ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:16.065 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:16.066 INFO 2981023 --- [host-check-agent-status3,b6c4abd270c3aa07,9b0d699cd9f878ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:16.066 INFO 2981023 --- [host-check-agent-status1,b6c4abd270c3aa07,7ca529b099312b44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:16.067 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:16.068 INFO 2981023 --- [host-check-agent-status1,b6c4abd270c3aa07,7ca529b099312b44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:16.071 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:27:16.071 INFO 2981023 --- [host-check-agent-status2,b6c4abd270c3aa07,5e2c227b4018a43b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:27:16.072 INFO 2981023 --- [host-check-agent-status2,b6c4abd270c3aa07,5e2c227b4018a43b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:27:16.072 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:27:16.074 INFO 2981023 --- [host-check-agent-status4,b6c4abd270c3aa07,6e56fd97f399c115] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:27:16.075 INFO 2981023 --- [host-check-agent-status4,b6c4abd270c3aa07,6e56fd97f399c115] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:27:16.075 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:16.078 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:16.107 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:27:16.110 INFO 2981023 --- [host-check-agent-status1,b6c4abd270c3aa07,7ca529b099312b44] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:27:16.110 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:27:16.109626755+08:00 2024-09-11 18:27:16.111 INFO 2981023 --- [host-check-agent-status2,b6c4abd270c3aa07,5e2c227b4018a43b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:27:16.111 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:27:16.110+08:00 2024-09-11 18:27:16.114 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:27:16.110+08:00 and status AVAILABLE 2024-09-11 18:27:16.114 INFO 2981023 --- [host-check-agent-status3,b6c4abd270c3aa07,9b0d699cd9f878ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:27:16.115 INFO 2981023 --- [host-check-agent-status2,b6c4abd270c3aa07,5e2c227b4018a43b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:27:16.115 INFO 2981023 --- [host-check-agent-status2,b6c4abd270c3aa07,5e2c227b4018a43b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:27:16.115 INFO 2981023 --- [host-check-agent-status2,b6c4abd270c3aa07,5e2c227b4018a43b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:27:16.121 INFO 2981023 --- [host-check-agent-status1,b6c4abd270c3aa07,7ca529b099312b44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:27:16.121 INFO 2981023 --- [host-check-agent-status1,b6c4abd270c3aa07,7ca529b099312b44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:27:16.121 INFO 2981023 --- [host-check-agent-status1,b6c4abd270c3aa07,7ca529b099312b44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:27:16.122 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:27:16.124 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:27:16.126 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:27:16.126109524+08:00 2024-09-11 18:27:16.126 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:27:16.126+08:00 2024-09-11 18:27:16.128 INFO 2981023 --- [host-check-agent-status3,b6c4abd270c3aa07,9b0d699cd9f878ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:27:16.128 INFO 2981023 --- [host-check-agent-status3,b6c4abd270c3aa07,9b0d699cd9f878ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:27:16.128 INFO 2981023 --- [host-check-agent-status3,b6c4abd270c3aa07,9b0d699cd9f878ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:27:16.135 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:27:16.141 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:27:16.142 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:27:16.126+08:00 and status AVAILABLE 2024-09-11 18:27:16.142 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:27:16.142204444+08:00 2024-09-11 18:27:16.143 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:27:16.142+08:00 2024-09-11 18:27:16.143 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:27:16.148268323+08:00 2024-09-11 18:27:16.145 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:27:16.142+08:00 and status AVAILABLE 2024-09-11 18:27:16.147 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:27:16.150 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:27:16.143+08:00 2024-09-11 18:27:16.151 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:27:16.152 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:27:16.143+08:00 and status AVAILABLE 2024-09-11 18:27:16.155 INFO 2981023 --- [host-check-agent-status4,b6c4abd270c3aa07,6e56fd97f399c115] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:27:16.166 INFO 2981023 --- [host-check-agent-status4,b6c4abd270c3aa07,6e56fd97f399c115] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:27:16.166 INFO 2981023 --- [host-check-agent-status4,b6c4abd270c3aa07,6e56fd97f399c115] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:27:16.166 INFO 2981023 --- [host-check-agent-status4,b6c4abd270c3aa07,6e56fd97f399c115] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:27:16.167 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:27:16.169 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:27:16.179 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:27:16.185 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:27:16.186 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:27:16.187 INFO 2981023 --- [ocp-schedules-30,cd9db55223813ca6,4423c914e4308c1b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:27:16.189 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:27:16.192 INFO 2981023 --- [ocp-schedules-20,ac32b162f62f4673,68cf4cf680cde2f2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:27:16.192 INFO 2981023 --- [ocp-schedules-23,498e7ac05d834347,c69ed1698d20dabb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:27:16.192 INFO 2981023 --- [ocp-schedules-21,1b616be8d2f28311,c56e224a63719fab] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:27:16.194 INFO 2981023 --- [ocp-schedules-30,cd9db55223813ca6,4423c914e4308c1b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:27:16.195 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:27:16.204 INFO 2981023 --- [ocp-schedules-23,498e7ac05d834347,c69ed1698d20dabb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:27:16.206 INFO 2981023 --- [ocp-schedules-20,ac32b162f62f4673,68cf4cf680cde2f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:27:16.206 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:27:16.229 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:16.229 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:16.232 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:27:16.235 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050363245000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:27:16.239 INFO 2981023 --- [ocp-schedules-28,86934a4bfa9f7cdb,6e5632b296b72a46] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:27:16.240 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:16.240 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:16.263 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:27:16.266 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:27:16.274 INFO 2981023 --- [ocp-schedules-20,ac32b162f62f4673,68cf4cf680cde2f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:27:16.283 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:27:16.297 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:27:16.302 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:27:16.303 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:27:16.304 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:27:16.327 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.327 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:27:16.329 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.329 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:27:16.329 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:16.330 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.330 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:16.338 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:27:16.342 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:27:16.343 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.343 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:16.356 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.359 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:27:16.362 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:27:16.360329+08:00, lastProcessTime=2024-09-11T18:27:06.351594+08:00 2024-09-11 18:27:16.362 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:27:16.365 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:27:16.365 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:27:16.366 INFO 2981023 --- [ocp-schedules-30,cd9db55223813ca6,4423c914e4308c1b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:27:16.373 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:27:16.375 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:27:16.376 INFO 2981023 --- [host-check-host-status4,bb5a8fb259fa1b72,d6e3013bb694f8a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:27:16.379 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:16.380 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.381 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:27:16.383 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:27:16.404 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:27:16.404 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050363245000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:27:16.405 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:27:16.407 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:27:16.416 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050360, nodeCount=480, affectRows=480 2024-09-11 18:27:16.423 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.424 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:16.425 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:27:16.425 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.427 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:27:16.428 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:27:16.437 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:27:16.439 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:27:16.444 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:27:16.444 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:27:16.449 INFO 2981023 --- [host-check-host-status2,bb5a8fb259fa1b72,86a7920f8c140bba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:27:16.450 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:27:16.463 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:27:16.472 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:27:16.481 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:16.482 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.484 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:27:16.485 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:27:16.490 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:27:16.494 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:16.496 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:16.506 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:16.507 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.508 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:27:16.510 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:27:16.515 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:27:16.517 INFO 2981023 --- [host-check-host-status3,bb5a8fb259fa1b72,2bb1e2ddfc1ee4db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:27:16.539 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:16.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.541 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:16.542 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:27:16.542 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.545 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:27:16.546 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:27:16.557 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:27:16.557 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:27:16.568 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:27:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:27:16.575 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:27:16.587 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:27:16.587 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:16.589 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.589 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:27:16.594 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:27:16.597 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:16.597 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:27:16.597 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.598 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:27:16.598 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:27:16.602 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:16.603 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:16.606 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:27:16.606 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:27:16.617 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:27:16.627 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:27:16.360329+08:00 2024-09-11 18:27:16.823 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:27:16.829 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:27:16.841 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:27:16.868 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:27:16.874 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:16.875 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:16.876 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:27:16.881 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:27:16.881 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:27:16.881 INFO 2981023 --- [ocp-schedules-23,498e7ac05d834347,c69ed1698d20dabb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:27:17.019 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:27:17.025 INFO 2981023 --- [host-check-host-status1,bb5a8fb259fa1b72,dbef065eecd36b06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:27:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,b51703db648288d0,07b4ae05b29c0462] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,b51703db648288d0,07b4ae05b29c0462] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:17.466 INFO 2981023 --- [ocp-async-3,5f75c521d72a924d,9676e9ac5eba06fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:17.797 INFO 2981023 --- [ocp-async-8,fa16264f67837ae3,3c7b28d09c2f1eb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:27:17.828 INFO 2981023 --- [host-check-clockdiff1,6a624c2abdb937a7,2eddb53fdc7072ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050437 0 0 , error= 2024-09-11 18:27:18.334 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:18.335 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:18.550 INFO 2981023 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050360, nodeCount=480, validCount=480 2024-09-11 18:27:19.466 INFO 2981023 --- [ocp-async-2,2f13f90620841664,8d21f89eded17fa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,3c00f7fc7effb58a,cd1c1327370c3f1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,3c00f7fc7effb58a,cd1c1327370c3f1f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:20.841 INFO 2981023 --- [host-check-clockdiff4,6a624c2abdb937a7,23de4460cb85dcf1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050440 0 0 , error= 2024-09-11 18:27:21.335 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:21.337 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:21.467 INFO 2981023 --- [ocp-async-5,096cd9123c852264,1feb1c4bc910f861] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:22.842 INFO 2981023 --- [ocp-schedules-28,93f4a3c3f98b8ed3,00be339d68bcb646] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:27:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,392011e825991457,3d8f07661d7446d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,392011e825991457,3d8f07661d7446d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:23.467 INFO 2981023 --- [ocp-async-9,e88bc84376fc2781,8fcabb2d68e57741] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:23.525 INFO 2981023 --- [ocp-schedules-8,8b9f5aedac5221c4,6c64502428e71b50] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:27:23.573 WARN 2981023 --- [scanner-13671930-1,8b9f5aedac5221c4,8ea763b1d6f61e6f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:27:23.597 INFO 2981023 --- [alarm-detect-18,8b9f5aedac5221c4,3f6b483bcd8d1f0c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646387, data is empty. 2024-09-11 18:27:23.599 INFO 2981023 --- [alarm-detect-18,8b9f5aedac5221c4,3f6b483bcd8d1f0c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646387, data is empty. 2024-09-11 18:27:23.603 INFO 2981023 --- [alarm-detect-18,8b9f5aedac5221c4,3f6b483bcd8d1f0c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646387, data is empty. 2024-09-11 18:27:23.628 INFO 2981023 --- [alarm-detect-12,8b9f5aedac5221c4,3b252466ea9bea8b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646387, data is empty. 2024-09-11 18:27:23.633 INFO 2981023 --- [alarm-detect-7,8b9f5aedac5221c4,ec4a869ad419b653] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646387, data is empty. 2024-09-11 18:27:23.635 INFO 2981023 --- [alarm-detect-12,8b9f5aedac5221c4,3b252466ea9bea8b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646387, data is empty. 2024-09-11 18:27:23.638 INFO 2981023 --- [alarm-detect-7,8b9f5aedac5221c4,ec4a869ad419b653] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646387, data is empty. 2024-09-11 18:27:23.640 INFO 2981023 --- [alarm-detect-12,8b9f5aedac5221c4,3b252466ea9bea8b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646387, data is empty. 2024-09-11 18:27:23.642 INFO 2981023 --- [alarm-detect-7,8b9f5aedac5221c4,ec4a869ad419b653] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646387, data is empty. 2024-09-11 18:27:23.707 INFO 2981023 --- [alarm-detect-13,8b9f5aedac5221c4,0eef62c7c0a5f168] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050405 ~ 1726050433 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050420, value=8.461)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.461, values={1726050420=8.461}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:23.707 INFO 2981023 --- [alarm-detect-13,8b9f5aedac5221c4,0eef62c7c0a5f168] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050405 ~ 1726050433 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050420, value=8.461)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.461, values={1726050420=8.461}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:27:23.755 INFO 2981023 --- [alarm-detect-19,8b9f5aedac5221c4,6381abb146ef8e74] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646387, data is empty. 2024-09-11 18:27:23.762 INFO 2981023 --- [alarm-detect-19,8b9f5aedac5221c4,6381abb146ef8e74] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646387, data is empty. 2024-09-11 18:27:23.770 INFO 2981023 --- [alarm-detect-19,8b9f5aedac5221c4,6381abb146ef8e74] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646387, data is empty. 2024-09-11 18:27:23.780 INFO 2981023 --- [alarm-detect-13,8b9f5aedac5221c4,c4bf36933f197529] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050405 ~ 1726050433 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050420, value=104.938)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.938, values={1726050420=104.938}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:24.337 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:24.337 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:24.509 INFO 2981023 --- [ocp-schedules-9,67772c1fc398100c,a84770adf901a590] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:27:25.467 INFO 2981023 --- [ocp-async-10,967f6f22f9c59b9b,3592587b4e7fe93a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,7a4f9e0744810070,e1ca342f3ea47447] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,7a4f9e0744810070,e1ca342f3ea47447] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:26.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:27:26.354361+08:00, lastProcessTime=2024-09-11T18:27:16.360329+08:00 2024-09-11 18:27:26.498 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:27:26.354361+08:00 2024-09-11 18:27:27.337 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:27.338 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:27.467 INFO 2981023 --- [ocp-async-2,0a99b2109dfb57a9,0e675707d4dc8b80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:27.797 INFO 2981023 --- [ocp-async-9,ae435d708d0fbaa9,5072bd151faacab0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:27:28.021 INFO 2981023 --- [ocp-schedules-13,82788ed153006b88,a45aadd718118cbd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:27:28.021 INFO 2981023 --- [ocp-schedules-13,82788ed153006b88,a45aadd718118cbd] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:27:28.021 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:27:28.022 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:27:28.022 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:27:28.047 INFO 2981023 --- [ocp-schedules-13,82788ed153006b88,a45aadd718118cbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:28.047 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:28.047 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:27:28.052 INFO 2981023 --- [ocp-schedules-13,82788ed153006b88,a45aadd718118cbd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:28.052 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:28.052 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:27:28.057 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:27:28.059 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:27:28.082 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:27:28.083 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:27:28.083 INFO 2981023 --- [ocp-schedules-13,82788ed153006b88,a45aadd718118cbd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:27:28.086 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:27:28.089 INFO 2981023 --- [ocp-schedules-13,82788ed153006b88,a45aadd718118cbd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:27:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,331270263e7d8a4b,483f7a87060fa6af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,331270263e7d8a4b,483f7a87060fa6af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:29.467 INFO 2981023 --- [ocp-async-7,c07cdd224eb64c1c,808707dddec88a69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:30.338 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:30.339 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:31.467 INFO 2981023 --- [ocp-async-8,c13471146157b910,ad88dc99c6a077ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,7278ab9c04a437d7,b0b904a45b386391] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,7278ab9c04a437d7,b0b904a45b386391] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:32.845 INFO 2981023 --- [ocp-schedules-24,9f74bcde3635f39b,bafb690e805dc734] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:27:33.339 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:33.339 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:33.468 INFO 2981023 --- [ocp-async-1,0af5c86d8deba4a9,474ccd128cb72367] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:33.801 INFO 2981023 --- [ocp-schedules-18,7021d948d597cbe4,fb5f387e30301619] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:27:33.832 WARN 2981023 --- [scanner-13672269-1,7021d948d597cbe4,ed179ac4da168347] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:27:33.848 INFO 2981023 --- [alarm-detect-13,7021d948d597cbe4,87f650687c8047e8] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050420 ~ 1726050443 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050435, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050435, value=98.776)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.776, values={1726050435=98.776}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050435=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:33.875 INFO 2981023 --- [alarm-detect-1,7021d948d597cbe4,b4b19a2971926a68] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050420 ~ 1726050443 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050435, value=98.776)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.776, values={1726050435=98.776}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:33.897 INFO 2981023 --- [alarm-detect-1,7021d948d597cbe4,484324f2dfbf5743] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050420 ~ 1726050443 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050435, value=7.481)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=7.481, values={1726050435=7.481}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:33.898 INFO 2981023 --- [alarm-detect-1,7021d948d597cbe4,484324f2dfbf5743] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050420 ~ 1726050443 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050435, value=7.481)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=7.481, values={1726050435=7.481}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:27:33.903 INFO 2981023 --- [alarm-detect-12,7021d948d597cbe4,c7ef210deb657de9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050420 ~ 1726050443 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050435, value=98.776)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.776, values={1726050435=98.776}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:33.961 INFO 2981023 --- [alarm-detect-15,7021d948d597cbe4,13d52fab0188a498] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050420 ~ 1726050443 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050435, value=104.4)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.4, values={1726050435=104.4}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,6ceaa696bc96c165,3eb1df7008cc79e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,6ceaa696bc96c165,3eb1df7008cc79e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:35.468 INFO 2981023 --- [ocp-async-5,b530fc4730504da9,52ac4985e4a44bf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:35.816 INFO 2981023 --- [ocp-schedules-21,76ad9341902d01fb,04505b73f9b90618] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:27:35.826 INFO 2981023 --- [ocp-schedules-21,76ad9341902d01fb,04505b73f9b90618] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:27:35.826 INFO 2981023 --- [ocp-schedules-21,76ad9341902d01fb,04505b73f9b90618] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:27:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:27:36.336944+08:00, lastProcessTime=2024-09-11T18:27:26.354361+08:00 2024-09-11 18:27:36.339 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:36.340 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:36.466 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:27:36.336944+08:00 2024-09-11 18:27:37.314 INFO 2981023 --- [prometheus-cache-monitor1,ea72ea069804afb2,ed4189a2a48442e5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:27:37.315 INFO 2981023 --- [prometheus-cache-monitor1,ea72ea069804afb2,ed4189a2a48442e5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:27:37.468 INFO 2981023 --- [ocp-async-9,590e99d568750c9e,9066ab9a58f1fd5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:37.797 INFO 2981023 --- [ocp-async-1,a880a6641a514f0b,e500a7d1ee21dc53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:27:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,d32ebd2755013aa1,133886c917c215ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,d32ebd2755013aa1,133886c917c215ee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:39.340 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:39.340 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:39.468 INFO 2981023 --- [ocp-async-2,dd675ead79ba6478,b48de9f88e40a990] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:40.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:27:40.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:27:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,d836e2df513cd52c,e047e0099182caff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,d836e2df513cd52c,e047e0099182caff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:41.468 INFO 2981023 --- [ocp-async-3,4a796e5745011443,8c2b19358c72d388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:42.341 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:42.341 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:42.848 INFO 2981023 --- [ocp-schedules-20,744574f0d81b1e59,33a174350bf2f95b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:27:43.468 INFO 2981023 --- [ocp-async-4,42d73351c43d163e,563f5e3293779c8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:43.979 INFO 2981023 --- [ocp-schedules-2,6c5d718f11a25452,196154c5ab872a69] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:27:44.043 INFO 2981023 --- [alarm-detect-15,6c5d718f11a25452,cf961b09a43016ae] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050435 ~ 1726050453 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050450, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050450, value=98.512)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.512, values={1726050450=98.512}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050450=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:44.046 INFO 2981023 --- [alarm-detect-12,6c5d718f11a25452,d2852794daa1f7d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646387, data is empty. 2024-09-11 18:27:44.049 INFO 2981023 --- [alarm-detect-12,6c5d718f11a25452,d2852794daa1f7d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646387, data is empty. 2024-09-11 18:27:44.052 INFO 2981023 --- [alarm-detect-12,6c5d718f11a25452,d2852794daa1f7d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646387, data is empty. 2024-09-11 18:27:44.058 WARN 2981023 --- [scanner-13672517-1,6c5d718f11a25452,4537b219227845db] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:27:44.077 INFO 2981023 --- [alarm-detect-9,6c5d718f11a25452,9a8ffa5a6e934c07] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646387, data is empty. 2024-09-11 18:27:44.078 INFO 2981023 --- [alarm-detect-2,6c5d718f11a25452,ef4fbb232c53a718] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646387, data is empty. 2024-09-11 18:27:44.080 INFO 2981023 --- [alarm-detect-9,6c5d718f11a25452,9a8ffa5a6e934c07] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646387, data is empty. 2024-09-11 18:27:44.082 INFO 2981023 --- [alarm-detect-2,6c5d718f11a25452,ef4fbb232c53a718] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646387, data is empty. 2024-09-11 18:27:44.084 INFO 2981023 --- [alarm-detect-9,6c5d718f11a25452,9a8ffa5a6e934c07] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646387, data is empty. 2024-09-11 18:27:44.086 INFO 2981023 --- [alarm-detect-2,6c5d718f11a25452,ef4fbb232c53a718] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646387, data is empty. 2024-09-11 18:27:44.100 INFO 2981023 --- [alarm-detect-17,6c5d718f11a25452,1af11cd055a2bc25] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050435 ~ 1726050454 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050450, value=98.512)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.512, values={1726050450=98.512}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:44.107 INFO 2981023 --- [alarm-detect-17,6c5d718f11a25452,45eb0fc74c711faf] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050435 ~ 1726050454 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050450, value=98.512)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.512, values={1726050450=98.512}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:44.192 INFO 2981023 --- [alarm-detect-10,6c5d718f11a25452,2f6bb34ea544f8e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646387, data is empty. 2024-09-11 18:27:44.198 INFO 2981023 --- [alarm-detect-10,6c5d718f11a25452,2f6bb34ea544f8e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646387, data is empty. 2024-09-11 18:27:44.204 INFO 2981023 --- [alarm-detect-10,6c5d718f11a25452,2f6bb34ea544f8e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646387, data is empty. 2024-09-11 18:27:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,bed821307b75bea8,68ff6f716f9b6e4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,bed821307b75bea8,68ff6f716f9b6e4e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:45.101 INFO 2981023 --- [prometheus-second-schedule-1,cfa02fa83d50ae26,de829c108e918c8d] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:27:45.341 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:45.343 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:45.468 INFO 2981023 --- [ocp-async-10,6ccf3af2828ec1fb,62d0f667a406e9de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:45.859 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:27:45.859 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:27:45.862 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:27:45.862 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:27:45.862 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:27:45.862 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:27:45.865 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:27:45.865 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:27:45.865 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:27:45.865 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:27:45.895 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:27:45.901 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:27:45.901 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:27:45.901 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:27:45.901 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:27:45.901 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:27:45.906 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:27:46.025 INFO 2981023 --- [ocp-schedules-18,d12f87f8ed00a5c6,780313af23e20378] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:27:46.027 INFO 2981023 --- [ocp-schedules-21,810277cdd6f7c5b3,782bf77de05c8509] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:27:46.027 INFO 2981023 --- [ocp-schedules-30,843eaa9067475b64,c04d49cd36138c08] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:27:46.030 INFO 2981023 --- [host-check-clockdiff1,d12f87f8ed00a5c6,ef4a417a34f1804f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:27:46.030 INFO 2981023 --- [host-check-agent-status1,843eaa9067475b64,dfdaddf4956a91f3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:27:46.031 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:27:46.032 INFO 2981023 --- [host-check-clockdiff2,d12f87f8ed00a5c6,637ab0c95dac7eed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:27:46.032 INFO 2981023 --- [host-check-clockdiff3,d12f87f8ed00a5c6,e0b18b208710ebb7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:27:46.034 INFO 2981023 --- [host-check-clockdiff4,d12f87f8ed00a5c6,ea747cc063eb3d75] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:27:46.035 INFO 2981023 --- [host-check-agent-status2,843eaa9067475b64,2484e046a6c7468f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:27:46.035 INFO 2981023 --- [host-check-agent-status3,843eaa9067475b64,9d668386b0e7c543] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:27:46.035 INFO 2981023 --- [host-check-agent-status4,843eaa9067475b64,ea586cbd10a1b2c8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:27:46.036 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:27:46.038 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:27:46.038 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:27:46.078 INFO 2981023 --- [host-check-agent-status2,843eaa9067475b64,2484e046a6c7468f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:27:46.079 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:46.079 INFO 2981023 --- [host-check-agent-status4,843eaa9067475b64,ea586cbd10a1b2c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:27:46.088 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:46.089 INFO 2981023 --- [host-check-agent-status1,843eaa9067475b64,dfdaddf4956a91f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:46.092 INFO 2981023 --- [host-check-agent-status3,843eaa9067475b64,9d668386b0e7c543] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:46.093 INFO 2981023 --- [host-check-agent-status3,843eaa9067475b64,9d668386b0e7c543] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:46.093 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:27:46.096 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:46.097 INFO 2981023 --- [host-check-agent-status4,843eaa9067475b64,ea586cbd10a1b2c8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:27:46.097 INFO 2981023 --- [host-check-agent-status1,843eaa9067475b64,dfdaddf4956a91f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:46.097 INFO 2981023 --- [host-check-agent-status2,843eaa9067475b64,2484e046a6c7468f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:27:46.097 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:46.100 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:27:46.122 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:27:46.129 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:27:46.171 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:27:46.187 INFO 2981023 --- [ocp-schedules-22,0492821407e948c6,1d139672a8a8a078] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:27:46.187 INFO 2981023 --- [ocp-schedules-11,408bc5708a3c1885,19b89bcc41dad826] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:27:46.188 INFO 2981023 --- [ocp-schedules-27,106b5ab579b21c97,21b6d58aed009d08] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:27:46.188 INFO 2981023 --- [ocp-schedules-19,2b3180ed6efaad71,b1b9895a4bb62cc3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:27:46.202 INFO 2981023 --- [host-check-clockdiff2,d12f87f8ed00a5c6,637ab0c95dac7eed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050466 5 5 , error= 2024-09-11 18:27:46.202 INFO 2981023 --- [host-check-clockdiff1,d12f87f8ed00a5c6,ef4a417a34f1804f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050466 0 0 , error= 2024-09-11 18:27:46.213 INFO 2981023 --- [host-check-clockdiff3,d12f87f8ed00a5c6,e0b18b208710ebb7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050466 0 0 , error= 2024-09-11 18:27:46.272 INFO 2981023 --- [ocp-schedules-1,8ba510ad834484a0,03987453b03671d4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:27:46.299 INFO 2981023 --- [ocp-schedules-27,106b5ab579b21c97,21b6d58aed009d08] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:27:46.313 INFO 2981023 --- [ocp-schedules-11,408bc5708a3c1885,19b89bcc41dad826] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:27:46.313 INFO 2981023 --- [ocp-schedules-22,0492821407e948c6,1d139672a8a8a078] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:27:46.314 INFO 2981023 --- [ocp-schedules-1,8ba510ad834484a0,03987453b03671d4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:27:46.335 INFO 2981023 --- [ocp-schedules-20,86d82d73ce8536d3,3f46b44009a3a841] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:27:46.338 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:27:46.363 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:27:46.386 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050363245000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:27:46.397 INFO 2981023 --- [ocp-schedules-11,408bc5708a3c1885,19b89bcc41dad826] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:27:46.404 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:27:46.403201+08:00, lastProcessTime=2024-09-11T18:27:36.336944+08:00 2024-09-11 18:27:46.420 INFO 2981023 --- [host-check-agent-status1,843eaa9067475b64,dfdaddf4956a91f3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:27:46.427 INFO 2981023 --- [host-check-agent-status3,843eaa9067475b64,9d668386b0e7c543] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:27:46.433 INFO 2981023 --- [host-check-agent-status2,843eaa9067475b64,2484e046a6c7468f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:27:46.434 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:27:46.435 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:27:46.440900196+08:00 2024-09-11 18:27:46.437 INFO 2981023 --- [host-check-agent-status1,843eaa9067475b64,dfdaddf4956a91f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:27:46.437 INFO 2981023 --- [host-check-agent-status1,843eaa9067475b64,dfdaddf4956a91f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:27:46.437 INFO 2981023 --- [host-check-agent-status1,843eaa9067475b64,dfdaddf4956a91f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:27:46.439 INFO 2981023 --- [host-check-agent-status2,843eaa9067475b64,2484e046a6c7468f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:27:46.439 INFO 2981023 --- [host-check-agent-status2,843eaa9067475b64,2484e046a6c7468f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:27:46.439 INFO 2981023 --- [host-check-agent-status2,843eaa9067475b64,2484e046a6c7468f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:27:46.445 INFO 2981023 --- [host-check-agent-status3,843eaa9067475b64,9d668386b0e7c543] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:27:46.445 INFO 2981023 --- [host-check-agent-status3,843eaa9067475b64,9d668386b0e7c543] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:27:46.446 INFO 2981023 --- [host-check-agent-status3,843eaa9067475b64,9d668386b0e7c543] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:27:46.486 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:27:46.435+08:00 2024-09-11 18:27:46.492 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:27:46.435+08:00 and status AVAILABLE 2024-09-11 18:27:46.500 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:27:46.502 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:27:46.502106573+08:00 2024-09-11 18:27:46.504 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:27:46.502+08:00 2024-09-11 18:27:46.504 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:27:46.506 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:27:46.502+08:00 and status AVAILABLE 2024-09-11 18:27:46.510 INFO 2981023 --- [host-check-agent-status4,843eaa9067475b64,ea586cbd10a1b2c8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:27:46.515 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:27:46.515 INFO 2981023 --- [host-check-agent-status4,843eaa9067475b64,ea586cbd10a1b2c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:27:46.515 INFO 2981023 --- [host-check-agent-status4,843eaa9067475b64,ea586cbd10a1b2c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:27:46.515 INFO 2981023 --- [host-check-agent-status4,843eaa9067475b64,ea586cbd10a1b2c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:27:46.517 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:27:46.517360842+08:00 2024-09-11 18:27:46.518 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:27:46.517+08:00 2024-09-11 18:27:46.518 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.520 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:27:46.521 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:27:46.517+08:00 and status AVAILABLE 2024-09-11 18:27:46.523 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:27:46.524 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:27:46.527 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:27:46.527027154+08:00 2024-09-11 18:27:46.527 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.528 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:27:46.527+08:00 2024-09-11 18:27:46.528 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:27:46.529 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:27:46.531 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:27:46.527+08:00 and status AVAILABLE 2024-09-11 18:27:46.531 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:27:46.531 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:27:46.532 INFO 2981023 --- [ocp-schedules-22,0492821407e948c6,1d139672a8a8a078] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:27:46.535 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:27:46.537 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:27:46.538 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:46.538 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:27:46.540 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:46.540 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:27:46.544 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:27:46.551 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.553 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.554 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:46.558 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:46.558 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:46.562 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:46.572 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:27:46.580 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:27:46.584 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.584 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.585 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:46.588 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:27:46.591 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:27:46.595 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:27:46.596 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:27:46.597 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:46.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.599 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:27:46.602 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.613 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:27:46.614 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:27:46.626 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:27:46.628 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:27:46.642 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:27:46.654 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:27:46.666 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:27:46.670 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:46.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:46.671 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:27:46.672 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.674 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:27:46.675 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:27:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:27:46.688 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:27:46.706 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:27:46.707 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:27:46.708 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:27:46.708 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050363245000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:27:46.714 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:27:46.715 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:27:46.715 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:27:46.726 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:27:46.727 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:46.735 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:27:46.737 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.740 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:27:46.743 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:46.744 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.744 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:27:46.745 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:27:46.745 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:27:46.749 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:27:46.749 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:27:46.749 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.752 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:27:46.755 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:27:46.756 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:27:46.767 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:27:46.781 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:27:46.788 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:27:46.789 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:27:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:27:46.792 INFO 2981023 --- [host-check-host-status2,810277cdd6f7c5b3,383a4681a3f12a35] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:27:46.798 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:27:46.813 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:27:46.814 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:27:46.819 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:27:46.820 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:27:46.823 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.957964157S 2024-09-11 18:27:46.825 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:27:46.827 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:27:46.841 INFO 2981023 --- [host-check-host-status4,810277cdd6f7c5b3,ba77e99d2b0999e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:27:46.842 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:46.848 INFO 2981023 --- [ocp-schedules-29,02c84e6ce745db55,5f4d28949dd94232] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:27:46.848 INFO 2981023 --- [ocp-schedules-29,02c84e6ce745db55,5f4d28949dd94232] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:27:46.849 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:46.852 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.854 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:27:46.855 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:27:46.863 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:46.863 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.869 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:27:46.876 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:27:46.403201+08:00 2024-09-11 18:27:46.879 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:27:46.890 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:46.891 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.891 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:27:46.892 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:27:46.892 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.893 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.894 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:27:46.902 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:27:46.903 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:27:46.905 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.911 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:27:46.918 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:27:46.918 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:27:46.922 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:27:46.923 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:27:46.928 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:27:46.937 INFO 2981023 --- [host-check-host-status3,810277cdd6f7c5b3,bd27cf00a7bd133c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:27:46.938 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:27:46.938 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:27:46.950 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:46.955 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.955 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:27:46.956 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:27:46.956 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:46.958 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:27:46.965 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:27:46.993 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:27:46.996 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:27:46.997 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:27:46.997 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:27:46.997 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:27:46.997 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:27:46.997 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:27:46.997 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:27:46.999 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:27:46.999 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:27:47.000 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:27:47.001 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:27:47.005 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:27:47.005 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:27:47.017 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:27:47.019 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:47.028 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:47.031 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:27:47.038 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:27:47.044 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:47.087 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:27:47.088 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] o.s.boot.SpringApplication : Started application in 1.226 seconds (JVM running for 86266.961) 2024-09-11 18:27:47.088 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.226379073S 2024-09-11 18:27:47.112 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:27:47.119 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:27:47.121 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:27:47.122 INFO 2981023 --- [ocp-schedules-25,686cb0b204080e55,f53a29b4830eafa1] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:27:47.129 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:27:47.147 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:27:47.259 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:27:47.266 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:27:47.279 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:47.284 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:27:47.296 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:27:47.296 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:27:47.297 INFO 2981023 --- [ocp-schedules-12,a260180777a00f88,7d6ea2569c74452e] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:27:47.303 INFO 2981023 --- [ocp-schedules-27,106b5ab579b21c97,21b6d58aed009d08] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:27:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,1f302b954908984d,73e91cefab991b82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:47.317 INFO 2981023 --- [prometheus-minute-schedule-1,1f302b954908984d,73e91cefab991b82] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:47.362 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:27:47.368 INFO 2981023 --- [host-check-host-status1,810277cdd6f7c5b3,aacbf580abfe8a8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:27:47.468 INFO 2981023 --- [ocp-async-3,3472c9f98ee4d960,db820c98629386b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:47.797 INFO 2981023 --- [ocp-async-2,5ef686edf647644c,78490f68cd5b6fa5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:27:48.254 INFO 2981023 --- [ocp-schedules-23,56f2e6518bafdb07,1f0878f4fbc20b5f] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:27:48.272 WARN 2981023 --- [ocp-schedules-23,56f2e6518bafdb07,1f0878f4fbc20b5f] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:27:48.343 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:48.344 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:49.469 INFO 2981023 --- [ocp-async-8,64e2097e616a5705,292ce3705d95360a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:27:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:27:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,e2f8e10b686bdf6e,00cf8affc8b3e3f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,e2f8e10b686bdf6e,00cf8affc8b3e3f1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:50.373 INFO 2981023 --- [ocp-schedules-26,da8b17b8424a171a,36e64d272c4d8921] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:27:51.344 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:51.346 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:51.469 INFO 2981023 --- [ocp-async-10,a62c892b54b38499,16b15a5f69fbfc2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:51.644 INFO 2981023 --- [ocp-schedules-30,c919dab770778829,6720c00d2aa888c1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:27:51.645 INFO 2981023 --- [ocp-schedules-30,c919dab770778829,6720c00d2aa888c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:27:51.646 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:27:51.646 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:27:51.646 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:27:51.646 INFO 2981023 --- [ocp-schedules-30,c919dab770778829,6720c00d2aa888c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:27:52.007 INFO 2981023 --- [host-check-clockdiff4,d12f87f8ed00a5c6,ea747cc063eb3d75] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050472 0 0 , error= 2024-09-11 18:27:52.850 INFO 2981023 --- [ocp-schedules-17,45282cc84358b5eb,69bbd9bcf1f4349e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:27:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,38c765eae562e40f,ec86ef8502499a71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,38c765eae562e40f,ec86ef8502499a71] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:53.388 INFO 2981023 --- [ocp-schedules-30,ca6c31f012defae4,f2410022f34372b3] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:27:53.388 INFO 2981023 --- [ocp-schedules-30,ca6c31f012defae4,f2410022f34372b3] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:27:53.469 INFO 2981023 --- [ocp-async-7,ee758d48cb12c235,a751d3815029e4d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:54.243 INFO 2981023 --- [ocp-schedules-24,bd8ffa2410f5b59b,88a7fda9bed68247] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:27:54.293 WARN 2981023 --- [scanner-13672823-1,bd8ffa2410f5b59b,2cc6a14623efae15] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:27:54.337 INFO 2981023 --- [alarm-detect-5,bd8ffa2410f5b59b,6cbd73b890082eea] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050435 ~ 1726050464 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050450, value=8.325)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.325, values={1726050450=8.325}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:54.337 INFO 2981023 --- [alarm-detect-5,bd8ffa2410f5b59b,6cbd73b890082eea] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050435 ~ 1726050464 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050450, value=8.325)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.325, values={1726050450=8.325}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:27:54.346 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:54.346 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:54.422 INFO 2981023 --- [alarm-detect-13,bd8ffa2410f5b59b,1d9f1c6d83bc3ffd] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050435 ~ 1726050464 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050450, value=104.442)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.442, values={1726050450=104.442}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:27:54.512 INFO 2981023 --- [ocp-schedules-21,7f804c3b319a8873,99ff4a9b0bc66ae8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:27:55.149 INFO 2981023 --- [ocp-schedules-25,d4a04f3867f6b370,142d1815a3e8779e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:27:55.149 INFO 2981023 --- [ocp-schedules-25,d4a04f3867f6b370,142d1815a3e8779e] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:27:55.155 INFO 2981023 --- [ocp-schedules-25,d4a04f3867f6b370,142d1815a3e8779e] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:27:55.155 INFO 2981023 --- [ocp-schedules-25,d4a04f3867f6b370,142d1815a3e8779e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:27:55.155 INFO 2981023 --- [ocp-schedules-25,d4a04f3867f6b370,142d1815a3e8779e] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:27:55.165 INFO 2981023 --- [ocp-schedules-25,d4a04f3867f6b370,142d1815a3e8779e] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:27:55.165 INFO 2981023 --- [ocp-schedules-25,d4a04f3867f6b370,142d1815a3e8779e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:27:55.165 INFO 2981023 --- [ocp-schedules-25,d4a04f3867f6b370,142d1815a3e8779e] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:27:55.176 INFO 2981023 --- [ocp-schedules-25,d4a04f3867f6b370,142d1815a3e8779e] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:27:55.452 INFO 2981023 --- [ocp-schedules-13,b50d418f73bd2cab,e67dd05be5938509] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:27:55.469 INFO 2981023 --- [ocp-async-4,633acb7f5c8774bf,36e2ad55f7ea5370] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:55.471 WARN 2981023 --- [ocp-schedules-13,b50d418f73bd2cab,e67dd05be5938509] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:27:55.829 INFO 2981023 --- [ocp-schedules-17,c351a2c9aa259b8a,f082fc2dd6d2cf3a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:27:55.838 INFO 2981023 --- [ocp-schedules-17,c351a2c9aa259b8a,f082fc2dd6d2cf3a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:27:55.838 INFO 2981023 --- [ocp-schedules-17,c351a2c9aa259b8a,f082fc2dd6d2cf3a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:27:55.918 INFO 2981023 --- [ocp-schedules-21,268d65e507b3abe7,2032b55d4cb302a0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:27:55.918 INFO 2981023 --- [ocp-schedules-21,268d65e507b3abe7,2032b55d4cb302a0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:27:55.922 INFO 2981023 --- [ocp-schedules-21,268d65e507b3abe7,2032b55d4cb302a0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:27:55.922 INFO 2981023 --- [ocp-schedules-21,268d65e507b3abe7,2032b55d4cb302a0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:27:55.925 INFO 2981023 --- [ocp-schedules-21,268d65e507b3abe7,2032b55d4cb302a0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:27:55.925 INFO 2981023 --- [ocp-schedules-21,268d65e507b3abe7,2032b55d4cb302a0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:27:56.148 INFO 2981023 --- [ocp-schedules-30,3198f6150324b91c,cd3e7f39502c43ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:27:56.156 INFO 2981023 --- [ocp-schedules-30,3198f6150324b91c,cd3e7f39502c43ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:27:56.157 INFO 2981023 --- [ocp-schedules-30,3198f6150324b91c,cd3e7f39502c43ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:27:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,4774f96cecd39873,cf33d10609a04ade] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:56.318 INFO 2981023 --- [prometheus-minute-schedule-1,4774f96cecd39873,cf33d10609a04ade] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:56.355 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:27:56.354594+08:00, lastProcessTime=2024-09-11T18:27:46.403201+08:00 2024-09-11 18:27:56.494 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:27:56.354594+08:00 2024-09-11 18:27:57.346 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:27:57.348 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:27:57.469 INFO 2981023 --- [ocp-async-6,7d001ca59863484f,52d327aca31b39a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:27:57.797 INFO 2981023 --- [ocp-async-9,29ce9edee3fc7981,8045ea40522bfed1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:27:58.891 INFO 2981023 --- [ocp-schedules-26,b95854f16b786fc8,9bb108a363d2bb15] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:27:58.891 INFO 2981023 --- [ocp-schedules-26,b95854f16b786fc8,9bb108a363d2bb15] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:27:58.924 INFO 2981023 --- [Collect all tenants major compactions1,b95854f16b786fc8,91f5775fd7c43695] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:27:58.924 INFO 2981023 --- [Collect all tenants major compactions5,b95854f16b786fc8,19a0fc103ef2e225] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:27:58.930 INFO 2981023 --- [Collect all tenants major compactions2,b95854f16b786fc8,b60e04cb88ec5ea8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:27:58.930 INFO 2981023 --- [Collect all tenants major compactions3,b95854f16b786fc8,41dbc052bde878ec] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:27:58.931 INFO 2981023 --- [Collect all tenants major compactions4,b95854f16b786fc8,c43f59a4168beedf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:27:58.931 INFO 2981023 --- [Collect all tenants major compactions6,b95854f16b786fc8,8cf099f8fcaef70c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:27:58.954 INFO 2981023 --- [Collect all tenants major compactions3,b95854f16b786fc8,41dbc052bde878ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:58.955 INFO 2981023 --- [Collect all tenants major compactions3,b95854f16b786fc8,41dbc052bde878ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:27:58.955 INFO 2981023 --- [Collect all tenants major compactions2,b95854f16b786fc8,b60e04cb88ec5ea8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:58.956 INFO 2981023 --- [Collect all tenants major compactions2,b95854f16b786fc8,b60e04cb88ec5ea8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:27:58.957 INFO 2981023 --- [Collect all tenants major compactions1,b95854f16b786fc8,91f5775fd7c43695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:58.958 INFO 2981023 --- [Collect all tenants major compactions1,b95854f16b786fc8,91f5775fd7c43695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:27:58.959 INFO 2981023 --- [Collect all tenants major compactions4,b95854f16b786fc8,c43f59a4168beedf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:58.959 INFO 2981023 --- [Collect all tenants major compactions3,b95854f16b786fc8,41dbc052bde878ec] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:27:58.960 INFO 2981023 --- [Collect all tenants major compactions5,b95854f16b786fc8,19a0fc103ef2e225] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:58.962 INFO 2981023 --- [Collect all tenants major compactions3,b95854f16b786fc8,41dbc052bde878ec] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:27:58.963 INFO 2981023 --- [Collect all tenants major compactions2,b95854f16b786fc8,b60e04cb88ec5ea8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:27:58.963 INFO 2981023 --- [Collect all tenants major compactions2,b95854f16b786fc8,b60e04cb88ec5ea8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:27:58.964 INFO 2981023 --- [Collect all tenants major compactions5,b95854f16b786fc8,19a0fc103ef2e225] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:27:58.965 INFO 2981023 --- [Collect all tenants major compactions1,b95854f16b786fc8,91f5775fd7c43695] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:27:58.965 INFO 2981023 --- [Collect all tenants major compactions1,b95854f16b786fc8,91f5775fd7c43695] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:27:58.965 INFO 2981023 --- [Collect all tenants major compactions4,b95854f16b786fc8,c43f59a4168beedf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:27:58.966 INFO 2981023 --- [Collect all tenants major compactions6,b95854f16b786fc8,8cf099f8fcaef70c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:27:58.967 INFO 2981023 --- [Collect all tenants major compactions6,b95854f16b786fc8,8cf099f8fcaef70c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:27:58.969 INFO 2981023 --- [Collect all tenants major compactions6,b95854f16b786fc8,8cf099f8fcaef70c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:27:58.969 INFO 2981023 --- [Collect all tenants major compactions6,b95854f16b786fc8,8cf099f8fcaef70c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:27:58.971 INFO 2981023 --- [Collect all tenants major compactions5,b95854f16b786fc8,19a0fc103ef2e225] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:27:58.971 INFO 2981023 --- [Collect all tenants major compactions4,b95854f16b786fc8,c43f59a4168beedf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:27:58.971 INFO 2981023 --- [Collect all tenants major compactions5,b95854f16b786fc8,19a0fc103ef2e225] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:27:58.971 INFO 2981023 --- [Collect all tenants major compactions4,b95854f16b786fc8,c43f59a4168beedf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:27:58.972 INFO 2981023 --- [ocp-schedules-26,b95854f16b786fc8,9bb108a363d2bb15] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:27:59.241 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.245 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName 2024-09-11 18:27:59.249 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.251 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.254 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.256 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.256 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.257 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.258 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.258 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.259 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.260 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.260 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.261 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.264 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.266 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.269 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.272 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.272 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.273 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.274 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.274 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.277 INFO 2981023 --- [pool-17-thread-1,,] .o.o.p.s.d.i.d.AbstractSqlIndexDiagnoser : INEFFECTIVE_HINT, Found out 3 targets of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), between :Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238},QuerySqlAuditRawStatList(timeout=30000000, level=0, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, startTimeUs=1726050119145000, endTimeUs=1726050419238000, sqlId=null, obServerId=null, obDbId=null, dbName=null, sqlIds=null, cpuTimeUs=20000, maxCpuTimeUs=null, execPs=5.0, tableScans=null, affectedRows=null, partitionCount=null, logicalReads=null, retryCount=null, retryCountRatio=null, maxRetryCount=null, rpcCount=null, elapsedTimeUs=null, cpuTimePercent=null, sqlTypes=null, obServerIdList=null, duration=null, limit=1000, batchSize=null, batchTime=null, dynamicSql=false) 2024-09-11 18:27:59.277 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 3 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.298 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.301 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.304 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:21:59.145, end=2024-09-11T18:26:59.238} 2024-09-11 18:27:59.304 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 18:27:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,c89acf6e1e417f5e,02904360701e6da8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050480, interval=60 2024-09-11 18:27:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,c89acf6e1e417f5e,02904360701e6da8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:27:59.469 INFO 2981023 --- [ocp-async-7,f39a6f62c8720ce7,ed53cf95fe1704be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:00.348 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:00.350 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:00.834 INFO 2981023 --- [ocp-schedules-7,58cc1772ab2b9d1a,f54ff11492f191ba] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:28:00.834+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:28:00.838 INFO 2981023 --- [ocp-schedules-7,58cc1772ab2b9d1a,f54ff11492f191ba] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:28:00.961 INFO 2981023 --- [ocp-schedules-7,58cc1772ab2b9d1a,f54ff11492f191ba] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 51 to SqlAggregateData 2024-09-11 18:28:00.963 INFO 2981023 --- [ocp-schedules-7,58cc1772ab2b9d1a,f54ff11492f191ba] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:28:01.469 INFO 2981023 --- [ocp-async-2,a804a98befd6e7a6,714d7c170f87e33a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,aa9a1625dd80a7a1,050a0deba91c6e60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:02.317 INFO 2981023 --- [prometheus-minute-schedule-1,aa9a1625dd80a7a1,050a0deba91c6e60] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:02.853 INFO 2981023 --- [ocp-schedules-23,1d39bf9bc7d40502,2183961d5cc83c17] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:28:03.255 INFO 2981023 --- [ocp-schedules-14,2d41056b61090a09,07003710211beb24] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:28:03.259 INFO 2981023 --- [ocp-schedules-14,2d41056b61090a09,07003710211beb24] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:28:03.286 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.287 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:28:03.295 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.297 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:28:03.298 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050483298000} 2024-09-11 18:28:03.303 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:28:03.307 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000238, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:28:03.307 INFO 2981023 --- [ocp-schedules-14,2d41056b61090a09,07003710211beb24] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:28:03.333 INFO 2981023 --- [ocp-schedules-27,e381042f30cdb774,dbe08ce07da35a56] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:28:03.334 INFO 2981023 --- [ocp-schedules-27,e381042f30cdb774,dbe08ce07da35a56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:28:03.350 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:03.351 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:03.364 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:28:03.364 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.367 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:28:03.367 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.367 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:28:03.369 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:28:03.370 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:28:03.370 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.373 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.373 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.373 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.373 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.377 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:28:03.379 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.379 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.384 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.384 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.384 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:28:03.385 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.390 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.391 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:28:03.395 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.396 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:28:03.397 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.397 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.401 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.401 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:28:03.407 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.408 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.409 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:28:03.409 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.410 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:28:03.410 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.411 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:28:03.411 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:28:03.413 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.413 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:28:03.414 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.414 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:28:03.415 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.415 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:28:03.415 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.416 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:28:03.416 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.417 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:28:03.417 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:28:03.417 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.418 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:28:03.418 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.419 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:28:03.419 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.420 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:28:03.421 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.421 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:28:03.421 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:28:03.423 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:28:03.432 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:03.434 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.434 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.434 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:03.435 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.436 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:28:03.441 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:28:03.443 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:28:03.443 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.444 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:28:03.445 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:28:03.446 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.449 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:28:03.451 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:28:03.451 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:28:03.451 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:28:03.451 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:28:03.451 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.451 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.456 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:28:03.457 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:28:03.457 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:28:03.457 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:28:03.469 INFO 2981023 --- [ocp-async-1,0d0a4422aa8c7450,9bedb46761620efe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:03.502 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.502 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.519 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.520 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.520 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:28:03.521 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE67E1D6E9-0-0] 2024-09-11 18:28:03.521 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:28:03.521 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.521 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:28:03.542 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.553 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:28:03.592 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.593 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:28:03.603 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:28:03.603 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6821D6D4-0-0] 2024-09-11 18:28:03.603 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:28:03.609 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.610 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:28:03.610 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.611 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:28:03.620 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.620 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:28:03.621 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.621 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:28:03.622 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.622 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:28:03.622 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.623 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:28:03.623 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:28:03.625 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:28:03.634 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:03.639 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.640 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:28:03.647 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:28:03.647 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:28:03.647 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:28:03.649 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:28:03.651 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000238, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:28:03.651 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:28:03.656 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:28:03.660 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:03.671 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:28:03.692 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:28:03.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:28:03.693 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:28:03.693 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:28:03.693 INFO 2981023 --- [ocp-schedules-27,e381042f30cdb774,dbe08ce07da35a56] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:28:04.415 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000238 2024-09-11 18:28:04.436 INFO 2981023 --- [ocp-schedules-3,d1a80df11e4e3370,dec94d07c29895ac] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:28:04.481 WARN 2981023 --- [scanner-13673047-1,d1a80df11e4e3370,3c15347ba20a58eb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:28:04.494 INFO 2981023 --- [alarm-detect-11,d1a80df11e4e3370,20d4f9b6befd8d4e] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050450 ~ 1726050474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050465, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050465, value=98.548)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.548, values={1726050465=98.548}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050465=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:04.494 INFO 2981023 --- [alarm-detect-11,d1a80df11e4e3370,20d4f9b6befd8d4e] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=os_tsar_cpu_util_hwm, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:28:04.514 INFO 2981023 --- [alarm-detect-1,d1a80df11e4e3370,bfd4b52c1e0e3259] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646387, data is empty. 2024-09-11 18:28:04.519 INFO 2981023 --- [alarm-detect-1,d1a80df11e4e3370,bfd4b52c1e0e3259] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646438, data is empty. 2024-09-11 18:28:04.523 INFO 2981023 --- [alarm-detect-1,d1a80df11e4e3370,bfd4b52c1e0e3259] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646438, data is empty. 2024-09-11 18:28:04.556 INFO 2981023 --- [alarm-detect-11,d1a80df11e4e3370,fa138a91f1b25247] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646387, data is empty. 2024-09-11 18:28:04.557 INFO 2981023 --- [alarm-detect-14,d1a80df11e4e3370,7863f3c9d2d521d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646387, data is empty. 2024-09-11 18:28:04.562 INFO 2981023 --- [alarm-detect-14,d1a80df11e4e3370,7863f3c9d2d521d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646438, data is empty. 2024-09-11 18:28:04.562 INFO 2981023 --- [alarm-detect-11,d1a80df11e4e3370,fa138a91f1b25247] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646438, data is empty. 2024-09-11 18:28:04.567 INFO 2981023 --- [alarm-detect-14,d1a80df11e4e3370,7863f3c9d2d521d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646438, data is empty. 2024-09-11 18:28:04.568 INFO 2981023 --- [alarm-detect-11,d1a80df11e4e3370,fa138a91f1b25247] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646438, data is empty. 2024-09-11 18:28:04.583 INFO 2981023 --- [alarm-detect-11,d1a80df11e4e3370,6f26c33bde1fa75d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050450 ~ 1726050474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050465, value=8.597)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.597, values={1726050465=8.597}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:04.583 INFO 2981023 --- [alarm-detect-11,d1a80df11e4e3370,6f26c33bde1fa75d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050450 ~ 1726050474 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050465, value=8.597)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.597, values={1726050465=8.597}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:28:04.602 INFO 2981023 --- [alarm-detect-18,d1a80df11e4e3370,f168e3f1928c5ccc] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050450 ~ 1726050474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050465, value=98.548)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.548, values={1726050465=98.548}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:04.613 INFO 2981023 --- [alarm-detect-17,d1a80df11e4e3370,b3d519ef5bc0da3c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050450 ~ 1726050474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050465, value=98.548)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.548, values={1726050465=98.548}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:04.668 INFO 2981023 --- [alarm-detect-6,d1a80df11e4e3370,9c6e529235b281d7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050450 ~ 1726050474 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050465, value=105.2)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.2, values={1726050465=105.2}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:04.676 INFO 2981023 --- [alarm-detect-4,d1a80df11e4e3370,b8d0e2e745b48446] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646387, data is empty. 2024-09-11 18:28:04.692 INFO 2981023 --- [alarm-detect-4,d1a80df11e4e3370,b8d0e2e745b48446] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646438, data is empty. 2024-09-11 18:28:04.699 INFO 2981023 --- [alarm-detect-4,d1a80df11e4e3370,b8d0e2e745b48446] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646438, data is empty. 2024-09-11 18:28:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,55cd0ce91bebc2f8,c5138c273a2675aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:05.318 INFO 2981023 --- [prometheus-minute-schedule-1,55cd0ce91bebc2f8,c5138c273a2675aa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:05.470 INFO 2981023 --- [ocp-async-5,37b9507cb5368be5,ec1287f582120182] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:28:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:28:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:28:05.986 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:28:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:28:05.996 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:28:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:28:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:28:05.999 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:05.999 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:28:06.000 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:06.000 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:28:06.050 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:25:41, start=2024-09-11T18:22:41, end=2024-09-11T18:25:11, updates=294, granule=120000ms, delay=180000ms 2024-09-11 18:28:06.062 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:25:41, target=2024-09-11T18:25:11, granule=120000ms, delay=30000ms) 2024-09-11 18:28:06.162 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T18:25:11, start=2024-09-11T18:10:51, end=2024-09-11T18:20:51, updates=343, granule=600000ms, delay=860000ms 2024-09-11 18:28:06.166 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T18:25:11, target=2024-09-11T18:20:51, granule=600000ms, delay=260000ms) 2024-09-11 18:28:06.166 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=196ms, period=120000ms 2024-09-11 18:28:06.351 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:06.352 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:28:06.352816+08:00, lastProcessTime=2024-09-11T18:27:56.354594+08:00 2024-09-11 18:28:06.483 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:28:06.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:28:06.352816+08:00 2024-09-11 18:28:07.470 INFO 2981023 --- [ocp-async-7,51fd13afa081fe1e,3fb53ffb95f68f68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:07.797 INFO 2981023 --- [ocp-async-1,073f76d41c10f9cf,1c0e80c126a6a6b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:28:08.223 INFO 2981023 --- [ocp-schedules-25,56d4c7d09b4231b1,4cbf86fb779a37fa] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:28:08.300 INFO 2981023 --- [ocp-schedules-29,ab0e1c6561b1dce0,55f2d6db94b22b98] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:28:08.300+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:28:08.300 INFO 2981023 --- [ocp-schedules-29,ab0e1c6561b1dce0,55f2d6db94b22b98] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:28:08.303 INFO 2981023 --- [ocp-schedules-29,ab0e1c6561b1dce0,55f2d6db94b22b98] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:28:08.303 INFO 2981023 --- [ocp-schedules-29,ab0e1c6561b1dce0,55f2d6db94b22b98] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:28:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,0be1babaaed9122d,29b6579e5bede6ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,0be1babaaed9122d,29b6579e5bede6ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:09.352 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:09.353 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:09.470 INFO 2981023 --- [ocp-async-10,1753466f8a07b6a4,af1faad4c32226cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:10.004 INFO 2981023 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:28:10.004 INFO 2981023 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:28:10.918 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:28:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:28:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,f6d227c6cb2704a2,316130fee50a545a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,f6d227c6cb2704a2,316130fee50a545a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:11.470 INFO 2981023 --- [ocp-async-3,10b4987f420f6885,1885fdb680a18ae9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:11.842 INFO 2981023 --- [ocp-schedules-5,67dca7c499b9e7fa,149c445276e4d254] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:28:12.353 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:12.354 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:12.855 INFO 2981023 --- [ocp-schedules-21,45c1fade87867452,1cafeecaa8b69fd0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:28:13.470 INFO 2981023 --- [ocp-async-4,646cfa8c8373ec8e,ad4ee8ef5340eefa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:13.598 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050420, nodeCount=480, affectRows=480 2024-09-11 18:28:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,9bf674faf932353e,e923b3e9d4e71c45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,789fc776fed0c87e,55de841412b99f2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:28:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,9bf674faf932353e,e923b3e9d4e71c45] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:14.737 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050420, nodeCount=49, affectRows=49 2024-09-11 18:28:14.738 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050420, nodeCount=47, validCount=47 2024-09-11 18:28:14.742 INFO 2981023 --- [ocp-schedules-10,7ccd7edf3862b9ea,626ff43f0b7058d4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:28:14.808 WARN 2981023 --- [scanner-13674860-1,7ccd7edf3862b9ea,fa6f5c073838d01d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:28:14.819 INFO 2981023 --- [alarm-detect-2,7ccd7edf3862b9ea,6a7937d279cedaa0] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050465 ~ 1726050484 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050480, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050480, value=98.174)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.174, values={1726050480=98.174}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050480=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:14.841 INFO 2981023 --- [alarm-detect-15,7ccd7edf3862b9ea,da14a953ad63667a] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050465 ~ 1726050484 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050480, value=98.174)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.174, values={1726050480=98.174}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:14.841 INFO 2981023 --- [alarm-detect-15,7ccd7edf3862b9ea,da14a953ad63667a] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=os_tsar_cpu_util_full, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:28:14.846 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050420, nodeCount=1000, affectRows=1000 2024-09-11 18:28:14.849 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050420, nodeCount=999, validCount=999 2024-09-11 18:28:14.892 INFO 2981023 --- [alarm-detect-11,7ccd7edf3862b9ea,9ed92f027283726b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050465 ~ 1726050484 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050480, value=98.174)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.174, values={1726050480=98.174}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:14.892 INFO 2981023 --- [alarm-detect-11,7ccd7edf3862b9ea,9ed92f027283726b] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_cpu_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:28:14.911 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050420, nodeCount=1000, affectRows=1000 2024-09-11 18:28:14.914 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050420, nodeCount=1000, validCount=1000 2024-09-11 18:28:15.341 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050420, nodeCount=480, validCount=480 2024-09-11 18:28:15.354 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:15.354 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:15.470 INFO 2981023 --- [ocp-async-8,6b5e79c444fc1f75,c90c99e321fcf50e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:15.841 INFO 2981023 --- [ocp-schedules-25,57fc9e97553d876a,60c361f79f7d9ec0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:28:15.851 INFO 2981023 --- [ocp-schedules-25,57fc9e97553d876a,60c361f79f7d9ec0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:28:15.851 INFO 2981023 --- [ocp-schedules-25,57fc9e97553d876a,60c361f79f7d9ec0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:28:16.015 INFO 2981023 --- [ocp-schedules-5,fd49fadbe4e6f331,673410ccda801987] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:28:16.015 INFO 2981023 --- [host-check-clockdiff1,fd49fadbe4e6f331,44f9a3e32fb30d2c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:28:16.016 INFO 2981023 --- [host-check-clockdiff2,fd49fadbe4e6f331,8a68bff914d3d576] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:28:16.016 INFO 2981023 --- [host-check-clockdiff3,fd49fadbe4e6f331,3bb92731d2a9fecf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:28:16.017 INFO 2981023 --- [host-check-clockdiff4,fd49fadbe4e6f331,cab9b525562f6a80] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:28:16.024 INFO 2981023 --- [ocp-schedules-12,db1c8e55c116a043,34da25df119e8c37] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:28:16.029 INFO 2981023 --- [host-check-clockdiff3,fd49fadbe4e6f331,3bb92731d2a9fecf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050496 0 0 , error= 2024-09-11 18:28:16.031 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:28:16.032 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:28:16.032 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:28:16.034 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:28:16.061 INFO 2981023 --- [ocp-schedules-13,d3f7dd84a9a16a4b,bdbe94ea7839587d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:28:16.062 INFO 2981023 --- [host-check-agent-status1,d3f7dd84a9a16a4b,76b01e69b626f28e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:28:16.062 INFO 2981023 --- [host-check-agent-status2,d3f7dd84a9a16a4b,8390f5f2c2e91b12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:28:16.062 INFO 2981023 --- [host-check-agent-status3,d3f7dd84a9a16a4b,92ae76b77b795a20] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:28:16.063 INFO 2981023 --- [host-check-agent-status4,d3f7dd84a9a16a4b,4c888e004a5a4a79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:28:16.067 INFO 2981023 --- [host-check-agent-status3,d3f7dd84a9a16a4b,92ae76b77b795a20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:28:16.068 INFO 2981023 --- [host-check-agent-status2,d3f7dd84a9a16a4b,8390f5f2c2e91b12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:28:16.069 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:28:16.073 INFO 2981023 --- [host-check-agent-status2,d3f7dd84a9a16a4b,8390f5f2c2e91b12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:28:16.075 INFO 2981023 --- [host-check-agent-status1,d3f7dd84a9a16a4b,76b01e69b626f28e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:28:16.076 INFO 2981023 --- [host-check-agent-status1,d3f7dd84a9a16a4b,76b01e69b626f28e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:28:16.077 INFO 2981023 --- [host-check-agent-status4,d3f7dd84a9a16a4b,4c888e004a5a4a79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:28:16.077 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:28:16.078 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:28:16.078 INFO 2981023 --- [host-check-agent-status4,d3f7dd84a9a16a4b,4c888e004a5a4a79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:28:16.079 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:28:16.080 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:28:16.081 INFO 2981023 --- [host-check-agent-status3,d3f7dd84a9a16a4b,92ae76b77b795a20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:28:16.093 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:28:16.188 INFO 2981023 --- [ocp-schedules-26,8e0380da0933e255,c99e812c6721b1a0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:28:16.188 INFO 2981023 --- [ocp-schedules-27,9fff51422977a017,c586bafc3d14152f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:28:16.189 INFO 2981023 --- [ocp-schedules-30,1956b9621938ee4f,2ebdc1db4109ce90] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:28:16.189 INFO 2981023 --- [ocp-schedules-6,87566b734c0d20e4,ab21e1e9188d5e53] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:28:16.195 INFO 2981023 --- [ocp-schedules-27,9fff51422977a017,c586bafc3d14152f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:28:16.195 INFO 2981023 --- [ocp-schedules-26,8e0380da0933e255,c99e812c6721b1a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:28:16.198 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:28:16.208 INFO 2981023 --- [ocp-schedules-30,1956b9621938ee4f,2ebdc1db4109ce90] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:28:16.210 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:28:16.259 INFO 2981023 --- [ocp-schedules-17,ba09d932ba717371,1d3df7ee7c1fb144] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:28:16.281 INFO 2981023 --- [host-check-agent-status1,d3f7dd84a9a16a4b,76b01e69b626f28e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:28:16.289 INFO 2981023 --- [host-check-agent-status1,d3f7dd84a9a16a4b,76b01e69b626f28e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:28:16.289 INFO 2981023 --- [host-check-agent-status1,d3f7dd84a9a16a4b,76b01e69b626f28e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:28:16.289 INFO 2981023 --- [host-check-agent-status1,d3f7dd84a9a16a4b,76b01e69b626f28e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:28:16.291 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:28:16.292 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:28:16.291683549+08:00 2024-09-11 18:28:16.292 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:28:16.292+08:00 2024-09-11 18:28:16.299 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:28:16.292+08:00 and status AVAILABLE 2024-09-11 18:28:16.300 INFO 2981023 --- [host-check-agent-status4,d3f7dd84a9a16a4b,4c888e004a5a4a79] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:28:16.303 INFO 2981023 --- [host-check-agent-status3,d3f7dd84a9a16a4b,92ae76b77b795a20] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:28:16.304 INFO 2981023 --- [host-check-agent-status2,d3f7dd84a9a16a4b,8390f5f2c2e91b12] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:28:16.305 INFO 2981023 --- [host-check-agent-status4,d3f7dd84a9a16a4b,4c888e004a5a4a79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:28:16.305 INFO 2981023 --- [host-check-agent-status4,d3f7dd84a9a16a4b,4c888e004a5a4a79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:28:16.305 INFO 2981023 --- [host-check-agent-status4,d3f7dd84a9a16a4b,4c888e004a5a4a79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:28:16.307 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:28:16.308 INFO 2981023 --- [host-check-agent-status2,d3f7dd84a9a16a4b,8390f5f2c2e91b12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:28:16.308 INFO 2981023 --- [host-check-agent-status2,d3f7dd84a9a16a4b,8390f5f2c2e91b12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:28:16.308 INFO 2981023 --- [host-check-agent-status2,d3f7dd84a9a16a4b,8390f5f2c2e91b12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:28:16.309 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:28:16.309301396+08:00 2024-09-11 18:28:16.309 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:28:16.313 INFO 2981023 --- [ocp-schedules-27,9fff51422977a017,c586bafc3d14152f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:28:16.313 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050483298000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:28:16.314 INFO 2981023 --- [host-check-agent-status3,d3f7dd84a9a16a4b,92ae76b77b795a20] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:28:16.314 INFO 2981023 --- [host-check-agent-status3,d3f7dd84a9a16a4b,92ae76b77b795a20] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:28:16.314 INFO 2981023 --- [host-check-agent-status3,d3f7dd84a9a16a4b,92ae76b77b795a20] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:28:16.315 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:28:16.316 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:28:16.321184729+08:00 2024-09-11 18:28:16.316 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:28:16.310256494+08:00 2024-09-11 18:28:16.317 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:28:16.316+08:00 2024-09-11 18:28:16.318 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:28:16.316+08:00 2024-09-11 18:28:16.318 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:28:16.322 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:28:16.316+08:00 and status AVAILABLE 2024-09-11 18:28:16.323 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:28:16.316+08:00 and status AVAILABLE 2024-09-11 18:28:16.325 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:28:16.309+08:00 2024-09-11 18:28:16.329 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:28:16.309+08:00 and status AVAILABLE 2024-09-11 18:28:16.335 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:28:16.368 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:28:16.375 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:28:16.376 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:28:16.376 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:28:16.377 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:28:16.377 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:28:16.379 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:28:16.380 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.381 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:28:16.381 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:28:16.385 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:28:16.383273+08:00, lastProcessTime=2024-09-11T18:28:06.352816+08:00 2024-09-11 18:28:16.386 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:28:16.390 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:28:16.392 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:28:16.392 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.393 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:28:16.395 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:28:16.396 INFO 2981023 --- [ocp-schedules-26,8e0380da0933e255,c99e812c6721b1a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:28:16.406 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.406 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:16.406 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:28:16.407 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.408 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:16.412 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:28:16.413 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:28:16.415 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.416 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:16.421 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:28:16.434 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:28:16.436 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:28:16.449 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:28:16.469 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:28:16.469 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:28:16.471 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:28:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.488 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:28:16.493 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:28:16.493 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050483298000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:28:16.498 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:28:16.514 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:28:16.519 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:28:16.544 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:28:16.555 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:28:16.557 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:28:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:16.560 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.560 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:16.561 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:28:16.561 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:28:16.564 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:28:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:28:16.573 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:28:16.580 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:28:16.580 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:28:16.585 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:28:16.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:16.589 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:28:16.593 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:28:16.593 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:28:16.596 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.600 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:28:16.601 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:28:16.602 INFO 2981023 --- [host-check-host-status4,db1c8e55c116a043,42393e58a4fec88a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:28:16.609 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:16.609 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.610 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:28:16.610 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:28:16.614 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:28:16.615 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:28:16.616 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:28:16.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:28:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:28:16.626 INFO 2981023 --- [host-check-host-status3,db1c8e55c116a043,c50d9ff5e7e062ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:28:16.638 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:28:16.641 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:28:16.642 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:16.647 INFO 2981023 --- [host-check-host-status2,db1c8e55c116a043,ec94354b5d6553f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:28:16.647 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:28:16.649 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.650 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:16.651 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:28:16.655 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.661 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:28:16.662 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:28:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:28:16.714 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:28:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:28:16.718 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:28:16.723 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:28:16.730 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=2, resultsCount=2 2024-09-11 18:28:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:28:16.736 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:16.736 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:28:16.383273+08:00 2024-09-11 18:28:16.737 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.737 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:28:16.739 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:16.742 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:28:16.742 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:28:16.750 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:16.751 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:28:16.752 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:28:16.754 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:28:16.757 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:28:16.758 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:28:16.768 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:28:16.930 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:28:16.935 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:28:16.958 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:28:17.038 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:28:17.043 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:17.053 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:17.058 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:28:17.063 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:28:17.063 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:28:17.064 INFO 2981023 --- [ocp-schedules-30,1956b9621938ee4f,2ebdc1db4109ce90] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:28:17.193 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:28:17.199 INFO 2981023 --- [host-check-host-status1,db1c8e55c116a043,2d7286930556692f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:28:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,e0664f852d6a8909,e783c187b11ac55d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:17.317 INFO 2981023 --- [prometheus-minute-schedule-1,e0664f852d6a8909,e783c187b11ac55d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:17.470 INFO 2981023 --- [ocp-async-1,998b8e5544d4c09d,6c2f7d2bb9249bc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:17.797 INFO 2981023 --- [ocp-async-4,24e07a68a5747d3b,c89bb190119504ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:28:17.816 INFO 2981023 --- [host-check-clockdiff1,fd49fadbe4e6f331,44f9a3e32fb30d2c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050497 0 0 , error= 2024-09-11 18:28:18.354 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:18.355 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:19.178 INFO 2981023 --- [host-check-clockdiff2,fd49fadbe4e6f331,8a68bff914d3d576] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050499 5 5 , error= 2024-09-11 18:28:19.471 INFO 2981023 --- [ocp-async-7,9a61ac33620469c4,a42aff28b73d5db4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,812902991bf37fc8,138f7b052963cbdc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,812902991bf37fc8,138f7b052963cbdc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:21.355 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:21.357 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:21.471 INFO 2981023 --- [ocp-async-6,b6b95ade0e669275,749fbcf35f7749b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:21.939 INFO 2981023 --- [host-check-clockdiff4,fd49fadbe4e6f331,cab9b525562f6a80] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050501 0 0 , error= 2024-09-11 18:28:22.858 INFO 2981023 --- [ocp-schedules-23,d0d013d8d9387b4b,587f16e6bdf07709] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:28:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,37c34eb873014b90,6661e36a7b0e0e88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,37c34eb873014b90,6661e36a7b0e0e88] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:23.471 INFO 2981023 --- [ocp-async-5,784a18326f4c3d59,27a827b5c687118a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:24.255 INFO 2981023 --- [ocp-schedules-14,bd28bf6d831c81dd,81be62c453bc3b5a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 18:28:24.255 INFO 2981023 --- [ocp-schedules-14,bd28bf6d831c81dd,81be62c453bc3b5a] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 18:28:24.281 INFO 2981023 --- [pool-619-thread-3,bd28bf6d831c81dd,567992aaa41ce7d7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:28:24.282 INFO 2981023 --- [pool-619-thread-1,bd28bf6d831c81dd,381b155284215c42] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:28:24.282 INFO 2981023 --- [pool-619-thread-2,bd28bf6d831c81dd,ea938b6a02ab0f46] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:28:24.317 INFO 2981023 --- [pool-619-thread-3,bd28bf6d831c81dd,567992aaa41ce7d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:24.318 INFO 2981023 --- [pool-619-thread-2,bd28bf6d831c81dd,ea938b6a02ab0f46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:24.319 INFO 2981023 --- [pool-619-thread-1,bd28bf6d831c81dd,381b155284215c42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:24.320 INFO 2981023 --- [pool-619-thread-3,bd28bf6d831c81dd,567992aaa41ce7d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726050203000000, 1726050504000000] 2024-09-11 18:28:24.321 INFO 2981023 --- [pool-619-thread-1,bd28bf6d831c81dd,381b155284215c42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726050203000000, 1726050504000000] 2024-09-11 18:28:24.324 INFO 2981023 --- [pool-619-thread-3,bd28bf6d831c81dd,567992aaa41ce7d7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:28:24.328 INFO 2981023 --- [pool-619-thread-2,bd28bf6d831c81dd,ea938b6a02ab0f46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726050203000000, 1726050504000000] 2024-09-11 18:28:24.357 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:24.357 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:24.403 INFO 2981023 --- [pool-619-thread-1,bd28bf6d831c81dd,381b155284215c42] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:28:24.515 INFO 2981023 --- [ocp-schedules-1,a1a8222565292b26,692559a76f4742d1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:28:24.652 INFO 2981023 --- [pool-619-thread-2,bd28bf6d831c81dd,ea938b6a02ab0f46] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:28:24.653 INFO 2981023 --- [ocp-schedules-14,bd28bf6d831c81dd,81be62c453bc3b5a] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 18:28:25.026 INFO 2981023 --- [ocp-schedules-16,8df133df530cb90d,988dfc5568adeefb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:28:25.082 WARN 2981023 --- [scanner-13675079-1,8df133df530cb90d,ee322f64d776b61d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:28:25.099 INFO 2981023 --- [alarm-detect-9,8df133df530cb90d,df312f3f31ce9666] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646438, data is empty. 2024-09-11 18:28:25.099 INFO 2981023 --- [alarm-detect-4,8df133df530cb90d,785014e5a22d9b0c] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050480 ~ 1726050495 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050495, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050495, value=98.938)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.938, values={1726050495=98.938}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050495=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:25.108 INFO 2981023 --- [alarm-detect-9,8df133df530cb90d,df312f3f31ce9666] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646438, data is empty. 2024-09-11 18:28:25.112 INFO 2981023 --- [alarm-detect-9,8df133df530cb90d,df312f3f31ce9666] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646438, data is empty. 2024-09-11 18:28:25.127 INFO 2981023 --- [alarm-detect-10,8df133df530cb90d,de946012c3de16ab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646438, data is empty. 2024-09-11 18:28:25.129 INFO 2981023 --- [alarm-detect-9,8df133df530cb90d,0874fbaa6eac9047] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646438, data is empty. 2024-09-11 18:28:25.133 INFO 2981023 --- [alarm-detect-10,8df133df530cb90d,de946012c3de16ab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646438, data is empty. 2024-09-11 18:28:25.134 INFO 2981023 --- [alarm-detect-9,8df133df530cb90d,0874fbaa6eac9047] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646438, data is empty. 2024-09-11 18:28:25.138 INFO 2981023 --- [alarm-detect-10,8df133df530cb90d,de946012c3de16ab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646438, data is empty. 2024-09-11 18:28:25.139 INFO 2981023 --- [alarm-detect-9,8df133df530cb90d,0874fbaa6eac9047] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646438, data is empty. 2024-09-11 18:28:25.151 INFO 2981023 --- [alarm-detect-3,8df133df530cb90d,d0c0c14c48156c31] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050480 ~ 1726050495 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050495, value=98.938)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.938, values={1726050495=98.938}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:25.170 INFO 2981023 --- [alarm-detect-5,8df133df530cb90d,612fd018b3a0afc3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050480 ~ 1726050495 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050495, value=98.938)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.938, values={1726050495=98.938}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:25.176 INFO 2981023 --- [alarm-detect-18,8df133df530cb90d,0f7c75c08ec1547e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050465 ~ 1726050495 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050480, value=10.14)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=10.14, values={1726050480=10.14}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:25.176 INFO 2981023 --- [alarm-detect-18,8df133df530cb90d,0f7c75c08ec1547e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050465 ~ 1726050495 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050480, value=10.14)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=10.14, values={1726050480=10.14}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:28:25.177 INFO 2981023 --- [alarm-detect-18,8df133df530cb90d,0f7c75c08ec1547e] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:28:25.249 INFO 2981023 --- [alarm-detect-18,8df133df530cb90d,544bf3bc73643caa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646438, data is empty. 2024-09-11 18:28:25.252 INFO 2981023 --- [alarm-detect-18,8df133df530cb90d,544bf3bc73643caa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646438, data is empty. 2024-09-11 18:28:25.254 INFO 2981023 --- [alarm-detect-18,8df133df530cb90d,544bf3bc73643caa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646438, data is empty. 2024-09-11 18:28:25.296 INFO 2981023 --- [alarm-detect-1,8df133df530cb90d,cb5b5fff099ac605] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050465 ~ 1726050495 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050480, value=105.795)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.795, values={1726050480=105.795}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:25.296 INFO 2981023 --- [alarm-detect-1,8df133df530cb90d,cb5b5fff099ac605] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_cpu_usage_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=rtdob-1718689631:tenant_name=linkreit:host=10.31.35.5-2] 2024-09-11 18:28:25.471 INFO 2981023 --- [ocp-async-9,d8fdc672c0f8b1f8,238cdc52ccb2ed0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,239a9660b26c6cf1,1d1b3068b74cd6fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,239a9660b26c6cf1,1d1b3068b74cd6fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:26.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:28:26.353910+08:00, lastProcessTime=2024-09-11T18:28:16.383273+08:00 2024-09-11 18:28:26.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=2, resultsCount=2 2024-09-11 18:28:26.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:28:26.353910+08:00 2024-09-11 18:28:27.358 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:27.359 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:27.471 INFO 2981023 --- [ocp-async-2,068397165d4c035d,f166ad53df3b1df9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:27.797 INFO 2981023 --- [ocp-async-5,2fb93aad51c9b9a4,f4273ec3ff503aea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:28:27.821 INFO 2981023 --- [ocp-async-5,2fb93aad51c9b9a4,f4273ec3ff503aea] c.o.o.a.s.OcpAlarmSnapshotServiceImpl : alarm type ob_tenant_cpu_usage_over_threshold 's snapshots already exist. 2024-09-11 18:28:28.093 INFO 2981023 --- [ocp-schedules-19,14684a382e3533e2,3fd0f1560cd3cc81] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:28:28.093 INFO 2981023 --- [ocp-schedules-19,14684a382e3533e2,3fd0f1560cd3cc81] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:28:28.093 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:28:28.093 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:28:28.094 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:28:28.113 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:28:28.116 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:28:28.118 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:28:28.120 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:28:28.123 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:28:28.126 INFO 2981023 --- [ocp-schedules-19,14684a382e3533e2,3fd0f1560cd3cc81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:28:28.127 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:28:28.135 INFO 2981023 --- [ocp-schedules-19,14684a382e3533e2,3fd0f1560cd3cc81] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:28:28.187 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:28:28.262 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:28:28.266 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:28:28.276 INFO 2981023 --- [ocp-schedules-19,14684a382e3533e2,3fd0f1560cd3cc81] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:28:28.278 INFO 2981023 --- [ocp-schedules-19,14684a382e3533e2,3fd0f1560cd3cc81] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:28:28.956 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.958 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obCl 2024-09-11 18:28:28.964 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.964 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:28:28.966 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.969 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.974 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.975 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.975 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:28:28.976 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.977 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.978 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.979 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.979 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.980 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.981 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.982 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.985 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.986 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:28:28.987 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:28.991 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 10 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.024 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.031 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.046 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 41 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.110 INFO 2981023 --- [sql_perf_diagnoser5,,] .s.d.i.d.PerformanceDegradationDiagnoser : Auto flushing plan cache is disabled 2024-09-11 18:28:29.123 INFO 2981023 --- [sql_perf_diagnoser7,,] .s.d.i.d.PerformanceDegradationDiagnoser : Auto flushing plan cache is disabled 2024-09-11 18:28:29.127 INFO 2981023 --- [sql_perf_diagnoser5,,] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007116 2024-09-11 18:28:29.137 INFO 2981023 --- [sql_perf_diagnoser7,,] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=6007117 2024-09-11 18:28:29.149 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 9 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.160 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 220 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.172 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.175 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.176 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.177 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.178 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.179 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:22:28.673, end=2024-09-11T18:27:28.951} 2024-09-11 18:28:29.179 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 11 dubious sqls 2024-09-11 18:28:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,bfc14da1fa90913c,a366a5ae54ecdf9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,bfc14da1fa90913c,a366a5ae54ecdf9f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:29.471 INFO 2981023 --- [ocp-async-1,61d9adee8e38d025,8c28df7f4d139bea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:30.360 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:30.360 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:31.471 INFO 2981023 --- [ocp-async-4,3a9a354ec47d41fb,d55775a78a9e212a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,9d47c20502c216c7,c67ce0935bd3961b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,9d47c20502c216c7,c67ce0935bd3961b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:32.860 INFO 2981023 --- [ocp-schedules-10,5b773bdfba17ae38,833391f23936030e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:28:33.360 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:33.361 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:33.471 INFO 2981023 --- [ocp-async-8,fa76be16e58f80b3,df0f9eb8288347a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,35675942dbb1da8d,ac5a291cab080a1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,35675942dbb1da8d,ac5a291cab080a1e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:35.331 INFO 2981023 --- [ocp-schedules-24,bd5eacfebc5e79f2,b4f2005ea68f3b18] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:28:35.371 WARN 2981023 --- [scanner-13675369-1,bd5eacfebc5e79f2,15169620c11450c0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:28:35.429 INFO 2981023 --- [alarm-detect-7,bd5eacfebc5e79f2,e09566ba4ced5bdb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050480 ~ 1726050505 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050495, value=9.003)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=9.003, values={1726050495=9.003}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:35.429 INFO 2981023 --- [alarm-detect-7,bd5eacfebc5e79f2,e09566ba4ced5bdb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050480 ~ 1726050505 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050495, value=9.003)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=9.003, values={1726050495=9.003}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:28:35.471 INFO 2981023 --- [ocp-async-10,d0591e045b878b59,6435d3c20d94c7ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:35.509 INFO 2981023 --- [alarm-detect-8,bd5eacfebc5e79f2,fcacb7664cf62373] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050480 ~ 1726050505 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050495, value=101.304)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=101.304, values={1726050495=101.304}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:35.854 INFO 2981023 --- [ocp-schedules-26,10fa4c5192372b0a,d81c32f512530902] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:28:35.862 INFO 2981023 --- [ocp-schedules-26,10fa4c5192372b0a,d81c32f512530902] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:28:35.862 INFO 2981023 --- [ocp-schedules-26,10fa4c5192372b0a,d81c32f512530902] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:28:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:28:36.337449+08:00, lastProcessTime=2024-09-11T18:28:26.353910+08:00 2024-09-11 18:28:36.361 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:36.362 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:36.458 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=2, resultsCount=2 2024-09-11 18:28:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:28:36.337449+08:00 2024-09-11 18:28:37.314 INFO 2981023 --- [prometheus-cache-monitor1,9553fef846d072bf,07a2c042d9590a03] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:28:37.315 INFO 2981023 --- [prometheus-cache-monitor1,9553fef846d072bf,07a2c042d9590a03] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:28:37.472 INFO 2981023 --- [ocp-async-5,b3c381ea77995a9f,400421c49d8e684b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:37.798 INFO 2981023 --- [ocp-async-8,a9ff1b9f979a0938,d31874fb4e305bf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:28:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,de8c35bde00312be,d2badfd829f32967] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,de8c35bde00312be,d2badfd829f32967] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:39.362 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:39.363 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:39.472 INFO 2981023 --- [ocp-async-2,4ddbfeb5df1ffd39,4ccf6d12ee3bd6b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:40.003 INFO 2981023 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:28:40.003 INFO 2981023 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:28:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,20aee96cf565297f,1483977e9e3579f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:41.318 INFO 2981023 --- [prometheus-minute-schedule-1,20aee96cf565297f,1483977e9e3579f3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:41.472 INFO 2981023 --- [ocp-async-3,89fc767a64966469,06ba50216147d07c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:42.363 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:42.364 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:42.863 INFO 2981023 --- [ocp-schedules-15,22e442cc27b8968c,a21756febf5c88a9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:28:43.472 INFO 2981023 --- [ocp-async-1,8effc5525ec81d34,9029b334e7295b18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,959039fa2aaa7cde,d1b05652e12cf67e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,959039fa2aaa7cde,d1b05652e12cf67e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:45.101 INFO 2981023 --- [prometheus-second-schedule-1,96f75ee98a2cc7bc,0e342f96785744c3] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:28:45.364 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:45.366 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:45.472 INFO 2981023 --- [ocp-async-4,368315114b1c41d1,51e08d5167d23e14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:45.518 INFO 2981023 --- [ocp-schedules-20,999cb8251950ac06,7bf6c5c359b50d8c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:28:45.567 WARN 2981023 --- [scanner-13675611-1,999cb8251950ac06,f662c4d587b9d035] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:28:45.596 INFO 2981023 --- [alarm-detect-1,999cb8251950ac06,536844ecb776f3e3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646438, data is empty. 2024-09-11 18:28:45.600 INFO 2981023 --- [alarm-detect-1,999cb8251950ac06,536844ecb776f3e3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646438, data is empty. 2024-09-11 18:28:45.604 INFO 2981023 --- [alarm-detect-1,999cb8251950ac06,536844ecb776f3e3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646438, data is empty. 2024-09-11 18:28:45.612 INFO 2981023 --- [alarm-detect-3,999cb8251950ac06,f9978f1ed672e630] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050495 ~ 1726050515 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050510, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050510, value=98.104)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.104, values={1726050510=98.104}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050510=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:45.623 INFO 2981023 --- [alarm-detect-9,999cb8251950ac06,608bcfea66cf62e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646438, data is empty. 2024-09-11 18:28:45.626 INFO 2981023 --- [alarm-detect-20,999cb8251950ac06,72447efb6444a1a7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646438, data is empty. 2024-09-11 18:28:45.627 INFO 2981023 --- [alarm-detect-9,999cb8251950ac06,608bcfea66cf62e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646438, data is empty. 2024-09-11 18:28:45.629 INFO 2981023 --- [alarm-detect-20,999cb8251950ac06,72447efb6444a1a7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646438, data is empty. 2024-09-11 18:28:45.630 INFO 2981023 --- [alarm-detect-9,999cb8251950ac06,608bcfea66cf62e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646438, data is empty. 2024-09-11 18:28:45.633 INFO 2981023 --- [alarm-detect-20,999cb8251950ac06,72447efb6444a1a7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646438, data is empty. 2024-09-11 18:28:45.635 INFO 2981023 --- [alarm-detect-3,999cb8251950ac06,8b07d043a62190bb] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050495 ~ 1726050515 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050510, value=98.104)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.104, values={1726050510=98.104}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:45.682 INFO 2981023 --- [alarm-detect-18,999cb8251950ac06,32da6650fa8d27a7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050495 ~ 1726050515 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050510, value=8.791)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.791, values={1726050510=8.791}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:45.682 INFO 2981023 --- [alarm-detect-18,999cb8251950ac06,32da6650fa8d27a7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050495 ~ 1726050515 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050510, value=8.791)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.791, values={1726050510=8.791}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:28:45.685 INFO 2981023 --- [alarm-detect-2,999cb8251950ac06,36448c94c381271c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050495 ~ 1726050515 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050510, value=98.104)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.104, values={1726050510=98.104}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:45.738 INFO 2981023 --- [alarm-detect-9,999cb8251950ac06,c01be5a303099ddd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646438, data is empty. 2024-09-11 18:28:45.740 INFO 2981023 --- [alarm-detect-13,999cb8251950ac06,48dd385b77893cc3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050495 ~ 1726050515 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050510, value=99.78)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=99.78, values={1726050510=99.78}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:45.744 INFO 2981023 --- [alarm-detect-9,999cb8251950ac06,c01be5a303099ddd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646438, data is empty. 2024-09-11 18:28:45.752 INFO 2981023 --- [alarm-detect-9,999cb8251950ac06,c01be5a303099ddd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646438, data is empty. 2024-09-11 18:28:46.022 INFO 2981023 --- [ocp-schedules-3,9c5e28fdf1b5d1d4,a14da1daa91cb26d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:28:46.022 INFO 2981023 --- [ocp-schedules-17,78e4ce728e2d6062,043279518e1e9884] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:28:46.023 INFO 2981023 --- [host-check-clockdiff1,78e4ce728e2d6062,6a4ba45b1279a882] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:28:46.023 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:28:46.025 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:28:46.028 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:28:46.028 INFO 2981023 --- [host-check-clockdiff2,78e4ce728e2d6062,761a338c4f183b5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:28:46.028 INFO 2981023 --- [host-check-clockdiff3,78e4ce728e2d6062,2cacbe90958d26d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:28:46.029 INFO 2981023 --- [host-check-clockdiff4,78e4ce728e2d6062,f6ce7c2eb4f38aa0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:28:46.031 INFO 2981023 --- [ocp-schedules-18,0ee30202757b2f18,db0105732432dc55] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:28:46.046 INFO 2981023 --- [host-check-agent-status1,0ee30202757b2f18,c9e860989e1a2e31] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:28:46.050 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:28:46.052 INFO 2981023 --- [host-check-clockdiff2,78e4ce728e2d6062,761a338c4f183b5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050526 6 6 , error= 2024-09-11 18:28:46.053 INFO 2981023 --- [host-check-clockdiff3,78e4ce728e2d6062,2cacbe90958d26d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050526 0 0 , error= 2024-09-11 18:28:46.053 INFO 2981023 --- [host-check-agent-status2,0ee30202757b2f18,9d475db0a33d086b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:28:46.060 INFO 2981023 --- [host-check-agent-status2,0ee30202757b2f18,9d475db0a33d086b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:28:46.064 INFO 2981023 --- [host-check-agent-status4,0ee30202757b2f18,670921000d1f8c51] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:28:46.064 INFO 2981023 --- [host-check-agent-status2,0ee30202757b2f18,9d475db0a33d086b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:28:46.064 INFO 2981023 --- [host-check-agent-status3,0ee30202757b2f18,a55cc9491c45a31c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:28:46.065 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:28:46.067 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:28:46.067 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:28:46.068 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:28:46.069 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:28:46.069 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:28:46.070 INFO 2981023 --- [host-check-agent-status1,0ee30202757b2f18,c9e860989e1a2e31] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:28:46.074 INFO 2981023 --- [host-check-agent-status1,0ee30202757b2f18,c9e860989e1a2e31] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:28:46.095 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:28:46.111 INFO 2981023 --- [host-check-agent-status4,0ee30202757b2f18,670921000d1f8c51] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:28:46.114 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:28:46.171 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:28:46.178 INFO 2981023 --- [host-check-agent-status3,0ee30202757b2f18,a55cc9491c45a31c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:28:46.187 INFO 2981023 --- [ocp-schedules-28,0d3302b480980e18,f2cc2f0d6f5bf3c2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:28:46.187 INFO 2981023 --- [ocp-schedules-7,4f51e50cb83ebe2e,747145645985838b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:28:46.194 INFO 2981023 --- [ocp-schedules-21,15c42fccfe8af254,a5d0142a021e55cf] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:28:46.195 INFO 2981023 --- [ocp-schedules-12,e19f316e46d522a7,fb1ce8ff5b82e460] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:28:46.205 INFO 2981023 --- [host-check-agent-status4,0ee30202757b2f18,670921000d1f8c51] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:28:46.269 INFO 2981023 --- [ocp-schedules-27,a0c1405c3f48bb08,2824e051e17b1511] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:28:46.271 INFO 2981023 --- [host-check-agent-status3,0ee30202757b2f18,a55cc9491c45a31c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:28:46.274 INFO 2981023 --- [ocp-schedules-21,15c42fccfe8af254,a5d0142a021e55cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:28:46.277 INFO 2981023 --- [ocp-schedules-28,0d3302b480980e18,f2cc2f0d6f5bf3c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:28:46.278 INFO 2981023 --- [ocp-schedules-7,4f51e50cb83ebe2e,747145645985838b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:28:46.279 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050483298000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:28:46.296 INFO 2981023 --- [ocp-schedules-7,4f51e50cb83ebe2e,747145645985838b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:28:46.380 INFO 2981023 --- [ocp-schedules-27,a0c1405c3f48bb08,2824e051e17b1511] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:28:46.456 INFO 2981023 --- [ocp-schedules-22,37f19405c5f5e40d,3db077496380694e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:28:46.462 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:28:46.465 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:28:46.469393691+08:00 2024-09-11 18:28:46.473 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:28:46.465+08:00 2024-09-11 18:28:46.476 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:28:46.478 INFO 2981023 --- [host-check-agent-status2,0ee30202757b2f18,9d475db0a33d086b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:28:46.478 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:28:46.465+08:00 and status AVAILABLE 2024-09-11 18:28:46.480 INFO 2981023 --- [host-check-agent-status3,0ee30202757b2f18,a55cc9491c45a31c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:28:46.484 INFO 2981023 --- [host-check-agent-status2,0ee30202757b2f18,9d475db0a33d086b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:28:46.484 INFO 2981023 --- [host-check-agent-status2,0ee30202757b2f18,9d475db0a33d086b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:28:46.484 INFO 2981023 --- [host-check-agent-status2,0ee30202757b2f18,9d475db0a33d086b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:28:46.486 INFO 2981023 --- [host-check-agent-status1,0ee30202757b2f18,c9e860989e1a2e31] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:28:46.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:28:46.485695+08:00, lastProcessTime=2024-09-11T18:28:36.337449+08:00 2024-09-11 18:28:46.489 INFO 2981023 --- [host-check-agent-status3,0ee30202757b2f18,a55cc9491c45a31c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:28:46.489 INFO 2981023 --- [host-check-agent-status3,0ee30202757b2f18,a55cc9491c45a31c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:28:46.489 INFO 2981023 --- [host-check-agent-status3,0ee30202757b2f18,a55cc9491c45a31c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:28:46.492 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:28:46.494 INFO 2981023 --- [host-check-agent-status1,0ee30202757b2f18,c9e860989e1a2e31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:28:46.494 INFO 2981023 --- [host-check-agent-status1,0ee30202757b2f18,c9e860989e1a2e31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:28:46.494 INFO 2981023 --- [host-check-agent-status1,0ee30202757b2f18,c9e860989e1a2e31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:28:46.501 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:28:46.504 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:28:46.503194165+08:00 2024-09-11 18:28:46.505 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:28:46.504+08:00 2024-09-11 18:28:46.509 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:28:46.504+08:00 and status AVAILABLE 2024-09-11 18:28:46.509 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:28:46.509 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:28:46.509762407+08:00 2024-09-11 18:28:46.518 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:28:46.522 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:28:46.522488744+08:00 2024-09-11 18:28:46.557 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:28:46.509+08:00 2024-09-11 18:28:46.559 INFO 2981023 --- [host-check-agent-status4,0ee30202757b2f18,670921000d1f8c51] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:28:46.560 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:28:46.522+08:00 2024-09-11 18:28:46.561 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:28:46.563 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:28:46.522+08:00 and status AVAILABLE 2024-09-11 18:28:46.563 INFO 2981023 --- [host-check-agent-status4,0ee30202757b2f18,670921000d1f8c51] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:28:46.563 INFO 2981023 --- [host-check-agent-status4,0ee30202757b2f18,670921000d1f8c51] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:28:46.563 INFO 2981023 --- [host-check-agent-status4,0ee30202757b2f18,670921000d1f8c51] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:28:46.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.564 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.564 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:28:46.564 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:28:46.509+08:00 and status AVAILABLE 2024-09-11 18:28:46.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:46.565 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:28:46.569 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:28:46.571 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:28:46.572 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:28:46.573 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:28:46.574 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:28:46.578 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.579 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.580 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:46.581 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:28:46.582 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:28:46.583 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:28:46.587 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:28:46.590 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:28:46.591 INFO 2981023 --- [ocp-schedules-28,0d3302b480980e18,f2cc2f0d6f5bf3c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:28:46.595 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:28:46.595 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:28:46.600 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:28:46.607 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:28:46.608 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.611 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:28:46.611 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.612 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.612 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:28:46.614 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:46.616 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.622 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:28:46.628 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:28:46.630 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:28:46.635 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:28:46.637 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.650 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:28:46.664 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:28:46.664 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050483298000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:28:46.664 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:28:46.667 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.668 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.668 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:46.669 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:28:46.669 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.672 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:28:46.673 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:28:46.673 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:28:46.674 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:28:46.691 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:28:46.691 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:28:46.697 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:28:46.712 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:28:46.714 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.720 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:28:46.726 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:28:46.729 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.732 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:28:46.738 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:28:46.741 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.743 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:28:46.762 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:28:46.763 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.765 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:28:46.765 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.766 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:28:46.766 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:28:46.766 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:28:46.771 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:28:46.772 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:28:46.781 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:28:46.782 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:28:46.789 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:28:46.797 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:28:46.798 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:28:46.801 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:28:46.803 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.803 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.804 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:28:46.807 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:28:46.809 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:28:46.810 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:28:46.817 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:28:46.820 INFO 2981023 --- [host-check-host-status4,9c5e28fdf1b5d1d4,f6dcc7479201cdd7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:28:46.822 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:28:46.840 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.846 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:28:46.851 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:28:46.859 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.860 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:28:46.861 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.878 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:28:46.880 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:28:46.880 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:28:46.882 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.886 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.886 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.887 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.887 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:28:46.889 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:28:46.890 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:28:46.891 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:28:46.891 INFO 2981023 --- [host-check-host-status3,9c5e28fdf1b5d1d4,e48ede44b87d91fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:28:46.891 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:28:46.894 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:28:46.894 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:28:46.895 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:28:46.895 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.895 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.902 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:28:46.904 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:28:46.908 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:28:46.908 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.908 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:28:46.908 INFO 2981023 --- [host-check-host-status2,9c5e28fdf1b5d1d4,db27647914380521] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:28:46.915 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:28:46.915 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:28:46.921 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:28:46.926 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:28:46.485695+08:00 2024-09-11 18:28:46.926 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:28:46.932 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.938 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.940 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:28:46.946 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:28:46.950 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:28:46.952 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:46.953 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:28:46.956 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:28:46.956 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:28:46.969 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:28:46.974 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.975 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:28:46.984 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:28:46.984 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:46.995 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:28:47.013 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:28:47.026 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:47.044 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:47.055 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:28:47.056 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:28:47.056 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:47.094 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:28:47.100 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:28:47.101 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:28:47.118 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:28:47.145 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:28:47.152 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:28:47.154 INFO 2981023 --- [ocp-schedules-29,6988f4061dfc79e4,f1528cb4b4562819] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:28:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,cfbcc91dc763a7f9,bf6306c18bf0eb7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:47.317 INFO 2981023 --- [prometheus-minute-schedule-1,cfbcc91dc763a7f9,bf6306c18bf0eb7c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:47.354 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:28:47.360 INFO 2981023 --- [host-check-host-status1,9c5e28fdf1b5d1d4,12e8531db96a6dd4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:28:47.415 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:28:47.420 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:28:47.428 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:47.433 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:28:47.442 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:28:47.442 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:28:47.442 INFO 2981023 --- [ocp-schedules-21,15c42fccfe8af254,a5d0142a021e55cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:28:47.472 INFO 2981023 --- [ocp-async-10,dc9ddda1decc6fdb,ea1c49e235caf7ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:47.798 INFO 2981023 --- [ocp-async-4,01843b5346d39cd7,bf8d568233870c12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:28:48.366 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:48.367 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:48.407 INFO 2981023 --- [ocp-schedules-3,3e799d249082a096,01c9c9e0e2c1dff4] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:28:48.426 WARN 2981023 --- [ocp-schedules-3,3e799d249082a096,01c9c9e0e2c1dff4] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:28:48.563 INFO 2981023 --- [host-check-clockdiff1,78e4ce728e2d6062,6a4ba45b1279a882] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050528 0 0 , error= 2024-09-11 18:28:49.472 INFO 2981023 --- [ocp-async-1,32f92ec0b2195fe8,6fac84b46143eef3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:28:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:28:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,2bec3f1d72c788ad,5b94dd663f2558e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,2bec3f1d72c788ad,5b94dd663f2558e8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:50.377 INFO 2981023 --- [ocp-schedules-29,d4dbb6e219c192d6,9caf505574cde6d1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:28:51.367 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:51.367 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:51.472 INFO 2981023 --- [ocp-async-7,11e3983025338501,51d37a5b1a51ac06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:51.646 INFO 2981023 --- [ocp-schedules-28,481be3c0413f2726,7c2d221ad1021a5b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:28:51.647 INFO 2981023 --- [ocp-schedules-28,481be3c0413f2726,7c2d221ad1021a5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:28:51.647 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:28:51.647 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:28:51.648 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:28:51.648 INFO 2981023 --- [ocp-schedules-28,481be3c0413f2726,7c2d221ad1021a5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:28:52.006 INFO 2981023 --- [host-check-clockdiff4,78e4ce728e2d6062,f6ce7c2eb4f38aa0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050532 0 0 , error= 2024-09-11 18:28:52.866 INFO 2981023 --- [ocp-schedules-21,2b5e8a97e1a10e69,a846f703e0948b17] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:28:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,437b548b7cbe29ad,0fe0d95ebae1a3f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,437b548b7cbe29ad,0fe0d95ebae1a3f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:53.395 INFO 2981023 --- [ocp-schedules-30,c2049194d893523b,18225a0deeb0fb54] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:28:53.395 INFO 2981023 --- [ocp-schedules-30,c2049194d893523b,18225a0deeb0fb54] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:28:53.473 INFO 2981023 --- [ocp-async-3,4f57533fae4579fa,41a6e037fd16ae44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:54.367 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:54.368 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:54.516 INFO 2981023 --- [ocp-schedules-14,f75f52e73a8667fa,b77b5537b0a68e21] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:28:55.000 INFO 2981023 --- [ocp-schedules-29,9391514b7f28a926,fb528bd1ea7ea69f] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T18:28:55+08:00, tenantSize=6 2024-09-11 18:28:55.002 INFO 2981023 --- [ocp-schedules-29,9391514b7f28a926,fb528bd1ea7ea69f] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726050535000000 2024-09-11 18:28:55.002 INFO 2981023 --- [ocp-schedules-29,9391514b7f28a926,fb528bd1ea7ea69f] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726050234960000 2024-09-11 18:28:55.016 INFO 2981023 --- [ocp-schedules-29,9391514b7f28a926,fb528bd1ea7ea69f] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 18:28:55.473 INFO 2981023 --- [ocp-async-5,8ce1710b52c0663d,96041b1c52c5bcec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:55.502 INFO 2981023 --- [ocp-schedules-1,5ce3148de2c41b73,244337566d9a51f8] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:28:55.519 WARN 2981023 --- [ocp-schedules-1,5ce3148de2c41b73,244337566d9a51f8] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:28:55.794 INFO 2981023 --- [ocp-schedules-4,1be946c43c03bd1e,8044eadd0910354f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:28:55.835 WARN 2981023 --- [scanner-13675937-1,1be946c43c03bd1e,008a2ca7eb6c0b70] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:28:55.859 INFO 2981023 --- [alarm-detect-20,1be946c43c03bd1e,d502991d6e93a759] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050510 ~ 1726050525 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050525, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050525, value=97.588)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=97.588, values={1726050525=97.588}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050525=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:55.871 INFO 2981023 --- [ocp-schedules-16,bc12759d171f3791,d42731169dcfa022] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:28:55.893 INFO 2981023 --- [ocp-schedules-16,bc12759d171f3791,d42731169dcfa022] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:28:55.893 INFO 2981023 --- [ocp-schedules-16,bc12759d171f3791,d42731169dcfa022] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:28:55.902 INFO 2981023 --- [alarm-detect-2,1be946c43c03bd1e,3dee40165ec15b94] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050510 ~ 1726050525 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050525, value=97.588)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=97.588, values={1726050525=97.588}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:28:55.927 INFO 2981023 --- [ocp-schedules-18,912607a1db061f9c,23d43a31f79aa24b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:28:55.927 INFO 2981023 --- [ocp-schedules-18,912607a1db061f9c,23d43a31f79aa24b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:28:55.930 INFO 2981023 --- [ocp-schedules-18,912607a1db061f9c,23d43a31f79aa24b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:28:55.930 INFO 2981023 --- [ocp-schedules-18,912607a1db061f9c,23d43a31f79aa24b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:28:55.930 INFO 2981023 --- [ocp-schedules-18,912607a1db061f9c,23d43a31f79aa24b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:28:55.930 INFO 2981023 --- [ocp-schedules-18,912607a1db061f9c,23d43a31f79aa24b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:28:56.157 INFO 2981023 --- [ocp-schedules-19,f07ebe14e4dfb8a9,e33f4e4dd8725b59] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:28:56.163 INFO 2981023 --- [ocp-schedules-19,f07ebe14e4dfb8a9,e33f4e4dd8725b59] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:28:56.163 INFO 2981023 --- [ocp-schedules-19,f07ebe14e4dfb8a9,e33f4e4dd8725b59] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:28:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,f6fe60ea7bf17ac2,3a862dfc977a42f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,f6fe60ea7bf17ac2,3a862dfc977a42f9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:56.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:28:56.337295+08:00, lastProcessTime=2024-09-11T18:28:46.485695+08:00 2024-09-11 18:28:56.464 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:28:56.337295+08:00 2024-09-11 18:28:57.368 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:28:57.369 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:28:57.473 INFO 2981023 --- [ocp-async-6,a01b289b5001d688,3256fa9e5ce6b820] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:28:57.798 INFO 2981023 --- [ocp-async-9,723d620496b3e0fc,d15a5804c5f6205f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:28:58.972 INFO 2981023 --- [ocp-schedules-19,13cfd94e0fe582ce,78a85c721b3490bc] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:28:58.972 INFO 2981023 --- [ocp-schedules-19,13cfd94e0fe582ce,78a85c721b3490bc] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:28:58.988 INFO 2981023 --- [Collect all tenants major compactions1,13cfd94e0fe582ce,9f77b99975669077] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:28:58.989 INFO 2981023 --- [Collect all tenants major compactions2,13cfd94e0fe582ce,852623c22d6cc6eb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:28:58.990 INFO 2981023 --- [Collect all tenants major compactions5,13cfd94e0fe582ce,45419518a36b6265] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:28:58.990 INFO 2981023 --- [Collect all tenants major compactions6,13cfd94e0fe582ce,5ff123df240ca6d2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:28:58.992 INFO 2981023 --- [Collect all tenants major compactions4,13cfd94e0fe582ce,a4287d36cf7121e7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:28:58.993 INFO 2981023 --- [Collect all tenants major compactions3,13cfd94e0fe582ce,5a7345805587dcbd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:28:59.003 INFO 2981023 --- [Collect all tenants major compactions6,13cfd94e0fe582ce,5ff123df240ca6d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:59.003 INFO 2981023 --- [Collect all tenants major compactions2,13cfd94e0fe582ce,852623c22d6cc6eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:59.004 INFO 2981023 --- [Collect all tenants major compactions2,13cfd94e0fe582ce,852623c22d6cc6eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:28:59.004 INFO 2981023 --- [Collect all tenants major compactions6,13cfd94e0fe582ce,5ff123df240ca6d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:28:59.005 INFO 2981023 --- [Collect all tenants major compactions3,13cfd94e0fe582ce,5a7345805587dcbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:59.005 INFO 2981023 --- [Collect all tenants major compactions3,13cfd94e0fe582ce,5a7345805587dcbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:28:59.006 INFO 2981023 --- [Collect all tenants major compactions1,13cfd94e0fe582ce,9f77b99975669077] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:59.007 INFO 2981023 --- [Collect all tenants major compactions2,13cfd94e0fe582ce,852623c22d6cc6eb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:28:59.007 INFO 2981023 --- [Collect all tenants major compactions2,13cfd94e0fe582ce,852623c22d6cc6eb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:28:59.008 INFO 2981023 --- [Collect all tenants major compactions6,13cfd94e0fe582ce,5ff123df240ca6d2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:28:59.008 INFO 2981023 --- [Collect all tenants major compactions6,13cfd94e0fe582ce,5ff123df240ca6d2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:28:59.008 INFO 2981023 --- [Collect all tenants major compactions3,13cfd94e0fe582ce,5a7345805587dcbd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:28:59.008 INFO 2981023 --- [Collect all tenants major compactions3,13cfd94e0fe582ce,5a7345805587dcbd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:28:59.008 INFO 2981023 --- [Collect all tenants major compactions1,13cfd94e0fe582ce,9f77b99975669077] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:28:59.009 INFO 2981023 --- [Collect all tenants major compactions1,13cfd94e0fe582ce,9f77b99975669077] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:28:59.009 INFO 2981023 --- [Collect all tenants major compactions1,13cfd94e0fe582ce,9f77b99975669077] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:28:59.016 INFO 2981023 --- [Collect all tenants major compactions5,13cfd94e0fe582ce,45419518a36b6265] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:59.026 INFO 2981023 --- [Collect all tenants major compactions4,13cfd94e0fe582ce,a4287d36cf7121e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:28:59.027 INFO 2981023 --- [Collect all tenants major compactions5,13cfd94e0fe582ce,45419518a36b6265] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:28:59.035 INFO 2981023 --- [Collect all tenants major compactions4,13cfd94e0fe582ce,a4287d36cf7121e7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:28:59.035 INFO 2981023 --- [Collect all tenants major compactions5,13cfd94e0fe582ce,45419518a36b6265] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:28:59.035 INFO 2981023 --- [Collect all tenants major compactions5,13cfd94e0fe582ce,45419518a36b6265] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:28:59.041 INFO 2981023 --- [Collect all tenants major compactions4,13cfd94e0fe582ce,a4287d36cf7121e7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:28:59.041 INFO 2981023 --- [Collect all tenants major compactions4,13cfd94e0fe582ce,a4287d36cf7121e7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:28:59.042 INFO 2981023 --- [ocp-schedules-19,13cfd94e0fe582ce,78a85c721b3490bc] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:28:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,91b3e985d738c148,9bc1bbcc2c6e015e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050540, interval=60 2024-09-11 18:28:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,91b3e985d738c148,9bc1bbcc2c6e015e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:28:59.473 INFO 2981023 --- [ocp-async-3,4db9bbc02e2bfdef,516cec1692e6968e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:00.369 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:00.370 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:00.982 INFO 2981023 --- [ocp-schedules-24,e3fbf0c4240c963e,3e6bd563835332b5] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:29:00.982+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:29:00.983 INFO 2981023 --- [ocp-schedules-24,e3fbf0c4240c963e,3e6bd563835332b5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:29:01.039 INFO 2981023 --- [ocp-schedules-24,e3fbf0c4240c963e,3e6bd563835332b5] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 53 to SqlAggregateData 2024-09-11 18:29:01.040 INFO 2981023 --- [ocp-schedules-24,e3fbf0c4240c963e,3e6bd563835332b5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:29:01.473 INFO 2981023 --- [ocp-async-5,4340098f9d64731b,075cbb0452116789] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,10f0cbe063ab0a72,a48bc12bba20339d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,10f0cbe063ab0a72,a48bc12bba20339d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:02.868 INFO 2981023 --- [ocp-schedules-10,60caa7225fda2214,e73de35f4cbd25cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:29:03.370 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:03.370 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:03.473 INFO 2981023 --- [ocp-async-7,f413cb1dda6855e9,82186cb2090977e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:03.694 INFO 2981023 --- [ocp-schedules-12,b8d9cf2cf45d61af,552c92a09140756d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:29:03.695 INFO 2981023 --- [ocp-schedules-12,b8d9cf2cf45d61af,552c92a09140756d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:29:03.722 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:29:03.723 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.725 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:29:03.725 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:29:03.726 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:29:03.726 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.730 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.730 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:29:03.735 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.735 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.735 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.735 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.736 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:29:03.740 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.740 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:29:03.747 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.747 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:29:03.750 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.751 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.752 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.753 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:29:03.757 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.758 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:29:03.760 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.761 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.768 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.768 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:29:03.769 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.769 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.769 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:29:03.769 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:29:03.770 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.770 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.770 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:29:03.770 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:29:03.771 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.771 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.771 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:29:03.771 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:29:03.772 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:29:03.772 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.773 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:29:03.773 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.773 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:29:03.774 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.775 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:29:03.775 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.775 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:29:03.776 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:29:03.776 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.777 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:29:03.777 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:29:03.778 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:29:03.789 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:03.789 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:03.790 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.790 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.790 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:29:03.791 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.791 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.795 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:29:03.796 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.798 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:29:03.798 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:29:03.803 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:29:03.804 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:29:03.804 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:29:03.805 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:29:03.805 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:29:03.806 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:29:03.806 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.807 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.811 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:29:03.812 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:29:03.812 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:29:03.812 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:29:03.817 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.822 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.839 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.844 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.848 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:29:03.849 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE69C1D6F2-0-0] 2024-09-11 18:29:03.849 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:29:03.857 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.860 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:29:03.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.889 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:29:03.909 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.910 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:29:03.910 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:29:03.910 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE69E1D6DD-0-0] 2024-09-11 18:29:03.911 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:29:03.919 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.920 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:29:03.928 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.929 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:29:03.929 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.930 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:29:03.931 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.931 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:29:03.932 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.932 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:29:03.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:29:03.934 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:29:03.935 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:29:03.944 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:03.957 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.957 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:29:03.967 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:29:03.968 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:29:03.968 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:03.969 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:29:03.978 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:29:03.979 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:29:03.979 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:29:03.979 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:29:03.979 INFO 2981023 --- [ocp-schedules-12,b8d9cf2cf45d61af,552c92a09140756d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:29:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,0d234de612d9fe65,cf801e3723ffccd2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,0d234de612d9fe65,cf801e3723ffccd2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:05.473 INFO 2981023 --- [ocp-async-10,947d35d5d3a1c1d9,8f719ddf8c1eff5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:06.002 INFO 2981023 --- [ocp-schedules-17,e6310f1ed9200069,083ae005db0383d5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:29:06.061 WARN 2981023 --- [scanner-13676155-1,e6310f1ed9200069,86019674821cfcb2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:29:06.062 INFO 2981023 --- [alarm-detect-14,e6310f1ed9200069,71e72cb05c0110f0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646438, data is empty. 2024-09-11 18:29:06.066 INFO 2981023 --- [alarm-detect-14,e6310f1ed9200069,71e72cb05c0110f0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646491, data is empty. 2024-09-11 18:29:06.070 INFO 2981023 --- [alarm-detect-14,e6310f1ed9200069,71e72cb05c0110f0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646491, data is empty. 2024-09-11 18:29:06.088 INFO 2981023 --- [alarm-detect-8,e6310f1ed9200069,da9528c95b7b87d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646438, data is empty. 2024-09-11 18:29:06.091 INFO 2981023 --- [alarm-detect-19,e6310f1ed9200069,8f180c96fe05fbde] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646438, data is empty. 2024-09-11 18:29:06.091 INFO 2981023 --- [alarm-detect-8,e6310f1ed9200069,da9528c95b7b87d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646491, data is empty. 2024-09-11 18:29:06.094 INFO 2981023 --- [alarm-detect-19,e6310f1ed9200069,8f180c96fe05fbde] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646491, data is empty. 2024-09-11 18:29:06.096 INFO 2981023 --- [alarm-detect-8,e6310f1ed9200069,da9528c95b7b87d6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646491, data is empty. 2024-09-11 18:29:06.098 INFO 2981023 --- [alarm-detect-19,e6310f1ed9200069,8f180c96fe05fbde] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646491, data is empty. 2024-09-11 18:29:06.118 INFO 2981023 --- [alarm-detect-20,e6310f1ed9200069,689bbb7eb386b15d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050510 ~ 1726050536 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050525, value=8.33)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.33, values={1726050525=8.33}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:06.118 INFO 2981023 --- [alarm-detect-20,e6310f1ed9200069,689bbb7eb386b15d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050510 ~ 1726050536 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050525, value=8.33)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.33, values={1726050525=8.33}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:29:06.200 INFO 2981023 --- [alarm-detect-4,e6310f1ed9200069,9e1783f45273c9e1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646438, data is empty. 2024-09-11 18:29:06.204 INFO 2981023 --- [alarm-detect-4,e6310f1ed9200069,9e1783f45273c9e1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646491, data is empty. 2024-09-11 18:29:06.208 INFO 2981023 --- [alarm-detect-4,e6310f1ed9200069,9e1783f45273c9e1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646491, data is empty. 2024-09-11 18:29:06.237 INFO 2981023 --- [alarm-detect-6,e6310f1ed9200069,332c13c2bf450340] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050510 ~ 1726050536 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050525, value=104.085)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.085, values={1726050525=104.085}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:29:06.337394+08:00, lastProcessTime=2024-09-11T18:28:56.337295+08:00 2024-09-11 18:29:06.370 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:06.371 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:06.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:29:06.337394+08:00 2024-09-11 18:29:07.473 INFO 2981023 --- [ocp-async-2,caed71897c664e5b,4406838a0a68411c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:07.798 INFO 2981023 --- [ocp-async-7,f7b4bf9aebfdf65c,6ee0a001a4848b16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:29:08.237 INFO 2981023 --- [ocp-schedules-16,222806a53f211ae7,2f12eabac1738423] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:29:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,a9fba31c775bd39e,0246a9b53a30a5b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,a9fba31c775bd39e,0246a9b53a30a5b2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:08.317 INFO 2981023 --- [ocp-schedules-8,082fb51ad26124e6,6b286a2cd8fda4bf] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:29:08.317+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:29:08.318 INFO 2981023 --- [ocp-schedules-8,082fb51ad26124e6,6b286a2cd8fda4bf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:29:08.324 INFO 2981023 --- [ocp-schedules-8,082fb51ad26124e6,6b286a2cd8fda4bf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:29:08.325 INFO 2981023 --- [ocp-schedules-8,082fb51ad26124e6,6b286a2cd8fda4bf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:29:09.371 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:09.371 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:09.474 INFO 2981023 --- [ocp-async-6,c3dd7b4041b2519e,84c20605ed24e503] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:10.003 INFO 2981023 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:29:10.003 INFO 2981023 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:29:10.913 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:29:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:29:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,6d99c54a406b4a08,c7daffea93fd747c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:11.319 INFO 2981023 --- [prometheus-minute-schedule-1,6d99c54a406b4a08,c7daffea93fd747c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:11.474 INFO 2981023 --- [ocp-async-9,f0d739c50cac4016,0a7fa66fe9dab4be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:11.893 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050480, nodeCount=1000, affectRows=1000 2024-09-11 18:29:11.895 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050480, nodeCount=999, validCount=999 2024-09-11 18:29:11.922 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050480, nodeCount=49, affectRows=49 2024-09-11 18:29:11.922 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050480, nodeCount=47, validCount=47 2024-09-11 18:29:11.935 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050480, nodeCount=1000, affectRows=1000 2024-09-11 18:29:11.937 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050480, nodeCount=1000, validCount=1000 2024-09-11 18:29:12.372 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:12.372 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:12.444 INFO 2981023 --- [ocp-schedules-3,2ec517e8909c8eff,fab4fd003f4c741b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:29:12.870 INFO 2981023 --- [ocp-schedules-5,24d781227a7f8f56,16a72550bb06fa01] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:29:13.233 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:25:12.892, end=2024-09-11T18:27:13.228} 2024-09-11 18:29:13.235 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId 2024-09-11 18:29:13.236 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:29:13.237 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:25:12.892, end=2024-09-11T18:27:13.228},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:29:13.237 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:29:13.237 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:29:13.246 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:29:13.265 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153265000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.265Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.266 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153266000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=AD63995D4A6E8C3787D81BDA88E06774, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.266Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.267 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:29:13.267 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.275 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.275 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.279 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.279 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.286 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.286 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.287 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:29:13.287 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.265, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.289 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.289 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.295 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.295 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.297 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.298 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.303 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.303 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=201001, sqlId=AD63995D4A6E8C3787D81BDA88E06774, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.305 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.310 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.266, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=4, sqlId=AD63995D4A6E8C3787D81BDA88E06774, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.311 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:29:13.313 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:25:12.892, end=2024-09-11T18:27:13.228},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:29:13.327 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:29:13.328 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:29:13.372 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153372000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.372Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.376 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153376000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.376Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.379 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153379000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.379Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.384 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153384000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.384Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.387 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153387000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.387Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.389 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153389000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=8A1B851989150B802AEEACA99152D145, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.389Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.390 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153390000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.390Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.392 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153392000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.392Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.397 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.397 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.397 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:29:13.397 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.403 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.403 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.403 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.403 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.419 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 70 2024-09-11 18:29:13.419 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.420 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:29:13.420 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.424 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 30 2024-09-11 18:29:13.424 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.425 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:29:13.425 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.428 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.428 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.430 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.430 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.433 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.433 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.436 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.436 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.436 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.436 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.436 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.436 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.439 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.439 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.440 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.440 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.443 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.443 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.444 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.444 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.447 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.447 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.447 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.447 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.449 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.449 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.451 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.452 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.452 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.453 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.453 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.454 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.454 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.454 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.455 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.455 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.458 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.458 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.458 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.459 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.459 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.460 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.460 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.465 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.465 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.384, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.470 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.389, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=8A1B851989150B802AEEACA99152D145, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.471 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.471 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.473 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.372, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.474 INFO 2981023 --- [ocp-async-1,0e54351de0698844,fbc03d2b88e7ceda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:13.475 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.475 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.478 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.478 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.478 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.480 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.480 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.480 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.481 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.481 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.376, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.489 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.492 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.379, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.492 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.39, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.494 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.392, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.509 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.387, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.512 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153512000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.512Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.523 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.523 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.525 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153525000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.525Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.536 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.536 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.536 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153536000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.536Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.540 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.540 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.541 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153541000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.541Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.546 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.546 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.546 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153546000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.546Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.548 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.548 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.549 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.550 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.550 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153550000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.550Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.552 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.552 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.557 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.557 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.558 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964153558000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:29:13.558Z, endTime=2024-09-11T10:27:13.228Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:29:13.559 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.559 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.562 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.562 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.563 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.563 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.564 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.564 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.567 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.592 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.592 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.594 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.594 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.595 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:29:13.595 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:29:13.596 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.512, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.597 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.597 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.599 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.599 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.599 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.599 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.601 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.601 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.601 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.601 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.602 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.602 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.605 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.606 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.606 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.607 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.607 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.609 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:29:13.609 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:29:13.611 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.612 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.612 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.612 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.525, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.613 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.613 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.614 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.615 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.615 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.617 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.536, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.618 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.619 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.620 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:29:13.620 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050433228000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:29:13.621 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.55, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.621 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:29:13.622 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.541, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.623 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.546, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.625 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:29:13.558, end=2024-09-11T18:27:13.228}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:29:13.625 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:29:13.626 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:25:12.892, end=2024-09-11T18:27:13.228},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:29:13.626 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:29:13.626 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:29:13.627 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:29:13.628 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:25:12.892, end=2024-09-11T18:27:13.228},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:29:13.628 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:29:13.628 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:29:13.629 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:29:13.629 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:25:12.892, end=2024-09-11T18:27:13.228},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:29:13.629 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:29:13.629 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:29:13.630 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:29:13.635 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:25:12.892, end=2024-09-11T18:27:13.228},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:29:13.636 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:29:13.636 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:29:13.646 INFO 2981023 --- [plan_change_detector19,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:29:13.646 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:29:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,ba7a7086a6d0aa02,16d370c681dd60b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,eceedb62a2c9ed5e,89ad20e30c3b525c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:29:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,ba7a7086a6d0aa02,16d370c681dd60b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:15.372 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:15.375 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:15.453 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050480, nodeCount=480, affectRows=480 2024-09-11 18:29:15.474 INFO 2981023 --- [ocp-async-4,006704af0e702a54,af4f3c05b79f35f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:15.898 INFO 2981023 --- [ocp-schedules-28,85e49592d560792d,d6ed61140024c0d9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:29:15.907 INFO 2981023 --- [ocp-schedules-28,85e49592d560792d,d6ed61140024c0d9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:29:15.907 INFO 2981023 --- [ocp-schedules-28,85e49592d560792d,d6ed61140024c0d9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:29:16.015 INFO 2981023 --- [ocp-schedules-10,35e2a8b8912080c0,3b208d5bf9ca68e3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:29:16.016 INFO 2981023 --- [host-check-clockdiff1,35e2a8b8912080c0,c3575010aac82e5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:29:16.016 INFO 2981023 --- [host-check-clockdiff2,35e2a8b8912080c0,d7a7dc43de3c129b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:29:16.016 INFO 2981023 --- [host-check-clockdiff3,35e2a8b8912080c0,85fdc0f1aa6f8263] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:29:16.018 INFO 2981023 --- [host-check-clockdiff4,35e2a8b8912080c0,33c2c12257b27e7d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:29:16.029 INFO 2981023 --- [ocp-schedules-4,7a3c085b44f0bcfa,4042fd72e74a1a0e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:29:16.031 INFO 2981023 --- [ocp-schedules-8,5cbbd52e9a6fb2d2,eef4b5a8324cd245] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:29:16.032 INFO 2981023 --- [host-check-clockdiff3,35e2a8b8912080c0,85fdc0f1aa6f8263] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050556 0 0 , error= 2024-09-11 18:29:16.038 INFO 2981023 --- [host-check-agent-status1,5cbbd52e9a6fb2d2,dc508c438b31bb9c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:29:16.039 INFO 2981023 --- [host-check-agent-status2,5cbbd52e9a6fb2d2,31511a07f4312935] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:29:16.039 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:29:16.039 INFO 2981023 --- [host-check-agent-status4,5cbbd52e9a6fb2d2,27dbc58ec29aa399] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:29:16.039 INFO 2981023 --- [host-check-clockdiff2,35e2a8b8912080c0,d7a7dc43de3c129b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050556 6 6 , error= 2024-09-11 18:29:16.039 INFO 2981023 --- [host-check-agent-status3,5cbbd52e9a6fb2d2,9b06891d2dd50689] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:29:16.032 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:29:16.039 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:29:16.040 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:29:16.065 INFO 2981023 --- [host-check-agent-status2,5cbbd52e9a6fb2d2,31511a07f4312935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:29:16.068 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:29:16.068 INFO 2981023 --- [host-check-agent-status1,5cbbd52e9a6fb2d2,dc508c438b31bb9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:29:16.070 INFO 2981023 --- [host-check-agent-status4,5cbbd52e9a6fb2d2,27dbc58ec29aa399] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:29:16.070 INFO 2981023 --- [host-check-agent-status2,5cbbd52e9a6fb2d2,31511a07f4312935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:29:16.072 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:29:16.073 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:29:16.073 INFO 2981023 --- [host-check-agent-status4,5cbbd52e9a6fb2d2,27dbc58ec29aa399] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:29:16.073 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:29:16.074 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:29:16.076 INFO 2981023 --- [host-check-agent-status1,5cbbd52e9a6fb2d2,dc508c438b31bb9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:29:16.078 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:29:16.180 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:29:16.184 INFO 2981023 --- [host-check-agent-status3,5cbbd52e9a6fb2d2,9b06891d2dd50689] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:29:16.187 INFO 2981023 --- [ocp-schedules-9,aa3bb13c1b659198,b3e24f91648ad627] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:29:16.192 INFO 2981023 --- [host-check-agent-status3,5cbbd52e9a6fb2d2,9b06891d2dd50689] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:29:16.197 INFO 2981023 --- [ocp-schedules-9,aa3bb13c1b659198,b3e24f91648ad627] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:29:16.197 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:29:16.198 INFO 2981023 --- [ocp-schedules-13,5374d330dbde7ce8,a6e176a507697828] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:29:16.198 INFO 2981023 --- [ocp-schedules-7,ac8d8b5857eabb77,255199935fdb5ce0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:29:16.198 INFO 2981023 --- [ocp-schedules-6,e3f121fed0d118f5,75d6ffe83d1d300a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:29:16.219 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050483298000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:29:16.256 INFO 2981023 --- [ocp-schedules-26,cd9d96f92f69fd25,ab4526be22d4f9ce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:29:16.260 INFO 2981023 --- [ocp-schedules-7,ac8d8b5857eabb77,255199935fdb5ce0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:29:16.262 INFO 2981023 --- [ocp-schedules-13,5374d330dbde7ce8,a6e176a507697828] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:29:16.309 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:29:16.311 INFO 2981023 --- [host-check-agent-status3,5cbbd52e9a6fb2d2,9b06891d2dd50689] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:29:16.312 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:29:16.311953720+08:00 2024-09-11 18:29:16.314 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:29:16.313+08:00 2024-09-11 18:29:16.316 INFO 2981023 --- [host-check-agent-status3,5cbbd52e9a6fb2d2,9b06891d2dd50689] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:29:16.316 INFO 2981023 --- [host-check-agent-status3,5cbbd52e9a6fb2d2,9b06891d2dd50689] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:29:16.316 INFO 2981023 --- [host-check-agent-status3,5cbbd52e9a6fb2d2,9b06891d2dd50689] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:29:16.316 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:29:16.313+08:00 and status AVAILABLE 2024-09-11 18:29:16.320 INFO 2981023 --- [host-check-agent-status2,5cbbd52e9a6fb2d2,31511a07f4312935] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:29:16.333 INFO 2981023 --- [host-check-agent-status1,5cbbd52e9a6fb2d2,dc508c438b31bb9c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:29:16.340 INFO 2981023 --- [host-check-agent-status1,5cbbd52e9a6fb2d2,dc508c438b31bb9c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:29:16.340 INFO 2981023 --- [host-check-agent-status1,5cbbd52e9a6fb2d2,dc508c438b31bb9c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:29:16.340 INFO 2981023 --- [host-check-agent-status1,5cbbd52e9a6fb2d2,dc508c438b31bb9c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:29:16.342 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:29:16.342 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:29:16.342545732+08:00 2024-09-11 18:29:16.343 INFO 2981023 --- [ocp-schedules-22,3a1536e1d334d6f8,72ed2b20737b926f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:29:16.344 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:29:16.345 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:29:16.350967249+08:00 2024-09-11 18:29:16.346 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:29:16.345+08:00 2024-09-11 18:29:16.348 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:29:16.345+08:00 and status AVAILABLE 2024-09-11 18:29:16.350 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:29:16.342+08:00 2024-09-11 18:29:16.351 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.353 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:29:16.342+08:00 and status AVAILABLE 2024-09-11 18:29:16.353 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:29:16.353 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:29:16.361 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:29:16.366 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:29:16.368 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.373 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:29:16.374 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:29:16.378 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:29:16.379 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:29:16.378789+08:00, lastProcessTime=2024-09-11T18:29:06.337394+08:00 2024-09-11 18:29:16.385 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:29:16.386 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:29:16.390 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:29:16.391 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:29:16.397 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:29:16.398 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:29:16.408 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:29:16.420 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:29:16.430 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:29:16.444 INFO 2981023 --- [host-check-agent-status2,5cbbd52e9a6fb2d2,31511a07f4312935] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:29:16.444 INFO 2981023 --- [host-check-agent-status2,5cbbd52e9a6fb2d2,31511a07f4312935] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:29:16.444 INFO 2981023 --- [host-check-agent-status2,5cbbd52e9a6fb2d2,31511a07f4312935] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:29:16.445 INFO 2981023 --- [host-check-agent-status4,5cbbd52e9a6fb2d2,27dbc58ec29aa399] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:29:16.447 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:29:16.450 INFO 2981023 --- [host-check-agent-status4,5cbbd52e9a6fb2d2,27dbc58ec29aa399] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:29:16.450 INFO 2981023 --- [host-check-agent-status4,5cbbd52e9a6fb2d2,27dbc58ec29aa399] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:29:16.450 INFO 2981023 --- [host-check-agent-status4,5cbbd52e9a6fb2d2,27dbc58ec29aa399] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:29:16.459 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:29:16.462 WARN 2981023 --- [scanner-13677954-1,cd9d96f92f69fd25,a4ca6f1c98425868] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:29:16.468 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:29:16.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.469 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:29:16.468873043+08:00 2024-09-11 18:29:16.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:16.472 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:29:16.469+08:00 2024-09-11 18:29:16.475 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:29:16.469+08:00 and status AVAILABLE 2024-09-11 18:29:16.477 INFO 2981023 --- [ocp-schedules-13,5374d330dbde7ce8,a6e176a507697828] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:29:16.482 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:29:16.485 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.485 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:29:16.485 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:16.486 INFO 2981023 --- [ocp-schedules-9,aa3bb13c1b659198,b3e24f91648ad627] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:29:16.491 INFO 2981023 --- [alarm-detect-17,cd9d96f92f69fd25,10fe7089f2415f5f] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050525 ~ 1726050546 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050540, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050540, value=98.981)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.981, values={1726050540=98.981}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050540=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:16.503 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.504 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:29:16.506 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:29:16.507 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:16.509 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:29:16.511 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:29:16.532 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:29:16.537 INFO 2981023 --- [alarm-detect-18,cd9d96f92f69fd25,5cdd362f232367db] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050525 ~ 1726050546 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050540, value=98.981)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.981, values={1726050540=98.981}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:16.557 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:29:16.572 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:29:16.576 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:29:16.576 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050483298000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:29:16.577 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:29:16.579 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:29:16.597 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:29:16.597 INFO 2981023 --- [alarm-detect-13,cd9d96f92f69fd25,f4390f29f0cefe7e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050525 ~ 1726050546 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050540, value=8.748)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.748, values={1726050540=8.748}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:16.598 INFO 2981023 --- [alarm-detect-13,cd9d96f92f69fd25,f4390f29f0cefe7e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050525 ~ 1726050546 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050540, value=8.748)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.748, values={1726050540=8.748}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:29:16.600 INFO 2981023 --- [alarm-detect-2,cd9d96f92f69fd25,d8ddd84e9d8eb653] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050525 ~ 1726050546 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050540, value=98.981)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.981, values={1726050540=98.981}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:16.609 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:29:16.611 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:29:16.619 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:29:16.621 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:29:16.633 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:29:16.636 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:16.640 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.643 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:16.644 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:29:16.646 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.649 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:29:16.650 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:29:16.651 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:29:16.653 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:29:16.659 INFO 2981023 --- [host-check-host-status4,7a3c085b44f0bcfa,c25ea528b3f7cb46] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:29:16.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:29:16.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:29:16.688 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:29:16.711 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:29:16.713 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:29:16.714 INFO 2981023 --- [alarm-detect-2,cd9d96f92f69fd25,4b52ec08a56f0ab2] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050525 ~ 1726050546 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050540, value=105.3)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.3, values={1726050540=105.3}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:16.724 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:16.727 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:16.727 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:29:16.730 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.731 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:29:16.732 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:29:16.735 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.739 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:29:16.739 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:29:16.745 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:29:16.747 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:29:16.750 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:29:16.757 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:29:16.770 INFO 2981023 --- [host-check-host-status2,7a3c085b44f0bcfa,054e69070688da2d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:29:16.772 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:29:16.778 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:29:16.780 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:16.787 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.796 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:16.798 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:29:16.799 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.801 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:29:16.805 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:29:16.807 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:29:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:29:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:29:16.827 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:29:16.835 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:16.838 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:29:16.841 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.842 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:29:16.843 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:29:16.848 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:29:16.852 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:29:16.853 INFO 2981023 --- [host-check-host-status3,7a3c085b44f0bcfa,656accbc60db4bac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:29:16.853 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:29:16.855 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:29:16.378789+08:00 2024-09-11 18:29:16.868 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:29:16.874 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:29:16.879 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:16.880 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:16.880 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:29:16.886 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:29:16.886 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:29:16.891 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:29:16.897 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:29:16.917 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:29:17.061 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:29:17.067 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:17.076 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:17.076 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:29:17.082 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:29:17.082 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:29:17.082 INFO 2981023 --- [ocp-schedules-7,ac8d8b5857eabb77,255199935fdb5ce0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:29:17.133 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:29:17.139 INFO 2981023 --- [host-check-host-status1,7a3c085b44f0bcfa,56201d9427a858a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:29:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,8fc0878eafbd6b09,8644e85bc80eed10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:17.317 INFO 2981023 --- [prometheus-minute-schedule-1,8fc0878eafbd6b09,8644e85bc80eed10] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:17.474 INFO 2981023 --- [ocp-async-8,d0c0d5bf52df2a9f,e8b86e31d2e1ea1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:17.798 INFO 2981023 --- [ocp-async-1,84e1f9c0dffe440e,a838218b390fff1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:29:17.804 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050480, nodeCount=480, validCount=480 2024-09-11 18:29:18.375 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:18.375 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:19.177 INFO 2981023 --- [host-check-clockdiff1,35e2a8b8912080c0,c3575010aac82e5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050559 0 0 , error= 2024-09-11 18:29:19.474 INFO 2981023 --- [ocp-async-2,8db2ac40b45e5518,760fb89b9b2fffe0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,d624993e59745c83,3dc1f9adc7412600] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,d624993e59745c83,3dc1f9adc7412600] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:21.375 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:21.376 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:21.474 INFO 2981023 --- [ocp-async-7,918c7ad228685dcc,6cb7db5405dc12f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:21.997 INFO 2981023 --- [host-check-clockdiff4,35e2a8b8912080c0,33c2c12257b27e7d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050561 0 0 , error= 2024-09-11 18:29:22.873 INFO 2981023 --- [ocp-schedules-21,94c61f71a31e9c24,f97147423df3a7e7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:29:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,0809bbef5f1da86c,22c64c2e127ff048] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,0809bbef5f1da86c,22c64c2e127ff048] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:23.474 INFO 2981023 --- [ocp-async-10,84b4c06013d90153,7c31b1c6f48a1d13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:24.376 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:24.376 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:24.519 INFO 2981023 --- [ocp-schedules-8,2e76985473a7f153,cfeab046f7d8a257] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:29:25.475 INFO 2981023 --- [ocp-async-3,b3153b6fa181bd50,4fa777298213b8eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,7ec3abd7db26fabc,d4b978bcd70af6af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,7ec3abd7db26fabc,d4b978bcd70af6af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:29:26.337750+08:00, lastProcessTime=2024-09-11T18:29:16.378789+08:00 2024-09-11 18:29:26.472 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:29:26.337750+08:00 2024-09-11 18:29:26.768 INFO 2981023 --- [ocp-schedules-11,80a326b75813424d,3e31bfcd0c333a08] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:29:26.822 WARN 2981023 --- [scanner-13678184-1,80a326b75813424d,1f420910700fc3d3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:29:26.830 INFO 2981023 --- [alarm-detect-2,80a326b75813424d,0393eb4fa72aea00] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050540 ~ 1726050556 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050555, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050555, value=99.316)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=99.316, values={1726050555=99.316}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050555=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:26.843 INFO 2981023 --- [alarm-detect-10,80a326b75813424d,b806c06cee149d8e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646491, data is empty. 2024-09-11 18:29:26.848 INFO 2981023 --- [alarm-detect-10,80a326b75813424d,b806c06cee149d8e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646491, data is empty. 2024-09-11 18:29:26.852 INFO 2981023 --- [alarm-detect-10,80a326b75813424d,b806c06cee149d8e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646491, data is empty. 2024-09-11 18:29:26.874 INFO 2981023 --- [alarm-detect-12,80a326b75813424d,ffb2cf47d3946052] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646491, data is empty. 2024-09-11 18:29:26.878 INFO 2981023 --- [alarm-detect-17,80a326b75813424d,b507bd4faf184dd5] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050540 ~ 1726050556 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050555, value=99.316)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=99.316, values={1726050555=99.316}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:26.879 INFO 2981023 --- [alarm-detect-5,80a326b75813424d,267903d3512f7731] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646491, data is empty. 2024-09-11 18:29:26.880 INFO 2981023 --- [alarm-detect-12,80a326b75813424d,ffb2cf47d3946052] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646491, data is empty. 2024-09-11 18:29:26.887 INFO 2981023 --- [alarm-detect-12,80a326b75813424d,ffb2cf47d3946052] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646491, data is empty. 2024-09-11 18:29:26.887 INFO 2981023 --- [alarm-detect-5,80a326b75813424d,267903d3512f7731] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646491, data is empty. 2024-09-11 18:29:26.895 INFO 2981023 --- [alarm-detect-5,80a326b75813424d,267903d3512f7731] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646491, data is empty. 2024-09-11 18:29:26.921 INFO 2981023 --- [alarm-detect-15,80a326b75813424d,0c5f50736b3caf60] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050540 ~ 1726050556 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050555, value=99.316)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=99.316, values={1726050555=99.316}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:26.990 INFO 2981023 --- [alarm-detect-4,80a326b75813424d,a3958e2dc5544c1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646491, data is empty. 2024-09-11 18:29:26.999 INFO 2981023 --- [alarm-detect-4,80a326b75813424d,a3958e2dc5544c1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646491, data is empty. 2024-09-11 18:29:27.003 INFO 2981023 --- [alarm-detect-4,80a326b75813424d,a3958e2dc5544c1c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646491, data is empty. 2024-09-11 18:29:27.376 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:27.377 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:27.475 INFO 2981023 --- [ocp-async-5,03bc2146b7b34d34,9727a3e3b33342a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:27.798 INFO 2981023 --- [ocp-async-10,c8445630b08f9487,a3ecfd83b93d69ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:29:28.281 INFO 2981023 --- [ocp-schedules-22,1c6ffface7b1cc84,ca810db6d63e0c22] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:29:28.281 INFO 2981023 --- [ocp-schedules-22,1c6ffface7b1cc84,ca810db6d63e0c22] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:29:28.281 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:29:28.281 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:29:28.282 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:29:28.286 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:29:28.286 INFO 2981023 --- [ocp-schedules-22,1c6ffface7b1cc84,ca810db6d63e0c22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:29:28.287 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:29:28.287 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:29:28.288 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:29:28.288 INFO 2981023 --- [ocp-schedules-22,1c6ffface7b1cc84,ca810db6d63e0c22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:29:28.288 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:29:28.288 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:29:28.305 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:29:28.307 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:29:28.308 INFO 2981023 --- [ocp-schedules-22,1c6ffface7b1cc84,ca810db6d63e0c22] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:29:28.309 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:29:28.312 INFO 2981023 --- [ocp-schedules-22,1c6ffface7b1cc84,ca810db6d63e0c22] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:29:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,cbeefa0f4d698afe,33de8b171da8bb60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,cbeefa0f4d698afe,33de8b171da8bb60] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:29.475 INFO 2981023 --- [ocp-async-8,b920873e7070e5b2,e9049050a001b2cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:30.377 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:30.377 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:31.475 INFO 2981023 --- [ocp-async-2,bcf7802478dcd9a4,a65cf9b89d615972] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,970a8d72723db55b,10ede49fcd0eb619] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,970a8d72723db55b,10ede49fcd0eb619] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:32.876 INFO 2981023 --- [ocp-schedules-1,f009f5e8e4567460,8bea788c7a0bb435] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:29:33.377 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:33.378 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:33.475 INFO 2981023 --- [ocp-async-4,bdecf65d1fc594db,4a76260bb88b7779] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,31411ac6250d8179,10ff47931be8965f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,31411ac6250d8179,10ff47931be8965f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:35.475 INFO 2981023 --- [ocp-async-6,f9b14de65481ae14,bc87f8c6154dde63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:35.909 INFO 2981023 --- [ocp-schedules-29,e9595eda2e92d58d,4b894c70fd0dbcc0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:29:35.915 INFO 2981023 --- [ocp-schedules-29,e9595eda2e92d58d,4b894c70fd0dbcc0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:29:35.915 INFO 2981023 --- [ocp-schedules-29,e9595eda2e92d58d,4b894c70fd0dbcc0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:29:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:29:36.336446+08:00, lastProcessTime=2024-09-11T18:29:26.337750+08:00 2024-09-11 18:29:36.378 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:36.379 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:36.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:29:36.336446+08:00 2024-09-11 18:29:37.037 INFO 2981023 --- [ocp-schedules-13,2a3857c31aa45aa3,ef55516d64e65fc5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:29:37.125 WARN 2981023 --- [scanner-13678490-1,2a3857c31aa45aa3,79c3c21881be42c1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:29:37.137 INFO 2981023 --- [alarm-detect-11,2a3857c31aa45aa3,bcb6aa7a50e8a1cf] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050540 ~ 1726050567 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050555, value=8.92)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.92, values={1726050555=8.92}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:37.137 INFO 2981023 --- [alarm-detect-11,2a3857c31aa45aa3,bcb6aa7a50e8a1cf] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050540 ~ 1726050567 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050555, value=8.92)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.92, values={1726050555=8.92}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:29:37.215 INFO 2981023 --- [alarm-detect-6,2a3857c31aa45aa3,de53ff7fd4829e57] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050540 ~ 1726050567 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050555, value=104.933)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.933, values={1726050555=104.933}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:37.314 INFO 2981023 --- [prometheus-cache-monitor1,74e812c5ee6fcf11,4a4cba0ce04a9e33] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:29:37.315 INFO 2981023 --- [prometheus-cache-monitor1,74e812c5ee6fcf11,4a4cba0ce04a9e33] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:29:37.475 INFO 2981023 --- [ocp-async-9,43e748f5e51d2145,d4c382e791e604d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:37.798 INFO 2981023 --- [ocp-async-4,546e9167cee4ca3a,092825f81713558f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:29:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,e36cbc9c5b41ebae,71ba13b09eb1c3fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,e36cbc9c5b41ebae,71ba13b09eb1c3fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:39.379 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:39.379 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:39.475 INFO 2981023 --- [ocp-async-5,677716fd7555bf55,fd96616f576fd3af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:40.002 INFO 2981023 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:29:40.002 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:29:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,80e3da83756045ba,f6642c8700de7dd1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,80e3da83756045ba,f6642c8700de7dd1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:41.475 INFO 2981023 --- [ocp-async-10,5991bf5368f165af,92d25ebc0c601256] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:42.379 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:42.380 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:42.878 INFO 2981023 --- [ocp-schedules-30,b91d8e154ac27b88,b8c943b39d741f5b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:29:43.475 INFO 2981023 --- [ocp-async-8,bd3ab4fc7e3feb90,8784ad9703f8afca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,331997756eccbacf,41b1e72314d37994] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,331997756eccbacf,41b1e72314d37994] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:45.101 INFO 2981023 --- [prometheus-second-schedule-1,172d3f938e178c1e,420fbf6d10773bd5] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:29:45.380 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:45.380 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:45.476 INFO 2981023 --- [ocp-async-2,7a71d05de7b0f1d1,79babd60cf07a3f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:45.997 INFO 2981023 --- [ocp-schedules-28,95fe86c12d7a295b,365b5255bf82350e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:29:45.998 INFO 2981023 --- [host-check-clockdiff1,95fe86c12d7a295b,7202d1a55092570a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:29:45.998 INFO 2981023 --- [host-check-clockdiff2,95fe86c12d7a295b,0e0d679715295b2e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:29:45.998 INFO 2981023 --- [host-check-clockdiff3,95fe86c12d7a295b,952c3426d8e62cd8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:29:45.999 INFO 2981023 --- [host-check-clockdiff4,95fe86c12d7a295b,f3892e11ded4c0ce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:29:46.010 INFO 2981023 --- [host-check-clockdiff3,95fe86c12d7a295b,952c3426d8e62cd8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050586 0 0 , error= 2024-09-11 18:29:46.013 INFO 2981023 --- [ocp-schedules-13,b75ecece1505e06d,44bf25bb934dced4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:29:46.013 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:29:46.013 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:29:46.014 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:29:46.014 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:29:46.017 INFO 2981023 --- [ocp-schedules-21,b410e8475ba574cf,8b8c54fc283901ee] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:29:46.017 INFO 2981023 --- [host-check-agent-status1,b410e8475ba574cf,352206350013f506] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:29:46.018 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:29:46.018 INFO 2981023 --- [host-check-agent-status2,b410e8475ba574cf,73aa4fa4637fa7f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:29:46.018 INFO 2981023 --- [host-check-agent-status3,b410e8475ba574cf,44714bbdb285345a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:29:46.019 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:29:46.019 INFO 2981023 --- [host-check-agent-status4,b410e8475ba574cf,8fa841a8c3b151bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:29:46.020 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:29:46.020 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:29:46.021 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:29:46.021 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:29:46.022 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:29:46.024 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:29:46.025 INFO 2981023 --- [host-check-agent-status1,b410e8475ba574cf,352206350013f506] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:29:46.027 INFO 2981023 --- [host-check-agent-status1,b410e8475ba574cf,352206350013f506] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:29:46.027 INFO 2981023 --- [host-check-agent-status2,b410e8475ba574cf,73aa4fa4637fa7f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:29:46.028 INFO 2981023 --- [host-check-agent-status4,b410e8475ba574cf,8fa841a8c3b151bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:29:46.029 INFO 2981023 --- [host-check-agent-status3,b410e8475ba574cf,44714bbdb285345a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:29:46.029 INFO 2981023 --- [host-check-agent-status2,b410e8475ba574cf,73aa4fa4637fa7f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:29:46.030 INFO 2981023 --- [host-check-agent-status4,b410e8475ba574cf,8fa841a8c3b151bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:29:46.031 INFO 2981023 --- [host-check-agent-status3,b410e8475ba574cf,44714bbdb285345a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:29:46.044 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:29:46.046 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:29:46.046424166+08:00 2024-09-11 18:29:46.048 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:29:46.046+08:00 2024-09-11 18:29:46.050 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:29:46.050 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:29:46.046+08:00 and status AVAILABLE 2024-09-11 18:29:46.052 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:29:46.053 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:29:46.058945349+08:00 2024-09-11 18:29:46.054 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:29:46.053388420+08:00 2024-09-11 18:29:46.054 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:29:46.053+08:00 2024-09-11 18:29:46.055 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:29:46.054+08:00 2024-09-11 18:29:46.056 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:29:46.053+08:00 and status AVAILABLE 2024-09-11 18:29:46.056 INFO 2981023 --- [host-check-agent-status2,b410e8475ba574cf,73aa4fa4637fa7f1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:29:46.058 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:29:46.054+08:00 and status AVAILABLE 2024-09-11 18:29:46.059 INFO 2981023 --- [host-check-agent-status2,b410e8475ba574cf,73aa4fa4637fa7f1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:29:46.059 INFO 2981023 --- [host-check-agent-status2,b410e8475ba574cf,73aa4fa4637fa7f1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:29:46.059 INFO 2981023 --- [host-check-agent-status2,b410e8475ba574cf,73aa4fa4637fa7f1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:29:46.059 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:29:46.059 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:29:46.063 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:29:46.063 INFO 2981023 --- [host-check-agent-status3,b410e8475ba574cf,44714bbdb285345a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:29:46.064 INFO 2981023 --- [host-check-agent-status1,b410e8475ba574cf,352206350013f506] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:29:46.065 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:29:46.065557219+08:00 2024-09-11 18:29:46.066 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:29:46.066 INFO 2981023 --- [host-check-agent-status3,b410e8475ba574cf,44714bbdb285345a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:29:46.066 INFO 2981023 --- [host-check-agent-status3,b410e8475ba574cf,44714bbdb285345a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:29:46.066 INFO 2981023 --- [host-check-agent-status3,b410e8475ba574cf,44714bbdb285345a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:29:46.066 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:29:46.066 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:29:46.065+08:00 2024-09-11 18:29:46.067 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:29:46.067 INFO 2981023 --- [host-check-agent-status4,b410e8475ba574cf,8fa841a8c3b151bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:29:46.068 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:29:46.065+08:00 and status AVAILABLE 2024-09-11 18:29:46.070 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:29:46.071 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:29:46.071 INFO 2981023 --- [host-check-agent-status4,b410e8475ba574cf,8fa841a8c3b151bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:29:46.071 INFO 2981023 --- [host-check-agent-status4,b410e8475ba574cf,8fa841a8c3b151bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:29:46.071 INFO 2981023 --- [host-check-agent-status4,b410e8475ba574cf,8fa841a8c3b151bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:29:46.073 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:29:46.077 INFO 2981023 --- [host-check-agent-status1,b410e8475ba574cf,352206350013f506] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:29:46.077 INFO 2981023 --- [host-check-agent-status1,b410e8475ba574cf,352206350013f506] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:29:46.077 INFO 2981023 --- [host-check-agent-status1,b410e8475ba574cf,352206350013f506] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:29:46.079 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:29:46.079 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:29:46.083 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:29:46.092 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:29:46.093 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:29:46.098 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:29:46.110 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:29:46.112 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:29:46.144 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:29:46.150 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:29:46.151 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:29:46.158 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:29:46.164 INFO 2981023 --- [ocp-schedules-20,ec82c37171ecd57a,af13f8c968aca8a5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:29:46.168 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:29:46.180 INFO 2981023 --- [ocp-schedules-20,ec82c37171ecd57a,af13f8c968aca8a5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:29:46.187 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:29:46.190 INFO 2981023 --- [ocp-schedules-5,63fd05f6ed4d1eee,ff46874b93afdbba] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:29:46.190 INFO 2981023 --- [ocp-schedules-2,7bdf45d8b5885a7f,a777a1adf1aee900] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:29:46.190 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:29:46.193 INFO 2981023 --- [ocp-schedules-2,7bdf45d8b5885a7f,a777a1adf1aee900] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:29:46.194 INFO 2981023 --- [ocp-schedules-27,bcd86abcc436de3e,c1982073b5e4ee7a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:29:46.194 INFO 2981023 --- [ocp-schedules-18,89deb71fcaf46abe,3bb384c9079fc850] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:29:46.197 INFO 2981023 --- [ocp-schedules-5,63fd05f6ed4d1eee,ff46874b93afdbba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:29:46.197 INFO 2981023 --- [ocp-schedules-27,bcd86abcc436de3e,c1982073b5e4ee7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:29:46.201 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:29:46.206 INFO 2981023 --- [ocp-schedules-10,a758c47bed8be4c6,6c934ead62c46baf] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:29:46.208 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:29:46.217 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:29:46.219 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:29:46.229 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:29:46.240 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.241 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:29:46.241 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:29:46.242 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:29:46.245 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:29:46.246 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.256 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:29:46.261 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050483298000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:29:46.261 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:29:46.264 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:29:46.272 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.273 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:29:46.273 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:29:46.274 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:29:46.279 INFO 2981023 --- [ocp-schedules-2,7bdf45d8b5885a7f,a777a1adf1aee900] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:29:46.280 INFO 2981023 --- [host-check-host-status4,b75ecece1505e06d,f439ce37e58706be] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:29:46.284 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:29:46.285 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:29:46.295 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:29:46.299 INFO 2981023 --- [host-check-host-status2,b75ecece1505e06d,082706a2c6214bf3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:29:46.314 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.323 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:29:46.327 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:29:46.329 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.329 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.329 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:46.330 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:46.330 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.330 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:29:46.331 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:46.332 INFO 2981023 --- [ocp-schedules-5,63fd05f6ed4d1eee,ff46874b93afdbba] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:29:46.356 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:29:46.357 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.366 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:29:46.364303+08:00, lastProcessTime=2024-09-11T18:29:36.336446+08:00 2024-09-11 18:29:46.369 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.370 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:29:46.371 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:29:46.371 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050483298000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:29:46.371 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:29:46.376 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:29:46.377 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:29:46.380 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.380 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.380 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.381 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:29:46.402 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:29:46.405 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.406 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.407 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:29:46.416 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.416 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.417 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:46.417 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:29:46.418 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.420 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:29:46.420 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:29:46.420 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:29:46.423 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:29:46.429 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:29:46.429 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:29:46.430 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:29:46.439 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.440 INFO 2981023 --- [host-check-host-status3,b75ecece1505e06d,74f7cc46fecb8402] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:29:46.442 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:29:46.450 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.451 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.451 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:29:46.452 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.452 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.452 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:29:46.455 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:29:46.458 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:29:46.459 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:29:46.462 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.462 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.463 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:29:46.466 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:29:46.472 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:29:46.473 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:29:46.473 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.473 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:29:46.473 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.480 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:29:46.483 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:29:46.485 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:29:46.485 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:29:46.486 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.491 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:29:46.491 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:29:46.491 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:29:46.502 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:29:46.514 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.525 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:29:46.532 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.538 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:29:46.539 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:29:46.540 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.540 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.543 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:29:46.545 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:29:46.545 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:29:46.547 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.547 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.547 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:29:46.548 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:29:46.549 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.552 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:29:46.553 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:29:46.553 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:29:46.559 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:29:46.559 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726049989295000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:29:46.567 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:29:46.568 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:29:46.568 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:29:46.576 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:29:46.364303+08:00 2024-09-11 18:29:46.583 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:29:46.584 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:29:46.595 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.606 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.610 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:29:46.615 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:29:46.619 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.672 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:29:46.692 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:29:46.698 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:29:46.701 INFO 2981023 --- [ocp-schedules-11,6466f4910f17034a,7f13549fe12683b3] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:29:46.849 INFO 2981023 --- [ocp-schedules-17,b4c1e4a05e06786e,9bea76a8a3c58e3e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:29:46.849 INFO 2981023 --- [ocp-schedules-17,b4c1e4a05e06786e,9bea76a8a3c58e3e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:29:46.879 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:29:46.885 INFO 2981023 --- [host-check-host-status1,b75ecece1505e06d,cae9a99232f3f1ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:29:46.927 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:29:46.933 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:29:46.937 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:46.943 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:29:46.953 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:29:46.953 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050012665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:29:46.954 INFO 2981023 --- [ocp-schedules-27,bcd86abcc436de3e,c1982073b5e4ee7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:29:47.228 INFO 2981023 --- [ocp-schedules-16,50092caea1e727fa,d6f7d849585a5d81] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:29:47.273 WARN 2981023 --- [scanner-13678716-1,50092caea1e727fa,3422264715b8711d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:29:47.285 INFO 2981023 --- [alarm-detect-11,50092caea1e727fa,2bf15c67c6a27cae] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050555 ~ 1726050577 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050570, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050570, value=98.464)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.464, values={1726050570=98.464}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050570=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:47.298 INFO 2981023 --- [alarm-detect-11,50092caea1e727fa,731f41775616b2ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646491, data is empty. 2024-09-11 18:29:47.304 INFO 2981023 --- [alarm-detect-11,50092caea1e727fa,731f41775616b2ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646491, data is empty. 2024-09-11 18:29:47.309 INFO 2981023 --- [alarm-detect-11,50092caea1e727fa,731f41775616b2ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646491, data is empty. 2024-09-11 18:29:47.314 INFO 2981023 --- [prometheus-minute-schedule-1,e0656c3b5bc242ba,bfbddbe197af3f8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:47.319 INFO 2981023 --- [prometheus-minute-schedule-1,e0656c3b5bc242ba,bfbddbe197af3f8c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:47.346 INFO 2981023 --- [alarm-detect-17,50092caea1e727fa,b40d0c71943279ae] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646491, data is empty. 2024-09-11 18:29:47.347 INFO 2981023 --- [alarm-detect-16,50092caea1e727fa,cec2dfbc8942f51c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646491, data is empty. 2024-09-11 18:29:47.350 INFO 2981023 --- [alarm-detect-17,50092caea1e727fa,b40d0c71943279ae] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646491, data is empty. 2024-09-11 18:29:47.382 INFO 2981023 --- [alarm-detect-17,50092caea1e727fa,b40d0c71943279ae] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646491, data is empty. 2024-09-11 18:29:47.384 INFO 2981023 --- [alarm-detect-16,50092caea1e727fa,cec2dfbc8942f51c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646491, data is empty. 2024-09-11 18:29:47.389 INFO 2981023 --- [alarm-detect-16,50092caea1e727fa,cec2dfbc8942f51c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646491, data is empty. 2024-09-11 18:29:47.403 INFO 2981023 --- [alarm-detect-1,50092caea1e727fa,5bc6ea7dd1d8d045] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050555 ~ 1726050577 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050570, value=98.464)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.464, values={1726050570=98.464}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:47.415 INFO 2981023 --- [alarm-detect-20,50092caea1e727fa,ba453a6764c8a86a] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050555 ~ 1726050577 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050570, value=98.464)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.464, values={1726050570=98.464}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:47.457 INFO 2981023 --- [alarm-detect-16,50092caea1e727fa,22e1160718576be9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050555 ~ 1726050577 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050570, value=8.737)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.737, values={1726050570=8.737}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:47.457 INFO 2981023 --- [alarm-detect-16,50092caea1e727fa,22e1160718576be9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050555 ~ 1726050577 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050570, value=8.737)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.737, values={1726050570=8.737}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:29:47.476 INFO 2981023 --- [ocp-async-6,040d3447f0b67420,9a15d00151067e99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:47.588 INFO 2981023 --- [alarm-detect-14,50092caea1e727fa,817a819c53c22c9a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646491, data is empty. 2024-09-11 18:29:47.591 INFO 2981023 --- [alarm-detect-14,50092caea1e727fa,817a819c53c22c9a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646491, data is empty. 2024-09-11 18:29:47.593 INFO 2981023 --- [alarm-detect-14,50092caea1e727fa,817a819c53c22c9a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646491, data is empty. 2024-09-11 18:29:47.776 INFO 2981023 --- [alarm-detect-19,50092caea1e727fa,d001c75327722ed2] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050555 ~ 1726050577 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050570, value=104.371)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=104.371, values={1726050570=104.371}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:47.798 INFO 2981023 --- [ocp-async-4,6c74a6db3ef4069d,1f5308e62f5026bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:29:47.821 INFO 2981023 --- [host-check-clockdiff2,95fe86c12d7a295b,0e0d679715295b2e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050587 5 5 , error= 2024-09-11 18:29:48.380 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:48.381 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:48.584 INFO 2981023 --- [ocp-schedules-20,cfefd8c3dbce7e9f,6d9b88dd927b3688] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:29:48.606 WARN 2981023 --- [ocp-schedules-20,cfefd8c3dbce7e9f,6d9b88dd927b3688] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:29:49.296 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,232e14aecf20db9f,f99f7e06f8071723] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&version=2&ObClusterId=1719994437, client=10.31.35.10, traceId=232e14aecf20db9f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkprdocp,1719994437,2,{"ObRegion":"linkprdocp","ObCluster":"linkprdocp","ObRegionId":1719994437,"ObClusterId":1719994437,"Type":"PRIMARY","timestamp":1726050589294414,"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:29:49.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,232e14aecf20db9f,f99f7e06f8071723] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkprdocp, obClusterId=1719994437, rootServerJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, groupName=linkprdocp, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:29:49.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,232e14aecf20db9f,f99f7e06f8071723] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:29:49.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,232e14aecf20db9f,f99f7e06f8071723] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkprdocp, clusterId=1719994437, affectRows=1 2024-09-11 18:29:49.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,232e14aecf20db9f,f99f7e06f8071723] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.10, traceId=232e14aecf20db9f, duration=10 ms] 2024-09-11 18:29:49.476 INFO 2981023 --- [ocp-async-8,08be68961cbbd8b0,43fb4ce49af47809] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:49.703 INFO 2981023 --- [host-check-clockdiff1,95fe86c12d7a295b,7202d1a55092570a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050589 0 0 , error= 2024-09-11 18:29:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:29:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:29:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,18e508fd8c4da12a,2a8318140e5060ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,18e508fd8c4da12a,2a8318140e5060ee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:50.369 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.374 INFO 2981023 --- [ocp-schedules-15,017ef66b41b5df46,408627810d972aea] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:29:50.380 INFO 2981023 --- [ocp-schedules-17,6f6151266940ce05,22d295b53d0ba938] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:29:51.381 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:51.383 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:51.476 INFO 2981023 --- [ocp-async-1,acbe529747ebf8e9,620974ce0facc25f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:51.648 INFO 2981023 --- [ocp-schedules-26,4f0420a8a29f342e,bd256636044faa82] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:29:51.649 INFO 2981023 --- [ocp-schedules-26,4f0420a8a29f342e,bd256636044faa82] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:29:51.649 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:29:51.649 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:29:51.650 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:29:51.650 INFO 2981023 --- [ocp-schedules-26,4f0420a8a29f342e,bd256636044faa82] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:29:51.948 INFO 2981023 --- [host-check-clockdiff4,95fe86c12d7a295b,f3892e11ded4c0ce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050591 0 0 , error= 2024-09-11 18:29:52.881 INFO 2981023 --- [ocp-schedules-12,ec644cbb7c988b2e,56e0d00c8b97fa53] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:29:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,55666c7e77438d19,cea04434b8f6ee9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,55666c7e77438d19,cea04434b8f6ee9f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:53.398 INFO 2981023 --- [ocp-schedules-19,74b5551c25a0a32b,f11c03cb5c9f6e88] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:29:53.398 INFO 2981023 --- [ocp-schedules-19,74b5551c25a0a32b,f11c03cb5c9f6e88] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:29:53.476 INFO 2981023 --- [ocp-async-10,09ad71dc0d55efcb,1bca7c97875c6c68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:54.383 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:54.383 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:54.523 INFO 2981023 --- [ocp-schedules-27,65c5ff02c2ac3d4d,4162629ecde82e16] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:29:55.476 INFO 2981023 --- [ocp-async-9,adcb9afc768f2c90,51422b05540ddf71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:55.547 INFO 2981023 --- [ocp-schedules-8,babf3b427b483d3c,62e7124b90ce7fd0] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:29:55.567 WARN 2981023 --- [ocp-schedules-8,babf3b427b483d3c,62e7124b90ce7fd0] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:29:55.917 INFO 2981023 --- [ocp-schedules-14,b58fa25a87b262a1,dfb16fb332b19fc0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:29:55.926 INFO 2981023 --- [ocp-schedules-14,b58fa25a87b262a1,dfb16fb332b19fc0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:29:55.926 INFO 2981023 --- [ocp-schedules-14,b58fa25a87b262a1,dfb16fb332b19fc0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:29:55.932 INFO 2981023 --- [ocp-schedules-10,4dbd0632f28cc85e,0ce353f8b5fe17ea] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:29:55.932 INFO 2981023 --- [ocp-schedules-10,4dbd0632f28cc85e,0ce353f8b5fe17ea] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:29:55.933 INFO 2981023 --- [ocp-schedules-10,4dbd0632f28cc85e,0ce353f8b5fe17ea] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:29:55.933 INFO 2981023 --- [ocp-schedules-10,4dbd0632f28cc85e,0ce353f8b5fe17ea] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:29:55.933 INFO 2981023 --- [ocp-schedules-10,4dbd0632f28cc85e,0ce353f8b5fe17ea] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:29:55.933 INFO 2981023 --- [ocp-schedules-10,4dbd0632f28cc85e,0ce353f8b5fe17ea] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:29:56.163 INFO 2981023 --- [ocp-schedules-4,bf527bbe77b4229c,6dce4bb8621f1b89] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:29:56.170 INFO 2981023 --- [ocp-schedules-4,bf527bbe77b4229c,6dce4bb8621f1b89] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:29:56.170 INFO 2981023 --- [ocp-schedules-4,bf527bbe77b4229c,6dce4bb8621f1b89] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:29:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,39309a67cd73eb97,6545e1ddff17b0d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,39309a67cd73eb97,6545e1ddff17b0d6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:29:56.353473+08:00, lastProcessTime=2024-09-11T18:29:46.364303+08:00 2024-09-11 18:29:56.493 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:29:56.353473+08:00 2024-09-11 18:29:57.383 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:29:57.385 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:29:57.476 INFO 2981023 --- [ocp-async-5,e809ff3849a1e3ea,889beb4d2abaa0d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:29:57.798 INFO 2981023 --- [ocp-async-3,5b8a2a195087ea21,e177f76338df8ebd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:29:57.831 INFO 2981023 --- [ocp-schedules-5,e7dfe48cf5f874bc,8b2af8e24eae887f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:29:57.875 WARN 2981023 --- [scanner-13679042-1,e7dfe48cf5f874bc,9e5b89fbda36f81a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:29:57.885 INFO 2981023 --- [alarm-detect-9,e7dfe48cf5f874bc,4346d6cc969ba4bd] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050570 ~ 1726050587 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050585, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050585, value=98.709)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.709, values={1726050585=98.709}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050585=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:57.916 INFO 2981023 --- [alarm-detect-12,e7dfe48cf5f874bc,f45b75b1aa591841] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050570 ~ 1726050587 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050585, value=98.709)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.709, values={1726050585=98.709}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:57.944 INFO 2981023 --- [alarm-detect-10,e7dfe48cf5f874bc,d20f515bc65964bf] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050570 ~ 1726050587 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050585, value=98.709)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.709, values={1726050585=98.709}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:29:59.043 INFO 2981023 --- [ocp-schedules-2,783aaa04d9f61402,450954da3ff685b2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:29:59.043 INFO 2981023 --- [ocp-schedules-2,783aaa04d9f61402,450954da3ff685b2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:29:59.075 INFO 2981023 --- [Collect all tenants major compactions4,783aaa04d9f61402,87fe868e93cf495a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:29:59.075 INFO 2981023 --- [Collect all tenants major compactions5,783aaa04d9f61402,56fb2cc2ea92a561] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:29:59.075 INFO 2981023 --- [Collect all tenants major compactions6,783aaa04d9f61402,64a4397083b5753d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:29:59.075 INFO 2981023 --- [Collect all tenants major compactions3,783aaa04d9f61402,ab860f08b3a671be] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:29:59.075 INFO 2981023 --- [Collect all tenants major compactions1,783aaa04d9f61402,7a6fc7d736fca04d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:29:59.082 INFO 2981023 --- [Collect all tenants major compactions2,783aaa04d9f61402,568205ad03ad0116] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:29:59.099 INFO 2981023 --- [Collect all tenants major compactions5,783aaa04d9f61402,56fb2cc2ea92a561] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:59.099 INFO 2981023 --- [Collect all tenants major compactions6,783aaa04d9f61402,64a4397083b5753d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:59.099 INFO 2981023 --- [Collect all tenants major compactions1,783aaa04d9f61402,7a6fc7d736fca04d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:59.099 INFO 2981023 --- [Collect all tenants major compactions4,783aaa04d9f61402,87fe868e93cf495a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:59.100 INFO 2981023 --- [Collect all tenants major compactions3,783aaa04d9f61402,ab860f08b3a671be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:59.100 INFO 2981023 --- [Collect all tenants major compactions5,783aaa04d9f61402,56fb2cc2ea92a561] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:29:59.100 INFO 2981023 --- [Collect all tenants major compactions1,783aaa04d9f61402,7a6fc7d736fca04d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:29:59.101 INFO 2981023 --- [Collect all tenants major compactions4,783aaa04d9f61402,87fe868e93cf495a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:29:59.101 INFO 2981023 --- [Collect all tenants major compactions6,783aaa04d9f61402,64a4397083b5753d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:29:59.101 INFO 2981023 --- [Collect all tenants major compactions2,783aaa04d9f61402,568205ad03ad0116] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:29:59.101 INFO 2981023 --- [Collect all tenants major compactions3,783aaa04d9f61402,ab860f08b3a671be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:29:59.102 INFO 2981023 --- [Collect all tenants major compactions2,783aaa04d9f61402,568205ad03ad0116] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:29:59.104 INFO 2981023 --- [Collect all tenants major compactions6,783aaa04d9f61402,64a4397083b5753d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:29:59.104 INFO 2981023 --- [Collect all tenants major compactions5,783aaa04d9f61402,56fb2cc2ea92a561] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:29:59.105 INFO 2981023 --- [Collect all tenants major compactions5,783aaa04d9f61402,56fb2cc2ea92a561] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:29:59.105 INFO 2981023 --- [Collect all tenants major compactions1,783aaa04d9f61402,7a6fc7d736fca04d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:29:59.105 INFO 2981023 --- [Collect all tenants major compactions1,783aaa04d9f61402,7a6fc7d736fca04d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:29:59.105 INFO 2981023 --- [Collect all tenants major compactions3,783aaa04d9f61402,ab860f08b3a671be] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:29:59.105 INFO 2981023 --- [Collect all tenants major compactions3,783aaa04d9f61402,ab860f08b3a671be] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:29:59.105 INFO 2981023 --- [Collect all tenants major compactions6,783aaa04d9f61402,64a4397083b5753d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:29:59.105 INFO 2981023 --- [Collect all tenants major compactions2,783aaa04d9f61402,568205ad03ad0116] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:29:59.105 INFO 2981023 --- [Collect all tenants major compactions2,783aaa04d9f61402,568205ad03ad0116] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:29:59.106 INFO 2981023 --- [Collect all tenants major compactions4,783aaa04d9f61402,87fe868e93cf495a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:29:59.106 INFO 2981023 --- [Collect all tenants major compactions4,783aaa04d9f61402,87fe868e93cf495a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:29:59.106 INFO 2981023 --- [ocp-schedules-2,783aaa04d9f61402,450954da3ff685b2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:29:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,de5ea3ca7692815f,ecddfb18e23cf1c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050600, interval=60 2024-09-11 18:29:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,de5ea3ca7692815f,ecddfb18e23cf1c0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:29:59.476 INFO 2981023 --- [ocp-async-10,c741cf4d0181273e,0466c120ad58f685] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:00.385 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:00.387 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:00.659 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:25:01.0, nextRun=2024-09-11 18:30:00.0, currentDate=Wed Sep 11 18:30:00 HKT 2024 2024-09-11 18:30:00.659 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:25:01.0, nextRunTime=2024-09-11 18:30:00.0} 2024-09-11 18:30:00.666 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 18:30:00.666 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:30:00.666 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:30:00.666+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:30:00.678 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:25:01.0, nextRun=2024-09-11 18:30:00.0, currentDate=Wed Sep 11 18:30:00 HKT 2024 2024-09-11 18:30:00.678 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:25:01.0, nextRunTime=2024-09-11 18:30:00.0} 2024-09-11 18:30:00.701 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 18:30:00.710 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 18:30:00.710 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 18:30:00.711 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:30:00.711 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,c04f64b8a0819717] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T18:30:00.711+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:30:01.057 INFO 2981023 --- [ocp-schedules-21,5f0e81d036216598,13798cd28924a591] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:30:01.057+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:30:01.062 INFO 2981023 --- [ocp-schedules-21,5f0e81d036216598,13798cd28924a591] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:30:01.357 INFO 2981023 --- [ocp-schedules-21,5f0e81d036216598,13798cd28924a591] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 54 to SqlAggregateData 2024-09-11 18:30:01.359 INFO 2981023 --- [ocp-schedules-21,5f0e81d036216598,13798cd28924a591] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:30:01.477 INFO 2981023 --- [ocp-async-9,02bc1923e160c558,1502fcba199b41ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:01.558 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,28bc1c8b1e28cf4a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048253, operation:EXECUTE, state: RUNNING 2024-09-11 18:30:01.573 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,28bc1c8b1e28cf4a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048254, operation:EXECUTE, state: RUNNING 2024-09-11 18:30:01.581 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048253, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048223, task_operation=execute, latest_execution_start_time=2024-09-11T18:30:01.551+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048253}, listMap={}}, executor=10.31.35.10 2024-09-11 18:30:01.597 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:30:01.600 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048254, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048224, task_operation=execute, latest_execution_start_time=2024-09-11T18:30:01.571+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048254}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 18:30:01.604 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 18:30:01.628 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 18:30:01.631 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 18:30:01.637 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:30:01.643 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 18:30:01.644 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 18:30:01.644 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 18:30:01.668 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 18:30:01.671 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 18:30:01.679 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,18a2a9c3cbeb74f9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048254, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:30:01.712 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:30:01.729 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:30:01.753 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:30:01.764 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 18:30:01.764 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 18:30:01.770 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:30:01.871 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:30:01.884 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048224, status:SUCCESS 2024-09-11 18:30:01.887 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048224, status:SUCCESS 2024-09-11 18:30:01.915 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:30:01.940 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:30:01.955 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:30:02.004 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:30:02.015 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:30:02.028 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:30:02.034 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 18:30:02.034 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3395.459579986], b:[-5.807309276841022E12] 2024-09-11 18:30:02.043 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:30:02.117 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:30:02.151 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:30:02.164 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:30:02.173 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:30:02.183 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:30:02.189 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:30:02.196 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:30:02.200 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 18:30:02.201 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:30:02.217 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:30:02.228 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:30:02.241 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,516412f0e8264233] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048253, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:30:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,c1823f8ed31d31f4,6ac6ed81253a16c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:02.317 INFO 2981023 --- [prometheus-minute-schedule-1,c1823f8ed31d31f4,6ac6ed81253a16c0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:02.332 INFO 2981023 --- [ocp-schedules-8,df5b668fb74abcb9,885b2954cafce371] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:30:02.332 INFO 2981023 --- [ocp-schedules-8,df5b668fb74abcb9,885b2954cafce371] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:30:02.884 INFO 2981023 --- [ocp-schedules-21,500b5508ceccefc3,a56491e37d30b21e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:30:02.897 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048223, status:SUCCESS 2024-09-11 18:30:02.897 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048223, status:SUCCESS 2024-09-11 18:30:03.307 INFO 2981023 --- [ocp-schedules-15,248ca1f742152f8e,640bb98ee71e93d1] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:30:03.308 INFO 2981023 --- [ocp-schedules-15,248ca1f742152f8e,640bb98ee71e93d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:30:03.329 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:03.329 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:30:03.334 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:03.336 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:30:03.337 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050603337000} 2024-09-11 18:30:03.343 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:30:03.346 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000239, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:30:03.347 INFO 2981023 --- [ocp-schedules-15,248ca1f742152f8e,640bb98ee71e93d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:30:03.387 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:03.387 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:03.477 INFO 2981023 --- [ocp-async-1,2902779582c5f940,83166be48b08682e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:03.834 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:30:03.834 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:30:03.834 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:30:03.838 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000239, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:30:03.840 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:30:03.979 INFO 2981023 --- [ocp-schedules-11,da9e2a5b51b13f54,08ad2fc37955944a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:30:03.980 INFO 2981023 --- [ocp-schedules-11,da9e2a5b51b13f54,08ad2fc37955944a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:30:04.005 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:30:04.005 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.009 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:30:04.009 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:30:04.009 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.012 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:30:04.013 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:30:04.013 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.016 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.017 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.017 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:30:04.020 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.021 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.022 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.022 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.027 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.027 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:30:04.033 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.034 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.034 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.035 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.035 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:30:04.039 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.040 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.040 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:30:04.045 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.045 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.047 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.047 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:30:04.052 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.052 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.060 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.061 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:30:04.061 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.062 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:30:04.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.064 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:30:04.065 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.065 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:30:04.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:30:04.067 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.067 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:30:04.068 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.068 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:30:04.069 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.069 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:30:04.069 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.070 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:30:04.070 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:30:04.074 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.074 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:30:04.075 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:30:04.076 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.080 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.086 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:04.087 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.088 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:30:04.088 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.089 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:30:04.089 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.090 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:30:04.090 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:30:04.092 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:30:04.092 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:30:04.093 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:30:04.095 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6BF1D6DC-0-0] 2024-09-11 18:30:04.095 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:30:04.096 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:30:04.096 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.097 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:30:04.099 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.102 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:30:04.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:30:04.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.106 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:04.108 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.108 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.109 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:30:04.110 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:30:04.112 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:30:04.112 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:30:04.112 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:30:04.118 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:30:04.119 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:30:04.119 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:30:04.119 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:30:04.130 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.136 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:30:04.149 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.155 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:30:04.156 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:30:04.156 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6BF1D6E6-0-0] 2024-09-11 18:30:04.156 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:30:04.161 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.162 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:30:04.162 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.163 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:30:04.163 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.164 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:30:04.164 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.165 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:30:04.165 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.166 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:30:04.166 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.167 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:30:04.167 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:30:04.169 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:30:04.179 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:04.227 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.251 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:30:04.272 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:30:04.274 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:30:04.277 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:04.298 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:30:04.320 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:30:04.322 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:30:04.322 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:30:04.322 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:30:04.322 INFO 2981023 --- [ocp-schedules-11,da9e2a5b51b13f54,08ad2fc37955944a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:30:04.447 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000239 2024-09-11 18:30:04.811 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2d70368f725f719] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:20:05.0, nextRun=2024-09-11 18:30:04.0, currentDate=Wed Sep 11 18:30:04 HKT 2024 2024-09-11 18:30:04.811 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2d70368f725f719] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-09-11 18:20:05.0, nextRunTime=2024-09-11 18:30:04.0} 2024-09-11 18:30:04.818 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2d70368f725f719] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-09-11 18:30:04.818 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2d70368f725f719] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:30:04.818 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,d2d70368f725f719] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:30:04.818+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:30:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,03294718f6ed67dc,d5ae0d44e786f2ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,03294718f6ed67dc,d5ae0d44e786f2ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:05.477 INFO 2981023 --- [ocp-async-6,4609909f3d182d88,d59f41431972e719] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:05.617 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,b301d9bebb02464d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048255, operation:EXECUTE, state: RUNNING 2024-09-11 18:30:05.642 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048255, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048225, task_operation=execute, latest_execution_start_time=2024-09-11T18:30:05.615+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=4048255}, listMap={}}, executor=10.31.35.10 2024-09-11 18:30:05.649 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 18:30:05.649 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 18:30:05.649 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.650 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:30:05.651 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.652 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.652 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.653 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 18:30:05.664 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.664 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.664 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.665 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 18:30:05.674 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.674 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.675 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.675 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 18:30:05.684 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.685 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.685 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.685 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 18:30:05.694 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.695 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.695 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.696 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 18:30:05.700 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.700 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.700 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.701 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 18:30:05.704 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.704 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.704 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.705 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 18:30:05.707 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.708 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.708 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.709 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 18:30:05.712 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.713 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.713 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.714 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 18:30:05.717 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.717 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.718 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.718 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 18:30:05.721 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.721 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.722 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.722 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 18:30:05.728 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.728 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.729 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.729 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 18:30:05.738 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.738 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.738 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.739 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 18:30:05.743 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.743 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.743 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.744 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 18:30:05.748 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.748 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.748 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.749 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 18:30:05.752 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.752 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.752 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.753 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 18:30:05.755 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.756 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.756 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.757 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 18:30:05.760 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.760 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.761 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.761 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 18:30:05.764 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.764 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.765 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.765 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 18:30:05.768 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.768 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.769 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.769 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 18:30:05.772 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.772 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.773 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.773 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 18:30:05.776 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.776 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.777 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.777 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 18:30:05.780 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.780 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.781 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.781 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 18:30:05.784 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.784 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.785 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.785 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 18:30:05.788 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.788 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.789 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.789 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 18:30:05.792 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.792 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.792 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.793 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 18:30:05.795 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.796 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.796 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.796 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 18:30:05.799 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.799 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.800 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.800 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 18:30:05.803 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.804 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.804 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.804 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 18:30:05.807 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.807 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.808 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.808 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 18:30:05.811 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.811 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.812 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.812 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 18:30:05.815 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.815 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.816 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.816 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 18:30:05.819 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.819 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.820 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.820 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 18:30:05.822 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.823 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.823 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.824 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 18:30:05.828 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.829 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.829 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.830 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 18:30:05.834 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.835 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.835 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.835 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 18:30:05.839 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.839 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.840 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.840 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 18:30:05.844 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.845 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.845 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.846 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 18:30:05.848 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.849 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.849 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.849 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 18:30:05.852 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.852 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.852 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.853 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 18:30:05.855 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.855 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.856 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.856 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 18:30:05.858 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.859 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.859 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.859 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 18:30:05.866 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.866 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.867 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.867 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 18:30:05.883 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.883 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.883 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.884 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 18:30:05.888 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.888 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.888 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.889 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 18:30:05.892 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.893 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.893 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.893 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 18:30:05.897 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.898 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.898 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.898 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 18:30:05.901 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.901 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.901 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.902 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 18:30:05.904 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.905 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.905 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.905 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 18:30:05.908 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 18:30:05.908 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 18:30:05.913 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.913 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:30:05.915 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.915 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.916 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.916 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 18:30:05.920 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.920 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:05.921 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:05.921 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 18:30:05.923 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 18:30:05.937 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,9da027f041decdfe] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048255, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:30:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:30:05.990 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:30:05.995 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:30:06.000 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:30:06.000 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:30:06.000 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:30:06.004 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:06.004 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:30:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:30:06.063 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:27:41, start=2024-09-11T18:25:11, end=2024-09-11T18:27:41, updates=304, granule=120000ms, delay=150000ms 2024-09-11 18:30:06.072 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:27:41, target=2024-09-11T18:27:41, granule=120000ms, delay=0ms) 2024-09-11 18:30:06.072 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:27:41, target=2024-09-11T18:20:51, granule=600000ms, delay=410000ms) 2024-09-11 18:30:06.072 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=105ms, period=120000ms 2024-09-11 18:30:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:30:06.353305+08:00, lastProcessTime=2024-09-11T18:29:56.353473+08:00 2024-09-11 18:30:06.387 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:06.388 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:06.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:30:06.353305+08:00 2024-09-11 18:30:06.934 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048225, status:SUCCESS 2024-09-11 18:30:06.934 INFO 2981023 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048225, status:SUCCESS 2024-09-11 18:30:07.477 INFO 2981023 --- [ocp-async-3,531db45ea70f5407,cbbe2008a9a9bb1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:07.799 INFO 2981023 --- [ocp-async-1,85a2d1b77bbd966c,7ee433d94b0918f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:30:08.014 INFO 2981023 --- [ocp-schedules-23,a7cd5ab1b7ab0e7c,2dd039fa3518cb97] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:30:08.042 WARN 2981023 --- [scanner-13679419-1,a7cd5ab1b7ab0e7c,2da5dd986f1ad446] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:30:08.088 INFO 2981023 --- [alarm-detect-19,a7cd5ab1b7ab0e7c,47adecf2f053c7e7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646491, data is empty. 2024-09-11 18:30:08.092 INFO 2981023 --- [alarm-detect-19,a7cd5ab1b7ab0e7c,47adecf2f053c7e7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646545, data is empty. 2024-09-11 18:30:08.097 INFO 2981023 --- [alarm-detect-19,a7cd5ab1b7ab0e7c,47adecf2f053c7e7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646545, data is empty. 2024-09-11 18:30:08.124 INFO 2981023 --- [alarm-detect-18,a7cd5ab1b7ab0e7c,06fd04aa117b2f90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646491, data is empty. 2024-09-11 18:30:08.131 INFO 2981023 --- [alarm-detect-7,a7cd5ab1b7ab0e7c,b103d7258ce76bb5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646491, data is empty. 2024-09-11 18:30:08.134 INFO 2981023 --- [alarm-detect-18,a7cd5ab1b7ab0e7c,06fd04aa117b2f90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646545, data is empty. 2024-09-11 18:30:08.136 INFO 2981023 --- [alarm-detect-7,a7cd5ab1b7ab0e7c,b103d7258ce76bb5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646545, data is empty. 2024-09-11 18:30:08.141 INFO 2981023 --- [alarm-detect-18,a7cd5ab1b7ab0e7c,06fd04aa117b2f90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646545, data is empty. 2024-09-11 18:30:08.141 INFO 2981023 --- [alarm-detect-7,a7cd5ab1b7ab0e7c,b103d7258ce76bb5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646545, data is empty. 2024-09-11 18:30:08.168 INFO 2981023 --- [alarm-detect-19,a7cd5ab1b7ab0e7c,25928104c9c5beac] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050570 ~ 1726050598 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050585, value=8.47)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.47, values={1726050585=8.47}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:08.168 INFO 2981023 --- [alarm-detect-19,a7cd5ab1b7ab0e7c,25928104c9c5beac] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050570 ~ 1726050598 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050585, value=8.47)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.47, values={1726050585=8.47}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:30:08.236 INFO 2981023 --- [alarm-detect-19,a7cd5ab1b7ab0e7c,d34dde510f7673fd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646491, data is empty. 2024-09-11 18:30:08.252 INFO 2981023 --- [alarm-detect-19,a7cd5ab1b7ab0e7c,d34dde510f7673fd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646545, data is empty. 2024-09-11 18:30:08.258 INFO 2981023 --- [alarm-detect-19,a7cd5ab1b7ab0e7c,d34dde510f7673fd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646545, data is empty. 2024-09-11 18:30:08.269 INFO 2981023 --- [ocp-schedules-10,33dbb1544fcb579c,83ecbbe594508465] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:30:08.280 INFO 2981023 --- [alarm-detect-15,a7cd5ab1b7ab0e7c,0c445e3bafe76f7c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050570 ~ 1726050598 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050585, value=105.071)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.071, values={1726050585=105.071}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,adf03dc2d9a47e29,5a9db757008c0a25] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,adf03dc2d9a47e29,5a9db757008c0a25] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:08.339 INFO 2981023 --- [ocp-schedules-8,75ddb2fb448c4b9e,0244e321ecb0481b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:30:08.339+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:30:08.339 INFO 2981023 --- [ocp-schedules-8,75ddb2fb448c4b9e,0244e321ecb0481b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:30:08.342 INFO 2981023 --- [ocp-schedules-8,75ddb2fb448c4b9e,0244e321ecb0481b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:30:08.343 INFO 2981023 --- [ocp-schedules-8,75ddb2fb448c4b9e,0244e321ecb0481b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:30:09.388 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:09.388 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:09.477 INFO 2981023 --- [ocp-async-5,13fe16832b3c5b17,970aad7d4c56fdd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:10.003 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:30:10.003 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:30:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:30:10.921 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:30:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,636161fa95fc1f9a,a93d7027c0b6da96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,636161fa95fc1f9a,a93d7027c0b6da96] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:11.477 INFO 2981023 --- [ocp-async-2,06ef20e9f85ba516,19d055de7b0c3558] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:12.047 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050540, nodeCount=1000, affectRows=1000 2024-09-11 18:30:12.050 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050540, nodeCount=999, validCount=999 2024-09-11 18:30:12.058 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050540, nodeCount=48, affectRows=48 2024-09-11 18:30:12.058 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050540, nodeCount=46, validCount=46 2024-09-11 18:30:12.070 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050540, nodeCount=1000, affectRows=1000 2024-09-11 18:30:12.072 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050540, nodeCount=1000, validCount=1000 2024-09-11 18:30:12.389 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:12.389 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:12.664 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,92e46a559d685b79,963f1e17d16d9eca] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&version=2&ObClusterId=1718689631, client=10.31.35.5, traceId=92e46a559d685b79, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=rtdob,1718689631,2,{"ObRegion":"rtdob","ObCluster":"rtdob","ObRegionId":1718689631,"ObClusterId":1718689631,"Type":"PRIMARY","timestamp":1726050612659351,"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:30:12.672 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,92e46a559d685b79,963f1e17d16d9eca] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=rtdob, obClusterId=1718689631, rootServerJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, groupName=rtdob, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:30:12.672 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,92e46a559d685b79,963f1e17d16d9eca] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:30:12.672 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,92e46a559d685b79,963f1e17d16d9eca] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=rtdob, clusterId=1718689631, affectRows=1 2024-09-11 18:30:12.672 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,92e46a559d685b79,963f1e17d16d9eca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.5, traceId=92e46a559d685b79, duration=8 ms] 2024-09-11 18:30:12.886 INFO 2981023 --- [ocp-schedules-14,ed20851f6c53351b,0cd474ff48470455] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:30:12.910 INFO 2981023 --- [ocp-schedules-28,95a0b9faad401b24,815245d8f520ab08] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:30:12.915 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050540, nodeCount=480, affectRows=480 2024-09-11 18:30:13.477 INFO 2981023 --- [ocp-async-8,b1055e63a0e6e211,2c769e7fde143a6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,b06cdf11c7b9e7d3,a240b9b4361dc895] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,fc0b055559dd6114,4c03513935dfa97d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:30:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,b06cdf11c7b9e7d3,a240b9b4361dc895] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:15.389 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:15.391 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:15.467 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050540, nodeCount=480, validCount=480 2024-09-11 18:30:15.477 INFO 2981023 --- [ocp-async-4,9a724993d14e7920,7161bc87a98867a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:15.930 INFO 2981023 --- [ocp-schedules-21,2ae49549fb7b5146,9332f6e2711a8f77] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:30:15.938 INFO 2981023 --- [ocp-schedules-21,2ae49549fb7b5146,9332f6e2711a8f77] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:30:15.938 INFO 2981023 --- [ocp-schedules-21,2ae49549fb7b5146,9332f6e2711a8f77] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:30:16.020 INFO 2981023 --- [ocp-schedules-11,56cdc8d606958916,a94543001c9efa9b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:30:16.021 INFO 2981023 --- [host-check-clockdiff1,56cdc8d606958916,30cb17a728b50669] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:30:16.026 INFO 2981023 --- [host-check-clockdiff2,56cdc8d606958916,164e3cf2c2f1d169] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:30:16.027 INFO 2981023 --- [ocp-schedules-4,5ef9be519e46fd11,5f9c3ebdbce2755d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:30:16.028 INFO 2981023 --- [host-check-clockdiff3,56cdc8d606958916,1aabb3a6ca8c6ec8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:30:16.029 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:30:16.030 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:30:16.030 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:30:16.030 INFO 2981023 --- [ocp-schedules-6,da87de3b074438d1,81614c1ec5ffe999] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:30:16.030 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:30:16.051 INFO 2981023 --- [host-check-clockdiff4,56cdc8d606958916,306a4f9be832e602] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:30:16.051 INFO 2981023 --- [host-check-agent-status1,da87de3b074438d1,30ef1c729170b16a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:30:16.052 INFO 2981023 --- [host-check-agent-status2,da87de3b074438d1,c0dfb73feba1fd74] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:30:16.052 INFO 2981023 --- [host-check-agent-status4,da87de3b074438d1,bb9def7422d099e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:30:16.053 INFO 2981023 --- [host-check-agent-status3,da87de3b074438d1,89d50a5a3879fa67] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:30:16.064 INFO 2981023 --- [host-check-clockdiff3,56cdc8d606958916,1aabb3a6ca8c6ec8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050616 0 0 , error= 2024-09-11 18:30:16.067 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:30:16.067 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:30:16.067 INFO 2981023 --- [host-check-agent-status2,da87de3b074438d1,c0dfb73feba1fd74] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:30:16.071 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:30:16.071 INFO 2981023 --- [host-check-agent-status2,da87de3b074438d1,c0dfb73feba1fd74] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:30:16.071 INFO 2981023 --- [host-check-agent-status3,da87de3b074438d1,89d50a5a3879fa67] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:30:16.073 INFO 2981023 --- [host-check-agent-status3,da87de3b074438d1,89d50a5a3879fa67] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:30:16.079 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:30:16.080 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:30:16.082 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:30:16.100 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:30:16.193 INFO 2981023 --- [ocp-schedules-12,4d7caab1bbc98302,be2a6ef34c23b194] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:30:16.193 INFO 2981023 --- [ocp-schedules-8,a71395393d91b7ed,3d816b2edf9c5ebf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:30:16.193 INFO 2981023 --- [ocp-schedules-22,e452275decb8796e,f097141d830f9bdd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:30:16.193 INFO 2981023 --- [ocp-schedules-24,0181c394ec40b345,c6189476af3c7811] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:30:16.264 INFO 2981023 --- [host-check-agent-status1,da87de3b074438d1,30ef1c729170b16a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:30:16.264 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:30:16.265 INFO 2981023 --- [ocp-schedules-22,e452275decb8796e,f097141d830f9bdd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:30:16.274 INFO 2981023 --- [ocp-schedules-8,a71395393d91b7ed,3d816b2edf9c5ebf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:30:16.275 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050603337000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:30:16.277 INFO 2981023 --- [ocp-schedules-12,4d7caab1bbc98302,be2a6ef34c23b194] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:30:16.282 INFO 2981023 --- [ocp-schedules-12,4d7caab1bbc98302,be2a6ef34c23b194] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:30:16.289 INFO 2981023 --- [host-check-agent-status1,da87de3b074438d1,30ef1c729170b16a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:30:16.289 INFO 2981023 --- [host-check-agent-status4,da87de3b074438d1,bb9def7422d099e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:30:16.293 INFO 2981023 --- [host-check-agent-status4,da87de3b074438d1,bb9def7422d099e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:30:16.386 INFO 2981023 --- [ocp-schedules-13,ac89360414fa0d4a,6eb93af2be05ee08] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:30:16.397 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:30:16.400 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:30:16.400190015+08:00 2024-09-11 18:30:16.455 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:30:16.400+08:00 2024-09-11 18:30:16.457 INFO 2981023 --- [host-check-agent-status2,da87de3b074438d1,c0dfb73feba1fd74] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:30:16.457 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:30:16.461 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:30:16.400+08:00 and status AVAILABLE 2024-09-11 18:30:16.462 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:30:16.461359059+08:00 2024-09-11 18:30:16.462 INFO 2981023 --- [host-check-agent-status3,da87de3b074438d1,89d50a5a3879fa67] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:30:16.463 INFO 2981023 --- [host-check-agent-status2,da87de3b074438d1,c0dfb73feba1fd74] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:30:16.463 INFO 2981023 --- [host-check-agent-status2,da87de3b074438d1,c0dfb73feba1fd74] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:30:16.463 INFO 2981023 --- [host-check-agent-status2,da87de3b074438d1,c0dfb73feba1fd74] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:30:16.465 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:30:16.462+08:00 2024-09-11 18:30:16.467 INFO 2981023 --- [host-check-agent-status3,da87de3b074438d1,89d50a5a3879fa67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:30:16.467 INFO 2981023 --- [host-check-agent-status3,da87de3b074438d1,89d50a5a3879fa67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:30:16.467 INFO 2981023 --- [host-check-agent-status3,da87de3b074438d1,89d50a5a3879fa67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:30:16.470 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:30:16.470 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:30:16.470640657+08:00 2024-09-11 18:30:16.471 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:30:16.462+08:00 and status AVAILABLE 2024-09-11 18:30:16.472 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:30:16.470+08:00 2024-09-11 18:30:16.474 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:30:16.472482+08:00, lastProcessTime=2024-09-11T18:30:06.353305+08:00 2024-09-11 18:30:16.475 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:30:16.476 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:30:16.470+08:00 and status AVAILABLE 2024-09-11 18:30:16.479 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.481 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:30:16.482 INFO 2981023 --- [host-check-agent-status1,da87de3b074438d1,30ef1c729170b16a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:30:16.483 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:30:16.486 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:30:16.494 INFO 2981023 --- [host-check-agent-status1,da87de3b074438d1,30ef1c729170b16a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:30:16.494 INFO 2981023 --- [host-check-agent-status1,da87de3b074438d1,30ef1c729170b16a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:30:16.495 INFO 2981023 --- [host-check-agent-status1,da87de3b074438d1,30ef1c729170b16a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:30:16.498 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:30:16.499 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.500 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:30:16.500 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.501 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:16.501 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:30:16.501 INFO 2981023 --- [ocp-schedules-8,a71395393d91b7ed,3d816b2edf9c5ebf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:30:16.504 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:30:16.505 INFO 2981023 --- [host-check-agent-status4,da87de3b074438d1,bb9def7422d099e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:30:16.506 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:30:16.507 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:30:16.509 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:30:16.513496661+08:00 2024-09-11 18:30:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.510 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:30:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:16.512 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:30:16.512 INFO 2981023 --- [host-check-agent-status4,da87de3b074438d1,bb9def7422d099e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:30:16.512 INFO 2981023 --- [host-check-agent-status4,da87de3b074438d1,bb9def7422d099e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:30:16.512 INFO 2981023 --- [host-check-agent-status4,da87de3b074438d1,bb9def7422d099e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:30:16.514 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:30:16.509+08:00 2024-09-11 18:30:16.554 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:30:16.509+08:00 and status AVAILABLE 2024-09-11 18:30:16.556 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:30:16.559 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:30:16.562 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:30:16.572 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.573 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:16.575 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:30:16.575 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:30:16.576 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:30:16.578 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:30:16.579 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:30:16.596 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:16.596 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:30:16.596 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.597 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:30:16.605 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:30:16.615 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:30:16.616 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:30:16.616 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:30:16.649 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:30:16.663 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:16.664 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:30:16.664 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.665 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:16.667 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:30:16.667 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.668 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:30:16.670 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:30:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:30:16.675 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:30:16.681 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:30:16.681 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050603337000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:30:16.684 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:30:16.685 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:30:16.686 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:30:16.695 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:30:16.708 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:30:16.708 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:30:16.713 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:30:16.719 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:30:16.721 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:30:16.723 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:16.724 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.724 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:30:16.725 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:30:16.725 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:30:16.727 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:30:16.728 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:30:16.730 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:16.731 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:30:16.732 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:30:16.733 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.733 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:30:16.740 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:30:16.742 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:30:16.747 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:30:16.748 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:30:16.754 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:30:16.759 INFO 2981023 --- [host-check-host-status4,5ef9be519e46fd11,6a82ae49c826b605] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:30:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:16.773 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.774 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:30:16.774 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:16.775 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:30:16.775 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:30:16.775 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.778 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:30:16.779 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:30:16.788 INFO 2981023 --- [host-check-host-status3,5ef9be519e46fd11,0a29f9655e7f9778] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:30:16.789 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:30:16.789 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:30:16.802 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:30:16.802 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:30:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.813 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:30:16.814 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:30:16.820 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:30:16.825 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:30:16.826 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:30:16.831 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:16.832 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:30:16.832 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:16.833 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:30:16.833 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:30:16.841 INFO 2981023 --- [host-check-host-status2,5ef9be519e46fd11,90424dec2f44646b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:30:16.842 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:30:16.842 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:30:16.862 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:30:16.878 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:30:16.878 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:30:16.472482+08:00 2024-09-11 18:30:16.884 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:30:16.896 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:30:17.089 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:30:17.097 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:17.107 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:17.112 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:30:17.124 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:30:17.124 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:30:17.124 INFO 2981023 --- [ocp-schedules-22,e452275decb8796e,f097141d830f9bdd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:30:17.136 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:30:17.142 INFO 2981023 --- [host-check-host-status1,5ef9be519e46fd11,5cacebb2fc718e60] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:30:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,8d12e40afd196e70,8f657ea18304577c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:17.317 INFO 2981023 --- [prometheus-minute-schedule-1,8d12e40afd196e70,8f657ea18304577c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:17.478 INFO 2981023 --- [ocp-async-5,dddc2a76c70ed6ee,4492d809814ff225] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:17.799 INFO 2981023 --- [ocp-async-8,f2efcdf5a77a1066,d1c48e76fa066cd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:30:17.813 INFO 2981023 --- [host-check-clockdiff2,56cdc8d606958916,164e3cf2c2f1d169] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050617 5 5 , error= 2024-09-11 18:30:18.293 INFO 2981023 --- [ocp-schedules-21,e294e1e81402db5e,0f8dc37d5d60bbf1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:30:18.341 INFO 2981023 --- [alarm-detect-1,e294e1e81402db5e,57d9eaefba03ebaf] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050585 ~ 1726050608 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050600, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050600, value=98.301)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.301, values={1726050600=98.301}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050600=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:18.391 INFO 2981023 --- [alarm-detect-10,e294e1e81402db5e,50ecea4cc03b7918] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050585 ~ 1726050608 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050600, value=98.301)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.301, values={1726050600=98.301}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:18.394 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:18.395 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:18.397 INFO 2981023 --- [alarm-detect-19,e294e1e81402db5e,481a919a4cc3d81f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050585 ~ 1726050608 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050600, value=9.392)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=9.392, values={1726050600=9.392}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:18.397 INFO 2981023 --- [alarm-detect-19,e294e1e81402db5e,481a919a4cc3d81f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050585 ~ 1726050608 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050600, value=9.392)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=9.392, values={1726050600=9.392}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:30:18.401 WARN 2981023 --- [scanner-13681264-1,e294e1e81402db5e,f57673684e5fbce6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:30:18.416 INFO 2981023 --- [alarm-detect-2,e294e1e81402db5e,476c276bde5e1fb6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050585 ~ 1726050608 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050600, value=98.301)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.301, values={1726050600=98.301}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:18.524 INFO 2981023 --- [alarm-detect-3,e294e1e81402db5e,7c3030624496f349] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050585 ~ 1726050608 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050600, value=105.428)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=105.428, values={1726050600=105.428}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:18.564 INFO 2981023 --- [host-check-clockdiff1,56cdc8d606958916,30cb17a728b50669] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050618 0 0 , error= 2024-09-11 18:30:19.478 INFO 2981023 --- [ocp-async-10,7d0220e58db00300,92669fad8e023d67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,023f89121519cf40,44ee1103b244b6a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,023f89121519cf40,44ee1103b244b6a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:21.395 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:21.397 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:21.478 INFO 2981023 --- [ocp-async-9,1ed778909471aa93,c2f87543a55dd072] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:22.009 INFO 2981023 --- [host-check-clockdiff4,56cdc8d606958916,306a4f9be832e602] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050622 0 0 , error= 2024-09-11 18:30:22.895 INFO 2981023 --- [ocp-schedules-27,a0d7513e50165ceb,a46cc56e5eb7028b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:30:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,1b54cf7a7fdac046,c2e3cea053cc3c50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,1b54cf7a7fdac046,c2e3cea053cc3c50] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:23.478 INFO 2981023 --- [ocp-async-7,26b0bbcd6fc4a92c,864681e838da0a88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:24.397 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:24.397 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:24.526 INFO 2981023 --- [ocp-schedules-3,ff4fac60a74204af,a41137cee9f56b78] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:30:25.478 INFO 2981023 --- [ocp-async-2,15d7178105045007,fc8ee9e3a1012aad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,7cfd508b4ce147a2,02b148c51a08be80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,7cfd508b4ce147a2,02b148c51a08be80] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:30:26.352315+08:00, lastProcessTime=2024-09-11T18:30:16.472482+08:00 2024-09-11 18:30:26.480 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:30:26.352315+08:00 2024-09-11 18:30:27.397 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:27.398 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:27.478 INFO 2981023 --- [ocp-async-10,6beb7db960c2c1a1,bb77e532c1bc613b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:27.799 INFO 2981023 --- [ocp-async-7,9646efeea0cab9e9,0f1fffb2fe74f6b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:30:28.315 INFO 2981023 --- [ocp-schedules-11,e2a329e537eb773d,0bf891f638ca57f7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:30:28.315 INFO 2981023 --- [ocp-schedules-11,e2a329e537eb773d,0bf891f638ca57f7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:30:28.315 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:30:28.315 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:30:28.316 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:30:28.341 INFO 2981023 --- [ocp-schedules-11,e2a329e537eb773d,0bf891f638ca57f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:30:28.343 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:30:28.345 INFO 2981023 --- [ocp-schedules-11,e2a329e537eb773d,0bf891f638ca57f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:30:28.345 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:30:28.346 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:30:28.347 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:30:28.347 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:30:28.348 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:30:28.375 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:30:28.375 INFO 2981023 --- [ocp-schedules-11,e2a329e537eb773d,0bf891f638ca57f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:30:28.375 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:30:28.376 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:30:28.380 INFO 2981023 --- [ocp-schedules-11,e2a329e537eb773d,0bf891f638ca57f7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:30:28.539 INFO 2981023 --- [ocp-schedules-1,35bf4848ae4763ab,5655a7927d4e0d32] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:30:28.582 INFO 2981023 --- [alarm-detect-3,35bf4848ae4763ab,04d50d20b29bbd0a] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_hwm detect time range 1726050600 ~ 1726050618 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_steal=[SeriesMetricValues(metric=Metric{name='cpu_steal', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050615, value=0.0)])], cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050615, value=98.779)])]}, metricEvaluateResults={cpu_used_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_used_percent, match=true, value=98.779, values={1726050615=98.779}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})], cpu_steal < 20=[ExpressionEvaluateResult(matchValue=20.0, metric=cpu_steal, match=true, value=0.0, values={1726050615=0.0}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:28.589 WARN 2981023 --- [scanner-13681440-1,35bf4848ae4763ab,d003382c2056ef2c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:30:28.600 INFO 2981023 --- [alarm-detect-2,35bf4848ae4763ab,6d72dce5f135433d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646545, data is empty. 2024-09-11 18:30:28.603 INFO 2981023 --- [alarm-detect-2,35bf4848ae4763ab,6d72dce5f135433d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646545, data is empty. 2024-09-11 18:30:28.608 INFO 2981023 --- [alarm-detect-2,35bf4848ae4763ab,6d72dce5f135433d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646545, data is empty. 2024-09-11 18:30:28.632 INFO 2981023 --- [alarm-detect-6,35bf4848ae4763ab,e3bd13f551fa8ad9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646545, data is empty. 2024-09-11 18:30:28.635 INFO 2981023 --- [alarm-detect-6,35bf4848ae4763ab,e3bd13f551fa8ad9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646545, data is empty. 2024-09-11 18:30:28.635 INFO 2981023 --- [alarm-detect-17,35bf4848ae4763ab,6225d4cf624d724e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646545, data is empty. 2024-09-11 18:30:28.642 INFO 2981023 --- [alarm-detect-6,35bf4848ae4763ab,e3bd13f551fa8ad9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646545, data is empty. 2024-09-11 18:30:28.645 INFO 2981023 --- [alarm-detect-17,35bf4848ae4763ab,6225d4cf624d724e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646545, data is empty. 2024-09-11 18:30:28.654 INFO 2981023 --- [alarm-detect-17,35bf4848ae4763ab,6225d4cf624d724e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646545, data is empty. 2024-09-11 18:30:28.660 INFO 2981023 --- [alarm-detect-4,35bf4848ae4763ab,3ccd65e4b330f6df] c.o.o.alarm.core.detect.MetricDetector : alarmType os_tsar_cpu_util_full detect time range 1726050600 ~ 1726050618 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_used_percent=[SeriesMetricValues(metric=Metric{name='cpu_used_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050615, value=98.779)])]}, metricEvaluateResults={cpu_used_percent >= 98=[ExpressionEvaluateResult(matchValue=98.0, metric=cpu_used_percent, match=true, value=98.779, values={1726050615=98.779}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:28.665 INFO 2981023 --- [alarm-detect-9,35bf4848ae4763ab,bc6fbe2280f3caf5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_cpu_percent_over_threshold detect time range 1726050600 ~ 1726050618 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={cpu_percent=[SeriesMetricValues(metric=Metric{name='cpu_percent', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050615, value=98.779)])]}, metricEvaluateResults={cpu_percent > 90=[ExpressionEvaluateResult(matchValue=90.0, metric=cpu_percent, match=true, value=98.779, values={1726050615=98.779}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:28.757 INFO 2981023 --- [alarm-detect-4,35bf4848ae4763ab,3f6e8446cbc31251] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646545, data is empty. 2024-09-11 18:30:28.761 INFO 2981023 --- [alarm-detect-4,35bf4848ae4763ab,3f6e8446cbc31251] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646545, data is empty. 2024-09-11 18:30:28.766 INFO 2981023 --- [alarm-detect-4,35bf4848ae4763ab,3f6e8446cbc31251] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646545, data is empty. 2024-09-11 18:30:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,24a399e200155112,7cf45b499c23bcf6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,24a399e200155112,7cf45b499c23bcf6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:29.478 INFO 2981023 --- [ocp-async-5,b57e8ed0d5ad74a1,44c9b8a738006912] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:30.398 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:30.399 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:31.478 INFO 2981023 --- [ocp-async-3,205a662fb851a62b,b83e21811372861e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,73ab1d89c5b396b6,eb136b2cb484e16d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,73ab1d89c5b396b6,eb136b2cb484e16d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:32.897 INFO 2981023 --- [ocp-schedules-8,501eb32ac78db9c1,0c6e9ed3f1e73faf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:30:33.399 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:33.400 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:33.479 INFO 2981023 --- [ocp-async-4,f4f30ac4d380ceb7,3ada0da8a3eff656] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,e5d92a105abf53ff,19104d8c6bbbd896] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,e5d92a105abf53ff,19104d8c6bbbd896] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:35.479 INFO 2981023 --- [ocp-async-9,9fef11ef80c91b78,bc9e20641f2b5a1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:35.941 INFO 2981023 --- [ocp-schedules-11,477d2057a2564923,850d89ce7c39da0e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:30:35.949 INFO 2981023 --- [ocp-schedules-11,477d2057a2564923,850d89ce7c39da0e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:30:35.949 INFO 2981023 --- [ocp-schedules-11,477d2057a2564923,850d89ce7c39da0e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:30:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:30:36.336904+08:00, lastProcessTime=2024-09-11T18:30:26.352315+08:00 2024-09-11 18:30:36.400 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:36.401 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:30:36.336904+08:00 2024-09-11 18:30:37.314 INFO 2981023 --- [prometheus-cache-monitor1,f9f91bbd8cf5e8bb,841d84d1a5ed3875] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8195,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:30:37.315 INFO 2981023 --- [prometheus-cache-monitor1,f9f91bbd8cf5e8bb,841d84d1a5ed3875] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:30:37.479 INFO 2981023 --- [ocp-async-7,db911b60a9d5a5ea,9422ad9a3f66d65c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:37.799 INFO 2981023 --- [ocp-async-4,1db7eee19b640e2b,34a6b7029b6246a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:30:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,b577143410be9e8d,2bf3e485ee2cbd71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,b577143410be9e8d,2bf3e485ee2cbd71] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:38.802 INFO 2981023 --- [ocp-schedules-4,091703dc4ca6fb4d,ab053ae308767524] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:30:38.848 WARN 2981023 --- [scanner-13681766-1,091703dc4ca6fb4d,be6d5bc7df507488] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:30:38.891 INFO 2981023 --- [alarm-detect-17,091703dc4ca6fb4d,8b2b1068e076ac92] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050600 ~ 1726050628 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050615, value=8.353)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=8.353, values={1726050615=8.353}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:38.891 INFO 2981023 --- [alarm-detect-17,091703dc4ca6fb4d,8b2b1068e076ac92] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050600 ~ 1726050628 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050615, value=8.353)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=8.353, values={1726050615=8.353}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:30:39.028 INFO 2981023 --- [alarm-detect-6,091703dc4ca6fb4d,98442c71cf15d505] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_cpu_usage_over_threshold detect time range 1726050600 ~ 1726050628 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_cpu_percent=[SeriesMetricValues(metric=Metric{name='ob_tenant_cpu_percent', labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit}}, data=[ValueData(timestamp=1726050615, value=102.523)])]}, metricEvaluateResults={ob_tenant_cpu_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_tenant_cpu_percent, match=true, value=102.523, values={1726050615=102.523}, valueCount=1, labels={app=OB, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, ob_tenant_id=1004, obzone=zone1, svr_ip=10.31.35.5, tenant_name=linkreit})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:39.401 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:39.401 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:39.479 INFO 2981023 --- [ocp-async-1,5cfd08d604a4f630,04607004e2f78add] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:40.003 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:30:40.004 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:30:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,7efdf4f5f4fbb02b,b2f7cbf9cc9db081] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:41.318 INFO 2981023 --- [prometheus-minute-schedule-1,7efdf4f5f4fbb02b,b2f7cbf9cc9db081] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:41.479 INFO 2981023 --- [ocp-async-6,70c9993c9e3b1928,c41dfa82c414491b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:42.401 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:42.403 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:42.900 INFO 2981023 --- [ocp-schedules-9,0e87838818bb5c5c,644691610fda9117] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:30:43.479 INFO 2981023 --- [ocp-async-8,d31359f055ef4cd4,ebf6d2035a486a36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,fbc197706db3d1aa,2afcf669db605697] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,fbc197706db3d1aa,2afcf669db605697] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:45.101 INFO 2981023 --- [prometheus-second-schedule-1,7596d8fbda1acab4,1bb0dddcdcb05534] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:30:45.403 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:45.404 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:45.479 INFO 2981023 --- [ocp-async-3,2ef96e251069aa51,2f09cafd74d57d9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:46.015 INFO 2981023 --- [ocp-schedules-14,f5795663a007f17c,67ca5e8172b2f1a3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:30:46.015 INFO 2981023 --- [host-check-clockdiff1,f5795663a007f17c,77f52642ed6f2c91] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:30:46.016 INFO 2981023 --- [host-check-clockdiff2,f5795663a007f17c,bee1100e57d6b087] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:30:46.016 INFO 2981023 --- [host-check-clockdiff3,f5795663a007f17c,b527c9ad2a5894d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:30:46.023 INFO 2981023 --- [ocp-schedules-28,ee85a987cf6ce3de,00e19d5aec2024fa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:30:46.024 INFO 2981023 --- [host-check-clockdiff4,f5795663a007f17c,61e3549a84aca22c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:30:46.027 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:30:46.027 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:30:46.028 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:30:46.034 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:30:46.035 INFO 2981023 --- [host-check-clockdiff3,f5795663a007f17c,b527c9ad2a5894d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050646 0 0 , error= 2024-09-11 18:30:46.048 INFO 2981023 --- [host-check-clockdiff2,f5795663a007f17c,bee1100e57d6b087] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050646 5 5 , error= 2024-09-11 18:30:46.050 INFO 2981023 --- [ocp-schedules-3,376de98302a500f1,276e59b1e24ca04c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:30:46.056 INFO 2981023 --- [host-check-agent-status1,376de98302a500f1,4730a244df948146] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:30:46.057 INFO 2981023 --- [host-check-agent-status2,376de98302a500f1,5af40679e852a9e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:30:46.057 INFO 2981023 --- [host-check-agent-status3,376de98302a500f1,c3fa4373ca5d5aac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:30:46.057 INFO 2981023 --- [host-check-agent-status4,376de98302a500f1,1c624d5d195d5eff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:30:46.063 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:30:46.064 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:30:46.065 INFO 2981023 --- [host-check-agent-status2,376de98302a500f1,5af40679e852a9e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:30:46.068 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:30:46.069 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:30:46.069 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:30:46.074 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:30:46.075 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:30:46.076 INFO 2981023 --- [host-check-agent-status2,376de98302a500f1,5af40679e852a9e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:30:46.079 INFO 2981023 --- [host-check-agent-status4,376de98302a500f1,1c624d5d195d5eff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:30:46.080 INFO 2981023 --- [host-check-agent-status4,376de98302a500f1,1c624d5d195d5eff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:30:46.081 INFO 2981023 --- [host-check-agent-status3,376de98302a500f1,c3fa4373ca5d5aac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:30:46.081 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:30:46.081 INFO 2981023 --- [host-check-agent-status1,376de98302a500f1,4730a244df948146] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:30:46.102 INFO 2981023 --- [host-check-agent-status3,376de98302a500f1,c3fa4373ca5d5aac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:30:46.109 INFO 2981023 --- [host-check-agent-status1,376de98302a500f1,4730a244df948146] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:30:46.183 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:30:46.197 INFO 2981023 --- [ocp-schedules-8,121bc2aa5addf4ce,1af2d862e83b8912] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:30:46.197 INFO 2981023 --- [ocp-schedules-23,454377718c2bc99c,57d3b216c4d4cd36] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:30:46.197 INFO 2981023 --- [ocp-schedules-17,814fdcef09d5027f,874527c00c7a3340] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:30:46.197 INFO 2981023 --- [ocp-schedules-13,3fd64db746079436,e3d489873da38bf0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:30:46.265 INFO 2981023 --- [ocp-schedules-8,121bc2aa5addf4ce,1af2d862e83b8912] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:30:46.273 INFO 2981023 --- [ocp-schedules-17,814fdcef09d5027f,874527c00c7a3340] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:30:46.289 INFO 2981023 --- [ocp-schedules-15,2511fa2124dfefad,bbd312e48d7bebdf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:30:46.291 INFO 2981023 --- [ocp-schedules-23,454377718c2bc99c,57d3b216c4d4cd36] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:30:46.316 INFO 2981023 --- [ocp-schedules-8,121bc2aa5addf4ce,1af2d862e83b8912] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:30:46.332 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050603337000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:30:46.357 INFO 2981023 --- [ocp-schedules-12,fd74c75bb9e4461a,46ad38e2419d2f0e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:30:46.357 INFO 2981023 --- [ocp-schedules-15,2511fa2124dfefad,bbd312e48d7bebdf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:30:46.375 INFO 2981023 --- [host-check-agent-status4,376de98302a500f1,1c624d5d195d5eff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:30:46.376 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:30:46.377 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:30:46.380 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:30:46.379365848+08:00 2024-09-11 18:30:46.380 INFO 2981023 --- [host-check-agent-status4,376de98302a500f1,1c624d5d195d5eff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:30:46.380 INFO 2981023 --- [host-check-agent-status4,376de98302a500f1,1c624d5d195d5eff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:30:46.380 INFO 2981023 --- [host-check-agent-status4,376de98302a500f1,1c624d5d195d5eff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:30:46.386 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:30:46.389 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:30:46.388987781+08:00 2024-09-11 18:30:46.389 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:30:46.391 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:30:46.391500572+08:00 2024-09-11 18:30:46.397 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:30:46.397 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:30:46.391+08:00 2024-09-11 18:30:46.398 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:30:46.389+08:00 2024-09-11 18:30:46.398 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:30:46.380+08:00 2024-09-11 18:30:46.399 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:30:46.404793001+08:00 2024-09-11 18:30:46.401 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:30:46.389+08:00 and status AVAILABLE 2024-09-11 18:30:46.401 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:30:46.391+08:00 and status AVAILABLE 2024-09-11 18:30:46.454 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:30:46.380+08:00 and status AVAILABLE 2024-09-11 18:30:46.459 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:30:46.399+08:00 2024-09-11 18:30:46.460 INFO 2981023 --- [host-check-agent-status2,376de98302a500f1,5af40679e852a9e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:30:46.460 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:30:46.459454+08:00, lastProcessTime=2024-09-11T18:30:36.336904+08:00 2024-09-11 18:30:46.463 INFO 2981023 --- [host-check-agent-status2,376de98302a500f1,5af40679e852a9e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:30:46.463 INFO 2981023 --- [host-check-agent-status2,376de98302a500f1,5af40679e852a9e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:30:46.463 INFO 2981023 --- [host-check-agent-status2,376de98302a500f1,5af40679e852a9e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:30:46.466 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:30:46.399+08:00 and status AVAILABLE 2024-09-11 18:30:46.468 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:30:46.476 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:30:46.477 INFO 2981023 --- [host-check-agent-status1,376de98302a500f1,4730a244df948146] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:30:46.479 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:30:46.480 INFO 2981023 --- [host-check-agent-status3,376de98302a500f1,c3fa4373ca5d5aac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:30:46.481 INFO 2981023 --- [host-check-agent-status1,376de98302a500f1,4730a244df948146] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:30:46.481 INFO 2981023 --- [host-check-agent-status1,376de98302a500f1,4730a244df948146] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:30:46.481 INFO 2981023 --- [host-check-agent-status1,376de98302a500f1,4730a244df948146] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:30:46.497 INFO 2981023 --- [host-check-agent-status3,376de98302a500f1,c3fa4373ca5d5aac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:30:46.497 INFO 2981023 --- [host-check-agent-status3,376de98302a500f1,c3fa4373ca5d5aac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:30:46.497 INFO 2981023 --- [host-check-agent-status3,376de98302a500f1,c3fa4373ca5d5aac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:30:46.499 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:30:46.501 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:30:46.501 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:30:46.503 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:30:46.503 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:30:46.558 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.558 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:46.562 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.562 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:46.563 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:30:46.565 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:30:46.569 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.569 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:30:46.570 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.570 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:30:46.570 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:30:46.571 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:30:46.571 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.571 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:30:46.574 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:30:46.578 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:30:46.578 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.579 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:30:46.580 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:30:46.581 INFO 2981023 --- [ocp-schedules-23,454377718c2bc99c,57d3b216c4d4cd36] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:30:46.582 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:30:46.589 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.589 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:46.593 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:30:46.597 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:30:46.598 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:30:46.603 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:30:46.611 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:30:46.622 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:30:46.623 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:30:46.630 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:30:46.664 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:30:46.664 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050603337000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:30:46.670 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:30:46.670 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.671 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.671 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.671 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:30:46.672 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:30:46.672 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.672 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:30:46.673 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:30:46.674 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.685 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:30:46.688 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:30:46.694 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:30:46.700 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:30:46.709 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:30:46.712 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.713 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.714 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:46.715 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:30:46.716 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.717 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:30:46.718 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:30:46.719 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:30:46.721 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:30:46.729 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:30:46.735 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:30:46.735 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:30:46.737 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:30:46.754 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.755 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:30:46.759 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:30:46.759 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:30:46.767 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.768 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:30:46.769 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:30:46.769 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.770 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:30:46.770 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:30:46.771 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.772 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:30:46.772 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:30:46.774 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:30:46.775 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.777 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.782 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:30:46.782 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.782 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.783 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.783 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:30:46.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:30:46.784 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:30:46.784 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:30:46.787 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:30:46.788 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:30:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:30:46.792 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:30:46.795 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:30:46.798 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:30:46.805 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:30:46.813 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:30:46.813 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.814 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.815 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:30:46.815 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:30:46.816 INFO 2981023 --- [host-check-host-status4,ee85a987cf6ce3de,45cb3c489826914a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:30:46.816 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:30:46.817 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.820 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:30:46.820 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:30:46.821 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:30:46.824 INFO 2981023 --- [host-check-host-status3,ee85a987cf6ce3de,f19337bd5347ba1e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:30:46.825 INFO 2981023 --- [host-check-host-status2,ee85a987cf6ce3de,9367ca64bf1b6bea] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:30:46.832 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:30:46.834 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.834 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:30:46.834 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:30:46.835 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.835 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:30:46.836 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:30:46.837 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.854 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:30:46.873 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:30:46.878 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:30:46.886 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.886 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.887 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:30:46.888 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:30:46.888 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:30:46.892 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:30:46.898 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:30:46.899 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:30:46.906 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:30:46.459454+08:00 2024-09-11 18:30:46.908 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.911 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.912 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:30:46.913 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:30:46.913 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.917 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:30:46.918 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:30:46.922 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:30:46.924 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:46.925 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:46.925 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:30:46.928 INFO 2981023 --- [host-check-host-status1,ee85a987cf6ce3de,051b4e956624c8ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:30:46.932 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:30:46.932 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:30:46.956 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:30:46.975 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:30:46.981 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:30:46.984 INFO 2981023 --- [ocp-schedules-16,6fbf3cc897089f1c,86766a196b549df2] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:30:47.007 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:30:47.013 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:30:47.013 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:47.014 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:30:47.019 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:30:47.019 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:30:47.019 INFO 2981023 --- [ocp-schedules-17,814fdcef09d5027f,874527c00c7a3340] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:30:47.297 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:30:47.297 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:30:47.300 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:30:47.300 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:30:47.300 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:30:47.300 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:30:47.302 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:30:47.302 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:30:47.302 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:30:47.302 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:30:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,403ba79d7b3d43bd,cdc9dd5d9eb20ff7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:47.319 INFO 2981023 --- [prometheus-minute-schedule-1,403ba79d7b3d43bd,cdc9dd5d9eb20ff7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:47.328 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:30:47.334 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:30:47.334 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:30:47.334 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:30:47.334 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:30:47.334 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:30:47.340 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:30:47.479 INFO 2981023 --- [ocp-async-6,f453d123297a23e7,d850a45631492d25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:47.578 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:30:47.799 INFO 2981023 --- [ocp-async-1,b6822685f90d50ad,9b8b6e0cb1e3f7e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:30:47.842 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.540446154S 2024-09-11 18:30:47.843 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:30:47.889 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:30:47.889 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:30:47.904 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:30:47.936 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:30:47.936 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:30:47.936 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:30:47.936 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:30:47.936 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:30:47.936 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:30:47.943 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:30:48.021 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] o.s.boot.SpringApplication : Started application in 0.721 seconds (JVM running for 86447.893) 2024-09-11 18:30:48.021 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.721170208S 2024-09-11 18:30:48.237 INFO 2981023 --- [ocp-schedules-2,cd29370fdd9fd9c1,855ce783898e7cd1] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:30:48.404 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:48.405 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:48.548 INFO 2981023 --- [host-check-clockdiff1,f5795663a007f17c,77f52642ed6f2c91] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050648 0 0 , error= 2024-09-11 18:30:48.749 INFO 2981023 --- [ocp-schedules-8,d4a237d0bcd36f8e,ededd9ea08e5675b] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:30:48.768 WARN 2981023 --- [ocp-schedules-8,d4a237d0bcd36f8e,ededd9ea08e5675b] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:30:49.040 INFO 2981023 --- [ocp-schedules-20,79dbc6d0dc69f714,53b4dabbf96e4247] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:30:49.126 WARN 2981023 --- [scanner-13682018-1,79dbc6d0dc69f714,44f6050ddb32990a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:30:49.138 INFO 2981023 --- [alarm-detect-14,79dbc6d0dc69f714,57f9e65be9a99ad9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646545, data is empty. 2024-09-11 18:30:49.143 INFO 2981023 --- [alarm-detect-14,79dbc6d0dc69f714,57f9e65be9a99ad9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646545, data is empty. 2024-09-11 18:30:49.148 INFO 2981023 --- [alarm-detect-14,79dbc6d0dc69f714,57f9e65be9a99ad9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646545, data is empty. 2024-09-11 18:30:49.189 INFO 2981023 --- [alarm-detect-15,79dbc6d0dc69f714,4d0ce46da88e6bab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646545, data is empty. 2024-09-11 18:30:49.192 INFO 2981023 --- [alarm-detect-19,79dbc6d0dc69f714,40e58290d0929f36] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646545, data is empty. 2024-09-11 18:30:49.194 INFO 2981023 --- [alarm-detect-15,79dbc6d0dc69f714,4d0ce46da88e6bab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646545, data is empty. 2024-09-11 18:30:49.199 INFO 2981023 --- [alarm-detect-19,79dbc6d0dc69f714,40e58290d0929f36] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646545, data is empty. 2024-09-11 18:30:49.203 INFO 2981023 --- [alarm-detect-19,79dbc6d0dc69f714,40e58290d0929f36] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646545, data is empty. 2024-09-11 18:30:49.205 INFO 2981023 --- [alarm-detect-15,79dbc6d0dc69f714,4d0ce46da88e6bab] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646545, data is empty. 2024-09-11 18:30:49.215 INFO 2981023 --- [alarm-detect-5,79dbc6d0dc69f714,c9c72eff921c81b7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050615 ~ 1726050639 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050630, value=6.968)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=6.968, values={1726050630=6.968}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:30:49.215 INFO 2981023 --- [alarm-detect-5,79dbc6d0dc69f714,c9c72eff921c81b7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050615 ~ 1726050639 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050630, value=6.968)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=6.968, values={1726050630=6.968}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:30:49.307 INFO 2981023 --- [alarm-detect-19,79dbc6d0dc69f714,fbc01031b7053207] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646545, data is empty. 2024-09-11 18:30:49.312 INFO 2981023 --- [alarm-detect-19,79dbc6d0dc69f714,fbc01031b7053207] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646545, data is empty. 2024-09-11 18:30:49.317 INFO 2981023 --- [alarm-detect-19,79dbc6d0dc69f714,fbc01031b7053207] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646545, data is empty. 2024-09-11 18:30:49.479 INFO 2981023 --- [ocp-async-2,21f7b4baadebdbff,dbb6e2ac536d81f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:30:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:30:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,32dd181d4abd02c1,8ee95b693b8756a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:50.318 INFO 2981023 --- [prometheus-minute-schedule-1,32dd181d4abd02c1,8ee95b693b8756a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:50.382 INFO 2981023 --- [ocp-schedules-12,ea2103f3f82c5aa4,1e4423c31102c802] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:30:51.405 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:51.407 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:51.480 INFO 2981023 --- [ocp-async-3,1eed5d75f78f3cb5,5f514b1cfefc8583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:51.650 INFO 2981023 --- [ocp-schedules-8,c42efecf6b5d6a70,9f8ab9adb3abae37] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:30:51.651 INFO 2981023 --- [ocp-schedules-8,c42efecf6b5d6a70,9f8ab9adb3abae37] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:30:51.652 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:30:51.652 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:30:51.652 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:30:51.652 INFO 2981023 --- [ocp-schedules-8,c42efecf6b5d6a70,9f8ab9adb3abae37] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:30:52.023 INFO 2981023 --- [host-check-clockdiff4,f5795663a007f17c,61e3549a84aca22c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050652 0 0 , error= 2024-09-11 18:30:52.902 INFO 2981023 --- [ocp-schedules-16,a3dd8b9bd6216716,7ae9f1e96a7b72e0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:30:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,d5dd274d986e78f6,0788bf7d683ec89c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,d5dd274d986e78f6,0788bf7d683ec89c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:53.404 INFO 2981023 --- [ocp-schedules-29,059e7950be6b6d1e,0b800510ffce1bb9] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:30:53.404 INFO 2981023 --- [ocp-schedules-29,059e7950be6b6d1e,0b800510ffce1bb9] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:30:53.480 INFO 2981023 --- [ocp-async-5,21475df658a5b4ef,af1f99d83596a86f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:54.407 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:54.408 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:54.528 INFO 2981023 --- [ocp-schedules-4,f8ff9fd86ee65467,4c999bedc35d886f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:30:55.176 INFO 2981023 --- [ocp-schedules-2,5308bc7fcd9523db,5506916513c0051c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:30:55.176 INFO 2981023 --- [ocp-schedules-2,5308bc7fcd9523db,5506916513c0051c] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:30:55.181 INFO 2981023 --- [ocp-schedules-2,5308bc7fcd9523db,5506916513c0051c] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:30:55.181 INFO 2981023 --- [ocp-schedules-2,5308bc7fcd9523db,5506916513c0051c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:30:55.181 INFO 2981023 --- [ocp-schedules-2,5308bc7fcd9523db,5506916513c0051c] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:30:55.183 INFO 2981023 --- [ocp-schedules-2,5308bc7fcd9523db,5506916513c0051c] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:30:55.183 INFO 2981023 --- [ocp-schedules-2,5308bc7fcd9523db,5506916513c0051c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:30:55.183 INFO 2981023 --- [ocp-schedules-2,5308bc7fcd9523db,5506916513c0051c] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:30:55.193 INFO 2981023 --- [ocp-schedules-2,5308bc7fcd9523db,5506916513c0051c] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:30:55.480 INFO 2981023 --- [ocp-async-7,f5a9f81e9da8306e,c333b914b841f458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:55.598 INFO 2981023 --- [ocp-schedules-26,8f9c89153689f99e,d8c578cebdc44a94] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:30:55.615 WARN 2981023 --- [ocp-schedules-26,8f9c89153689f99e,d8c578cebdc44a94] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:30:55.934 INFO 2981023 --- [ocp-schedules-7,71540b25c0fd56dd,f66ca65889529d02] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:30:55.934 INFO 2981023 --- [ocp-schedules-7,71540b25c0fd56dd,f66ca65889529d02] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:30:55.936 INFO 2981023 --- [ocp-schedules-7,71540b25c0fd56dd,f66ca65889529d02] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:30:55.936 INFO 2981023 --- [ocp-schedules-7,71540b25c0fd56dd,f66ca65889529d02] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:30:55.939 INFO 2981023 --- [ocp-schedules-7,71540b25c0fd56dd,f66ca65889529d02] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:30:55.939 INFO 2981023 --- [ocp-schedules-7,71540b25c0fd56dd,f66ca65889529d02] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:30:55.952 INFO 2981023 --- [ocp-schedules-20,f205c653a2a24248,4b62b9764ab80c0b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:30:55.961 INFO 2981023 --- [ocp-schedules-20,f205c653a2a24248,4b62b9764ab80c0b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:30:55.961 INFO 2981023 --- [ocp-schedules-20,f205c653a2a24248,4b62b9764ab80c0b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:30:56.170 INFO 2981023 --- [ocp-schedules-15,2650bcefe009f382,c43ec62c2fa81b18] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:30:56.180 INFO 2981023 --- [ocp-schedules-15,2650bcefe009f382,c43ec62c2fa81b18] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:30:56.180 INFO 2981023 --- [ocp-schedules-15,2650bcefe009f382,c43ec62c2fa81b18] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:30:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,d6d4bf95fad7d698,0770f7fc8383b3d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,d6d4bf95fad7d698,0770f7fc8383b3d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:30:56.351910+08:00, lastProcessTime=2024-09-11T18:30:46.459454+08:00 2024-09-11 18:30:56.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:30:56.351910+08:00 2024-09-11 18:30:57.408 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:30:57.409 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:30:57.480 INFO 2981023 --- [ocp-async-4,fdbb28eff0b32ebf,98058403475d62d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:30:57.799 INFO 2981023 --- [ocp-async-8,6c097ec084aca9a8,9e562655e6ba1e99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:30:59.106 INFO 2981023 --- [ocp-schedules-5,a58b3c8b52a7b0d7,01e189ce89340302] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:30:59.106 INFO 2981023 --- [ocp-schedules-5,a58b3c8b52a7b0d7,01e189ce89340302] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:30:59.129 INFO 2981023 --- [Collect all tenants major compactions1,a58b3c8b52a7b0d7,24b9a5beb75c8fb4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:30:59.130 INFO 2981023 --- [Collect all tenants major compactions2,a58b3c8b52a7b0d7,c66594234e085c74] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:30:59.131 INFO 2981023 --- [Collect all tenants major compactions5,a58b3c8b52a7b0d7,ee6eb7667c3c03cc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:30:59.131 INFO 2981023 --- [Collect all tenants major compactions3,a58b3c8b52a7b0d7,6e041dd4b517cee6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:30:59.132 INFO 2981023 --- [Collect all tenants major compactions6,a58b3c8b52a7b0d7,8629c7836ab9e8b0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:30:59.134 INFO 2981023 --- [Collect all tenants major compactions4,a58b3c8b52a7b0d7,93f4551e4d2ef165] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:30:59.153 INFO 2981023 --- [Collect all tenants major compactions4,a58b3c8b52a7b0d7,93f4551e4d2ef165] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:59.154 INFO 2981023 --- [Collect all tenants major compactions2,a58b3c8b52a7b0d7,c66594234e085c74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:59.154 INFO 2981023 --- [Collect all tenants major compactions3,a58b3c8b52a7b0d7,6e041dd4b517cee6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:59.155 INFO 2981023 --- [Collect all tenants major compactions3,a58b3c8b52a7b0d7,6e041dd4b517cee6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:30:59.155 INFO 2981023 --- [Collect all tenants major compactions5,a58b3c8b52a7b0d7,ee6eb7667c3c03cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:59.155 INFO 2981023 --- [Collect all tenants major compactions1,a58b3c8b52a7b0d7,24b9a5beb75c8fb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:59.156 INFO 2981023 --- [Collect all tenants major compactions2,a58b3c8b52a7b0d7,c66594234e085c74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:30:59.156 INFO 2981023 --- [Collect all tenants major compactions5,a58b3c8b52a7b0d7,ee6eb7667c3c03cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:30:59.156 INFO 2981023 --- [Collect all tenants major compactions4,a58b3c8b52a7b0d7,93f4551e4d2ef165] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:30:59.158 INFO 2981023 --- [Collect all tenants major compactions1,a58b3c8b52a7b0d7,24b9a5beb75c8fb4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:30:59.158 INFO 2981023 --- [Collect all tenants major compactions3,a58b3c8b52a7b0d7,6e041dd4b517cee6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:30:59.159 INFO 2981023 --- [Collect all tenants major compactions4,a58b3c8b52a7b0d7,93f4551e4d2ef165] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:30:59.160 INFO 2981023 --- [Collect all tenants major compactions6,a58b3c8b52a7b0d7,8629c7836ab9e8b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:30:59.160 INFO 2981023 --- [Collect all tenants major compactions1,a58b3c8b52a7b0d7,24b9a5beb75c8fb4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:30:59.162 INFO 2981023 --- [Collect all tenants major compactions3,a58b3c8b52a7b0d7,6e041dd4b517cee6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:30:59.162 INFO 2981023 --- [Collect all tenants major compactions6,a58b3c8b52a7b0d7,8629c7836ab9e8b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:30:59.162 INFO 2981023 --- [Collect all tenants major compactions5,a58b3c8b52a7b0d7,ee6eb7667c3c03cc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:30:59.162 INFO 2981023 --- [Collect all tenants major compactions2,a58b3c8b52a7b0d7,c66594234e085c74] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:30:59.162 INFO 2981023 --- [Collect all tenants major compactions5,a58b3c8b52a7b0d7,ee6eb7667c3c03cc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:30:59.162 INFO 2981023 --- [Collect all tenants major compactions2,a58b3c8b52a7b0d7,c66594234e085c74] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:30:59.163 INFO 2981023 --- [Collect all tenants major compactions1,a58b3c8b52a7b0d7,24b9a5beb75c8fb4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:30:59.164 INFO 2981023 --- [Collect all tenants major compactions6,a58b3c8b52a7b0d7,8629c7836ab9e8b0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:30:59.164 INFO 2981023 --- [Collect all tenants major compactions6,a58b3c8b52a7b0d7,8629c7836ab9e8b0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:30:59.164 INFO 2981023 --- [Collect all tenants major compactions4,a58b3c8b52a7b0d7,93f4551e4d2ef165] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:30:59.165 INFO 2981023 --- [ocp-schedules-5,a58b3c8b52a7b0d7,01e189ce89340302] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:30:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,75d461a74856b2b6,21b8d52c4011f23e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050660, interval=60 2024-09-11 18:30:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,75d461a74856b2b6,21b8d52c4011f23e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:30:59.357 INFO 2981023 --- [ocp-schedules-19,bf9d59872634af8f,f2cbc9383957fe45] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:30:59.405 WARN 2981023 --- [scanner-13682314-1,bf9d59872634af8f,e8bd82f67ea45164] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:30:59.480 INFO 2981023 --- [ocp-async-6,b7d2a93914a1c5b1,c95a192f1925d971] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:00.409 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:00.410 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:01.394 INFO 2981023 --- [ocp-schedules-28,08990d51cdeae152,8928bf3d157333c9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:31:01.394+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:31:01.410 INFO 2981023 --- [ocp-schedules-28,08990d51cdeae152,8928bf3d157333c9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:31:01.481 INFO 2981023 --- [ocp-async-1,7e6da76ab9cc8caf,db530c5ae0d0bedd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:01.608 INFO 2981023 --- [ocp-schedules-28,08990d51cdeae152,8928bf3d157333c9] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 52 to SqlAggregateData 2024-09-11 18:31:01.610 INFO 2981023 --- [ocp-schedules-28,08990d51cdeae152,8928bf3d157333c9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:31:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,c949b261e0440497,3b61cf2b111c269b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,c949b261e0440497,3b61cf2b111c269b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:02.905 INFO 2981023 --- [ocp-schedules-4,63f9cb361b732136,7c9cfa80525584f8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:31:03.410 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:03.411 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:03.481 INFO 2981023 --- [ocp-async-10,5d5d2373b08bef74,74f2d9b3568b6660] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:04.322 INFO 2981023 --- [ocp-schedules-24,5f4f67394cfcd918,61078d08e4b136a5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:31:04.323 INFO 2981023 --- [ocp-schedules-24,5f4f67394cfcd918,61078d08e4b136a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:31:04.355 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:31:04.355 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.358 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:31:04.358 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.359 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:31:04.360 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:31:04.360 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.361 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:31:04.362 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:31:04.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.368 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.369 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.369 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.370 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.371 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.379 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.383 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.384 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.386 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.386 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:31:04.386 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.387 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.395 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.395 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:31:04.400 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.401 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.401 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.402 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:31:04.408 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.409 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:31:04.411 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.412 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.412 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:31:04.415 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6D41D6EF-0-0] 2024-09-11 18:31:04.416 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:31:04.416 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.416 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:31:04.417 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.417 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:31:04.418 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.418 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:31:04.419 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:31:04.421 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.422 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:31:04.422 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.423 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:31:04.423 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.424 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:31:04.424 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.425 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:31:04.425 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:31:04.425 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.426 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:31:04.426 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.427 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:31:04.427 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.428 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:31:04.428 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.429 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:31:04.429 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.429 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:31:04.430 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:31:04.431 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.431 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:31:04.434 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:31:04.439 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:04.440 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.441 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.442 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:04.443 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.443 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:31:04.450 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:31:04.451 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.452 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:31:04.452 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:31:04.454 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:31:04.454 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.454 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:31:04.454 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:31:04.454 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:31:04.454 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:31:04.460 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:31:04.463 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:31:04.463 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.463 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.468 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:31:04.469 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:31:04.469 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:31:04.469 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:31:04.476 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.476 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:31:04.490 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:31:04.490 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6D71D6CE-0-0] 2024-09-11 18:31:04.490 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:31:04.496 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.497 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:31:04.511 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.514 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:31:04.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.515 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:31:04.516 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.516 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:31:04.517 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.517 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:31:04.517 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.518 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:31:04.518 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:31:04.520 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:31:04.529 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:04.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.549 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:31:04.562 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:31:04.563 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:31:04.568 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:04.572 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:31:04.588 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:31:04.589 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:31:04.589 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:31:04.589 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:31:04.590 INFO 2981023 --- [ocp-schedules-24,5f4f67394cfcd918,61078d08e4b136a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:31:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f99ae74bdf831ce8,af8e81e2affc633e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,f99ae74bdf831ce8,af8e81e2affc633e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:05.481 INFO 2981023 --- [ocp-async-9,37ed5ab7e7e16cad,e57eeca19529f80b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:31:06.337655+08:00, lastProcessTime=2024-09-11T18:30:56.351910+08:00 2024-09-11 18:31:06.411 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:06.411 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:06.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:31:06.337655+08:00 2024-09-11 18:31:07.481 INFO 2981023 --- [ocp-async-6,f8428cef6eecd073,49275b078b3f7de7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:07.799 INFO 2981023 --- [ocp-async-10,18ce997d544b1bdd,5b6b627774aea857] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:31:08.284 INFO 2981023 --- [ocp-schedules-13,f11d282080be9699,c1502ef509d55891] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:31:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,9ce296525890803f,edc64ec29281d389] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,9ce296525890803f,edc64ec29281d389] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:08.356 INFO 2981023 --- [ocp-schedules-21,a6e90ac76f837ac6,89e245e2d81e2c83] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:31:08.356+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:31:08.357 INFO 2981023 --- [ocp-schedules-21,a6e90ac76f837ac6,89e245e2d81e2c83] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:31:08.359 INFO 2981023 --- [ocp-schedules-21,a6e90ac76f837ac6,89e245e2d81e2c83] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:31:08.360 INFO 2981023 --- [ocp-schedules-21,a6e90ac76f837ac6,89e245e2d81e2c83] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:31:09.412 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:09.412 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:09.481 INFO 2981023 --- [ocp-async-3,a1ba29d723f50147,f666f473cf5858e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:09.548 INFO 2981023 --- [ocp-schedules-25,b2de0407f3d34e58,d45b269608464f5a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:31:09.559 WARN 2981023 --- [scanner-13682508-1,b2de0407f3d34e58,39383109cd381ab7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:31:09.615 INFO 2981023 --- [alarm-detect-9,b2de0407f3d34e58,d697e5838c46f99a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646545, data is empty. 2024-09-11 18:31:09.618 INFO 2981023 --- [alarm-detect-9,b2de0407f3d34e58,d697e5838c46f99a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646597, data is empty. 2024-09-11 18:31:09.620 INFO 2981023 --- [alarm-detect-9,b2de0407f3d34e58,d697e5838c46f99a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646597, data is empty. 2024-09-11 18:31:09.649 INFO 2981023 --- [alarm-detect-2,b2de0407f3d34e58,4feb6044aac1e38e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646545, data is empty. 2024-09-11 18:31:09.651 INFO 2981023 --- [alarm-detect-6,b2de0407f3d34e58,a402f22614150ab1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646545, data is empty. 2024-09-11 18:31:09.655 INFO 2981023 --- [alarm-detect-2,b2de0407f3d34e58,4feb6044aac1e38e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646597, data is empty. 2024-09-11 18:31:09.657 INFO 2981023 --- [alarm-detect-6,b2de0407f3d34e58,a402f22614150ab1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646597, data is empty. 2024-09-11 18:31:09.661 INFO 2981023 --- [alarm-detect-2,b2de0407f3d34e58,4feb6044aac1e38e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646597, data is empty. 2024-09-11 18:31:09.664 INFO 2981023 --- [alarm-detect-6,b2de0407f3d34e58,a402f22614150ab1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646597, data is empty. 2024-09-11 18:31:09.692 INFO 2981023 --- [alarm-detect-8,b2de0407f3d34e58,acf6d56bd21b51eb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050630 ~ 1726050659 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050645, value=5.577)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=5.577, values={1726050645=5.577}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:31:09.692 INFO 2981023 --- [alarm-detect-8,b2de0407f3d34e58,acf6d56bd21b51eb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050630 ~ 1726050659 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050645, value=5.577)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=5.577, values={1726050645=5.577}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:31:09.779 INFO 2981023 --- [alarm-detect-7,b2de0407f3d34e58,cc60cea03676a57e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646545, data is empty. 2024-09-11 18:31:09.784 INFO 2981023 --- [alarm-detect-7,b2de0407f3d34e58,cc60cea03676a57e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646597, data is empty. 2024-09-11 18:31:09.792 INFO 2981023 --- [alarm-detect-7,b2de0407f3d34e58,cc60cea03676a57e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646597, data is empty. 2024-09-11 18:31:10.002 INFO 2981023 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:31:10.002 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:31:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:31:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:31:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,0be327c36794f45f,fa3b19fa9c47a727] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,0be327c36794f45f,fa3b19fa9c47a727] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:11.389 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050600, nodeCount=285, affectRows=285 2024-09-11 18:31:11.482 INFO 2981023 --- [ocp-async-8,6e1ac7c69a509295,7222fe795b3311e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:12.169 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050600, nodeCount=1000, affectRows=1000 2024-09-11 18:31:12.172 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050600, nodeCount=999, validCount=999 2024-09-11 18:31:12.176 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050600, nodeCount=1000, affectRows=1000 2024-09-11 18:31:12.178 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050600, nodeCount=48, affectRows=48 2024-09-11 18:31:12.179 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050600, nodeCount=1000, validCount=1000 2024-09-11 18:31:12.179 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050600, nodeCount=46, validCount=46 2024-09-11 18:31:12.412 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:12.413 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:12.907 INFO 2981023 --- [ocp-schedules-11,b4d20f948c047f1e,96608cc685c419d8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:31:13.027 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050600, nodeCount=285, validCount=285 2024-09-11 18:31:13.073 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050600, nodeCount=195, affectRows=195 2024-09-11 18:31:13.428 INFO 2981023 --- [ocp-schedules-19,6f8c1bba8c9ead2c,46a9b00672bf2ff7] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:31:13.482 INFO 2981023 --- [ocp-async-2,163d4dfa430d1de6,37a35f5133150e18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:13.673 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:27:13.228, end=2024-09-11T18:29:13.665} 2024-09-11 18:31:13.679 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkpr 2024-09-11 18:31:13.679 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:31:13.681 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:13.228, end=2024-09-11T18:29:13.665},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:31:13.681 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:31:13.681 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:31:13.691 INFO 2981023 --- [plan_change_detector18,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:31:13.712 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273712000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.712Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.714 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:31:13.714 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.720 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.720 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.722 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.722 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.728 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.728 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.729 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:31:13.729 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.712, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.729 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:31:13.730 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:13.228, end=2024-09-11T18:29:13.665},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:31:13.730 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:31:13.730 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:31:13.731 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:31:13.734 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:13.228, end=2024-09-11T18:29:13.665},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:31:13.734 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:31:13.734 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:31:13.746 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:31:13.746 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:31:13.747 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:13.228, end=2024-09-11T18:29:13.665},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:31:13.747 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:31:13.747 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:31:13.748 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:31:13.748 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:13.228, end=2024-09-11T18:29:13.665},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:31:13.748 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:31:13.749 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:31:13.750 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:31:13.752 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:13.228, end=2024-09-11T18:29:13.665},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:31:13.752 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:31:13.752 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:31:13.794 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273794000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.794Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.802 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050600, nodeCount=195, validCount=195 2024-09-11 18:31:13.803 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273803000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.803Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.805 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273805000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.805Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.813 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273812000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.812Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.829 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273829000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.829Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.840 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.840 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.845 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:31:13.845 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.848 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273848000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.848Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.849 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273848000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.848Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.850 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 70 2024-09-11 18:31:13.850 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.851 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273851000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=8A1B851989150B802AEEACA99152D145, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.851Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.851 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.851 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.852 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.852 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.853 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:31:13.853 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.860 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 30 2024-09-11 18:31:13.860 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.862 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.862 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.864 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.864 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.864 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.864 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.866 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.866 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.869 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.869 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.872 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.872 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.873 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.873 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.875 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.875 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.876 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.876 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.876 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.876 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.876 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.876 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.877 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.877 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.878 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:13.879 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.880 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.883 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.883 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.887 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.887 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.889 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.889 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.890 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.890 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.890 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.794, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.891 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:13.891 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.891 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.891 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.891 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.892 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:13.894 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:13.894 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:13.896 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.896 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.896 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.896 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.899 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.899 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.901 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.805, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.901 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.812, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.901 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.901 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.906 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:13.908 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.829, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.908 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.803, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.910 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.910 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.912 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.914 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:13.914 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.851, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=8A1B851989150B802AEEACA99152D145, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.915 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:13.920 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.848, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.922 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.848, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.930 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273930000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.930Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.940 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:31:13.940 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.943 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273943000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.943Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.953 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.953 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.954 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.954 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.956 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273956000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.956Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.956 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.956 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.958 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273958000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.958Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.961 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273961000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=9DC649D7324D0017757D3894A9682422, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.961Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.964 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273964000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.964Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.968 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.968 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.969 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273969000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=563D19C678935F4895C494E06A5CA13F, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.969Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.971 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.971 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964273971000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:31:13.971Z, endTime=2024-09-11T10:29:13.665Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:31:13.971 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.975 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.975 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.975 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:13.979 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.979 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.982 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.982 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.982 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.982 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.986 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.986 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.988 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:13.988 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:13.989 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.990 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.990 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.990 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.992 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.992 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:13.993 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:31:13.993 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:31:13.994 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.93, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:13.995 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:13.995 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:14.001 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.001 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.002 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.002 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9DC649D7324D0017757D3894A9682422, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.003 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.003 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.005 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:14.006 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:14.006 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:14.006 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.006 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.007 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.007 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.008 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.008 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=563D19C678935F4895C494E06A5CA13F, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.011 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:14.011 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:14.011 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:14.011 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:14.011 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:14.011 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:31:14.011 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:31:14.011 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.011 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.013 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.956, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:14.015 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:14.017 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.943, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:14.021 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.021 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.022 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.958, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:14.022 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.022 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9DC649D7324D0017757D3894A9682422, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.023 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.023 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=563D19C678935F4895C494E06A5CA13F, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.023 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:31:14.023 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050553665000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:31:14.023 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:14.024 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:14.024 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:14.025 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:31:14.029 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.971, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:14.030 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.964, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:14.030 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.969, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=563D19C678935F4895C494E06A5CA13F, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:14.030 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:31:13.961, end=2024-09-11T18:29:13.665}, clusterId=2, tenantId=6, sqlId=9DC649D7324D0017757D3894A9682422, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:31:14.030 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:31:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,75b579d83470291b,5c39d50535fef25b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,c73ca575e8a90d37,afe87bd721b41770] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:31:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,75b579d83470291b,5c39d50535fef25b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:15.413 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:15.415 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:15.482 INFO 2981023 --- [ocp-async-1,c316853bfa36f1a8,d25d716bb952e268] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:15.964 INFO 2981023 --- [ocp-schedules-12,b83430093e80fcd9,5ed918f601f3d24b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:31:15.972 INFO 2981023 --- [ocp-schedules-12,b83430093e80fcd9,5ed918f601f3d24b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:31:15.972 INFO 2981023 --- [ocp-schedules-12,b83430093e80fcd9,5ed918f601f3d24b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:31:16.020 INFO 2981023 --- [ocp-schedules-9,88286daea6df33a1,fdbc0c1bab8e89b2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:31:16.020 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:31:16.021 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:31:16.021 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:31:16.022 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:31:16.060 INFO 2981023 --- [ocp-schedules-29,32eecfc67340a9a9,b83e470d830c5df0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:31:16.061 INFO 2981023 --- [host-check-clockdiff1,32eecfc67340a9a9,a2f455b06ce67972] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:31:16.061 INFO 2981023 --- [ocp-schedules-20,123ea87b4e3860cd,9abb40043dff6ad3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:31:16.061 INFO 2981023 --- [host-check-clockdiff2,32eecfc67340a9a9,eba4d1efdb5d56ca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:31:16.062 INFO 2981023 --- [host-check-clockdiff3,32eecfc67340a9a9,269078e6af24d1ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:31:16.063 INFO 2981023 --- [host-check-agent-status1,123ea87b4e3860cd,4b7fd204f63512e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:31:16.064 INFO 2981023 --- [host-check-agent-status2,123ea87b4e3860cd,8316d142c4060046] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:31:16.064 INFO 2981023 --- [host-check-clockdiff4,32eecfc67340a9a9,1b4d061e7a72a470] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:31:16.066 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:31:16.067 INFO 2981023 --- [host-check-agent-status3,123ea87b4e3860cd,d14ba31cb4ffd750] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:31:16.068 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:31:16.068 INFO 2981023 --- [host-check-agent-status4,123ea87b4e3860cd,710e65f1264300bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:31:16.072 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:31:16.075 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:31:16.087 INFO 2981023 --- [host-check-agent-status2,123ea87b4e3860cd,8316d142c4060046] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:31:16.087 INFO 2981023 --- [host-check-agent-status3,123ea87b4e3860cd,d14ba31cb4ffd750] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:31:16.088 INFO 2981023 --- [host-check-agent-status1,123ea87b4e3860cd,4b7fd204f63512e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:31:16.089 INFO 2981023 --- [host-check-agent-status3,123ea87b4e3860cd,d14ba31cb4ffd750] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:31:16.089 INFO 2981023 --- [host-check-agent-status1,123ea87b4e3860cd,4b7fd204f63512e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:31:16.089 INFO 2981023 --- [host-check-agent-status2,123ea87b4e3860cd,8316d142c4060046] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:31:16.090 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:31:16.091 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:31:16.093 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:31:16.097 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:31:16.097 INFO 2981023 --- [host-check-agent-status4,123ea87b4e3860cd,710e65f1264300bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:31:16.098 INFO 2981023 --- [host-check-agent-status4,123ea87b4e3860cd,710e65f1264300bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:31:16.109 INFO 2981023 --- [host-check-clockdiff1,32eecfc67340a9a9,a2f455b06ce67972] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050676 0 0 , error= 2024-09-11 18:31:16.115 INFO 2981023 --- [host-check-clockdiff3,32eecfc67340a9a9,269078e6af24d1ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050676 0 0 , error= 2024-09-11 18:31:16.189 INFO 2981023 --- [ocp-schedules-8,364beb0bcb76559e,aa89067717313a56] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:31:16.189 INFO 2981023 --- [ocp-schedules-25,599a075b81940c1c,e25b4c4b78b608e5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:31:16.189 INFO 2981023 --- [ocp-schedules-18,f198d30716332097,e88cf7d292d5273a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:31:16.189 INFO 2981023 --- [ocp-schedules-1,84d5271475465aa1,e5fcdbfb6c3226d7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:31:16.277 INFO 2981023 --- [ocp-schedules-8,364beb0bcb76559e,aa89067717313a56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:31:16.278 INFO 2981023 --- [ocp-schedules-25,599a075b81940c1c,e25b4c4b78b608e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:31:16.286 INFO 2981023 --- [ocp-schedules-18,f198d30716332097,e88cf7d292d5273a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:31:16.301 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050603337000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:31:16.301 INFO 2981023 --- [ocp-schedules-8,364beb0bcb76559e,aa89067717313a56] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:31:16.386 INFO 2981023 --- [ocp-schedules-21,64df7b188bc276cf,a8e754f86b372f86] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:31:16.461 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:31:16.464 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:31:16.464311678+08:00 2024-09-11 18:31:16.465 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:31:16.465 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:31:16.467 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:31:16.467358323+08:00 2024-09-11 18:31:16.468 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:31:16.467+08:00 2024-09-11 18:31:16.469 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:31:16.464+08:00 2024-09-11 18:31:16.471 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:31:16.469842356+08:00 2024-09-11 18:31:16.474 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:31:16.467+08:00 and status AVAILABLE 2024-09-11 18:31:16.476 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:31:16.471+08:00 2024-09-11 18:31:16.476 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:31:16.464+08:00 and status AVAILABLE 2024-09-11 18:31:16.478 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:31:16.471+08:00 and status AVAILABLE 2024-09-11 18:31:16.479 INFO 2981023 --- [host-check-agent-status2,123ea87b4e3860cd,8316d142c4060046] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:31:16.481 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:31:16.483 INFO 2981023 --- [host-check-agent-status4,123ea87b4e3860cd,710e65f1264300bd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:31:16.484 INFO 2981023 --- [host-check-agent-status1,123ea87b4e3860cd,4b7fd204f63512e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:31:16.484 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:31:16.489103364+08:00 2024-09-11 18:31:16.484 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:31:16.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:31:16.484161+08:00, lastProcessTime=2024-09-11T18:31:06.337655+08:00 2024-09-11 18:31:16.487 INFO 2981023 --- [host-check-agent-status4,123ea87b4e3860cd,710e65f1264300bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:31:16.487 INFO 2981023 --- [host-check-agent-status4,123ea87b4e3860cd,710e65f1264300bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:31:16.487 INFO 2981023 --- [host-check-agent-status4,123ea87b4e3860cd,710e65f1264300bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:31:16.487 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.488 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:31:16.488 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:31:16.489 INFO 2981023 --- [host-check-agent-status3,123ea87b4e3860cd,d14ba31cb4ffd750] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:31:16.491 INFO 2981023 --- [host-check-agent-status2,123ea87b4e3860cd,8316d142c4060046] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:31:16.491 INFO 2981023 --- [host-check-agent-status2,123ea87b4e3860cd,8316d142c4060046] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:31:16.491 INFO 2981023 --- [host-check-agent-status2,123ea87b4e3860cd,8316d142c4060046] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:31:16.491 INFO 2981023 --- [host-check-agent-status1,123ea87b4e3860cd,4b7fd204f63512e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:31:16.491 INFO 2981023 --- [host-check-agent-status1,123ea87b4e3860cd,4b7fd204f63512e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:31:16.491 INFO 2981023 --- [host-check-agent-status1,123ea87b4e3860cd,4b7fd204f63512e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:31:16.492 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:31:16.484+08:00 2024-09-11 18:31:16.494 INFO 2981023 --- [host-check-agent-status3,123ea87b4e3860cd,d14ba31cb4ffd750] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:31:16.494 INFO 2981023 --- [host-check-agent-status3,123ea87b4e3860cd,d14ba31cb4ffd750] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:31:16.494 INFO 2981023 --- [host-check-agent-status3,123ea87b4e3860cd,d14ba31cb4ffd750] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:31:16.497 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:31:16.484+08:00 and status AVAILABLE 2024-09-11 18:31:16.498 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:31:16.499 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:31:16.502 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.502 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:31:16.503 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:31:16.511 INFO 2981023 --- [ocp-schedules-25,599a075b81940c1c,e25b4c4b78b608e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:31:16.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:16.559 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:31:16.566 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:31:16.567 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:31:16.568 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:31:16.569 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:31:16.570 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:16.578 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:31:16.579 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:31:16.579 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:31:16.584 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:31:16.592 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:31:16.595 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:31:16.596 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.596 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:16.606 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:16.607 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:31:16.610 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.611 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:31:16.617 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:31:16.622 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:31:16.625 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:31:16.637 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:31:16.654 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:31:16.661 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:31:16.662 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:31:16.663 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:31:16.672 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:16.672 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.673 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:16.673 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:31:16.677 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.679 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:31:16.680 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:31:16.680 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:31:16.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:31:16.698 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:31:16.707 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:31:16.707 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050603337000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:31:16.708 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:31:16.714 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:31:16.714 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:31:16.718 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:31:16.729 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:31:16.733 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:31:16.735 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:31:16.736 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:16.736 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.737 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:31:16.738 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:31:16.741 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:31:16.742 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:31:16.742 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:31:16.744 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:31:16.745 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:31:16.748 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:31:16.750 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:31:16.753 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:31:16.762 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:31:16.763 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:31:16.772 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:31:16.772 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:31:16.776 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:16.776 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.777 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:31:16.780 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:31:16.788 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:31:16.795 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:31:16.806 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:16.806 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:31:16.806 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.807 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:16.807 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:31:16.808 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:31:16.808 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.810 INFO 2981023 --- [host-check-host-status4,88286daea6df33a1,1bd053938ec8e2f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:31:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:31:16.812 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:31:16.816 INFO 2981023 --- [host-check-host-status3,88286daea6df33a1,4a68cffacbe45082] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:31:16.816 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:31:16.817 INFO 2981023 --- [host-check-host-status2,88286daea6df33a1,c19a378ceb4fa1e7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:31:16.823 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:31:16.823 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:31:16.838 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:31:16.842 INFO 2981023 --- [host-check-host-status1,88286daea6df33a1,98c5dfc0649f23a2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:31:16.848 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:16.850 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.850 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:31:16.851 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:31:16.856 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:31:16.860 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:31:16.861 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:31:16.881 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:31:16.484161+08:00 2024-09-11 18:31:16.884 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:31:16.908 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:31:16.915 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:16.916 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:31:16.918 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.919 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:31:16.923 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:16.923 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:16.924 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:31:16.925 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:31:16.925 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:31:16.929 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:31:16.929 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:31:16.929 INFO 2981023 --- [ocp-schedules-18,f198d30716332097,e88cf7d292d5273a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:31:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,0c90344ef86f146f,e9a205fc8767c5f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:17.317 INFO 2981023 --- [prometheus-minute-schedule-1,0c90344ef86f146f,e9a205fc8767c5f5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:17.482 INFO 2981023 --- [ocp-async-10,d4b9acda02648459,db2e7078803e075b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:17.800 INFO 2981023 --- [ocp-async-2,9e8a27d5f87271df,16801999e7dd00a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:31:17.851 INFO 2981023 --- [host-check-clockdiff2,32eecfc67340a9a9,eba4d1efdb5d56ca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050677 5 5 , error= 2024-09-11 18:31:18.415 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:18.416 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:19.482 INFO 2981023 --- [ocp-async-6,a3b234912ce506f0,dd9fa55974720ba6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:19.834 INFO 2981023 --- [ocp-schedules-17,b9db55441bb9805b,4c4ac0098cb57e07] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:31:19.880 WARN 2981023 --- [scanner-13684361-1,b9db55441bb9805b,3cf065d2597bd2e8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:31:19.916 INFO 2981023 --- [alarm-detect-13,b9db55441bb9805b,8877ae58fd1d7dbf] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=os_tsar_cpu_util_full, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-2, alarm_template_id=0:host=10.31.35.6-2, alarm_template_id=0:host=10.31.35.5-2, alarm_template_id=0:host=10.31.35.7-2] 2024-09-11 18:31:19.927 INFO 2981023 --- [alarm-detect-9,b9db55441bb9805b,f1a4a47703366688] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050645 ~ 1726050669 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050660, value=5.957)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=5.957, values={1726050660=5.957}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:31:19.927 INFO 2981023 --- [alarm-detect-9,b9db55441bb9805b,f1a4a47703366688] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050645 ~ 1726050669 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050660, value=5.957)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=5.957, values={1726050660=5.957}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:31:19.943 INFO 2981023 --- [alarm-detect-4,b9db55441bb9805b,f93d6b7d107bd573] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_cpu_percent_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-2, alarm_template_id=0:host=10.31.35.6-2, alarm_template_id=0:host=10.31.35.5-2, alarm_template_id=0:host=10.31.35.7-2] 2024-09-11 18:31:19.962 INFO 2981023 --- [alarm-detect-11,b9db55441bb9805b,8699bb942a627e22] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=os_tsar_cpu_util_hwm, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-2, alarm_template_id=0:host=10.31.35.6-2, alarm_template_id=0:host=10.31.35.5-2, alarm_template_id=0:host=10.31.35.7-2] 2024-09-11 18:31:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,52f951bb03097909,6b1ef672b9f14915] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,52f951bb03097909,6b1ef672b9f14915] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:21.416 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:21.418 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:21.482 INFO 2981023 --- [ocp-async-4,175a41beb058c816,2f6f1808ecf20541] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:22.059 INFO 2981023 --- [host-check-clockdiff4,32eecfc67340a9a9,1b4d061e7a72a470] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050682 0 0 , error= 2024-09-11 18:31:22.910 INFO 2981023 --- [ocp-schedules-6,d8f868b262713feb,58cdf0f13a408ff9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:31:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,715f5e2ec7b2ab6c,ed7a6757a1693e56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,715f5e2ec7b2ab6c,ed7a6757a1693e56] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:23.482 INFO 2981023 --- [ocp-async-9,b67d508b339f8467,00fd7f9c90fe915d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:24.418 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:24.418 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:24.466 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b7b0df179d7ec681,80640061c4675b7a] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkrtd&version=2&ObClusterId=1719994443, client=10.31.35.6, traceId=b7b0df179d7ec681, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkrtd,1719994443,2,{"ObRegion":"linkrtd","ObCluster":"linkrtd","ObRegionId":1719994443,"ObClusterId":1719994443,"Type":"PRIMARY","timestamp":1726050684469993,"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:31:24.474 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b7b0df179d7ec681,80640061c4675b7a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkrtd, obClusterId=1719994443, rootServerJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050684467000}, groupName=linkrtd, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:31:24.474 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b7b0df179d7ec681,80640061c4675b7a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:31:24.474 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b7b0df179d7ec681,80640061c4675b7a] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkrtd, clusterId=1719994443, affectRows=1 2024-09-11 18:31:24.474 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b7b0df179d7ec681,80640061c4675b7a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.6, traceId=b7b0df179d7ec681, duration=9 ms] 2024-09-11 18:31:24.531 INFO 2981023 --- [ocp-schedules-30,e1528f9532f267e3,cdffe062b672e3e3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:31:25.482 INFO 2981023 --- [ocp-async-5,530fe4426c931819,ea02b2bb6c1a8873] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,d0fede53aa33c3f6,a6c4dfa92cfd78d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,d0fede53aa33c3f6,a6c4dfa92cfd78d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:31:26.353005+08:00, lastProcessTime=2024-09-11T18:31:16.484161+08:00 2024-09-11 18:31:26.486 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=3, resultsCount=3 2024-09-11 18:31:26.495 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:31:26.353005+08:00 2024-09-11 18:31:27.419 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:27.421 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:27.483 INFO 2981023 --- [ocp-async-7,797d23632a083151,339a4c497e4eed44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:27.800 INFO 2981023 --- [ocp-async-9,a6ed972cacc9a8ed,a39f3fc841991b52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:31:28.383 INFO 2981023 --- [ocp-schedules-14,c0d6e24946b0d51d,21c1d425f00dbb47] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:31:28.383 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:31:28.383 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:31:28.384 INFO 2981023 --- [ocp-schedules-14,c0d6e24946b0d51d,21c1d425f00dbb47] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:31:28.384 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:31:28.408 INFO 2981023 --- [ocp-schedules-14,c0d6e24946b0d51d,21c1d425f00dbb47] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:31:28.409 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:31:28.412 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:31:28.412 INFO 2981023 --- [ocp-schedules-14,c0d6e24946b0d51d,21c1d425f00dbb47] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:31:28.419 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:31:28.420 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:31:28.488 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:31:28.559 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:31:28.582 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:31:28.583 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:31:28.587 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:31:28.589 INFO 2981023 --- [ocp-schedules-14,c0d6e24946b0d51d,21c1d425f00dbb47] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:31:28.589 INFO 2981023 --- [ocp-schedules-14,c0d6e24946b0d51d,21c1d425f00dbb47] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:31:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,91e8bb908809d173,658ae2071b7bfd90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,91e8bb908809d173,658ae2071b7bfd90] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:29.483 INFO 2981023 --- [ocp-async-10,fd4d28996143ce86,6ceecd627c69352d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:30.081 INFO 2981023 --- [ocp-schedules-29,3954f5ee6d8f4672,e88e6479c9a823d5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:31:30.154 WARN 2981023 --- [scanner-13684557-1,3954f5ee6d8f4672,a98506189ecb68d5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:31:30.160 INFO 2981023 --- [alarm-detect-3,3954f5ee6d8f4672,84e8c05dcec3a56c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646597, data is empty. 2024-09-11 18:31:30.169 INFO 2981023 --- [alarm-detect-3,3954f5ee6d8f4672,84e8c05dcec3a56c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646597, data is empty. 2024-09-11 18:31:30.173 INFO 2981023 --- [alarm-detect-3,3954f5ee6d8f4672,84e8c05dcec3a56c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646597, data is empty. 2024-09-11 18:31:30.190 INFO 2981023 --- [alarm-detect-20,3954f5ee6d8f4672,70ecafd64b2db4fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646597, data is empty. 2024-09-11 18:31:30.194 INFO 2981023 --- [alarm-detect-20,3954f5ee6d8f4672,70ecafd64b2db4fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646597, data is empty. 2024-09-11 18:31:30.194 INFO 2981023 --- [alarm-detect-5,3954f5ee6d8f4672,6a7376ca5b73da8a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646597, data is empty. 2024-09-11 18:31:30.197 INFO 2981023 --- [alarm-detect-20,3954f5ee6d8f4672,70ecafd64b2db4fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646597, data is empty. 2024-09-11 18:31:30.202 INFO 2981023 --- [alarm-detect-5,3954f5ee6d8f4672,6a7376ca5b73da8a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646597, data is empty. 2024-09-11 18:31:30.212 INFO 2981023 --- [alarm-detect-5,3954f5ee6d8f4672,6a7376ca5b73da8a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646597, data is empty. 2024-09-11 18:31:30.234 INFO 2981023 --- [alarm-detect-3,3954f5ee6d8f4672,7190489851e30482] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050660 ~ 1726050680 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050675, value=4.682)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=4.682, values={1726050675=4.682}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:31:30.234 INFO 2981023 --- [alarm-detect-3,3954f5ee6d8f4672,7190489851e30482] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050660 ~ 1726050680 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050675, value=4.682)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=4.682, values={1726050675=4.682}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:31:30.234 INFO 2981023 --- [alarm-detect-3,3954f5ee6d8f4672,7190489851e30482] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.5-2] 2024-09-11 18:31:30.316 INFO 2981023 --- [alarm-detect-17,3954f5ee6d8f4672,d73fa5d15d801c89] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646597, data is empty. 2024-09-11 18:31:30.320 INFO 2981023 --- [alarm-detect-17,3954f5ee6d8f4672,d73fa5d15d801c89] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646597, data is empty. 2024-09-11 18:31:30.325 INFO 2981023 --- [alarm-detect-17,3954f5ee6d8f4672,d73fa5d15d801c89] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646597, data is empty. 2024-09-11 18:31:30.347 INFO 2981023 --- [alarm-detect-19,3954f5ee6d8f4672,16f4ae7da56c1c7f] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_cpu_usage_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=linkrtd-1719994443:tenant_name=sys:host=10.31.35.6-2, alarm_template_id=0:ob_cluster=rtdob-1718689631:tenant_name=sys:host=10.31.35.5-2, alarm_template_id=0:ob_cluster=rtdob-1718689631:tenant_name=linkreit:host=10.31.35.5-2] 2024-09-11 18:31:30.421 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:30.422 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:31.483 INFO 2981023 --- [ocp-async-2,52a574fd21780523,97966b0f0e5cb180] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,7a0f011dedc3ffce,120412ec395e8e05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,7a0f011dedc3ffce,120412ec395e8e05] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:32.913 INFO 2981023 --- [ocp-schedules-15,563b6a1e49a4738a,9f41e87ca686c1e2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:31:33.422 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:33.422 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:33.483 INFO 2981023 --- [ocp-async-1,0176bdba4c186ae1,67d7387932ed0df1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,bcd415c9c624e71c,4d58aaa2774b0287] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,bcd415c9c624e71c,4d58aaa2774b0287] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:35.483 INFO 2981023 --- [ocp-async-3,15fb172d8b37848d,2f2ab6aaffbde351] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:35.975 INFO 2981023 --- [ocp-schedules-16,930dbdd5a92a9d8c,72fafc2f79f39919] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:31:35.984 INFO 2981023 --- [ocp-schedules-16,930dbdd5a92a9d8c,72fafc2f79f39919] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:31:35.984 INFO 2981023 --- [ocp-schedules-16,930dbdd5a92a9d8c,72fafc2f79f39919] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:31:36.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:31:36.351868+08:00, lastProcessTime=2024-09-11T18:31:26.353005+08:00 2024-09-11 18:31:36.423 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:36.423 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:36.475 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=2, resultsCount=2 2024-09-11 18:31:36.479 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:31:36.351868+08:00 2024-09-11 18:31:37.315 INFO 2981023 --- [prometheus-cache-monitor1,3a6b84cab110ae3d,b087f502805af13e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:31:37.315 INFO 2981023 --- [prometheus-cache-monitor1,3a6b84cab110ae3d,b087f502805af13e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:31:37.483 INFO 2981023 --- [ocp-async-8,4fad00dcac65d5d6,5c4fa66497e4365c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:37.800 INFO 2981023 --- [ocp-async-1,95220592b0576c7e,1c289cadc688b8f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:31:37.811 INFO 2981023 --- [ocp-async-1,95220592b0576c7e,1c289cadc688b8f9] c.o.o.a.s.OcpAlarmSnapshotServiceImpl : alarm type ob_tenant_cpu_usage_over_threshold 's snapshots already exist. 2024-09-11 18:31:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,899f91f4f37137cf,885a2c6f199ae201] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,899f91f4f37137cf,885a2c6f199ae201] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:39.423 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:39.424 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:39.483 INFO 2981023 --- [ocp-async-7,a264df2bb733324a,b2e2f5afc6b87854] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:40.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:31:40.004 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:31:40.371 INFO 2981023 --- [ocp-schedules-2,2f736460fe615d67,57b68950d40ebca9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:31:40.427 WARN 2981023 --- [scanner-13684890-1,2f736460fe615d67,95cb0ab8f05067de] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:31:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,51d657dd84a7466a,eb4956da8b494b39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,51d657dd84a7466a,eb4956da8b494b39] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:41.483 INFO 2981023 --- [ocp-async-9,4c5cb96a1ba263d8,7b5791d12847e440] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:42.424 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:42.425 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:42.916 INFO 2981023 --- [ocp-schedules-2,01376e936a81693b,62f9e61285a04fbe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:31:43.483 INFO 2981023 --- [ocp-async-5,afe65170414fdad1,f956accaa1c73caa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,aadad40072447811,54c3675be8f776fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,aadad40072447811,54c3675be8f776fb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:45.101 INFO 2981023 --- [prometheus-second-schedule-1,bf95f11406baf9c3,4bca1dcc9e7e5986] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:31:45.425 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:45.426 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:45.484 INFO 2981023 --- [ocp-async-6,60e9526b8f7d7f4b,dd583616f52817a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:46.018 INFO 2981023 --- [ocp-schedules-13,301a4ac43c02d75d,a9c1cdbb2a0bd682] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:31:46.018 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:31:46.019 INFO 2981023 --- [ocp-schedules-5,ad4b8b3d045dc857,7a31c63ac0589630] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:31:46.019 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:31:46.020 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:31:46.020 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:31:46.021 INFO 2981023 --- [host-check-clockdiff1,ad4b8b3d045dc857,1efeaeb081a2d872] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:31:46.021 INFO 2981023 --- [host-check-clockdiff2,ad4b8b3d045dc857,1f26dd481e4337a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:31:46.023 INFO 2981023 --- [host-check-clockdiff3,ad4b8b3d045dc857,5fdd45e018405e87] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:31:46.024 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:31:46.026 INFO 2981023 --- [host-check-clockdiff4,ad4b8b3d045dc857,f92f6400495d879b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:31:46.026 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:31:46.029 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:31:46.030 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:31:46.031 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:31:46.031 INFO 2981023 --- [ocp-schedules-19,d0ab8d79bf49c89a,1d0059133e77876a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:31:46.031 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:31:46.032 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:31:46.033 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:31:46.036 INFO 2981023 --- [host-check-clockdiff2,ad4b8b3d045dc857,1f26dd481e4337a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050706 6 6 , error= 2024-09-11 18:31:46.038 INFO 2981023 --- [host-check-agent-status1,d0ab8d79bf49c89a,c0d7dd7cc434f3f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:31:46.038 INFO 2981023 --- [host-check-agent-status3,d0ab8d79bf49c89a,0f7d03c4a27b04a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:31:46.040 INFO 2981023 --- [host-check-clockdiff3,ad4b8b3d045dc857,5fdd45e018405e87] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050706 0 0 , error= 2024-09-11 18:31:46.041 INFO 2981023 --- [host-check-agent-status2,d0ab8d79bf49c89a,3016b15254abe47a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:31:46.042 INFO 2981023 --- [host-check-agent-status4,d0ab8d79bf49c89a,b5cb07c6b25a165c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:31:46.049 INFO 2981023 --- [host-check-agent-status2,d0ab8d79bf49c89a,3016b15254abe47a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:31:46.051 INFO 2981023 --- [host-check-agent-status2,d0ab8d79bf49c89a,3016b15254abe47a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:31:46.051 INFO 2981023 --- [host-check-agent-status1,d0ab8d79bf49c89a,c0d7dd7cc434f3f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:31:46.051 INFO 2981023 --- [host-check-agent-status3,d0ab8d79bf49c89a,0f7d03c4a27b04a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:31:46.052 INFO 2981023 --- [host-check-agent-status1,d0ab8d79bf49c89a,c0d7dd7cc434f3f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:31:46.053 INFO 2981023 --- [host-check-agent-status3,d0ab8d79bf49c89a,0f7d03c4a27b04a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:31:46.055 INFO 2981023 --- [host-check-agent-status4,d0ab8d79bf49c89a,b5cb07c6b25a165c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:31:46.056 INFO 2981023 --- [host-check-agent-status4,d0ab8d79bf49c89a,b5cb07c6b25a165c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:31:46.058 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:31:46.060 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:31:46.061 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:31:46.065932301+08:00 2024-09-11 18:31:46.061 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:31:46.062 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:31:46.061+08:00 2024-09-11 18:31:46.064 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:31:46.063916264+08:00 2024-09-11 18:31:46.064 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:31:46.063501941+08:00 2024-09-11 18:31:46.065 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:31:46.064+08:00 2024-09-11 18:31:46.066 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:31:46.061+08:00 and status AVAILABLE 2024-09-11 18:31:46.066 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:31:46.065+08:00 2024-09-11 18:31:46.068 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:31:46.064+08:00 and status AVAILABLE 2024-09-11 18:31:46.068 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:31:46.069 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:31:46.065+08:00 and status AVAILABLE 2024-09-11 18:31:46.070 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:31:46.070324856+08:00 2024-09-11 18:31:46.074 INFO 2981023 --- [host-check-agent-status2,d0ab8d79bf49c89a,3016b15254abe47a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:31:46.076 INFO 2981023 --- [host-check-agent-status3,d0ab8d79bf49c89a,0f7d03c4a27b04a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:31:46.076 INFO 2981023 --- [host-check-agent-status2,d0ab8d79bf49c89a,3016b15254abe47a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:31:46.076 INFO 2981023 --- [host-check-agent-status2,d0ab8d79bf49c89a,3016b15254abe47a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:31:46.076 INFO 2981023 --- [host-check-agent-status2,d0ab8d79bf49c89a,3016b15254abe47a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:31:46.076 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:31:46.076 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:31:46.077 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:31:46.070+08:00 2024-09-11 18:31:46.078 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:31:46.094 INFO 2981023 --- [host-check-agent-status3,d0ab8d79bf49c89a,0f7d03c4a27b04a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:31:46.094 INFO 2981023 --- [host-check-agent-status3,d0ab8d79bf49c89a,0f7d03c4a27b04a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:31:46.094 INFO 2981023 --- [host-check-agent-status3,d0ab8d79bf49c89a,0f7d03c4a27b04a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:31:46.100 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:31:46.104 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:31:46.070+08:00 and status AVAILABLE 2024-09-11 18:31:46.104 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:31:46.104 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:31:46.105 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:31:46.105 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:31:46.106 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:31:46.114 INFO 2981023 --- [host-check-agent-status4,d0ab8d79bf49c89a,b5cb07c6b25a165c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:31:46.117 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:31:46.117 INFO 2981023 --- [host-check-agent-status1,d0ab8d79bf49c89a,c0d7dd7cc434f3f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:31:46.118 INFO 2981023 --- [host-check-agent-status4,d0ab8d79bf49c89a,b5cb07c6b25a165c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:31:46.118 INFO 2981023 --- [host-check-agent-status4,d0ab8d79bf49c89a,b5cb07c6b25a165c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:31:46.118 INFO 2981023 --- [host-check-agent-status4,d0ab8d79bf49c89a,b5cb07c6b25a165c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:31:46.120 INFO 2981023 --- [host-check-agent-status1,d0ab8d79bf49c89a,c0d7dd7cc434f3f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:31:46.120 INFO 2981023 --- [host-check-agent-status1,d0ab8d79bf49c89a,c0d7dd7cc434f3f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:31:46.120 INFO 2981023 --- [host-check-agent-status1,d0ab8d79bf49c89a,c0d7dd7cc434f3f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:31:46.127 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:31:46.128 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:31:46.136 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:31:46.137 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:31:46.139 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:31:46.148 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:31:46.158 INFO 2981023 --- [ocp-schedules-27,70672c4a4599d564,b3b5f4997ac2318b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:31:46.162 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:31:46.165 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:31:46.166 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:31:46.171 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:31:46.185 INFO 2981023 --- [ocp-schedules-27,70672c4a4599d564,b3b5f4997ac2318b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:31:46.188 INFO 2981023 --- [ocp-schedules-20,80ab464ebe81c826,ee401932f332d0b6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:31:46.188 INFO 2981023 --- [ocp-schedules-30,e2f1f1d85e62f33f,187a2946bd208de6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:31:46.188 INFO 2981023 --- [ocp-schedules-10,1838b5a4574b3098,3dde0d82be9eab38] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:31:46.188 INFO 2981023 --- [ocp-schedules-16,1498e9c648c42c22,b1baf76238cebc7d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:31:46.189 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:31:46.190 INFO 2981023 --- [ocp-schedules-10,1838b5a4574b3098,3dde0d82be9eab38] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:31:46.190 INFO 2981023 --- [ocp-schedules-30,e2f1f1d85e62f33f,187a2946bd208de6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:31:46.195 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:31:46.196 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:31:46.197 INFO 2981023 --- [ocp-schedules-20,80ab464ebe81c826,ee401932f332d0b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:31:46.217 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:31:46.229 INFO 2981023 --- [ocp-schedules-23,8889205dcd81355e,e7c185b9e154e1a6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:31:46.234 INFO 2981023 --- [ocp-schedules-30,e2f1f1d85e62f33f,187a2946bd208de6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:31:46.234 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050684467000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:31:46.274 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:31:46.274 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:31:46.274 INFO 2981023 --- [ocp-schedules-6,130d1d75a58fb0bc,bfe2cb465901117a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 18:31:46.276 INFO 2981023 --- [ocp-schedules-6,130d1d75a58fb0bc,bfe2cb465901117a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:31:46.276 INFO 2981023 --- [ocp-schedules-12,72de0b7e8ba97f54,65a7a6a88e4b6d08] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 18:31:46.276 INFO 2981023 --- [ocp-schedules-12,72de0b7e8ba97f54,65a7a6a88e4b6d08] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 18:31:46.284 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:31:46.294 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:31:46.296 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:31:46.304 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:31:46.315 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:31:46.319 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:31:46.320 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:31:46.321 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:31:46.321 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.321 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:31:46.327 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.327 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:31:46.328 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:31:46.332 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1ac58cd3be2b51b6,3e7a3da7cc4c5eb5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=1ac58cd3be2b51b6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 18:31:46.332 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,1ac58cd3be2b51b6,3e7a3da7cc4c5eb5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=1ac58cd3be2b51b6, duration=1 ms] 2024-09-11 18:31:46.334 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.334 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:31:46.337 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.337 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:46.343 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:31:46.350 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.351 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:31:46.351 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,484e375fa4584c5c,ac6604ff88521c6e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=484e375fa4584c5c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 18:31:46.352 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,484e375fa4584c5c,ac6604ff88521c6e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=484e375fa4584c5c, duration=1 ms] 2024-09-11 18:31:46.352 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.353 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:31:46.353 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.354 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:31:46.354 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:31:46.354 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.354 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:31:46.357 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.361 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 18:31:46.361 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.363 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 18:31:46.364 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:31:46.358452+08:00, lastProcessTime=2024-09-11T18:31:36.351868+08:00 2024-09-11 18:31:46.367 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.367 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:31:46.367 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:31:46.368 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:31:46.368 INFO 2981023 --- [ocp-schedules-20,80ab464ebe81c826,ee401932f332d0b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:31:46.371 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.371 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:46.372 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.373 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:46.373 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:31:46.373 INFO 2981023 --- [ocp-schedules-6,130d1d75a58fb0bc,bfe2cb465901117a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 18:31:46.388 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:31:46.389 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:31:46.389 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:31:46.393 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:31:46.398 INFO 2981023 --- [host-check-host-status4,301a4ac43c02d75d,fc766dd3d537c702] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:31:46.398 INFO 2981023 --- [host-check-host-status2,301a4ac43c02d75d,a9861843a249ca10] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:31:46.417 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:31:46.420 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.421 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.421 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:31:46.422 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.422 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:31:46.422 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.423 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.423 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:31:46.441 INFO 2981023 --- [host-check-host-status1,301a4ac43c02d75d,90e9bee7db32e743] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:31:46.444 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:31:46.447 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:31:46.456 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:31:46.460 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:31:46.462 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:31:46.462 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050684467000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:31:46.463 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.463 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.464 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:46.464 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:31:46.465 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.467 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:31:46.468 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:31:46.479 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.488 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.488 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:31:46.488 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:31:46.488 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.489 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:31:46.489 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:31:46.492 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:31:46.493 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.494 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:31:46.499 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:31:46.499 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.506 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:31:46.517 INFO 2981023 --- [pool-10-thread-4,eacb31e27a8afc18,c50656e29b0c2977] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 18:31:46.520 INFO 2981023 --- [pool-10-thread-2,eacb31e27a8afc18,5d43766f85f85401] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 18:31:46.528 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:31:46.528 INFO 2981023 --- [ForkJoinPool-6-worker-15,eacb31e27a8afc18,398876e0567ad0f5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 18:31:46.528 INFO 2981023 --- [ForkJoinPool-6-worker-15,eacb31e27a8afc18,398876e0567ad0f5] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 18:31:46.531 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:31:46.533 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 18:31:46.533 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 18:31:46.541 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:31:46.546 INFO 2981023 --- [pool-10-thread-4,eacb31e27a8afc18,c50656e29b0c2977] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 18:31:46.546 INFO 2981023 --- [pool-10-thread-4,eacb31e27a8afc18,c50656e29b0c2977] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 18:31:46.558 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 18:31:46.558 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 18:31:46.558 INFO 2981023 --- [host-check-host-status3,301a4ac43c02d75d,439c35b70d65a341] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:31:46.559 INFO 2981023 --- [pool-10-thread-3,eacb31e27a8afc18,77def4a5b9bc1712] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 18:31:46.569 WARN 2981023 --- [ForkJoinPool-6-worker-15,eacb31e27a8afc18,398876e0567ad0f5] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 18:31:46.573 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.573 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:31:46.574 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:31:46.579 INFO 2981023 --- [pool-10-thread-1,eacb31e27a8afc18,946b6c8a79ec4c82] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 18:31:46.579 INFO 2981023 --- [pool-10-thread-1,eacb31e27a8afc18,946b6c8a79ec4c82] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 18:31:46.586 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:31:46.587 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.587 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.587 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:31:46.589 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.589 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:31:46.589 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.590 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:31:46.592 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:31:46.592 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.598 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.598 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.599 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:31:46.607 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.608 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:31:46.608 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:31:46.609 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.609 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:31:46.610 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 18:31:46.611 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:31:46.612 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.614 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:31:46.616 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:31:46.616 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.616 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:31:46.618 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.619 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:31:46.620 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 18:31:46.633 INFO 2981023 --- [ForkJoinPool-6-worker-3,eacb31e27a8afc18,2a7fb37108bc0702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.635 INFO 2981023 --- [ForkJoinPool-6-worker-3,eacb31e27a8afc18,2a7fb37108bc0702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:31:46.644 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:31:46.644 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:31:46.649 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.649 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:31:46.651 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:31:46, 2024-09-11 18:31:46] 2024-09-11 18:31:46.652 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 18:31:46.652 INFO 2981023 --- [ForkJoinPool-6-worker-8,eacb31e27a8afc18,94c7688eca2f134e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 18:31:46.654 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.655 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.655 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:31:46.656 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:31:46.656 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:31:46.657 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.659 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:31:46.661 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:31:46.661 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.662 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.663 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:31:46.665 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:31:46, 2024-09-11 18:31:46] 2024-09-11 18:31:46.665 INFO 2981023 --- [ForkJoinPool-6-worker-3,eacb31e27a8afc18,2a7fb37108bc0702] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:31:46.666 INFO 2981023 --- [ForkJoinPool-6-worker-3,eacb31e27a8afc18,2a7fb37108bc0702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.667 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.667 INFO 2981023 --- [ForkJoinPool-6-worker-3,eacb31e27a8afc18,2a7fb37108bc0702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:31:46.668 INFO 2981023 --- [ForkJoinPool-6-worker-3,eacb31e27a8afc18,2a7fb37108bc0702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.669 INFO 2981023 --- [ForkJoinPool-6-worker-3,eacb31e27a8afc18,2a7fb37108bc0702] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:31:46.670 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:31:46.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:31:46.684 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.685 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:31:46.685 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:31:46.688 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 18:31:46.689 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.690 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.691 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:31:46.691 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:31:46.691 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.700 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:31:46.701 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.702 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:31:46, 2024-09-11 18:31:46] 2024-09-11 18:31:46.710 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 18:31:46.710 INFO 2981023 --- [ForkJoinPool-6-worker-6,eacb31e27a8afc18,9e8726c42ad23263] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 18:31:46.712 INFO 2981023 --- [pool-10-thread-2,eacb31e27a8afc18,5d43766f85f85401] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 18:31:46.721 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.723 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:31:46.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:31:46.745 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:31:46.746 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.749 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:31:46.750 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.751 INFO 2981023 --- [ForkJoinPool-6-worker-1,eacb31e27a8afc18,4bbdf154ff5cfc8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:31:46.753 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.756 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:31:46.756 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.757 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:31:46.757 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:31:46.757 INFO 2981023 --- [pool-10-thread-3,eacb31e27a8afc18,77def4a5b9bc1712] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 18:31:46.762 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:31:46.767 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.767 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:31:46.769 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:31:46.771 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.775 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 18:31:46.797 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:31:46.800 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:31:46.803 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:31:46.811 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.812 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.812 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:31:46.814 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:31:46.819 INFO 2981023 --- [ocp-schedules-15,debf198d0851f154,726e17887d6a08a6] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:31:46.825 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:31:46.825 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:31:46.834 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:31:46.839 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:31:46.358452+08:00 2024-09-11 18:31:46.841 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:31:46.841 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:46.842 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:31:46.848 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:31:46.848 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:31:46.848 INFO 2981023 --- [ocp-schedules-10,1838b5a4574b3098,3dde0d82be9eab38] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:31:46.849 INFO 2981023 --- [ocp-schedules-18,7cd964b868078d20,f967287f5e57525d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":11798,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:31:46.849 INFO 2981023 --- [ocp-schedules-18,7cd964b868078d20,f967287f5e57525d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:31:46.927 INFO 2981023 --- [pool-10-thread-1,eacb31e27a8afc18,946b6c8a79ec4c82] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 18:31:46.931 INFO 2981023 --- [pool-10-thread-1,eacb31e27a8afc18,946b6c8a79ec4c82] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 18:31:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,ac5809ebcee9b3ad,23d2430985d683d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,ac5809ebcee9b3ad,23d2430985d683d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:47.484 INFO 2981023 --- [ocp-async-9,e47cdb572534a868,f12e6aee669b4585] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:47.800 INFO 2981023 --- [ocp-async-2,e574eb229cf85dac,a9c0a6b3a90be84f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:31:47.811 INFO 2981023 --- [host-check-clockdiff1,ad4b8b3d045dc857,1efeaeb081a2d872] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050707 0 0 , error= 2024-09-11 18:31:48.426 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:48.426 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:48.902 INFO 2981023 --- [ocp-schedules-12,4e507c639e232af3,31d63fc154918acb] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:31:48.921 WARN 2981023 --- [ocp-schedules-12,4e507c639e232af3,31d63fc154918acb] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:31:49.484 INFO 2981023 --- [ocp-async-6,57be5d169b3045a4,8e4a676e781f40b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:31:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:31:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,a2736b7b3125cb4c,d6129058d67fb798] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,a2736b7b3125cb4c,d6129058d67fb798] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:50.385 INFO 2981023 --- [ocp-schedules-4,5b729babebb5f6b8,d232696b90a35d86] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:31:50.572 INFO 2981023 --- [ocp-schedules-8,3000055655875020,6ad0546187f0f268] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:31:50.619 WARN 2981023 --- [scanner-13685079-1,3000055655875020,0e1f95fba6ba6808] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 36 2024-09-11 18:31:50.639 INFO 2981023 --- [alarm-detect-14,3000055655875020,906bc8082d00064b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646597, data is empty. 2024-09-11 18:31:50.645 INFO 2981023 --- [alarm-detect-14,3000055655875020,906bc8082d00064b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646597, data is empty. 2024-09-11 18:31:50.650 INFO 2981023 --- [alarm-detect-14,3000055655875020,906bc8082d00064b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646597, data is empty. 2024-09-11 18:31:50.666 INFO 2981023 --- [alarm-detect-5,3000055655875020,1977d6ac83929ca3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646597, data is empty. 2024-09-11 18:31:50.667 INFO 2981023 --- [alarm-detect-2,3000055655875020,cb60c7ab7a7a3817] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646597, data is empty. 2024-09-11 18:31:50.669 INFO 2981023 --- [alarm-detect-5,3000055655875020,1977d6ac83929ca3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646597, data is empty. 2024-09-11 18:31:50.669 INFO 2981023 --- [alarm-detect-2,3000055655875020,cb60c7ab7a7a3817] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646597, data is empty. 2024-09-11 18:31:50.673 INFO 2981023 --- [alarm-detect-5,3000055655875020,1977d6ac83929ca3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646597, data is empty. 2024-09-11 18:31:50.673 INFO 2981023 --- [alarm-detect-2,3000055655875020,cb60c7ab7a7a3817] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646597, data is empty. 2024-09-11 18:31:50.744 INFO 2981023 --- [alarm-detect-10,3000055655875020,ffeb342e88c66229] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050675 ~ 1726050700 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050690, value=3.73)])]}, metricEvaluateResults={load1_per_cpu > 3=[ExpressionEvaluateResult(matchValue=3.0, metric=load1_per_cpu, match=true, value=3.73, values={1726050690=3.73}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-09-11 18:31:50.744 INFO 2981023 --- [alarm-detect-10,3000055655875020,ffeb342e88c66229] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050675 ~ 1726050700 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050690, value=3.73)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=3.73, values={1726050690=3.73}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:31:50.791 INFO 2981023 --- [alarm-detect-13,3000055655875020,e6b072a0a70da146] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646597, data is empty. 2024-09-11 18:31:50.797 INFO 2981023 --- [alarm-detect-13,3000055655875020,e6b072a0a70da146] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646597, data is empty. 2024-09-11 18:31:50.801 INFO 2981023 --- [alarm-detect-13,3000055655875020,e6b072a0a70da146] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646597, data is empty. 2024-09-11 18:31:51.426 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:51.428 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:51.484 INFO 2981023 --- [ocp-async-7,0f78d76a1db5566a,37de4f4ce0a730d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:51.652 INFO 2981023 --- [ocp-schedules-27,14eda8a40c287003,1c087bc94b10f71b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:31:51.653 INFO 2981023 --- [ocp-schedules-27,14eda8a40c287003,1c087bc94b10f71b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:31:51.654 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:31:51.654 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:31:51.654 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:31:51.655 INFO 2981023 --- [ocp-schedules-27,14eda8a40c287003,1c087bc94b10f71b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:31:52.012 WARN 2981023 --- [host-check-clockdiff4,ad4b8b3d045dc857,f92f6400495d879b] c.o.ocp.core.task.util.ClockDiff : Clock-diff failed. args=[10.31.35.7], exception={}. Please try other ocp.host.check.clock-diff.mode in System Parameters org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:153) at com.oceanbase.ocp.core.util.LocalCommandExecutor.exec(LocalCommandExecutor.java:50) at com.oceanbase.ocp.core.task.util.ClockDiff.execute(ClockDiff.java:95) at com.oceanbase.ocp.core.task.util.ClockDiff.diffWithIcmpTimestamp(ClockDiff.java:52) at com.oceanbase.ocp.core.task.util.ClockDiff$ClockDiffMode.doRun(ClockDiff.java:172) at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostClockDiff(HostCheckManagerImpl.java:353) at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostClockDiff(HostCheckManagerImpl.java:342) at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$doExecute$0(DistributedExclusiveExecutor.java:73) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 18:31:52.012 ERROR 2981023 --- [host-check-clockdiff4,ad4b8b3d045dc857,f92f6400495d879b] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.UnexpectedException occurred with code error.common.unexpected, and args [Execute clock diff failed., 10.31.35.7] 2024-09-11 18:31:52.012 WARN 2981023 --- [host-check-clockdiff4,ad4b8b3d045dc857,f92f6400495d879b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host clock diff fail. hostId=2000010, ip=10.31.35.7, exception={} com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=COMMON_UNEXPECTED, args=Execute clock diff failed.,10.31.35.7 at sun.reflect.GeneratedConstructorAccessor732.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) at com.oceanbase.ocp.core.util.ExceptionUtils.unExpected(ExceptionUtils.java:77) at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.doCheckHostClockDiff(HostCheckManagerImpl.java:355) at com.oceanbase.ocp.compute.host.manager.HostCheckManagerImpl.checkHostClockDiff(HostCheckManagerImpl.java:342) at com.oceanbase.ocp.core.task.engine.util.DistributedExclusiveExecutor.lambda$doExecute$0(DistributedExclusiveExecutor.java:73) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-09-11 18:31:52.237 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdCleanService : Clean series by rules started. 2024-09-11 18:31:52.243 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdCleanService : HostIpSize=4, clusterSize=3, obproxyClusterSize=0 2024-09-11 18:31:52.374 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdCleanService : Delete seriesKeyIds, size=261 2024-09-11 18:31:52.422 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdCleanService : Delete seriesKeyIds, size=976 2024-09-11 18:31:52.515 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdCleanService : Delete seriesKeyIds, size=170 2024-09-11 18:31:52.599 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdCleanService : Delete no data series key, size=118 2024-09-11 18:31:52.605 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdCleanService : Clean series by rules finished. 2024-09-11 18:31:52.605 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=11798 2024-09-11 18:31:52.633 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/update|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097360396288 2024-09-11 18:31:52.633 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=sdc|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112392781824 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_service_name_status|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|svr_ip=10.31.35.10|tenant_name=sys, id=690363204738772992 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/info|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363099512074240 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114544459776 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=OB|env_type=host|name=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174678196224 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983521902592 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|status=200|svr_ip=10.31.35.7, id=687520374890721280 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sda15|fstype=vfat|is_ob_disk=0|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419435556864 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_trans_stat_v2, id=690363398020689920 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_stat_0, id=690363221927030784 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249844318208 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=input_collect_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=hostCustomInput|process=ocp_monagent|svr_ip=10.31.35.6|type=input, id=690363097372979200 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_sync|app=OB|failed_reason=-|ob_cluster_id=1719994440|ob_cluster_name=linkrtd|ob_cluster_status=CREATING|svr_ip=10.31.35.10, id=690357920003813376 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_num|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|server_ips=|status=inactive|svr_ip=10.31.35.6, id=690363174686584832 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdb1|device=sdb1|env_type=host|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116704526336 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=ocp_agentd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118852009984 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=sdd|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112413753344 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/extra|process=ocp_monagent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363121003687936 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983538679808 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114565431296 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=OB|name=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174699167744 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|status=200|svr_ip=10.31.35.7, id=687520374878138368 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/info|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099537240064 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994440|ob_cluster_name=linkrtd|process=ocp_mgragent|process_status=running|svr_ip=10.31.35.6, id=690357919987036160 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30066|svr_ip=10.31.35.6|tenant_name=sys, id=690363247717806080 2024-09-11 18:31:52.634 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249865289728 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|is_ob_disk=0|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419460722688 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983551262720 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_stat_0, id=690363221952196608 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sql_transaction, id=690363398041661440 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_available|app=HOST|failed_reason=success|ob_cluster_id=1719994440|ob_cluster_name=linkrtd|svr_ip=10.31.35.6, id=690357920028979200 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40019|svr_ip=10.31.35.6|tenant_name=sys, id=690363176859234304 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247726194688 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_bytes_total|app=HOST|device=enP61571s1|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578965471232 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/reinstall|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=687517214017089536 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/extra|process=ocp_monagent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363121028853760 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_num|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|server_ips=10.31.35.6|status=active|svr_ip=10.31.35.6, id=690363174720139264 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=0|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983563845632 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-0|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112443113472 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=output_write_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mysqlTableOutput|process=ocp_monagent|svr_ip=10.31.35.6|type=output, id=690363398062632960 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=clock_diff_milliseconds|app=HOST|ob_cluster_id=1719994440|ob_cluster_name=linkrtd|svr_ip=10.31.35.6, id=690357920016396288 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_status_check|app=OB|check_item=cluster connection|failed_reason=-|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_cluster_status=RUNNING|svr_ip=10.31.35.10, id=690363204789104640 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249886261248 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=OB|env_type=host|name=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174728527872 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_stat_0, id=690363221973168128 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114598985728 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40030|svr_ip=10.31.35.6|tenant_name=sys, id=690363176880205824 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983576428544 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/update|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097423310848 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/info|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099570794496 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_drop_total|app=HOST|device=enP61571s1|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511579007414272 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc|device=sdc|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116754857984 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_stopped_duration_seconds|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|status=ACTIVE|svr_ip=10.31.35.6, id=690363174741110784 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247755554817 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50021|svr_ip=10.31.35.6|tenant_name=sys, id=690363247755554816 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520385707831296 2024-09-11 18:31:52.635 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983589011456 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249907232768 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=input_collect_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=nodeExporterInput|process=ocp_monagent|svr_ip=10.31.35.6|type=input, id=690363097435893760 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-1|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112468279296 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/extra|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363121058213888 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=OB|name=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174749499392 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/run|mountpoint=/run|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114619957248 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=ocp_mgragent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118914924544 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_stat_0, id=690363221998333952 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40116|svr_ip=10.31.35.6|tenant_name=sys, id=690363176905371648 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/update|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097448476672 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/stop|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363099595960320 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc1|device=sdc1|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116775829504 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_session_active_num|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174762082304 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-2|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112485056512 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_sync|app=OB|failed_reason=-|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_cluster_status=RUNNING|svr_ip=10.31.35.10, id=690363204831047680 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=input_collect_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=processInput|process=ocp_monagent|svr_ip=10.31.35.6|type=input, id=690363097456865280 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=OB|name=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174770470912 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249932398592 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.7, id=687520374999773184 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/extra|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363121083379712 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40117|svr_ip=10.31.35.6|tenant_name=sys, id=690363176922148864 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_stat_0, id=690363222019305472 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=ERROR|logSourceType=rootservice|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690366383064903680 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994440|ob_cluster_name=linkrtd|process=ocp_monagent|process_status=running|svr_ip=10.31.35.6, id=690357919961870336 2024-09-11 18:31:52.636 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_session_all_num|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174778859520 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247793303553 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50022|svr_ip=10.31.35.6|tenant_name=sys, id=690363247793303552 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/update|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097469448192 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994440|ob_cluster_name=linkrtd|process=ocp_agentd|process_status=running|svr_ip=10.31.35.6, id=690357919974453248 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sda1|fstype=xfs|is_ob_disk=0|mount_point=/boot|mountpoint=/boot|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419406196736 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/stop|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363099621126144 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-3|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112506028032 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/run/user/0|mountpoint=/run/user/0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114653511680 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc14|device=sdc14|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116800995328 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.7, id=687520374982995968 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=OB|name=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174791442432 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=log_tailer_count|app=HOST|log_file_name=all|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097482031104 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_status_check|app=OB|check_item=cluster connection|failed_reason=-|ob_cluster_id=1719994440|ob_cluster_name=linkrtd|ob_cluster_status=CREATING|svr_ip=10.31.35.10, id=690357919945093120 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample, id=690363222040276992 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249957564416 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=EDIAG|logSourceType=rootservice|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690366383052320768 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=input_collect_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mysqlTableInput|process=ocp_monagent|svr_ip=10.31.35.6|type=input, id=690363121108545536 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40118|svr_ip=10.31.35.6|tenant_name=sys, id=690363176947314688 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983643537408 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/update|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097490419712 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=10000|svr_ip=10.31.35.6|tenant_name=sys, id=690363174804025344 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/stop|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363099642097664 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983383490560 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=log_tailer_processed_file_count|app=HOST|log_file_name=monagent.log|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097498808320 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_clog|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108236226560 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114678677504 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc15|device=sdc15|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116826161152 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=OB|name=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174812413952 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/nicSpeed|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=687517390211411968 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247826857984 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_errs_total|app=HOST|device=enP61571s1|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511579066134528 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-4|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112535388160 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=input_collect_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=sqlTransactionInput|process=ocp_monagent|svr_ip=10.31.35.6|type=input, id=690363121125322752 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample, id=690363222061248512 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50030|svr_ip=10.31.35.6|tenant_name=sys, id=690363247831052288 2024-09-11 18:31:52.637 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/bootstrap|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097507196928 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983396073472 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit_config|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249982730240 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc2|device=sdc2|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116838744064 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=ocp_monagent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118986227712 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50000|svr_ip=10.31.35.6|tenant_name=sys, id=690363176972480512 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/stop|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099663069184 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_clog|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108253003776 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=10001|svr_ip=10.31.35.6|tenant_name=sys, id=690363174829191168 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/nicSpeed|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=687517390228189184 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_gv_database|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097523974144 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=custom_default_sql|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121146294272 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983412850688 2024-09-11 18:31:52.638 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample, id=690363222082220032 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/bootstrap|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097528168448 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=watch_ob_dir_exists|app=OB|name=block_file|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363108265586688 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114708037632 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50001|svr_ip=10.31.35.6|tenant_name=sys, id=690363176989257728 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit_config|process=ocp_monagent|svr_ip=10.31.35.6, id=690363250003701760 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=sda|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112564748288 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/nicSpeed|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687517390244966400 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=10002|svr_ip=10.31.35.6|tenant_name=sys, id=690363174850162688 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=0|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983425433600 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant500_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247864606720 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/stop|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099688235008 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdd|device=sdd|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116868104192 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50039|svr_ip=10.31.35.6|tenant_name=sys, id=690363247868801024 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_gv_database|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097544945664 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_clog|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108282363904 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/nicSpeed|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687517390265937920 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114729009152 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=custom_default_sql|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121171460096 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50004|svr_ip=10.31.35.6|tenant_name=sys, id=690363177010229248 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983438016512 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520385510699008 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/bootstrap|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097553334272 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=watch_ob_dir_exists|app=OB|name=ob_data_dir|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363108290752512 2024-09-11 18:31:52.666 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=10003|svr_ip=10.31.35.6|tenant_name=sys, id=690363174866939904 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample, id=690363222111580160 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit_config|process=ocp_monagent|svr_ip=10.31.35.6, id=690363250028867584 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=user_row_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690366823487795200 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_gv_database|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097561722880 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=sdb|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112594108416 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=ocp_agentd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119036559360 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_clog|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108303335424 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/reinstall|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=687517214054838272 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50005|svr_ip=10.31.35.6|tenant_name=sys, id=690363177027006464 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983454793728 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.7, id=687520374827806720 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|is_ob_disk=0|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419506860032 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/bootstrap|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097574305792 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/snapshot/collect|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363099721789440 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=watch_ob_dir_exists|app=OB|name=ob_install_dir|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363108311724032 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=custom_default_sql|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121196625920 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50040|svr_ip=10.31.35.6|tenant_name=sys, id=690363247902355457 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant500_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247902355456 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit_config|process=ocp_monagent|svr_ip=10.31.35.6, id=690363250049839104 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_bytes_total|app=HOST|device=enP61571s1|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511579124854784 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=10004|svr_ip=10.31.35.6|tenant_name=sys, id=690363174892105728 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983467376640 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample, id=690363222136745984 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_clog|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108320112640 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/reinstall|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=687517214071615488 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdc1|fstype=ext4|is_ob_disk=0|mount_point=/data|mountpoint=/data|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419485888512 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_gv_database|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097586888704 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=sdc|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112619274240 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114770952192 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983479959552 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/nicSpeed|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=687517390186246144 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=watch_ob_dir_exists|app=OB|name=ob_log_dir|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363108332695552 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50008|svr_ip=10.31.35.6|tenant_name=sys, id=690363177056366592 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_drop_total|app=HOST|device=enP61571s1|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511579179380736 2024-09-11 18:31:52.667 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/bootstrap|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097599471616 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/snapshot/collect|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363099746955264 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=custom_default_sql|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121221791744 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/reinstall|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687517214088392704 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=10005|svr_ip=10.31.35.6|tenant_name=sys, id=690363174913077248 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_clog|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108341084160 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983492542464 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.7, id=687520374861361152 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample_statistics, id=690363222166106112 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant500_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247935909888 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.7, id=687520374848778240 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60031|svr_ip=10.31.35.6|tenant_name=sys, id=690363247940104192 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_gv_database|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097616248832 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=watch_ob_dir_exists|app=OB|name=ob_run_dir|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363108353667072 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=sdd|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112648634368 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=ocp_mgragent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119091085312 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/reinstall|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687517214105169920 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit_config|process=ocp_monagent|svr_ip=10.31.35.6, id=690363250091782144 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114800312320 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdd1|device=sdd1|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116947795968 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=custom_default_sql|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121242763264 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=10006|svr_ip=10.31.35.6|tenant_name=sys, id=690363174934048768 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50009|svr_ip=10.31.35.6|tenant_name=sys, id=690363177081532416 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983509319680 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/checkConnect|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097624637440 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_available|app=HOST|failed_reason=success|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|svr_ip=10.31.35.7, id=690369747949481984 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498675597312 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247415816192 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server_data_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249563299840 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_virtual_memory_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363099239444480 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_report_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mysqlTableOutput|process=ocp_monagent|svr_ip=10.31.35.6, id=690363397743865856 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=INFO|logSourceType=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174405566464 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983786143744 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_status_check|app=OB|check_item=cluster connection|failed_reason=-|ob_cluster_id=1719994442|ob_cluster_name=linkrtd|ob_cluster_status=CREATING|svr_ip=10.31.35.10, id=690374026076647424 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=chronyd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118570991616 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=120002|svr_ip=10.31.35.6|tenant_name=sys, id=690363247424204800 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/checkDirectoryPermission|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097100349440 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=6|mode=idle|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112132734976 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114280218624 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/uninstall|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363099252027392 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=sqlTransactionInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363122874347520 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983798726656 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247432593408 2024-09-11 18:31:52.668 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=INFO|logSourceType=rootservice|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174422343680 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server_data_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249584271360 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994442|ob_cluster_name=linkrtd|process=ocp_monagent|process_status=running|svr_ip=10.31.35.6, id=690374026093424640 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan, id=690363221671178240 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/notify|process=ocp_monagent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097117126656 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_virtual_memory_max_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363099264610304 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=ntpd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118591963136 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=log_kv_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174430732288 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=0|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983811309568 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=120005|svr_ip=10.31.35.6|tenant_name=sys, id=690363247445176320 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=6|mode=softirq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112153706496 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-1|device=dm-1|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116448673792 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=TRACE|logSourceType=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174434926592 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|process=ocp_agentd|process_status=running|svr_ip=10.31.35.6, id=690361694848053248 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/checkDirectoryPermission|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097125515264 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247453564928 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/uninstall|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099277193216 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/run|mountpoint=/run|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114309578752 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983823892480 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/notify|process=ocp_monagent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097133903872 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=cardinalityStat|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363122903707648 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan, id=690363221692149760 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|process=ocp_mgragent|process_status=running|svr_ip=10.31.35.6, id=690361694835470336 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=processor_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=aggregateProcessor|process=ocp_monagent|svr_ip=10.31.35.6|type=processor, id=690363099285581824 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=clock_diff_milliseconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|svr_ip=10.31.35.7, id=690369747932704768 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=120006|svr_ip=10.31.35.6|tenant_name=sys, id=690363247466147840 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=7|mode=idle|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112174678016 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=opt_ds_stat_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174455898112 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983836475392 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server_data_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249617825792 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/download|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097146486784 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/uninstall|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099293970432 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247474536448 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/run/user/0|mountpoint=/run/user/0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114330550272 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_status_check|app=OB|check_item=cluster connection|failed_reason=-|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_cluster_status=CREATING|svr_ip=10.31.35.10, id=690361694818693120 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/notify|process=ocp_monagent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097154875392 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mysqlTableOutput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=output, id=690363397798391808 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|process=ocp_mgragent|process_status=running|svr_ip=10.31.35.7, id=690369747882373120 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=WARN|logSourceType=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174468481024 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983849058304 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan, id=690363221713121280 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=processor_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=attrProcessor|process=ocp_monagent|svr_ip=10.31.35.6|type=processor, id=690363099306553344 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=jointable|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363122928873472 2024-09-11 18:31:52.669 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=opt_system_stat_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174472675328 2024-09-11 18:31:52.670 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_sync|app=OB|failed_reason=-|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_cluster_status=CREATING|svr_ip=10.31.35.10, id=690361694877413376 2024-09-11 18:31:52.670 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-2|device=dm-2|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116490616832 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/ocaenbase/redo|mountpoint=/ocaenbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578730590208 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=clock_diff_milliseconds|app=HOST|ob_cluster_id=1719994442|ob_cluster_name=linkrtd|svr_ip=10.31.35.6, id=690374026147950592 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/exists|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363099314941952 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|process=ocp_agentd|process_status=running|svr_ip=10.31.35.7, id=690369747894956032 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=WDIAG|logSourceType=election|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174481063936 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983861641216 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=200001|svr_ip=10.31.35.6|tenant_name=sys, id=690363247495507968 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server_data_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249642991616 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/download|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097171652608 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114351521792 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_available|app=HOST|failed_reason=success|ob_cluster_id=1719994442|ob_cluster_name=linkrtd|svr_ip=10.31.35.6, id=690374026160533504 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|process=ocp_monagent|process_status=running|svr_ip=10.31.35.6, id=690361694860636160 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/notify|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363097180041216 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=processor_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mountLabelProcessor|process=ocp_monagent|svr_ip=10.31.35.6|type=processor, id=690363099327524864 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mysqlTableInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363122949844992 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|process=ocp_monagent|process_status=running|svr_ip=10.31.35.7, id=690369747907538944 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=opt_table_stat_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174493646848 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/run|mountpoint=/run|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983874224128 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/download|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097188429824 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=7|mode=softirq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112220815360 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=WDIAG|logSourceType=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174502035456 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan, id=690363221746675712 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server_data_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249663963136 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994442|ob_cluster_name=linkrtd|process=ocp_mgragent|process_status=running|svr_ip=10.31.35.6, id=690374026106007552 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_available|app=HOST|failed_reason=success|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|svr_ip=10.31.35.6, id=690361694915162112 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114372493312 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mysqlTableOutput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=output, id=690363397844529152 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=schema_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174510424064 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/run/user/0|mountpoint=/run/user/0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983891001344 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=host_agent_process_status|app=HOST|ob_cluster_id=1719994442|ob_cluster_name=linkrtd|process=ocp_agentd|process_status=running|svr_ip=10.31.35.6, id=690374026114396160 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/module/config/notify|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363097201012736 2024-09-11 18:31:52.690 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=retagProcessor|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363122970816512 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=WDIAG|logSourceType=rootservice|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174514618368 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/download|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097209401344 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-0|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112241786880 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-3|device=dm-3|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116536754176 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_trans_stat_v2, id=690363397861306368 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=ocp_agentd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118688432128 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=storage_meta_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174527201280 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983907778560 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_explain, id=690363221771841536 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249689128960 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_sync|app=OB|failed_reason=-|ob_cluster_id=1719994442|ob_cluster_name=linkrtd|ob_cluster_status=CREATING|svr_ip=10.31.35.10, id=690374026131173376 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=clock_diff_milliseconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|svr_ip=10.31.35.6, id=690361694889996288 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_monagent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097217789952 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114397659136 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40012|svr_ip=10.31.35.6|tenant_name=sys, id=690363176678879232 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=observer_ulimit_max_fd_count|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174535589888 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/exists|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363099373662208 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-4|device=dm-4|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116553531392 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=sqlTransactionInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363122995982336 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/download|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097230372864 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983656120320 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-1|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112266952704 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=vtable_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174548172800 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|status=200|svr_ip=10.31.35.7, id=687520375029133312 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578789310464 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_monagent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097242955776 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=oceanbase_connectivity|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|target=monitordb, id=690363174560755712 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/run|mountpoint=/run|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983672897536 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/getRealPath|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097251344384 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/exists|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363099398828032 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114431213568 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_bytes_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/extra|plugin=ob_extra|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363120873664512 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=processor_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=cardinalityStat|process=ocp_monagent|svr_ip=10.31.35.6|type=processor, id=690363123021148160 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.7, id=687520375012356096 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-2|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112287924224 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_plan_cache_access_total|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174569144320 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_monagent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097259732992 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249735266304 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_bytes_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/extra|plugin=ob_extra_custom|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363120886247424 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=oceanbase_connectivity|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|target=oceanbase, id=690363174577532928 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/run/user/0|mountpoint=/run/user/0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983693869056 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/getRealPath|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097272315904 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/exists|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099419799552 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-3|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112304701440 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=ocp_mgragent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118747152384 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_trans_stat_v2, id=690363397915832320 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_explain, id=690363221830561792 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_plan_cache_hit_total|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174590115840 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|status=200|svr_ip=10.31.35.7, id=687520375041716224 2024-09-11 18:31:52.691 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363097284898816 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114464768000 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sda|device=sda|env_type=host|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116612251648 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=processor_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=jointable|process=ocp_monagent|svr_ip=10.31.35.6|type=processor, id=690363123054702592 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249760432128 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_boot_time_seconds|app=OB|name=observer|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174602698752 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/getRealPath|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097293287424 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/exists|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099440771072 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_explain, id=690363221851533312 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-4|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112329867264 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=ocp_monagent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118772318208 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_trans_stat_v2, id=690363397953581056 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363097305870336 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/extra|plugin=ob_extra|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363120928190464 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=processor_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=retagProcessor|process=ocp_monagent|svr_ip=10.31.35.6|type=processor, id=690363123075674112 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249781403648 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sda1|device=sda1|env_type=host|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116637417472 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/getRealPath|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097314258944 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=1|is_ob_volume=1|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114494128128 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983740006400 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_explain, id=690363221872504832 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=sda|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112350838784 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/info|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363099470131200 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_status_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363097326841856 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_trans_stat_v2, id=690363397978746880 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247654891520 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/extra|plugin=ob_extra_custom|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363120953356288 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40018|svr_ip=10.31.35.6|tenant_name=sys, id=690363176792125440 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983756783616 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249806569472 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/getRealPath|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097335230464 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=sdb|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112371810304 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=1|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114519293952 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_explain, id=690363221897670656 2024-09-11 18:31:52.692 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/process/info|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363099491102720 2024-09-11 18:31:52.716 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdb|device=sdb|env_type=host|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116670971904 2024-09-11 18:31:52.716 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983769366528 2024-09-11 18:31:52.716 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_status_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363097352007680 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_plan_cache_memory_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363174665613312 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_unit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249827540992 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/extra|process=ocp_monagent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363120978522112 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_trans_stat_v2, id=690363397999718400 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60019|svr_ip=10.31.35.6|tenant_name=sys, id=690363177354162176 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/ob|process=ocp_monagent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363183796613120 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_assigned_mem_total|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248221122560 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-2|device=dm-2|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117224620032 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=140005|svr_ip=10.31.35.6|tenant_name=sys, id=690363175210872832 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_assigned|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248225316864 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_status_check|app=OB|check_item=cluster connection|failed_reason=-|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_cluster_status=OPERATING|svr_ip=10.31.35.10, id=690369622057447424 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition_replica|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097901461504 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/checkPortOccupied|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363100048945152 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|mount_label=|mount_point=/run/user/1002|mountpoint=/run/user/1002|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687517866290081792 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095758172161 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_bytes_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/host|plugin=node_info|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363095758172160 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=tcp_retrans|app=HOST|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119380492288 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984062967808 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanDataFiles|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097909850112 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_plan_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121532170240 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248242094080 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_max_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108655656960 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115098107904 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=140006|svr_ip=10.31.35.6|tenant_name=sys, id=690363175231844352 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60020|svr_ip=10.31.35.6|tenant_name=sys, id=690363177379328000 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/ob|process=ocp_monagent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363183821778944 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_stat_server_progress, id=690363222476484608 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/checkPortOccupied|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363100069916672 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-3|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112954818560 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984079745024 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_disk_total_size|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248250482688 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-3|device=dm-3|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117258174464 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_goroutines|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095787532289 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/host|plugin=custom_info|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363095787532288 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=140012|svr_ip=10.31.35.6|tenant_name=sys, id=690363175248621568 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_max_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108676628480 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60021|svr_ip=10.31.35.6|tenant_name=sys, id=690363177400299520 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984096522240 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248267259904 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/checkPortOccupied|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100090888192 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_plan_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121565724672 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578420211712 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-4|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112979984384 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|device=tmpfs|fstype=tmpfs|install_path=|is_ob_disk=0|mount_label=|mount_point=/run/user/1002|mountpoint=/run/user/1002|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687517866327830528 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=1|is_ob_volume=1|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115131662336 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984109105152 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=140013|svr_ip=10.31.35.6|tenant_name=sys, id=690363175269593088 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/ob|process=ocp_monagent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363183859527680 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_stat_server_progress, id=690363222518427648 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_disk_v4_data_size|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248288231424 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_info|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6|version=go1.19.6, id=690363095816892417 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/host|plugin=node_info|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363095816892416 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition_replica|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097964376064 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_max_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108701794304 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=sda|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112996761600 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60022|svr_ip=10.31.35.6|tenant_name=sys, id=690363177425465344 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/checkPortOccupied|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100116054016 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_plan_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121590890496 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984125882368 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=20000|svr_ip=10.31.35.6|tenant_name=sys, id=690363175286370304 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248300814336 2024-09-11 18:31:52.717 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanDataFiles|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097976958976 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=1|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115156828160 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_stat_server_progress, id=690363222539399168 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520385154183168 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0|svr_ip=10.31.35.6, id=690363095837863937 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_alloc_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095837863936 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition_replica|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097985347584 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_max_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108722765824 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=sdb|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113017733120 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60023|svr_ip=10.31.35.6|tenant_name=sys, id=690363177446436864 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_session|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121611862016 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/ob|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363183893082112 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=0|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984146853888 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/commandExists|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363100141219840 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanDataFiles|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097997930496 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=200002|svr_ip=10.31.35.6|tenant_name=sys, id=690363175311536128 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.25|svr_ip=10.31.35.6, id=690363095858835457 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_alloc_bytes_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095858835456 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_role|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098006319104 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_max_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108743737344 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60024|svr_ip=10.31.35.6|tenant_name=sys, id=690363177467408384 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/ob|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363183909859328 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984163631104 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115190382592 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_session|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121632833536 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_stat_server_progress, id=690363222568759296 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanDataFiles|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098014707712 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/commandExists|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363100162191360 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=sdc|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113047093248 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_disk_v4_log_disk_size|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248342757376 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984176214016 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=200005|svr_ip=10.31.35.6|tenant_name=sys, id=690363175336701952 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248351145984 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60087|svr_ip=10.31.35.6|tenant_name=sys, id=690363177488379904 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6, id=690363095884001281 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_buck_hash_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095884001280 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_role|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098031484928 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_max_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108768903168 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=sdd|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113063870464 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115211354112 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983920361472 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/commandExists|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363100183162880 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_session|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121657999360 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_stat_server_progress, id=690363222593925120 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/getProcessInfo|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363098039873536 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_disk_v4_log_size|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248367923200 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520384961245184 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=200006|svr_ip=10.31.35.6|tenant_name=sys, id=690363175357673472 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983932944384 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-4|device=dm-4|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117375614976 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_role|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098052456448 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-0|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113084841984 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_min_iops|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=config_sys_zone1_sys1c4g_kid, id=690367021567995904 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=80001|svr_ip=10.31.35.6|tenant_name=sys, id=690363177513545728 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108794068992 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_session|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121678970880 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983945527296 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/getProcessInfo|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363098060845056 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/commandExists|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100208328704 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115240714240 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sample_sql, id=690363222619090944 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=20001|svr_ip=10.31.35.6|tenant_name=sys, id=690363175378644992 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/basic|process=ocp_monagent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363183968579584 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983958110208 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_frees_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095925944321 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.75|svr_ip=10.31.35.6, id=690363095925944320 2024-09-11 18:31:52.718 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_role|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098073427968 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248401477632 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108815040512 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=80002|svr_ip=10.31.35.6|tenant_name=sys, id=690363177538711552 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/getProcessInfo|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363098081816576 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115261685760 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=200010|svr_ip=10.31.35.6|tenant_name=sys, id=690363175395422208 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_log_disk_total_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248409866240 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/basic|process=ocp_monagent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363183989551104 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sample_statistics, id=690363222644256768 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/ocaenbase/redo|mountpoint=/ocaenbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578546040832 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=1|svr_ip=10.31.35.6, id=690363095942721537 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_gc_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095942721536 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_role|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098090205184 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248418254848 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=0|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983983276032 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578604761088 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/getProcessInfo|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098098593792 2024-09-11 18:31:52.739 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/commandExists|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100246077440 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108836012032 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-1|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113130979328 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_session|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121720913920 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_load1|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115282657280 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_max_cpu|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=config_sys_zone1_sys1c4g_kid, id=690367021484109824 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=200013|svr_ip=10.31.35.6|tenant_name=sys, id=690363175420588032 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=80003|svr_ip=10.31.35.6|tenant_name=sys, id=690363177568071680 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/basic|process=ocp_monagent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363184010522624 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983995858944 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sda|device=sda|env_type=host|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117438529536 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_role|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098115371008 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_log_disk_used_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248443420672 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578592178176 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363095972081665 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_alloc_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095972081664 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108856983552 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-2|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113151950848 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sql_audit, id=690363222677811200 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_max_iops|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=config_sys_zone1_sys1c4g_kid, id=690367021505081344 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984012636160 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/getProcessInfo|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098127953920 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_max_memory_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=config_sys_zone1_sys1c4g_kid, id=690367021526052864 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/basic|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363184031494144 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk_v4|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248456003584 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=80007|svr_ip=10.31.35.6|tenant_name=sys, id=690363177593237504 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_load15|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115316211712 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=20002|svr_ip=10.31.35.6|tenant_name=sys, id=690363175449948160 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984025219072 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_sync|app=OB|failed_reason=-|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_cluster_status=OPERATING|svr_ip=10.31.35.10, id=690369622095196160 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_table|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098140536832 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108877955072 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_session|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121762856960 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sql_audit_progress, id=690363222698782720 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363095997247489 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_idle_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095997247488 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-3|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113177116672 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_min_cpu|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=config_sys_zone1_sys1c4g_kid, id=690367021542830080 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_memory_hold_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248472780800 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanCGroupEnv|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363098148925440 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/hostInfo|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363100296409088 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sda1|device=sda1|env_type=host|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117476278272 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk_v4|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248481169408 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_load5|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115337183232 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30000|svr_ip=10.31.35.6|tenant_name=sys, id=690363175470919680 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=80008|svr_ip=10.31.35.6|tenant_name=sys, id=690363177618403328 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/basic|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363184060854272 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984046190592 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_table|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098161508352 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108898926592 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_sysstat|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121783828480 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdb|device=sdb|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984318820352 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108366249984 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/snapshot/collect|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363099780509696 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_unit_config|process=ocp_monagent|svr_ip=10.31.35.6, id=690363250108559360 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_gv_database|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097637220352 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-0|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112669605888 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdb1|device=sdb1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984331403264 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=custom_default_sql|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121263734784 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=120000|svr_ip=10.31.35.6|tenant_name=sys, id=690363174955020288 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant500_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247969464320 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/checkConnect|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097645608960 2024-09-11 18:31:52.740 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50010|svr_ip=10.31.35.6|tenant_name=sys, id=690363177106698240 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108387221504 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc|device=sdc|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984343986176 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample_statistics, id=690363222208049152 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-1|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112686383104 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60057|svr_ip=10.31.35.6|tenant_name=sys, id=690363247982047232 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097658191872 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=1|is_ob_volume=1|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114838061056 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc1|device=sdc1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984356569088 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=ocp_monagent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119141416960 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/checkConnect|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097670774784 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=120001|svr_ip=10.31.35.6|tenant_name=sys, id=690363174984380416 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-2|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112707354624 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50011|svr_ip=10.31.35.6|tenant_name=sys, id=690363177136058368 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc14|device=sdc14|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984369152000 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample_statistics, id=690363222233214976 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108416581632 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=1|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114867421184 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant500_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248015601664 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578147581952 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097691746304 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=120008|svr_ip=10.31.35.6|tenant_name=sys, id=690363175005351936 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc15|device=sdc15|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984385929216 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=ocp_agentd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119170777088 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121318260736 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50037|svr_ip=10.31.35.6|tenant_name=sys, id=690363177157029888 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample_statistics, id=690363222254186496 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-3|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112732520448 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/checkConnect|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097704329216 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/snapshot/collect|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099851812864 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108441747456 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdc2|device=sdc2|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984398512128 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60083|svr_ip=10.31.35.6|tenant_name=sys, id=690363248032378880 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520385431007232 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578197913600 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114888392704 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=120009|svr_ip=10.31.35.6|tenant_name=sys, id=690363175026323456 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121339232256 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdd|device=sdd|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984411095040 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097721106432 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-4|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112753491968 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=50038|svr_ip=10.31.35.6|tenant_name=sys, id=690363177182195712 2024-09-11 18:31:52.741 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108462718976 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-0|device=dm-0|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117052653568 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/checkConnect|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097729495040 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=687511966930202624 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant500_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248057544704 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=130000|svr_ip=10.31.35.6|tenant_name=sys, id=690363175047294976 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sdd1|device=sdd1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984427872256 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/snapshot/collect|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099885367296 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114917752832 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=fuse_row_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363237328515072 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant500_memory_hold_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363248065933312 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097742077952 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=sda|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112774463488 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121364398080 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanAllFiles|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097750466560 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=ocp_mgragent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119225303040 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60000|svr_ip=10.31.35.6|tenant_name=sys, id=690363177211555840 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-0|device=dm-0|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984444649472 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_assigned|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248078516224 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108492079104 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=130001|svr_ip=10.31.35.6|tenant_name=sys, id=690363175068266496 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363099906338816 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|quantile=0|svr_ip=10.31.35.6, id=690363095615565824 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=sdb|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112795435008 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114942918656 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121385369600 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984188796928 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample_statistics, id=690363222321295360 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097767243776 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=687511966959562752 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60001|svr_ip=10.31.35.6|tenant_name=sys, id=690363177228333056 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant500_memory_used_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363248095293440 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=index_block_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363237362069504 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanAllFiles|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097775632384 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984201379840 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|quantile=0.25|svr_ip=10.31.35.6, id=690363095632343040 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_latch|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108517244928 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=130002|svr_ip=10.31.35.6|tenant_name=sys, id=690363175093432320 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_assigned|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248107876352 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=sdc|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112816406528 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121406341120 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984213962752 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363099939893248 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60002|svr_ip=10.31.35.6|tenant_name=sys, id=690363177253498880 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=user_block_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363237383041024 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520385267429376 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_assigned_cpu_assigned|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248124653568 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|quantile=0.5|svr_ip=10.31.35.6, id=690363095653314560 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_latch|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108538216448 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-1|device=dm-1|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117128151040 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=687511966993117184 2024-09-11 18:31:52.762 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_bytes_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/ob|plugin=custom_info|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363183704338432 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_sql_audit_stat_0, id=690363222359044096 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=sdd|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112837378048 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=ocp_monagent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119279828992 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=130004|svr_ip=10.31.35.6|tenant_name=sys, id=690363175118598144 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-0|device=dm-0|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984230739968 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363099956670464 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121431506944 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097813381120 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60003|svr_ip=10.31.35.6|tenant_name=sys, id=690363177274470400 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114997444608 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-1|device=dm-1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984243322880 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_assigned|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248145625088 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_sql_audit_stat_0, id=690363222380015616 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|quantile=0.75|svr_ip=10.31.35.6, id=690363095678480384 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanAllFiles|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097825964032 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_latch|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108563382272 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-2|device=dm-2|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984251711488 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_assigned_cpu_total|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248154013696 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_bytes_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/basic|plugin=ob_basic|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363183733698560 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition_replica|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097834352640 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=ocp_agentd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119309189120 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=140002|svr_ip=10.31.35.6|tenant_name=sys, id=690363175147958272 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099986030592 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115018416128 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_plan_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121460867072 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_assigned|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248166596608 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanAllFiles|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097842741248 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-0|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112875126784 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60004|svr_ip=10.31.35.6|tenant_name=sys, id=690363177303830528 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-3|device=dm-3|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984268488704 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_latch|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108584353792 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/ob|plugin=custom_info|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363183750475776 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_sql_audit_stat_0, id=690363222405181440 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687511967009894400 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|quantile=1|svr_ip=10.31.35.6, id=690363095707840512 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_plan_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121477644288 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-4|device=dm-4|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984281071616 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/ocaenbase/redo|mountpoint=/ocaenbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578315354112 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition_replica|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097859518464 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=ocp_mgragent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119334354944 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578369880064 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/status|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100011196416 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/ob/basic|plugin=ob_basic|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363183767252992 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_assigned_mem_assigned|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248191762432 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_latch|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108605325312 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115047776256 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=60005|svr_ip=10.31.35.6|tenant_name=sys, id=690363177328996352 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sda|device=sda|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984293654528 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_audit_stat_0, id=690363222426152960 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=exporter_export_metrics_bytes_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/host|plugin=custom_info|process=ocp_monagent|svr_ip=10.31.35.6|type=exporter, id=690363095724617728 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanAllFiles|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363097872101376 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-1|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112904486912 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=140003|svr_ip=10.31.35.6|tenant_name=sys, id=690363175185707008 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_assigned|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248200151040 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_gc_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363095728812032 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_partition_replica|process=ocp_monagent|svr_ip=10.31.35.6, id=690363097880489984 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=ocp_monagent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363119355326464 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_plan_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121502810112 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=sda1|device=sda1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984306237440 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/checkPortOccupied|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363100032167936 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setClockSource|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687511967043448832 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/cleanDataFiles|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097888878592 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_audit_stat_0, id=690363222447124480 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_latch|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108630491136 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-2|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112925458432 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115072942080 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdb|device=sdb|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984587255808 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=arbitration_status|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246614704128 2024-09-11 18:31:52.763 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30013|svr_ip=10.31.35.6|tenant_name=sys, id=690363175751938048 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sda15|fstype=vfat|is_ob_disk=0|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420509298688 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_next_gc_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096295043072 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_time_waited|app=OB|event_group=USER_IO|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177903616000 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=arbitration_status_inactive_server_count|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363246623092736 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248770576384 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.7, id=687520373821173760 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/preCheck|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363098446721024 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/task/status|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363100594204672 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982456549376 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sda1|fstype=xfs|is_ob_disk=0|mount_point=/boot|mountpoint=/boot|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420484132864 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=sda|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113483300864 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdb1|device=sdb1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984608227328 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096307625984 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_sync|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122077429760 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=arbitration_status|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246635675648 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=node_info|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363098459303936 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_drop_total|app=HOST|device=lo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115639173120 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982469132288 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30080|svr_ip=10.31.35.6|tenant_name=sys, id=690363175777103872 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_total_waits|app=OB|event_group=APPLICATION|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177924587520 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc|device=sdc|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984620810240 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=log_stream_degraded_count|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246644064256 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|is_ob_disk=0|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420534464512 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_other_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096320208896 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_sync|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122090012672 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=sdb|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113504272384 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/preCheck|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098476081152 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-2|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982485909504 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_unit_max_cpu|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|svr_port=2882|tenant_name=sys|unit_id=1, id=690363248804130816 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_last_gc_time_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096332791808 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_errs_total|app=HOST|device=enP41243s1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115660144640 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_sync|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122102595584 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc1|device=sdc1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984637587456 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=arbitration_status|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246660841472 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.7, id=687520373842145280 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=custom_info|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363098484469760 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event_detail|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109221888000 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc15|device=sdc15|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117811822592 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30081|svr_ip=10.31.35.6|tenant_name=sys, id=690363175798075392 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248812519424 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_total_waits|app=OB|event_group=COMMIT|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177949753344 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-3|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982498492416 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_stack_inuse_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096345374720 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/task/status|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363100640342016 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_compaction_current_timestamp|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246673424384 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/preCheck|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098497052672 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=sdc|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113529438208 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc14|device=sdc14|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984654364672 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=687511803423649792 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_sync|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122123567104 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=dm-4|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982511075328 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_unit_max_memory|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|svr_port=2882|tenant_name=sys|unit_id=1, id=690363248829296640 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|status=200|svr_ip=10.31.35.7, id=687520373896671232 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_lookups_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096357957632 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=node_info|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363098505441280 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_errs_total|app=HOST|device=eth0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115685310464 2024-09-11 18:31:52.781 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=arbitration_status|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246686007296 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687517300889513984 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc2|device=sdc2|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117836988416 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30082|svr_ip=10.31.35.6|tenant_name=sys, id=690363175823241216 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_total_waits|app=OB|event_group=CONCURRENCY|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177970724864 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc15|device=sdc15|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984666947584 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event_detail|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109251248128 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=sdd|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113546215424 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=sda|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982523658240 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|status=200|svr_ip=10.31.35.7, id=687520373875699712 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687517893611778048 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareCGroupEnv|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363098518024192 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/task/status|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363100665507840 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_compaction_frozen_time|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246698590208 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_sync|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122144538624 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc2|device=sdc2|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984679530496 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248850268160 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=687511803398483968 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdd|device=sdd|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117853765632 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/setSecurityLimit|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=687511891441119232 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_errs_total|app=HOST|device=lo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115710476288 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_total_waits|app=OB|event_group=CONFIGURATION|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177991696384 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=sdb|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982540435456 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=arbitration_status|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246711173120 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event_detail|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109272219648 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-0|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113567186944 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdd|device=sdd|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984692113408 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=custom_info|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098538995712 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/task/status|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100686479360 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_sync|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122161315840 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30083|svr_ip=10.31.35.6|tenant_name=sys, id=690363175852601344 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_compaction_frozen_version|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246719561728 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_stack_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096395706368 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_unit_min_cpu|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|svr_port=2882|tenant_name=sys|unit_id=1, id=690363248871239680 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdd1|device=sdd1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984704696320 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareCGroupEnv|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363098551578624 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=arbitration_status|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246732144640 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248879628288 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-1|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113588158464 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_total_waits|app=OB|event_group=NETWORK|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178016862208 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=sdc|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982565601280 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/setSecurityLimit|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=687511891478867968 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520384638283776 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mallocs_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096412483584 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/task/status|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100707450880 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109297385472 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122182287360 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=log_tailer_processed_file_count|app=HOST|log_file_name=observer.log|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363510876827648 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=host_log_tailer_to_alarm|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098564161536 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdd1|device=sdd1|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117891514368 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30084|svr_ip=10.31.35.6|tenant_name=sys, id=690363175877767168 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_compaction_is_error|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246744727552 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_unit_min_memory|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|svr_port=2882|tenant_name=sys|unit_id=1, id=690363248892211200 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687511803490758656 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096420872192 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-2|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113609129984 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_total_waits|app=OB|event_group=OTHER|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178037833728 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=log_stream|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246757310464 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248904794112 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mcache_inuse_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096433455104 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareCGroupEnv|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363098580938752 2024-09-11 18:31:52.782 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_status_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100728422400 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_bytes_total|app=HOST|device=enP41243s1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115760807936 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109322551296 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-1|device=dm-1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984474009600 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/setSecurityLimit|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=687511891516616704 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=687511803469787136 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sql_plan_monitor, id=690363598978183168 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_threads|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096441843712 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30085|svr_ip=10.31.35.6|tenant_name=sys, id=690363175902932992 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_compaction_is_suspended|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246769893376 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=mlog_oblog_level_count|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098593521664 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_max_cpu|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=sys_unit_config, id=690363248921571328 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mcache_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096454426624 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-3|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113634295808 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=log_stream|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246782476288 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareCGroupEnv|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098606104576 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122232619008 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30086|svr_ip=10.31.35.6|tenant_name=sys, id=690363175923904512 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-2|device=dm-2|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984499175424 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248938348544 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=ob_sql_plan_monitor|process=ocp_monagent|svr_ip=10.31.35.6, id=690363598953017344 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/status|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363096467009536 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=mlog_oblog_location_cache|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098614493184 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/setSecurityLimit|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687511891554365440 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_compaction_last_version|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246795059200 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687511803507535872 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109356105728 2024-09-11 18:31:52.783 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_net_bandwidth_bps|app=HOST|device=enP41243s1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117946040320 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mspan_inuse_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096475398144 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareCGroupEnv|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098622881792 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-4|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113655267328 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-3|device=dm-3|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984511758336 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30200|svr_ip=10.31.35.6|tenant_name=sys, id=690363175940681728 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=log_stream|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246807642112 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_max_iops|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=sys_unit_config, id=690363248955125760 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/status|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363096483786752 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/ocaenbase/redo|mountpoint=/ocaenbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511578067890176 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=node_info|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098635464704 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122257784832 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=dm-4|device=dm-4|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984524341248 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|is_ob_disk=0|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420572213248 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_bytes_total|app=HOST|device=eth0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115819528192 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mspan_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096496369664 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareDirectories|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363098643853312 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_status_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100791336960 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109381271552 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=sda|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113676238848 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_total_waits|app=OB|event_group=SYSTEM_IO|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178104942592 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_compaction_start_time|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246824419328 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248971902976 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/setSecurityLimit|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687511891587919872 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_net_bandwidth_bps|app=HOST|device=eth0|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117979594752 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30201|svr_ip=10.31.35.6|tenant_name=sys, id=690363175965847552 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sda|device=sda|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984541118464 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_size_bytes|app=HOST|check_readonly=|device=/dev/sdc1|fstype=ext4|is_ob_disk=0|mount_point=/data|mountpoint=/data|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420555436032 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/status|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363096508952576 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=ob_resource_trends|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098656436224 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122278756352 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=log_stream|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246837002240 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_next_gc_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096517341184 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109402243072 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=sdb|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113697210368 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_bytes_total|app=HOST|device=lo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115844694016 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_total_waits|app=OB|event_group=USER_IO|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178125914112 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sda1|device=sda1|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371984553701376 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_max_memory_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=sys_unit_config, id=690363248992874496 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareDirectories|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363098669019136 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_status_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|status=500|svr_ip=10.31.35.6, id=690363100816502784 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_database_disk_data_size|app=OB|database_id=201001|database_name=oceanbase|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246849585152 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520384533426176 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/netInstall|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.7, id=687520373796007936 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/status|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363096525729792 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122299727872 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=log_stream|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246857973760 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_report_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=prometheusExporter|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098681602048 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30204|svr_ip=10.31.35.6|tenant_name=sys, id=690363175995207680 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_other_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096538312704 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=sdc|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113718181888 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_database_disk_data_size|app=OB|database_id=201003|database_name=mysql|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246866362368 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249013846016 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363100837474304 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109427408896 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/agent/status|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363096546701312 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareDirectories|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363098694184960 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_boot_time_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111579086848 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_table_num|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178155274240 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=log_stream|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246874750976 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_drop_total|app=HOST|device=enP41243s1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115878248448 2024-09-11 18:31:52.802 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memstore|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122320699392 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=sdc|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982708207616 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-4|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113198088192 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_sync|app=OB|failed_reason=-|ob_cluster_id=1718689631|ob_cluster_name=rtdob|ob_cluster_status=OPERATING|svr_ip=10.31.35.10, id=687845388844298240 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_goroutines|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096022413312 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanCGroupEnv|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363098169896960 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_inuse_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096022413313 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/hostInfo|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363100317380608 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdb|device=sdb|env_type=host|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117497249792 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sql_plan_explain, id=690363222728142848 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_memory_used_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248497946624 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sda1|fstype=xfs|is_ob_disk=0|mount_point=/boot|mountpoint=/boot|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420240863232 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_table|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098178285568 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_sysstat|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121800605696 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=input_collect_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=dbConnectivityInput|process=ocp_monagent|svr_ip=10.31.35.6|type=input, id=690363184081825792 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=sdd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982724984832 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520384814444544 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108919898112 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=sda|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113214865408 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_memory_Buffers_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115362349056 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30001|svr_ip=10.31.35.6|tenant_name=sys, id=690363175496085504 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanCGroupEnv|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363098186674176 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk_v4|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248514723840 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_objects|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096043384832 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/hostInfo|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363100338352128 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982737567744 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_info|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|version=go1.19.6, id=690363096047579136 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=80009|svr_ip=10.31.35.6|tenant_name=sys, id=690363177656152064 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=metric_cardinality_count|app=HOST|metric=ob_sysstat|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363184098603008 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_status_check|app=OB|check_item=cluster connection|failed_reason=-|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_cluster_status=DELETING|svr_ip=10.31.35.10, id=690372012806533120 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_table|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098199257088 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_sysstat|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121821577216 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sql_plan_meta, id=690363222757502976 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=sdb|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113235836928 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdb1|device=sdb1|env_type=host|is_ob_volume=1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117530804224 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30002|svr_ip=10.31.35.6|tenant_name=sys, id=690363175517057024 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982750150656 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sda15|fstype=vfat|is_ob_disk=0|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420274417664 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanCGroupEnv|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098207645696 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/hostInfo|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100355129344 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108945063936 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_memory_Cached_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115387514880 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_resource_max_cpu|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248539889664 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_released_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096068550656 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_table|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098216034304 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_alloc_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096068550657 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=80057|svr_ip=10.31.35.6|tenant_name=sys, id=690363177677123584 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-2|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982762733568 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_sysstat|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121842548736 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=metric_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline=|process=ocp_monagent|svr_ip=10.31.35.6, id=690363184123768832 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk_v4|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248548278272 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/ocaenbase/redo|mountpoint=/ocaenbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511577606516736 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30005|svr_ip=10.31.35.6|tenant_name=sys, id=690363175538028544 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanCGroupEnv|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098228617216 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/hostInfo|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100376100864 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108966035456 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sql_plan_stat, id=690363222786863104 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-3|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982779510784 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_alloc_bytes_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096089522176 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096089522177 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=sdc|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113269391360 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_time_waited|app=OB|event_group=APPLICATION|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177698095104 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_sysstat|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121863520256 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_basic|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363184144740352 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_sync|app=OB|failed_reason=-|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_cluster_status=DELETING|svr_ip=10.31.35.10, id=690372012852670464 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_plan_monitor, id=690363611535929344 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=dm-4|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982792093696 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_resource_max_memory|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248573444096 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363108987006976 2024-09-11 18:31:52.803 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_memory_MemFree_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115433652224 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_table|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098257977344 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc|device=sdc|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117585330176 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=sda|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982804676608 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=sql_audit_input_collect_lost_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obproxy_cluster=|obproxy_cluster_id=0|plugin=sql_audit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363222816223232 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_last_gc_time_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096114688000 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_buck_hash_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096114688001 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/userExists|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363100409655296 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_sysstat|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121884491776 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_basic|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363184165711872 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk_v4|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248590221312 2024-09-11 18:31:52.804 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_time_waited|app=OB|event_group=COMMIT|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177727455232 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanDirectories|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363098270560256 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109007978496 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=sdb|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982817259520 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=sdd|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113307140096 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_memory_MemTotal_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115454623744 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30006|svr_ip=10.31.35.6|tenant_name=sys, id=690363175588360192 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tablegroup|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098278948864 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=ob_basic|process=ocp_monagent|svr_ip=10.31.35.6, id=690363184182489088 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=sdc|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982829842432 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=sql_audit_input_collect_total_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obproxy_cluster=|obproxy_cluster_id=0|plugin=sql_audit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363222841389056 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109024755712 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_resource_min_cpu|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248615387136 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_frees_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096144048128 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanDirectories|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363098291531776 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121913851904 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-0|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113328111616 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30007|svr_ip=10.31.35.6|tenant_name=sys, id=690363175609331712 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=687517893527891968 2024-09-11 18:31:52.821 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_sql_plan_monitor, id=690363611607232512 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_lookups_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096152436736 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/userExists|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363100447404032 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_memory_SReclaimable_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115479789568 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=dbConnectivityInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363184203460608 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_io_time_weighted_seconds_total|app=HOST|device=sdd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982578184192 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tablegroup|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098304114688 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.7, id=687520384688615424 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanDirectories|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363098308308992 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_time_waited|app=OB|event_group=CONCURRENCY|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177769398272 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_disk_v4|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248636358656 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_gc_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096165019648 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event_detail|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109049921536 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982590767104 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|is_ob_disk=0|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420349915136 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121939017728 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30008|svr_ip=10.31.35.6|tenant_name=sys, id=690363175630303232 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=687517893540474880 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tablegroup|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098320891904 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/userExists|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363100468375552 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-1|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113353277440 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=dbConnectivityInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363184224432128 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mallocs_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096177602560 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_bytes_total|app=HOST|device=enP41243s1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115504955392 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982603350016 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_sql_plan_monitor, id=690363611586260992 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanDirectories|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098329280512 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_time_waited|app=OB|event_group=CONFIGURATION|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177790369792 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_resource_min_memory|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363248657330176 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_alloc_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096185991168 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event_detail|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109070893056 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc1|device=sdc1|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117660827648 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=687517893557252096 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event|process=ocp_monagent|svr_ip=10.31.35.6, id=690363121959989248 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tablegroup|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098341863424 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248669913088 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mcache_inuse_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096198574080 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30009|svr_ip=10.31.35.6|tenant_name=sys, id=690363175659663360 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_sql_plan_monitor, id=690363611556900864 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/cleanDirectories|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098350252032 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_bytes_total|app=HOST|device=eth0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115530121216 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_time_waited|app=OB|event_group=NETWORK|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177811341312 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setTransparentHugePage|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687517893574029312 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_idle_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096206962688 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-2|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982632710144 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511577744928768 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tablegroup|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098362834944 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511577799454720 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_net_bandwidth_bps|app=HOST|device=enP61571s1|env_type=host|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511579946938368 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|is_ob_disk=0|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420299583488 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mcache_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096219545600 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30010|svr_ip=10.31.35.6|tenant_name=sys, id=690363175680634880 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-3|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982645293056 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_server_data_index_data_size|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|svr_port=2882|tenant_name=sys, id=690363248695078912 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/preCheck|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363098371223552 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-2|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113403609088 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_bytes_total|app=HOST|device=lo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115551092736 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687517300755296256 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/userExists|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100522901504 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_util|app=HOST|dev=sdc14|device=sdc14|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363117702770688 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tablegroup|process=ocp_monagent|svr_ip=10.31.35.6, id=690363098379612160 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_time_waited|app=OB|event_group=OTHER|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177840701440 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=dm-4|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982657875968 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_inuse_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096236322816 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122006126592 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248711856128 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mspan_inuse_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096244711424 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event_detail|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109129613312 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/preCheck|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363098396389376 2024-09-11 18:31:52.822 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-3|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113428774912 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_drop_total|app=HOST|device=enP41243s1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115576258560 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30011|svr_ip=10.31.35.6|tenant_name=sys, id=690363175709995008 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=sda|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982674653184 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_plan_monitor, id=690363611640786944 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_server_data_size|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|svr_port=2882|tenant_name=sys, id=690363248728633344 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_objects|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096257294336 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=custom_info|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363098408972288 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_log_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363248737021952 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_mspan_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363096265682944 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/userExists|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363100560650240 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122035486720 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30012|svr_ip=10.31.35.6|tenant_name=sys, id=690363175730966528 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_system_event_time_waited|app=OB|event_group=SYSTEM_IO|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363177878450176 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_bytes_total|app=HOST|device=sdb|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982695624704 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_plan_monitor, id=690363611628204032 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_readonly|app=HOST|check_readonly=|device=/dev/sdc1|fstype=ext4|is_ob_disk=0|mount_point=/data|mountpoint=/data|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420324749312 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_heap_released_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096278265856 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-4|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113458135040 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_receive_drop_total|app=HOST|device=eth0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115605618688 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_server_required_size|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|svr_port=2882|tenant_name=sys, id=690363248753799168 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_system_event|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122052263936 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498407161856 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249294864384 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.99|svr_ip=10.31.35.6|table=ob_hist_sqltext, id=690363109708427264 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30222|svr_ip=10.31.35.6|tenant_name=sys, id=690363176284614656 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_database_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247151575040 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_boot_time_seconds|app=HOST|name=ocp_monagent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118302556160 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=sdd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982985031680 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/start|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098983591936 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=0|mode=softirq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111868493824 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_extra|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363122605912064 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498419744768 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/host|process=ocp_monagent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363096844496896 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=nodeExporterInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363098991980544 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sqltext, id=690363109729398784 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40000|svr_ip=10.31.35.6|tenant_name=sys, id=690363176305586176 2024-09-11 18:31:52.823 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249320030208 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983010197504 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_extra_custom|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363122626883584 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sqltext, id=690363109746176000 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40001|svr_ip=10.31.35.6|tenant_name=sys, id=690363176322363392 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sda15|fstype=vfat|is_ob_disk=0|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420005982208 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/disk/usage|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363096865468416 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=1|device=/dev/mapper/rootvg-rootlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/|mountpoint=/|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114045337600 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983022780416 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498444910592 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/start|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099017146368 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498457493504 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sda1|fstype=xfs|is_ob_disk=0|mount_point=/boot|mountpoint=/boot|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419980816384 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=1|mode=idle|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111906242560 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_memstore_freeze_times|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363247201906688 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249349390336 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/host|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363096878051328 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setSelinux|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=687511916644691968 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_errs_total|app=HOST|device=lo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116209598464 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_extra|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363122652049408 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40002|svr_ip=10.31.35.6|tenant_name=sys, id=690363176343334912 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-2|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983039557632 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/disk/usage|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363096886439936 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_task_size|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|task_name=sql_text, id=690363109771341824 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114066309120 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setSelinux|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687511916703412224 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk_invalid|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247214489600 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=processInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363099042312192 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=1|mode=softirq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111927214080 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498482659328 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_memstore_freeze_trigger_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363247222878208 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/host|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363096899022848 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_extra_custom|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363122668826624 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-3|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983056334848 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=ob_sql_plan|process=ocp_monagent|svr_ip=10.31.35.6, id=690363221457268736 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=/home/admin/oceanbase|is_ob_disk=0|mount_label=install_path|mount_point=/home|mountpoint=/home|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511577392607232 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249378750464 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114087280640 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40003|svr_ip=10.31.35.6|tenant_name=sys, id=690363176368500736 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk_invalid|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247235461120 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=ocp_agentd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118386442240 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=dm-4|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983068917760 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setSelinux|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=687511916678246400 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/install|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363099063283712 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_memstore_total_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363247243849728 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=2|mode=idle|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111956574208 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=sda|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983081500672 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan, id=690363221482434560 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=prometheusExporter|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=exporter, id=690363099075866624 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40004|svr_ip=10.31.35.6|tenant_name=sys, id=690363176389472256 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk_invalid|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247256432640 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687517300382003200 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/install|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363099084255232 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_time_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116264124416 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=ob_extra|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122706575360 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=sdb|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983094083584 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_resource_cpu|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247264821248 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249412304896 2024-09-11 18:31:52.842 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=ocp_mgragent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118415802368 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/disk/usage|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363096945160192 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_explain, id=690363221503406080 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk_invalid|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247273209856 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363099096838144 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=2|mode=softirq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111981740032 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40005|svr_ip=10.31.35.6|tenant_name=sys, id=690363176410443776 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=sdc|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983106666496 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_server_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498528796672 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_read_time_seconds_total|app=HOST|device=sdd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982842425344 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/install|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363099105226752 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=ob_extra_custom|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122727546880 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_resource_cpu_assigned|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247285792768 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_monagent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363096961937408 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114141806592 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=ocp_monagent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118436773888 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982855008256 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan_stat_0, id=690363221524377600 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk_invalid|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247294181376 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687517300549775360 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_max_fds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363099117809664 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40006|svr_ip=10.31.35.6|tenant_name=sys, id=690363176431415296 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498558156800 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/disk/usage|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363096974520320 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=3|mode=idle|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112006905856 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114154389504 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249450053632 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982867591168 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_resource_cpu_assigned_percent|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247306764288 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=1|is_ob_volume=1|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114166972416 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample, id=690363221545349120 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_disk_invalid|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247315152896 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_monagent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363096991297536 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40007|svr_ip=10.31.35.6|tenant_name=sys, id=690363176452386816 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-2|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982880174080 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=3|mode=softirq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112027877376 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498583322624 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_resource_disk_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247327735808 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249475219456 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/checkDirectoryPermission|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363097003880448 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/install|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099155558400 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-3|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982896951296 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_audit_sample_statistics, id=690363221566320640 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_monagent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097012269056 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=cardinalityStat|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363122782072832 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cluster_status_check|app=OB|check_item=cluster connection|failed_reason=-|ob_cluster_id=1718689631|ob_cluster_name=rtdob|ob_cluster_status=OPERATING|svr_ip=10.31.35.10, id=687845388965933056 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40008|svr_ip=10.31.35.6|tenant_name=sys, id=690363176473358336 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247340318720 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=4|mode=idle|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112048848896 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=cpu_count|app=HOST|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116343816192 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498595905536 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/checkDirectoryPermission|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363097020657664 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_open_fds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363099168141312 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=ocp_agentd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118495494144 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=dm-4|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982909534208 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_resource_memory_assigned_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247348707328 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=1|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114204721152 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249500385280 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363097029046272 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_sql_transaction|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363397672562688 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sql_audit_stat_0, id=690363221587292160 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498616877056 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=sda|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982922117120 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247361290240 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/install|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363099184918528 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=4|mode=softirq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112069820416 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=io_await|app=HOST|dev=dm-0|device=dm-0|env_type=host|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116364787712 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=jointable|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363122807238656 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40009|svr_ip=10.31.35.6|tenant_name=sys, id=690363176498524160 2024-09-11 18:31:52.843 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/file/checkDirectoryPermission|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363097041629184 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=ocp_mgragent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118520659968 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_sql_transaction|process=ocp_monagent|status=successful|svr_ip=10.31.35.6, id=690363397689339904 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=sdb|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982934700032 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_resource_memory_assigned_percent|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247373873152 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_resident_memory_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363099197501440 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114229886976 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|is_ob_disk=0|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420026953728 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_task|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498642042880 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=5|mode=idle|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112090791936 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mysqlTableInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363122828210176 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40010|svr_ip=10.31.35.6|tenant_name=sys, id=690363176519495680 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_reads_completed_total|app=HOST|device=sdc|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371982947282944 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247386456064 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/uninstall|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363099210084352 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_stat_server_progress, id=690363221620846592 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_server_data_index|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249538134016 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|is_ob_disk=0|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420073091072 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_start_time_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363099218472960 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=ocp_monagent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118545825792 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files_free|app=HOST|check_readonly=|device=/dev/sdc1|fstype=ext4|is_ob_disk=0|mount_point=/data|mountpoint=/data|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516420047925248 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363097075183616 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363114255052800 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_server_resource_memory_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247403233280 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=5|mode=softirq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363112111763456 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=ob_sql_transaction|process=ocp_monagent|svr_ip=10.31.35.6, id=690363397722894336 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_log_level_count|app=OB|level=INFO|logSourceType=election|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363174392983552 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=40011|svr_ip=10.31.35.6|tenant_name=sys, id=690363176540467200 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setSelinux|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=687511916846018560 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/uninstall|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363099231055872 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=retagProcessor|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363122853376000 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_plan_cache_plan, id=690363221641818112 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_min_cpu|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=sys_unit_config, id=690363249026428928 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|is_ob_disk=0|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419695603712 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30205|svr_ip=10.31.35.6|tenant_name=sys, id=690363176016179200 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_stack_inuse_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096559284224 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=aggregateProcessor|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363098706767872 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983253467136 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_database_disk_data_size|app=OB|database_id=500003|database_name=ocp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246887333888 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=sdd|is_ob_volume=|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113743347712 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/time|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363096567672832 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareDirectories|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098715156480 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total|app=HOST|mode=idle|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111600058368 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_memory|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249043206144 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_max_fds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363100866834432 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_request_queue|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122341670912 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983266050048 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_compaction|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246899916800 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_stack_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096580255744 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=attrProcessor|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363098727739392 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_xa_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109465157632 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_drop_total|app=HOST|device=eth0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115907608576 2024-09-11 18:31:52.862 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_database_disk_required_size|app=OB|database_id=201001|database_name=oceanbase|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246912499712 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/time|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363096588644352 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_net_bandwidth_bps|app=HOST|device=lo|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118063480832 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-2|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983282827264 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/install/prepareDirectories|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098740322304 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_open_fds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363100887805952 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_compaction|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246920888320 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/info|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690369588469460992 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_errs_total|app=HOST|device=enP61571s1|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687511579233906688 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total|app=HOST|mode=iowait|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111629418496 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_request_queue|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122366836736 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-3|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983291215872 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|is_ob_disk=0|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419741741056 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_memstats_sys_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096601227264 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_xa_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109486129152 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_status_count|app=HOST|method=GET|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|status=500|svr_ip=10.31.35.6, id=687831522596909056 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113785290752 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_database_disk_required_size|app=OB|database_id=201003|database_name=mysql|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246933471232 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=0|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687517300205842432 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sdc1|fstype=ext4|is_ob_disk=0|mount_point=/data|mountpoint=/data|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419716575232 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/time|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363096609615872 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=hostCustomInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363098757099520 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30208|svr_ip=10.31.35.6|tenant_name=sys, id=690363176070705152 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=dm-4|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983303798784 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_resident_memory_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363100908777472 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_request_queue|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122383613952 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_compaction|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246941859840 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/info|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690369588482043904 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=go_threads|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6, id=690363096618004480 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_xa_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109502906368 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/access|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363098769682432 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_ntp_offset_seconds|app=HOST|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118097035264 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=sda|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983316381696 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_database_disk_required_size|app=OB|database_id=500003|database_name=ocp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363246950248448 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/info|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690369588440100864 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/time|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363096626393088 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total|app=HOST|mode=irq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111658778624 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-tmplv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/tmp|mountpoint=/tmp|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113810456576 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_drop_total|app=HOST|device=lo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363115957940224 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_request_queue|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122400391168 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sda1|fstype=xfs|is_ob_disk=0|mount_point=/boot|mountpoint=/boot|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419641077760 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mountLabelProcessor|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363098782265344 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_start_time_seconds|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363100929748992 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30209|svr_ip=10.31.35.6|tenant_name=sys, id=690363176095870976 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_log_sync_delay|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178243354624 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=sdb|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983328964608 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_compaction|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246962831360 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_xa_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109523877888 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_config_min_iops|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys|unit_config_name=sys_unit_config, id=690363249114509312 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/log/alarms|process=ocp_monagent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363096643170304 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/access|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363098790653952 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_disk_free_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363246971219968 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/info|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690369588452683776 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=sdc|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983341547520 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/time|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363096651558912 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249127092224 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/package/info|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690369588410740736 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_files|app=HOST|check_readonly=|device=/dev/sda15|fstype=vfat|is_ob_disk=0|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|obregion=|obzone=|svr_ip=10.31.35.6, id=687516419662049280 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=nodeExporterInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363098803236864 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_request_queue|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122425556992 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_request_queue_large_size|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178264326144 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_compaction|process=ocp_monagent|svr_ip=10.31.35.6, id=690363246983802880 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/log/alarms|process=ocp_monagent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363096659947520 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_xa_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109544849408 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-usrlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/usr|mountpoint=/usr|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113839816704 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30212|svr_ip=10.31.35.6|tenant_name=sys, id=690363176121036800 2024-09-11 18:31:52.863 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_written_bytes_total|app=HOST|device=sdd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983354130432 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_virtual_memory_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363100959109120 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total|app=HOST|mode=nice|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111696527360 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_backup_storage_capacity_bytes|app=BACKUP|backup_type=LOG_BACKUP|cluster_id=2|ob_cluster_id=1718689631|ob_cluster_name=rtdob|ob_tenant_id=1004|storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p59|svr_ip=10.31.35.10, id=686706949860126720 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363096668336128 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_disk_invalid_count|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363246996385792 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_cache_size_bytes|app=OB|cache_name=tablet_table_cache|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363551943258112 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/access|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363098820014080 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ntp_service_available|app=HOST|env_type=host|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118147366912 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_request_queue|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122446528512 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-homelv|fstype=xfs|install_path=|is_ob_disk=0|log_disk_path=|mount_label=|mount_point=/home|mountpoint=/home|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983370907648 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_unit_num|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363249152258048 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/log/alarms|process=ocp_monagent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363096680919040 2024-09-11 18:31:52.864 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=processInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363098828402688 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_xa_trans|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109565820928 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/mapper/rootvg-varlv|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/var|mountpoint=/var|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113860788224 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_tenant_request_queue_total_size|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178289491968 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_compaction|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247008968704 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_virtual_memory_max_bytes|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_mgragent|svr_ip=10.31.35.6, id=690363100980080640 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total|app=HOST|mode=softirq|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111717498880 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363096689307648 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_write_time_seconds_total|app=HOST|device=sdd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983115055104 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/access|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098840985600 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_errs_total|app=HOST|device=enP41243s1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116020854784 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/log/alarms|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363096697696256 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sda1|fstype=ext4|install_path=/data/rtdob/oceanbase|is_ob_disk=1|is_ob_volume=1|log_disk_path=|mount_label=install_path|mount_point=/data|mountpoint=/data|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_data|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113877565440 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_disk_log_assigned_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247025745920 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=ob_log_tailer_to_alarm|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109586792448 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_waitevent|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122471694336 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memtable|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498289721344 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_seconds_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=prometheusExporter|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=exporter, id=690363098853568512 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30213|svr_ip=10.31.35.6|tenant_name=sys, id=690363176167174144 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983131832320 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_memtable_max_snapshot_duration_seconds|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363498285527040 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.05|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_database_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247034134528 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363096710279168 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_waitevent_wait_seconds_total|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178318852096 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249185812480 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_boot_time_seconds|app=HOST|name=ocp_agentd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118189309952 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_disk_log_total_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247042523136 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total|app=HOST|mode=steal|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111751053312 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/log/alarms|process=ocp_monagent|status=200|svr_ip=10.31.35.6, id=690363096722862080 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/access|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363098870345728 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_pipeline_process_metrics_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|pipeline_name=ob_sql_audit|process=ocp_monagent|svr_ip=10.31.35.6, id=690363109607763968 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_waitevent|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122492665856 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983148609536 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memtable|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498314887168 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.1|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_database_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247055106048 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_sum|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363096731250688 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdb1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=1|log_disk_path=/oceanbase/redo|mount_label=log_disk_path|mount_point=/oceanbase/redo|mountpoint=/oceanbase/redo|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=ob_log|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113911119872 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30216|svr_ip=10.31.35.6|tenant_name=sys, id=690363176192339968 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=aggregateProcessor|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363098882928640 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-2|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983161192448 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_waitevent_wait_total|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363178348212224 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249215172608 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_waitevent|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122513637376 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memtable|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498331664384 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_disk_log_used_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247071883264 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|svr_ip=10.31.35.6|table=ob_hist_sqltext, id=690363109632929792 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total|app=HOST|mode=system|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111780413440 2024-09-11 18:31:52.889 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-3|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983173775360 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/start|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363098899705856 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdc1|fstype=xfs|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot|mountpoint=/boot|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113932091392 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30217|svr_ip=10.31.35.6|tenant_name=sys, id=690363176213311488 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=0.5|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_database_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247084466176 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/host|process=ocp_monagent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363096760610816 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_boot_time_seconds|app=HOST|name=ocp_mgragent|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363118235447296 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=dm-4|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983186358272 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_disk_total_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247088660480 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=attrProcessor|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363098912288768 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_network_transmit_errs_total|app=HOST|device=eth0|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363116092157952 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memtable|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498344247296 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memtable|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498356830208 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary_count|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/stat|process=ocp_mgragent|status=200|svr_ip=10.31.35.6, id=690363096773193728 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/start|process=ocp_mgragent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363098920677376 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.5|svr_ip=10.31.35.6|table=ob_hist_sqltext, id=690363109658095616 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdc15|fstype=vfat|install_path=|is_ob_disk=0|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/boot/efi|mountpoint=/boot/efi|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113953062912 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_waitevent|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122542997504 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30220|svr_ip=10.31.35.6|tenant_name=sys, id=690363176234283008 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=sda|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983203135488 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_database_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247105437696 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_bucket|app=HOST|le=+Inf|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_tenant_resource|process=ocp_monagent|svr_ip=10.31.35.6, id=690363249252921344 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=hostCustomInput|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=input, id=690363098933260288 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=0|ob_cluster_name=|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/system/setSelinux|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=687511916602748928 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=GET|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/metrics/node/host|process=ocp_monagent|quantile=0.9|status=200|svr_ip=10.31.35.6, id=690363096789970944 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total|app=HOST|mode=user|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111822356480 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=sdb|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983215718400 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_index_error_num|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363247118020608 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_sysstat|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|stat_id=30221|svr_ip=10.31.35.6|tenant_name=sys, id=690363176255254528 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=mysql_output_write_sql_milliseconds_summary|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|process=ocp_monagent|quantile=0.9|svr_ip=10.31.35.6|table=ob_hist_sqltext, id=690363109683261440 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_waitevent|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122568163328 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/disk/usage|process=ocp_mgragent|quantile=0.5|status=200|svr_ip=10.31.35.6, id=690363096802553856 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=http_request_milliseconds_summary|app=HOST|method=POST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|path=/api/v1/ob/observer/start|process=ocp_mgragent|quantile=0.99|status=200|svr_ip=10.31.35.6, id=690363098950037504 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_filesystem_avail_bytes|app=HOST|check_readonly=|device=/dev/sdd1|fstype=ext4|install_path=|is_ob_disk=1|is_ob_volume=|log_disk_path=|mount_label=|mount_point=/mnt|mountpoint=/mnt|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_device=|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363113982423040 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=sdc|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983228301312 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_memtable|process=ocp_monagent|svr_ip=10.31.35.6, id=690363498390384640 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_sum|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_database_disk|process=ocp_monagent|svr_ip=10.31.35.6, id=690363247130603520 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_plugin_execute_total|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=mountLabelProcessor|process=ocp_monagent|status=successful|svr_ip=10.31.35.6|type=processor, id=690363098958426112 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_cpu_seconds_total_native|app=HOST|cpu=0|mode=idle|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690363111843328000 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=monagent_input_sql_duration_seconds_count|app=HOST|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|plugin=ob_waitevent|process=ocp_monagent|svr_ip=10.31.35.6, id=690363122584940544 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=node_disk_writes_completed_total|app=HOST|device=sdd|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|obproxy_cluster=|obproxy_cluster_id=0|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6, id=690371983240884224 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=ob_memstore_active_bytes|app=OB|ob_cluster_id=1719994441|ob_cluster_name=linkrtd|ob_tenant_id=1|obregion=linkrtd|obzone=zone1|svr_ip=10.31.35.6|tenant_name=sys, id=690363247143186432 2024-09-11 18:31:52.890 INFO 2981023 --- [ocp-schedules-16,ffaacc0ae37c4933,03f4c9a2f2186d1b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=10273 2024-09-11 18:31:52.918 INFO 2981023 --- [ocp-schedules-22,98da0094cad0bd34,66648aa407850b34] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:31:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,2f3678c7dc9ae573,b5aefcad1a8fb2a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,2f3678c7dc9ae573,b5aefcad1a8fb2a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:53.411 INFO 2981023 --- [ocp-schedules-30,e4123f0eb13bdbe3,195b36662afe2a71] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:31:53.411 INFO 2981023 --- [ocp-schedules-30,e4123f0eb13bdbe3,195b36662afe2a71] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:31:53.484 INFO 2981023 --- [ocp-async-3,10465514d5024d24,f9a24d6a50967842] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:54.428 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:54.428 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:54.535 INFO 2981023 --- [ocp-schedules-28,bd1654abcdb0229c,aab4e06272862084] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:31:55.484 INFO 2981023 --- [ocp-async-8,8676fc72a1e91334,814b493df0a01d29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:55.644 INFO 2981023 --- [ocp-schedules-15,fcdb83cf3e8535e8,7174c152b9aa99f6] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:31:55.662 WARN 2981023 --- [ocp-schedules-15,fcdb83cf3e8535e8,7174c152b9aa99f6] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:31:55.942 INFO 2981023 --- [ocp-schedules-10,895fadc12f40c0a9,d18dc19e54ed8613] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:31:55.942 INFO 2981023 --- [ocp-schedules-10,895fadc12f40c0a9,d18dc19e54ed8613] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:31:55.943 INFO 2981023 --- [ocp-schedules-10,895fadc12f40c0a9,d18dc19e54ed8613] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:31:55.943 INFO 2981023 --- [ocp-schedules-10,895fadc12f40c0a9,d18dc19e54ed8613] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:31:55.943 INFO 2981023 --- [ocp-schedules-10,895fadc12f40c0a9,d18dc19e54ed8613] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:31:55.943 INFO 2981023 --- [ocp-schedules-10,895fadc12f40c0a9,d18dc19e54ed8613] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:31:55.987 INFO 2981023 --- [ocp-schedules-22,7de7f95c4869dd13,5c28223cf9b89a1b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:31:55.995 INFO 2981023 --- [ocp-schedules-22,7de7f95c4869dd13,5c28223cf9b89a1b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:31:55.995 INFO 2981023 --- [ocp-schedules-22,7de7f95c4869dd13,5c28223cf9b89a1b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:31:56.180 INFO 2981023 --- [ocp-schedules-1,3f1371fc3a0e4dbf,fc9ee9691794f475] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:31:56.187 INFO 2981023 --- [ocp-schedules-1,3f1371fc3a0e4dbf,fc9ee9691794f475] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:31:56.187 INFO 2981023 --- [ocp-schedules-1,3f1371fc3a0e4dbf,fc9ee9691794f475] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:31:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,cb669c119c4f379f,5064055a7cb7f769] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,cb669c119c4f379f,5064055a7cb7f769] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:56.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:31:56.353563+08:00, lastProcessTime=2024-09-11T18:31:46.358452+08:00 2024-09-11 18:31:56.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:31:56.353563+08:00 2024-09-11 18:31:57.428 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:31:57.430 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:31:57.484 INFO 2981023 --- [ocp-async-1,408f30992c83738f,2428d2488ab5e9be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:31:57.800 INFO 2981023 --- [ocp-async-5,a1f24d6027a4990f,163196f9f9ce70d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:31:59.165 INFO 2981023 --- [ocp-schedules-23,1cb95755c7f945f6,c8e33f918cc77dd3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:31:59.165 INFO 2981023 --- [ocp-schedules-23,1cb95755c7f945f6,c8e33f918cc77dd3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:31:59.196 INFO 2981023 --- [Collect all tenants major compactions1,1cb95755c7f945f6,dd15ad422a14bc6e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:31:59.197 INFO 2981023 --- [Collect all tenants major compactions2,1cb95755c7f945f6,0b7262846e252c3d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:31:59.197 INFO 2981023 --- [Collect all tenants major compactions3,1cb95755c7f945f6,3893accc745e3e34] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:31:59.203 INFO 2981023 --- [Collect all tenants major compactions4,1cb95755c7f945f6,ade6015947f8cbc7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:31:59.204 INFO 2981023 --- [Collect all tenants major compactions5,1cb95755c7f945f6,a0fca9673085c1dc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:31:59.205 INFO 2981023 --- [Collect all tenants major compactions6,1cb95755c7f945f6,b4da35a9929dec4d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:31:59.218 INFO 2981023 --- [Collect all tenants major compactions1,1cb95755c7f945f6,dd15ad422a14bc6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:59.219 INFO 2981023 --- [Collect all tenants major compactions2,1cb95755c7f945f6,0b7262846e252c3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:59.219 INFO 2981023 --- [Collect all tenants major compactions3,1cb95755c7f945f6,3893accc745e3e34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:59.220 INFO 2981023 --- [Collect all tenants major compactions2,1cb95755c7f945f6,0b7262846e252c3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:31:59.220 INFO 2981023 --- [Collect all tenants major compactions1,1cb95755c7f945f6,dd15ad422a14bc6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:31:59.221 INFO 2981023 --- [Collect all tenants major compactions3,1cb95755c7f945f6,3893accc745e3e34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:31:59.221 INFO 2981023 --- [Collect all tenants major compactions6,1cb95755c7f945f6,b4da35a9929dec4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:59.222 INFO 2981023 --- [Collect all tenants major compactions4,1cb95755c7f945f6,ade6015947f8cbc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:59.222 INFO 2981023 --- [Collect all tenants major compactions6,1cb95755c7f945f6,b4da35a9929dec4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:31:59.223 INFO 2981023 --- [Collect all tenants major compactions1,1cb95755c7f945f6,dd15ad422a14bc6e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:31:59.223 INFO 2981023 --- [Collect all tenants major compactions1,1cb95755c7f945f6,dd15ad422a14bc6e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:31:59.223 INFO 2981023 --- [Collect all tenants major compactions2,1cb95755c7f945f6,0b7262846e252c3d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:31:59.223 INFO 2981023 --- [Collect all tenants major compactions2,1cb95755c7f945f6,0b7262846e252c3d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:31:59.223 INFO 2981023 --- [Collect all tenants major compactions4,1cb95755c7f945f6,ade6015947f8cbc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:31:59.224 INFO 2981023 --- [Collect all tenants major compactions3,1cb95755c7f945f6,3893accc745e3e34] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:31:59.224 INFO 2981023 --- [Collect all tenants major compactions3,1cb95755c7f945f6,3893accc745e3e34] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:31:59.224 INFO 2981023 --- [Collect all tenants major compactions5,1cb95755c7f945f6,a0fca9673085c1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:31:59.225 INFO 2981023 --- [Collect all tenants major compactions4,1cb95755c7f945f6,ade6015947f8cbc7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:31:59.225 INFO 2981023 --- [Collect all tenants major compactions4,1cb95755c7f945f6,ade6015947f8cbc7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:31:59.225 INFO 2981023 --- [Collect all tenants major compactions5,1cb95755c7f945f6,a0fca9673085c1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:31:59.226 INFO 2981023 --- [Collect all tenants major compactions6,1cb95755c7f945f6,b4da35a9929dec4d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:31:59.226 INFO 2981023 --- [Collect all tenants major compactions6,1cb95755c7f945f6,b4da35a9929dec4d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:31:59.227 INFO 2981023 --- [Collect all tenants major compactions5,1cb95755c7f945f6,a0fca9673085c1dc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:31:59.227 INFO 2981023 --- [Collect all tenants major compactions5,1cb95755c7f945f6,a0fca9673085c1dc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:31:59.228 INFO 2981023 --- [ocp-schedules-23,1cb95755c7f945f6,c8e33f918cc77dd3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:31:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,73f5517375f61283,bb0a8eca006131f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050720, interval=60 2024-09-11 18:31:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,73f5517375f61283,bb0a8eca006131f2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:31:59.484 INFO 2981023 --- [ocp-async-9,47e93affe57dea08,e573c931cdebe92d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:00.430 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:00.431 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:00.850 INFO 2981023 --- [ocp-schedules-22,380063ef7679876f,a74f7863361e7b94] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:32:00.864 WARN 2981023 --- [scanner-13685408-1,380063ef7679876f,501a5f34bce99f21] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:32:01.045 INFO 2981023 --- [alarm-detect-4,380063ef7679876f,1c7e9800024704fb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050690 ~ 1726050710 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050705, value=2.912)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.912, values={1726050705=2.912}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:32:01.485 INFO 2981023 --- [ocp-async-2,f66c7bffaabe7ba3,aa8589e754f624a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:01.625 INFO 2981023 --- [ocp-schedules-14,4f71a554151c3032,df134ec547f84e07] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:32:01.625+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:32:01.630 INFO 2981023 --- [ocp-schedules-14,4f71a554151c3032,df134ec547f84e07] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:32:01.776 INFO 2981023 --- [ocp-schedules-14,4f71a554151c3032,df134ec547f84e07] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 49 to SqlAggregateData 2024-09-11 18:32:01.778 INFO 2981023 --- [ocp-schedules-14,4f71a554151c3032,df134ec547f84e07] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:32:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,83718d7c57e07053,8a33a3f390b5ab0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,83718d7c57e07053,8a33a3f390b5ab0a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:02.921 INFO 2981023 --- [ocp-schedules-4,bf168fcac096d3be,772e0033b9677b2c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:32:03.347 INFO 2981023 --- [ocp-schedules-21,2af6298982279d5e,efa92c5ce256259e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:32:03.349 INFO 2981023 --- [ocp-schedules-21,2af6298982279d5e,efa92c5ce256259e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:03.369 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:03.370 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:32:03.374 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:03.376 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:32:03.377 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050723377000} 2024-09-11 18:32:03.382 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:32:03.386 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000240, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:32:03.386 INFO 2981023 --- [ocp-schedules-21,2af6298982279d5e,efa92c5ce256259e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:32:03.431 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:03.431 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:03.485 INFO 2981023 --- [ocp-async-7,dfbb5ae3eb3c0973,f4a1426dc860e06f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:03.984 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:32:03.984 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:32:03.984 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:32:03.988 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000240, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:32:03.990 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:32:04.474 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000240 2024-09-11 18:32:04.590 INFO 2981023 --- [ocp-schedules-8,43d549ae14c0f078,637170e04db740ce] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:32:04.591 INFO 2981023 --- [ocp-schedules-8,43d549ae14c0f078,637170e04db740ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:04.621 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:32:04.622 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.624 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:32:04.624 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.625 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:32:04.626 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:32:04.627 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:32:04.628 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.629 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.630 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.631 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.631 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.634 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:32:04.635 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.635 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.640 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:32:04.643 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.643 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.644 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:32:04.656 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.656 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:32:04.658 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.659 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.659 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.659 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.664 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:32:04.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:32:04.670 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.671 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:32:04.671 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.671 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:32:04.672 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:32:04.679 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.680 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:32:04.680 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.681 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:32:04.681 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.682 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:32:04.682 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:32:04.683 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:32:04.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:32:04.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.685 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:32:04.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:32:04.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.687 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:32:04.687 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.688 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:32:04.688 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:32:04.689 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:32:04.692 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.697 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:04.697 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:32:04.697 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6E91D6DE-0-0] 2024-09-11 18:32:04.698 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:32:04.698 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.698 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:32:04.699 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.699 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.700 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:04.701 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.701 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:32:04.706 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:32:04.708 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:32:04.708 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.709 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:32:04.713 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.713 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:32:04.713 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:32:04.714 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:32:04.714 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:32:04.714 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:32:04.714 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:32:04.715 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:32:04.715 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.715 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.719 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:32:04.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:32:04.721 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:32:04.721 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:32:04.722 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:32:04.723 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:32:04.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6E91D6E8-0-0] 2024-09-11 18:32:04.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:32:04.728 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:32:04.729 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.730 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:32:04.730 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.730 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:32:04.731 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.731 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:32:04.732 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.732 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:32:04.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.733 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:32:04.734 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:32:04.735 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:32:04.744 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:04.745 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.746 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:32:04.754 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:32:04.756 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:32:04.756 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:04.756 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:32:04.766 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:32:04.767 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:32:04.767 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:32:04.767 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:32:04.767 INFO 2981023 --- [ocp-schedules-8,43d549ae14c0f078,637170e04db740ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:32:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f0a39e20d04e7589,e997503118cb3955] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,f0a39e20d04e7589,e997503118cb3955] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:05.485 INFO 2981023 --- [ocp-async-10,50e7722dc6fd4e5a,d51d94f2e1d5a910] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:32:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:32:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:32:05.984 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:32:05.989 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:32:05.994 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:32:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:32:05.994 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:32:05.996 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - not enough cached interval (source=2024-09-11T18:29:41, target=2024-09-11T18:27:41, granule=120000ms, delay=120000ms) 2024-09-11 18:32:05.996 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:27:41, target=2024-09-11T18:20:51, granule=600000ms, delay=410000ms) 2024-09-11 18:32:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=26ms, period=120000ms 2024-09-11 18:32:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:32:06.337453+08:00, lastProcessTime=2024-09-11T18:31:56.353563+08:00 2024-09-11 18:32:06.431 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:06.433 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:06.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:32:06.337453+08:00 2024-09-11 18:32:07.485 INFO 2981023 --- [ocp-async-9,a5bd3ee1a4cde4e5,c1a0384b2ceec29e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:07.800 INFO 2981023 --- [ocp-async-7,d37eaee8056f21f7,635424abd2d888c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:32:08.298 INFO 2981023 --- [ocp-schedules-12,c203b7abf44f28e3,f279802b7ea0fafd] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:32:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,98b8598a203a07a3,622130248c85b00c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,98b8598a203a07a3,622130248c85b00c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:08.374 INFO 2981023 --- [ocp-schedules-8,c74e3aeacdd5b6cc,e5787d02bf257186] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:32:08.374+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:32:08.375 INFO 2981023 --- [ocp-schedules-8,c74e3aeacdd5b6cc,e5787d02bf257186] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:32:08.376 INFO 2981023 --- [ocp-schedules-8,c74e3aeacdd5b6cc,e5787d02bf257186] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:32:08.376 INFO 2981023 --- [ocp-schedules-8,c74e3aeacdd5b6cc,e5787d02bf257186] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:32:09.433 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:09.434 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:09.485 INFO 2981023 --- [ocp-async-4,1294fe693e5d0f4b,edfd9995e613c059] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:32:10.003 INFO 2981023 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:32:10.574 INFO 2981023 --- [ocp-schedules-21,074e09869192f4ab,2b6240eba30c909b] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-09-11 18:32:10.574 INFO 2981023 --- [ocp-schedules-21,074e09869192f4ab,2b6240eba30c909b] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-09-11 18:32:10.574 INFO 2981023 --- [ocp-schedules-21,074e09869192f4ab,2b6240eba30c909b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=10273 2024-09-11 18:32:10.792 INFO 2981023 --- [ocp-schedules-21,074e09869192f4ab,2b6240eba30c909b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=10273 2024-09-11 18:32:10.913 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:32:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:32:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,129e6d7d5384e4a1,db978703d29ca39d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:11.318 INFO 2981023 --- [prometheus-minute-schedule-1,129e6d7d5384e4a1,db978703d29ca39d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:11.485 INFO 2981023 --- [ocp-async-5,a50046c22aa020b3,5820e13a067d14e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:11.511 INFO 2981023 --- [ocp-schedules-19,6b67797af63b8a3b,bf366f41b75f34a9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:32:11.560 WARN 2981023 --- [scanner-13686198-1,6b67797af63b8a3b,c50ff58640438be1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:32:11.599 INFO 2981023 --- [alarm-detect-13,6b67797af63b8a3b,e7dc0c95290deba6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646597, data is empty. 2024-09-11 18:32:11.605 INFO 2981023 --- [alarm-detect-13,6b67797af63b8a3b,e7dc0c95290deba6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646646, data is empty. 2024-09-11 18:32:11.608 INFO 2981023 --- [alarm-detect-13,6b67797af63b8a3b,e7dc0c95290deba6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646646, data is empty. 2024-09-11 18:32:11.623 INFO 2981023 --- [alarm-detect-9,6b67797af63b8a3b,f13a529446bccee1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646597, data is empty. 2024-09-11 18:32:11.624 INFO 2981023 --- [alarm-detect-13,6b67797af63b8a3b,13f1442105c576fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646597, data is empty. 2024-09-11 18:32:11.627 INFO 2981023 --- [alarm-detect-9,6b67797af63b8a3b,f13a529446bccee1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646646, data is empty. 2024-09-11 18:32:11.631 INFO 2981023 --- [alarm-detect-13,6b67797af63b8a3b,13f1442105c576fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646646, data is empty. 2024-09-11 18:32:11.639 INFO 2981023 --- [alarm-detect-13,6b67797af63b8a3b,13f1442105c576fb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646646, data is empty. 2024-09-11 18:32:11.640 INFO 2981023 --- [alarm-detect-9,6b67797af63b8a3b,f13a529446bccee1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646646, data is empty. 2024-09-11 18:32:11.758 INFO 2981023 --- [alarm-detect-1,6b67797af63b8a3b,893fc60d3563aa54] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646597, data is empty. 2024-09-11 18:32:11.761 INFO 2981023 --- [alarm-detect-1,6b67797af63b8a3b,893fc60d3563aa54] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646646, data is empty. 2024-09-11 18:32:11.767 INFO 2981023 --- [alarm-detect-1,6b67797af63b8a3b,893fc60d3563aa54] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646646, data is empty. 2024-09-11 18:32:12.336 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050660, nodeCount=1000, affectRows=1000 2024-09-11 18:32:12.338 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050660, nodeCount=999, validCount=999 2024-09-11 18:32:12.339 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050660, nodeCount=1000, affectRows=1000 2024-09-11 18:32:12.341 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050660, nodeCount=1000, validCount=1000 2024-09-11 18:32:12.344 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050660, nodeCount=48, affectRows=48 2024-09-11 18:32:12.344 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050660, nodeCount=46, validCount=46 2024-09-11 18:32:12.434 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:12.434 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:12.924 INFO 2981023 --- [ocp-schedules-20,e5ff91c4fd44400a,17f6b4ee8cc13d0f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:32:13.485 INFO 2981023 --- [ocp-async-8,d9a406687a49f0c7,0e5f55eac5424478] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:13.704 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 18:32:13.714 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:13.717 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:13.769 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:32:13.770 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:32:13.781 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:13.782 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:13.794 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:32:13.795 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:32:13.806 INFO 2981023 --- [ocp-schedules-21,7fd5de57a8f8a695,534856292d10ec62] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2024-09-11 18:32:13.908 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050660, nodeCount=480, affectRows=480 2024-09-11 18:32:14.118 INFO 2981023 --- [ocp-schedules-4,063301526e1b30b5,51f021afd60b7932] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:32:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,7ec97a0b125c6701,59432cef109980e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,69e696f6907eb455,90abf26657598120] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:32:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,7ec97a0b125c6701,59432cef109980e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:15.434 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:15.436 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:15.485 INFO 2981023 --- [ocp-async-2,9330aa209c3e10cf,0a236dccdfa087e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:15.663 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050660, nodeCount=480, validCount=480 2024-09-11 18:32:15.998 INFO 2981023 --- [ocp-schedules-29,18cbc04c89c64685,f04eecc559cfa340] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:32:16.020 INFO 2981023 --- [ocp-schedules-4,fff77eaf2c7698c7,7d8118c69b6e14a9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:32:16.020 INFO 2981023 --- [ocp-schedules-7,91a8e0b4666bc872,b85528f9c09efd1f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:32:16.020 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:32:16.020 INFO 2981023 --- [host-check-clockdiff1,fff77eaf2c7698c7,3a8ffedc3bd20dfa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:32:16.021 INFO 2981023 --- [host-check-clockdiff2,fff77eaf2c7698c7,4e13186ce80f3b6e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:32:16.021 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:32:16.032 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:32:16.038 INFO 2981023 --- [ocp-schedules-27,d964a5687cd9bfbe,69c3e9a06660dda8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:32:16.041 INFO 2981023 --- [host-check-clockdiff3,fff77eaf2c7698c7,95ee1ee721a5cd44] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:32:16.044 INFO 2981023 --- [ocp-schedules-29,18cbc04c89c64685,f04eecc559cfa340] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:32:16.044 INFO 2981023 --- [ocp-schedules-29,18cbc04c89c64685,f04eecc559cfa340] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:32:16.044 INFO 2981023 --- [host-check-clockdiff4,fff77eaf2c7698c7,a800b604958f56f6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:32:16.045 INFO 2981023 --- [host-check-agent-status1,d964a5687cd9bfbe,792013692fd6da54] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:32:16.045 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:32:16.050 INFO 2981023 --- [host-check-agent-status2,d964a5687cd9bfbe,466a12ff4a686e82] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:32:16.051 INFO 2981023 --- [host-check-agent-status3,d964a5687cd9bfbe,dae62eb93314f7ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:32:16.051 INFO 2981023 --- [host-check-agent-status4,d964a5687cd9bfbe,7ca5f160703539aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:32:16.059 INFO 2981023 --- [host-check-clockdiff3,fff77eaf2c7698c7,95ee1ee721a5cd44] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050736 0 0 , error= 2024-09-11 18:32:16.061 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:16.063 INFO 2981023 --- [host-check-agent-status2,d964a5687cd9bfbe,466a12ff4a686e82] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:32:16.063 INFO 2981023 --- [host-check-agent-status4,d964a5687cd9bfbe,7ca5f160703539aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:32:16.066 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:32:16.066 INFO 2981023 --- [host-check-agent-status4,d964a5687cd9bfbe,7ca5f160703539aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:32:16.067 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:16.067 INFO 2981023 --- [host-check-agent-status2,d964a5687cd9bfbe,466a12ff4a686e82] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:32:16.071 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:32:16.075 INFO 2981023 --- [host-check-agent-status1,d964a5687cd9bfbe,792013692fd6da54] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:16.077 INFO 2981023 --- [host-check-agent-status1,d964a5687cd9bfbe,792013692fd6da54] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:16.081 INFO 2981023 --- [host-check-agent-status3,d964a5687cd9bfbe,dae62eb93314f7ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:16.082 INFO 2981023 --- [host-check-agent-status3,d964a5687cd9bfbe,dae62eb93314f7ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:16.097 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:16.098 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:16.106 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:32:16.106 INFO 2981023 --- [host-check-agent-status4,d964a5687cd9bfbe,7ca5f160703539aa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:32:16.107 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:32:16.108 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:32:16.108341301+08:00 2024-09-11 18:32:16.108 INFO 2981023 --- [host-check-agent-status2,d964a5687cd9bfbe,466a12ff4a686e82] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:32:16.108 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:32:16.108+08:00 2024-09-11 18:32:16.109 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:32:16.114479393+08:00 2024-09-11 18:32:16.110 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:32:16.109+08:00 2024-09-11 18:32:16.110 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:32:16.108+08:00 and status AVAILABLE 2024-09-11 18:32:16.113 INFO 2981023 --- [host-check-agent-status2,d964a5687cd9bfbe,466a12ff4a686e82] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:32:16.113 INFO 2981023 --- [host-check-agent-status2,d964a5687cd9bfbe,466a12ff4a686e82] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:32:16.113 INFO 2981023 --- [host-check-agent-status2,d964a5687cd9bfbe,466a12ff4a686e82] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:32:16.113 INFO 2981023 --- [host-check-agent-status1,d964a5687cd9bfbe,792013692fd6da54] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:32:16.115 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:32:16.109+08:00 and status AVAILABLE 2024-09-11 18:32:16.116 INFO 2981023 --- [host-check-agent-status1,d964a5687cd9bfbe,792013692fd6da54] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:32:16.116 INFO 2981023 --- [host-check-agent-status1,d964a5687cd9bfbe,792013692fd6da54] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:32:16.116 INFO 2981023 --- [host-check-agent-status1,d964a5687cd9bfbe,792013692fd6da54] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:32:16.118 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:32:16.120 INFO 2981023 --- [host-check-agent-status4,d964a5687cd9bfbe,7ca5f160703539aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:32:16.121 INFO 2981023 --- [host-check-agent-status4,d964a5687cd9bfbe,7ca5f160703539aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:32:16.121 INFO 2981023 --- [host-check-agent-status4,d964a5687cd9bfbe,7ca5f160703539aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:32:16.127 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:32:16.129 INFO 2981023 --- [host-check-agent-status3,d964a5687cd9bfbe,dae62eb93314f7ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:32:16.133 INFO 2981023 --- [host-check-agent-status3,d964a5687cd9bfbe,dae62eb93314f7ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:32:16.133 INFO 2981023 --- [host-check-agent-status3,d964a5687cd9bfbe,dae62eb93314f7ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:32:16.133 INFO 2981023 --- [host-check-agent-status3,d964a5687cd9bfbe,dae62eb93314f7ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:32:16.140 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:32:16.140 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:32:16.187 INFO 2981023 --- [ocp-schedules-21,d1897a25c73e6fb3,e420e43e1dd75479] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:32:16.188 INFO 2981023 --- [ocp-schedules-9,be46a02752de4058,50fdf95b34f5c97a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:32:16.193 INFO 2981023 --- [ocp-schedules-15,7559d9e2fbd93adf,017708125c567bd8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:32:16.193 INFO 2981023 --- [ocp-schedules-5,20bce79a6df99bca,c86c58239adb7f53] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:32:16.195 INFO 2981023 --- [ocp-schedules-9,be46a02752de4058,50fdf95b34f5c97a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:16.195 INFO 2981023 --- [ocp-schedules-21,d1897a25c73e6fb3,e420e43e1dd75479] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:16.197 INFO 2981023 --- [ocp-schedules-15,7559d9e2fbd93adf,017708125c567bd8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:16.199 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:32:16.199 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:32:16.199575867+08:00 2024-09-11 18:32:16.200 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:32:16.199+08:00 2024-09-11 18:32:16.201 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:32:16.199+08:00 and status AVAILABLE 2024-09-11 18:32:16.203 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:32:16.205 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:32:16.206 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:16.207 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:16.212 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:32:16.216 INFO 2981023 --- [ocp-schedules-8,0cadc26bcaaa5590,d0bf172869435919] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:32:16.225 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:16.226 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:16.228 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:32:16.228 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:32:16.230 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:32:16.229317620+08:00 2024-09-11 18:32:16.231 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:32:16.230+08:00 2024-09-11 18:32:16.238 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:32:16.230+08:00 and status AVAILABLE 2024-09-11 18:32:16.253 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:32:16.254 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050723377000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:32:16.256 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:32:16.268 INFO 2981023 --- [ocp-schedules-9,be46a02752de4058,50fdf95b34f5c97a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:32:16.275 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:32:16.286 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.289 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:32:16.291 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:32:16.291 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:32:16.295 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.296 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:32:16.296 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:32:16.296 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:32:16.296 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:32:16.297 INFO 2981023 --- [ocp-schedules-21,d1897a25c73e6fb3,e420e43e1dd75479] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:32:16.313 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:32:16.328 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.328 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:16.342 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.345 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.346 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:16.346 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:32:16.347 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:32:16.347 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:32:16.348 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:16.354 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:32:16.356 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:32:16.364 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:32:16.363531+08:00, lastProcessTime=2024-09-11T18:32:06.337453+08:00 2024-09-11 18:32:16.367 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:32:16.367 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:32:16.374 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:32:16.384 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:32:16.385 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:32:16.385 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:16.386 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.386 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:32:16.401 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:32:16.401 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050723377000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:32:16.405 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:32:16.414 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:32:16.426 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:32:16.426 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:16.427 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.427 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:16.428 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:32:16.428 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.429 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:32:16.431 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:32:16.431 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:32:16.434 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:32:16.454 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:32:16.454 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:32:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:32:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:32:16.465 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:32:16.471 INFO 2981023 --- [host-check-host-status1,91a8e0b4666bc872,eb1dfab35c7c58df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:32:16.476 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:32:16.478 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:32:16.478 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:32:16.481 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:32:16.490 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:32:16.490 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:16.491 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.492 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:32:16.493 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:32:16.497 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:32:16.498 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:16.499 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.500 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:32:16.501 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:16.502 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:16.505 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:32:16.509 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:32:16.513 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:32:16.519 INFO 2981023 --- [host-check-host-status2,91a8e0b4666bc872,a0f468e2f452810c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:32:16.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:32:16.526 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:16.527 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.527 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:16.528 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:32:16.528 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.530 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:32:16.531 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:32:16.534 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:32:16.538 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:32:16.538 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:32:16.543 INFO 2981023 --- [host-check-host-status4,91a8e0b4666bc872,861d44ef46ea088f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:32:16.548 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:32:16.552 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:32:16.558 INFO 2981023 --- [host-check-host-status3,91a8e0b4666bc872,9229bc68ef02080d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:32:16.561 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:16.562 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.563 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:32:16.563 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:32:16.572 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:32:16.575 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:16.577 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:16.594 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:32:16.602 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:32:16.608 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:16.608 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.609 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:32:16.615 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:32:16.615 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:32:16.624 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:32:16.631 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:16.631 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:16.632 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:32:16.639 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:32:16.639 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:32:16.639 INFO 2981023 --- [ocp-schedules-15,7559d9e2fbd93adf,017708125c567bd8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:32:16.645 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:32:16.363531+08:00 2024-09-11 18:32:16.967 INFO 2981023 --- [host-check-clockdiff1,fff77eaf2c7698c7,3a8ffedc3bd20dfa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050736 0 0 , error= 2024-09-11 18:32:16.968 INFO 2981023 --- [host-check-clockdiff2,fff77eaf2c7698c7,4e13186ce80f3b6e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050736 5 5 , error= 2024-09-11 18:32:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,9e0a2764642544cb,dacff332897cc14d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,9e0a2764642544cb,dacff332897cc14d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:17.486 INFO 2981023 --- [ocp-async-7,7efbc8c10c4c51e1,3711057a8a3274f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:17.800 INFO 2981023 --- [ocp-async-8,149be166279e1cd9,4844f906d30430bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:32:18.436 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:18.437 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:19.486 INFO 2981023 --- [ocp-async-9,fa65150db61cce06,7de8de39a062fadb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,db97391172552b5c,7a8b290fce0605db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,db97391172552b5c,7a8b290fce0605db] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:21.437 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:21.439 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:21.460 INFO 2981023 --- [host-check-clockdiff4,fff77eaf2c7698c7,a800b604958f56f6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050741 0 0 , error= 2024-09-11 18:32:21.486 INFO 2981023 --- [ocp-async-1,21f39a97ad1acf84,b6490f4095e70894] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:21.845 INFO 2981023 --- [ocp-schedules-14,8ffc22f9ce580440,14ee7bfe05b54099] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:32:21.888 WARN 2981023 --- [scanner-13687449-1,8ffc22f9ce580440,309181b3c84cc655] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:32:21.946 INFO 2981023 --- [alarm-detect-9,8ffc22f9ce580440,f9e57f5b23bdd9f0] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050705 ~ 1726050731 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050720, value=2.607)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.607, values={1726050720=2.607}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:32:22.926 INFO 2981023 --- [ocp-schedules-26,13bc08413683f344,4a6e44da759a9fc2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:32:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,2d898b8e204e7c72,1a9ce6618cc18b57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,2d898b8e204e7c72,1a9ce6618cc18b57] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:23.486 INFO 2981023 --- [ocp-async-10,2420660af17ab2be,1c5d3c62a4be1eab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:24.439 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:24.439 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:24.538 INFO 2981023 --- [ocp-schedules-20,8a244cf3f9bf426d,090de6d6d43c19f3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:32:25.486 INFO 2981023 --- [ocp-async-3,a5e8c5f007e82f30,c8a40d88ebb87a9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,6ac437bf257bf671,ff4e71a7c75c5ffa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,6ac437bf257bf671,ff4e71a7c75c5ffa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:32:26.353262+08:00, lastProcessTime=2024-09-11T18:32:16.363531+08:00 2024-09-11 18:32:26.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:32:26.353262+08:00 2024-09-11 18:32:27.440 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:27.441 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:27.486 INFO 2981023 --- [ocp-async-6,3a096758a327cfd3,829af9cff8feaf90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:27.800 INFO 2981023 --- [ocp-async-10,f58a949beef08356,6a9c18d30c783f1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:32:28.593 INFO 2981023 --- [ocp-schedules-7,8d20cde7e2dc4bc4,b21186b8d418d691] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:32:28.593 INFO 2981023 --- [ocp-schedules-7,8d20cde7e2dc4bc4,b21186b8d418d691] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:32:28.593 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:32:28.593 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:32:28.593 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:32:28.618 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:32:28.622 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:32:28.626 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:28.627 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:32:28.627 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:28.628 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:32:28.628 INFO 2981023 --- [ocp-schedules-7,8d20cde7e2dc4bc4,b21186b8d418d691] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:28.633 INFO 2981023 --- [ocp-schedules-7,8d20cde7e2dc4bc4,b21186b8d418d691] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:28.775 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:32:28.776 INFO 2981023 --- [ocp-schedules-7,8d20cde7e2dc4bc4,b21186b8d418d691] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:32:28.798 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:32:28.798 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:32:28.801 INFO 2981023 --- [ocp-schedules-7,8d20cde7e2dc4bc4,b21186b8d418d691] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:32:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,1145832f69a4ce08,25302f34001529b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,1145832f69a4ce08,25302f34001529b3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:29.486 INFO 2981023 --- [ocp-async-7,bc89f2fdac7f2583,540ceee1dc29c3d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:30.441 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:30.443 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:31.486 INFO 2981023 --- [ocp-async-8,a372b2c400c6bd0b,535f7c402fc2aa13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:32.030 INFO 2981023 --- [ocp-schedules-6,3e7cdb77e5ef6246,39682eb4cf0be6be] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:32:32.081 WARN 2981023 --- [scanner-13687647-1,3e7cdb77e5ef6246,6bedc96c7599074d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:32:32.096 INFO 2981023 --- [alarm-detect-7,3e7cdb77e5ef6246,d52526d4ec339320] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646646, data is empty. 2024-09-11 18:32:32.103 INFO 2981023 --- [alarm-detect-7,3e7cdb77e5ef6246,d52526d4ec339320] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646646, data is empty. 2024-09-11 18:32:32.106 INFO 2981023 --- [alarm-detect-7,3e7cdb77e5ef6246,d52526d4ec339320] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646646, data is empty. 2024-09-11 18:32:32.124 INFO 2981023 --- [alarm-detect-11,3e7cdb77e5ef6246,3e59762c16afec5a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646646, data is empty. 2024-09-11 18:32:32.127 INFO 2981023 --- [alarm-detect-14,3e7cdb77e5ef6246,45a4675eeb30ec1b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646646, data is empty. 2024-09-11 18:32:32.128 INFO 2981023 --- [alarm-detect-11,3e7cdb77e5ef6246,3e59762c16afec5a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646646, data is empty. 2024-09-11 18:32:32.136 INFO 2981023 --- [alarm-detect-14,3e7cdb77e5ef6246,45a4675eeb30ec1b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646646, data is empty. 2024-09-11 18:32:32.139 INFO 2981023 --- [alarm-detect-11,3e7cdb77e5ef6246,3e59762c16afec5a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646646, data is empty. 2024-09-11 18:32:32.144 INFO 2981023 --- [alarm-detect-14,3e7cdb77e5ef6246,45a4675eeb30ec1b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646646, data is empty. 2024-09-11 18:32:32.201 INFO 2981023 --- [alarm-detect-2,3e7cdb77e5ef6246,4f9145acc1b8510c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050720 ~ 1726050742 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050735, value=2.04)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.04, values={1726050735=2.04}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:32:32.239 INFO 2981023 --- [alarm-detect-8,3e7cdb77e5ef6246,97760d6be965abd7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646646, data is empty. 2024-09-11 18:32:32.242 INFO 2981023 --- [alarm-detect-8,3e7cdb77e5ef6246,97760d6be965abd7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646646, data is empty. 2024-09-11 18:32:32.246 INFO 2981023 --- [alarm-detect-8,3e7cdb77e5ef6246,97760d6be965abd7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646646, data is empty. 2024-09-11 18:32:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,6518a51acabfde46,1b4ff8f1c2d25ab1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,6518a51acabfde46,1b4ff8f1c2d25ab1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:32.929 INFO 2981023 --- [ocp-schedules-30,42ac230e77ba82ad,c0d085a7f55a5600] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:32:33.443 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:33.444 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:33.486 INFO 2981023 --- [ocp-async-2,a8ee1b970a372ee3,aaea8e84cdf9380b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,7a05fc894fdab8b1,cf08e45199c8bb0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:35.318 INFO 2981023 --- [prometheus-minute-schedule-1,7a05fc894fdab8b1,cf08e45199c8bb0e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:35.486 INFO 2981023 --- [ocp-async-4,d8e319a4e13c4b77,c1de349f5e3ba2a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:36.047 INFO 2981023 --- [ocp-schedules-11,069ef592479dd4f4,5ff85086f47bbcfb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:32:36.055 INFO 2981023 --- [ocp-schedules-11,069ef592479dd4f4,5ff85086f47bbcfb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:32:36.055 INFO 2981023 --- [ocp-schedules-11,069ef592479dd4f4,5ff85086f47bbcfb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:32:36.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:32:36.351253+08:00, lastProcessTime=2024-09-11T18:32:26.353262+08:00 2024-09-11 18:32:36.444 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:36.444 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:36.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:32:36.351253+08:00 2024-09-11 18:32:37.314 INFO 2981023 --- [prometheus-cache-monitor1,82a14d8c8b254919,3f74d1a07d24670e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:32:37.315 INFO 2981023 --- [prometheus-cache-monitor1,82a14d8c8b254919,3f74d1a07d24670e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:32:37.486 INFO 2981023 --- [ocp-async-7,95bf574e7a743f79,b5c2d65bbe499fdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:37.800 INFO 2981023 --- [ocp-async-2,2f5c76b04988ae2f,7170e84c1dd1056b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:32:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,ff3d6b80d4e37945,9e3ed74d8f0148da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,ff3d6b80d4e37945,9e3ed74d8f0148da] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:39.444 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:39.445 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:39.487 INFO 2981023 --- [ocp-async-9,98c516f2bd911193,7974871cf34e7a7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:40.003 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:32:40.003 INFO 2981023 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:32:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,2db61005a269a39a,27530bd98a348d7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,2db61005a269a39a,27530bd98a348d7a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:41.487 INFO 2981023 --- [ocp-async-1,ea2ac2c781e2c5fc,9c8a9adfc91986c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:42.291 INFO 2981023 --- [ocp-schedules-2,0324df78da4fa52e,63699d809779c80b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:32:42.343 WARN 2981023 --- [scanner-13688006-1,0324df78da4fa52e,fe9726095eab60b9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:32:42.445 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:42.445 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:42.931 INFO 2981023 --- [ocp-schedules-12,06bf5f77941605ee,ba1c7e2bfd780886] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:32:43.487 INFO 2981023 --- [ocp-async-5,28d65fbb1e7f4e67,a67f4bc98d53f844] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,87ac230ee5289fcc,806cb41074bea58f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,87ac230ee5289fcc,806cb41074bea58f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:45.101 INFO 2981023 --- [prometheus-second-schedule-1,0a6f91fd0a1aaeb4,9c5a684a3f073219] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:32:45.445 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:45.446 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:45.487 INFO 2981023 --- [ocp-async-8,1d6ff8c0e0c54238,c018ccf52ee1aa71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:46.022 INFO 2981023 --- [ocp-schedules-27,7a35c76ccc00a326,227bc0172a26529f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:32:46.022 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:32:46.022 INFO 2981023 --- [ocp-schedules-7,186a2549818f67ab,926ec99064132b81] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:32:46.023 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:32:46.023 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:32:46.024 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:32:46.024 INFO 2981023 --- [host-check-clockdiff1,186a2549818f67ab,65492a29b30edda1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:32:46.024 INFO 2981023 --- [host-check-clockdiff2,186a2549818f67ab,48d87c8d73a097fc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:32:46.027 INFO 2981023 --- [host-check-clockdiff3,186a2549818f67ab,810c514e5e4b5694] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:32:46.034 INFO 2981023 --- [host-check-clockdiff4,186a2549818f67ab,a9e03942b571aaad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:32:46.051 INFO 2981023 --- [ocp-schedules-6,f505a6013c562012,f3db9a9ff66f0bb0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:32:46.051 INFO 2981023 --- [host-check-agent-status1,f505a6013c562012,caed46c63ad6af63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:32:46.055 INFO 2981023 --- [host-check-agent-status2,f505a6013c562012,25b1afe787eb687d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:32:46.057 INFO 2981023 --- [host-check-agent-status3,f505a6013c562012,67e8accaac4b3d55] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:32:46.057 INFO 2981023 --- [host-check-agent-status4,f505a6013c562012,20c59ba2e81d0556] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:32:46.061 INFO 2981023 --- [host-check-clockdiff3,186a2549818f67ab,810c514e5e4b5694] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050766 0 0 , error= 2024-09-11 18:32:46.062 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:32:46.065 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:46.066 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:46.067 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:32:46.069 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:46.070 INFO 2981023 --- [host-check-agent-status3,f505a6013c562012,67e8accaac4b3d55] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:46.070 INFO 2981023 --- [host-check-agent-status4,f505a6013c562012,20c59ba2e81d0556] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:32:46.071 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:46.072 INFO 2981023 --- [host-check-agent-status3,f505a6013c562012,67e8accaac4b3d55] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:46.074 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:32:46.075 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:32:46.076 INFO 2981023 --- [host-check-agent-status1,f505a6013c562012,caed46c63ad6af63] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:46.077 INFO 2981023 --- [host-check-agent-status1,f505a6013c562012,caed46c63ad6af63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:46.082 INFO 2981023 --- [host-check-agent-status4,f505a6013c562012,20c59ba2e81d0556] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:32:46.085 INFO 2981023 --- [host-check-agent-status2,f505a6013c562012,25b1afe787eb687d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:32:46.170 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:32:46.183 INFO 2981023 --- [host-check-agent-status2,f505a6013c562012,25b1afe787eb687d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:32:46.188 INFO 2981023 --- [ocp-schedules-3,bad581540b504f1e,e09eebb7af19c8c9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:32:46.188 INFO 2981023 --- [ocp-schedules-15,93092d7f55c7880c,b0e76ef992b097e8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:32:46.188 INFO 2981023 --- [ocp-schedules-11,5d0a23e8204153dd,29b5f31bd7b187cb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:32:46.198 INFO 2981023 --- [ocp-schedules-16,6d4e0132310b1959,d93966ed65d15ec4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:32:46.266 INFO 2981023 --- [ocp-schedules-18,3a0c02c41b56dbd7,f14a5122943c9858] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:32:46.267 INFO 2981023 --- [ocp-schedules-3,bad581540b504f1e,e09eebb7af19c8c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:46.275 INFO 2981023 --- [ocp-schedules-11,5d0a23e8204153dd,29b5f31bd7b187cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:46.275 INFO 2981023 --- [ocp-schedules-15,93092d7f55c7880c,b0e76ef992b097e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:46.302 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050723377000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:32:46.305 INFO 2981023 --- [ocp-schedules-18,3a0c02c41b56dbd7,f14a5122943c9858] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:32:46.306 INFO 2981023 --- [ocp-schedules-15,93092d7f55c7880c,b0e76ef992b097e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:32:46.327 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:32:46.366 INFO 2981023 --- [ocp-schedules-17,ad831690714fbd2a,100b0e8f229be888] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:32:46.370 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:32:46.372 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:32:46.373 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:32:46.372877854+08:00 2024-09-11 18:32:46.373 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:32:46.373738653+08:00 2024-09-11 18:32:46.373 INFO 2981023 --- [host-check-agent-status1,f505a6013c562012,caed46c63ad6af63] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:32:46.374 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:32:46.373+08:00 2024-09-11 18:32:46.374 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:32:46.373+08:00 2024-09-11 18:32:46.377 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:32:46.373+08:00 and status AVAILABLE 2024-09-11 18:32:46.382 INFO 2981023 --- [host-check-agent-status1,f505a6013c562012,caed46c63ad6af63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:32:46.382 INFO 2981023 --- [host-check-agent-status1,f505a6013c562012,caed46c63ad6af63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:32:46.382 INFO 2981023 --- [host-check-agent-status1,f505a6013c562012,caed46c63ad6af63] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:32:46.384 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:32:46.373+08:00 and status AVAILABLE 2024-09-11 18:32:46.387 INFO 2981023 --- [host-check-agent-status3,f505a6013c562012,67e8accaac4b3d55] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:32:46.387 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:32:46.389 INFO 2981023 --- [host-check-agent-status4,f505a6013c562012,20c59ba2e81d0556] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:32:46.389 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:32:46.389261498+08:00 2024-09-11 18:32:46.390 INFO 2981023 --- [host-check-agent-status2,f505a6013c562012,25b1afe787eb687d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:32:46.391 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:32:46.391 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:32:46.389+08:00 2024-09-11 18:32:46.392 INFO 2981023 --- [host-check-agent-status3,f505a6013c562012,67e8accaac4b3d55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:32:46.392 INFO 2981023 --- [host-check-agent-status3,f505a6013c562012,67e8accaac4b3d55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:32:46.392 INFO 2981023 --- [host-check-agent-status3,f505a6013c562012,67e8accaac4b3d55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:32:46.393 INFO 2981023 --- [host-check-agent-status4,f505a6013c562012,20c59ba2e81d0556] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:32:46.393 INFO 2981023 --- [host-check-agent-status4,f505a6013c562012,20c59ba2e81d0556] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:32:46.393 INFO 2981023 --- [host-check-agent-status4,f505a6013c562012,20c59ba2e81d0556] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:32:46.396 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:32:46.391937+08:00, lastProcessTime=2024-09-11T18:32:36.351253+08:00 2024-09-11 18:32:46.396 INFO 2981023 --- [host-check-agent-status2,f505a6013c562012,25b1afe787eb687d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:32:46.396 INFO 2981023 --- [host-check-agent-status2,f505a6013c562012,25b1afe787eb687d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:32:46.396 INFO 2981023 --- [host-check-agent-status2,f505a6013c562012,25b1afe787eb687d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:32:46.398 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:32:46.389+08:00 and status AVAILABLE 2024-09-11 18:32:46.401 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.402 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:32:46.402 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:32:46.403 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:32:46.404 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:32:46.405 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:32:46.410568931+08:00 2024-09-11 18:32:46.405 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:32:46.405+08:00 2024-09-11 18:32:46.407 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:32:46.405+08:00 and status AVAILABLE 2024-09-11 18:32:46.412 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:32:46.413 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:32:46.466 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:46.467 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:46.469 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:32:46.469 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:46.470 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:32:46.470 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:32:46.470 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:46.472 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:32:46.473 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.474 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:32:46.474 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:32:46.475 INFO 2981023 --- [ocp-schedules-3,bad581540b504f1e,e09eebb7af19c8c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:32:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.492 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:46.493 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:32:46.494 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:32:46.495 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:32:46.497 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.498 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:46.498 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:32:46.502 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.504 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:46.506 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.507 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:32:46.508 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:32:46.509 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.515 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:32:46.520 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:32:46.523 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:32:46.527 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:32:46.531 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:32:46.539 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:32:46.542 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:32:46.556 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:32:46.557 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:32:46.581 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:32:46.585 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:32:46.585 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050723377000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:32:46.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.590 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.591 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:32:46.591 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:32:46.591 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:32:46.592 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:32:46.609 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:32:46.612 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.613 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:32:46.613 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.614 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:32:46.614 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:32:46.615 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.616 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:32:46.621 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:32:46.621 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:32:46.624 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:32:46.629 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:32:46.635 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:32:46.645 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:32:46.653 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.654 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.654 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:46.655 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:32:46.655 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.656 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:32:46.658 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:32:46.658 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:32:46.659 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:32:46.663 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:32:46.670 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:32:46.672 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:32:46.681 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:32:46.681 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:32:46.683 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:32:46.684 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:32:46.689 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.694 INFO 2981023 --- [host-check-host-status1,7a35c76ccc00a326,244a3960a2e14262] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:32:46.696 INFO 2981023 --- [host-check-host-status4,7a35c76ccc00a326,4d414484c2c279fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:32:46.701 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:32:46.701 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:32:46.702 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.703 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.704 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:32:46.705 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:32:46.705 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.706 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:32:46.706 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:32:46.706 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.711 INFO 2981023 --- [host-check-host-status2,7a35c76ccc00a326,15e99d0c375428ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:32:46.712 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:32:46.714 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:32:46.715 INFO 2981023 --- [host-check-host-status3,7a35c76ccc00a326,90770c46fd79c923] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:32:46.728 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.729 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.729 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:32:46.730 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:32:46.734 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.734 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:32:46.735 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:32:46.735 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.736 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:32:46.738 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:32:46.739 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:32:46.755 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.756 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.756 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:32:46.757 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:32:46.757 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.759 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.760 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.760 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:32:46.761 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:32:46.761 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:32:46.761 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.771 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:32:46.771 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:32:46.772 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:32:46.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:32:46.778 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:32:46.791 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:32:46.792 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:32:46.801 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.802 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.802 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:32:46.803 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:32:46.807 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:32:46.810 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.810 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:32:46.811 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.811 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:32:46.811 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:32:46.812 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:32:46.812 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.820 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:32:46.391937+08:00 2024-09-11 18:32:46.828 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:32:46.835 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:32:46.841 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.841 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.842 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:32:46.848 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:32:46.848 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:32:46.858 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:32:46.861 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:32:46.863 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:46.863 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:46.864 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:32:46.869 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:32:46.869 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:32:46.869 INFO 2981023 --- [ocp-schedules-11,5d0a23e8204153dd,29b5f31bd7b187cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:32:46.878 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:32:46.883 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:32:46.886 INFO 2981023 --- [ocp-schedules-2,48d8a1db23187316,c94fd7438f4d2d27] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:32:46.974 INFO 2981023 --- [host-check-clockdiff2,186a2549818f67ab,48d87c8d73a097fc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050766 5 5 , error= 2024-09-11 18:32:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,724ec85865551545,8c6b51624bd58c19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,724ec85865551545,8c6b51624bd58c19] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:47.487 INFO 2981023 --- [ocp-schedules-30,4505176071b432ba,79457eb04ad19765] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-09-11 18:32:47.487 INFO 2981023 --- [ocp-schedules-30,4505176071b432ba,79457eb04ad19765] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-09-11 18:32:47.487 INFO 2981023 --- [ocp-async-10,8790cc64f1952d0b,13d447f737664d21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:47.800 INFO 2981023 --- [ocp-async-8,33e0b1aa19100fed,9f924ab38560e2d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:32:48.446 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:48.447 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:49.055 INFO 2981023 --- [ocp-schedules-15,8e776968355ac7df,3817b82404c13fce] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:32:49.073 WARN 2981023 --- [ocp-schedules-15,8e776968355ac7df,3817b82404c13fce] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:32:49.487 INFO 2981023 --- [ocp-async-5,682ef94fac2b46a7,017653be1ba39560] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:49.745 INFO 2981023 --- [ocp-schedules-21,db9bcbf4fa712e53,0f0254f78ef361ca] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-09-11 18:32:49.747 INFO 2981023 --- [ocp-schedules-21,db9bcbf4fa712e53,0f0254f78ef361ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:49.780 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:49.784 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 18:00:06.461339, 2024-09-11 18:32:49.753, 1000] 2024-09-11 18:32:49.784 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000006, startTime 2024-09-11 18:00:06.461339, endTime 2024-09-11 18:32:49.753 2024-09-11 18:32:49.798 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:49.799 INFO 2981023 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:49.799 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:49.802 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 02:03:06.281634, 2024-09-11 18:32:49.784, 1000] 2024-09-11 18:32:49.802 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 10:40:18.658505, 2024-09-11 18:32:49.784, 1000] 2024-09-11 18:32:49.803 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-09-11 02:03:06.281634, endTime 2024-09-11 18:32:49.784 2024-09-11 18:32:49.803 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:49.804 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2024-09-11 10:40:18.658505, endTime 2024-09-11 18:32:49.784 2024-09-11 18:32:49.823 INFO 2981023 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 18:12:34.59165, 2024-09-11 18:32:49.789, 1000] 2024-09-11 18:32:49.824 INFO 2981023 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000006, startTime 2024-09-11 18:12:34.59165, endTime 2024-09-11 18:32:49.789 2024-09-11 18:32:49.837 INFO 2981023 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:49.837 INFO 2981023 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:32:49.837 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:49.837 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:49.867 INFO 2981023 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 18:01:30.332457, 2024-09-11 18:32:49.83, 1000] 2024-09-11 18:32:49.867 INFO 2981023 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2024-09-11 18:01:30.332457, endTime 2024-09-11 18:32:49.83 2024-09-11 18:32:49.868 INFO 2981023 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-09-11 08:00:10.321716, 2024-09-11 18:32:49.83, 1000] 2024-09-11 18:32:49.868 INFO 2981023 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-09-11 08:00:10.321716, endTime 2024-09-11 18:32:49.83 2024-09-11 18:32:49.868 INFO 2981023 --- [ocp-schedules-21,db9bcbf4fa712e53,0f0254f78ef361ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, tryCount=3, successCount=3 2024-09-11 18:32:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:32:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:32:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,6bfd16bc3066f3a8,d50290fc5b230519] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,6bfd16bc3066f3a8,d50290fc5b230519] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:50.377 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.382 INFO 2981023 --- [ocp-schedules-17,19c7b315d9032672,b262896647e5acd8] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:32:50.387 INFO 2981023 --- [ocp-schedules-19,c41c9c7c1de834c6,80c6bcbac89aac0a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:32:51.118 INFO 2981023 --- [host-check-clockdiff1,186a2549818f67ab,65492a29b30edda1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050771 0 0 , error= 2024-09-11 18:32:51.447 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:51.449 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:51.487 INFO 2981023 --- [ocp-async-6,dbd243bf8315a4f3,3d279192d2fddd95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:51.655 INFO 2981023 --- [ocp-schedules-13,820d5ad8c761ed31,9979125c56a342f4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:32:51.656 INFO 2981023 --- [ocp-schedules-13,820d5ad8c761ed31,9979125c56a342f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:32:51.657 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:32:51.657 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:32:51.657 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:32:51.657 INFO 2981023 --- [ocp-schedules-13,820d5ad8c761ed31,9979125c56a342f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:32:52.004 INFO 2981023 --- [host-check-clockdiff4,186a2549818f67ab,a9e03942b571aaad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050772 0 0 , error= 2024-09-11 18:32:52.482 INFO 2981023 --- [ocp-schedules-20,9e253b09c210004a,a634738e58c3d6e6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:32:52.559 WARN 2981023 --- [scanner-13688235-1,9e253b09c210004a,9ef5eb4656483304] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:32:52.560 INFO 2981023 --- [alarm-detect-16,9e253b09c210004a,60a1b717ec24e88c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646646, data is empty. 2024-09-11 18:32:52.563 INFO 2981023 --- [alarm-detect-16,9e253b09c210004a,60a1b717ec24e88c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646646, data is empty. 2024-09-11 18:32:52.566 INFO 2981023 --- [alarm-detect-16,9e253b09c210004a,60a1b717ec24e88c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646646, data is empty. 2024-09-11 18:32:52.590 INFO 2981023 --- [alarm-detect-4,9e253b09c210004a,834741ae8ea6f7e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646646, data is empty. 2024-09-11 18:32:52.593 INFO 2981023 --- [alarm-detect-2,9e253b09c210004a,82bcc35e5126e283] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646646, data is empty. 2024-09-11 18:32:52.595 INFO 2981023 --- [alarm-detect-4,9e253b09c210004a,834741ae8ea6f7e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646646, data is empty. 2024-09-11 18:32:52.600 INFO 2981023 --- [alarm-detect-2,9e253b09c210004a,82bcc35e5126e283] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646646, data is empty. 2024-09-11 18:32:52.601 INFO 2981023 --- [alarm-detect-4,9e253b09c210004a,834741ae8ea6f7e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646646, data is empty. 2024-09-11 18:32:52.604 INFO 2981023 --- [alarm-detect-2,9e253b09c210004a,82bcc35e5126e283] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646646, data is empty. 2024-09-11 18:32:52.634 INFO 2981023 --- [alarm-detect-2,9e253b09c210004a,e6d3d11f41ff3695] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050735 ~ 1726050762 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050750, value=1.661)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.661, values={1726050750=1.661}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:32:52.712 INFO 2981023 --- [alarm-detect-3,9e253b09c210004a,6843b853c219c7b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646646, data is empty. 2024-09-11 18:32:52.716 INFO 2981023 --- [alarm-detect-3,9e253b09c210004a,6843b853c219c7b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646646, data is empty. 2024-09-11 18:32:52.720 INFO 2981023 --- [alarm-detect-3,9e253b09c210004a,6843b853c219c7b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646646, data is empty. 2024-09-11 18:32:52.933 INFO 2981023 --- [ocp-schedules-26,749cd6e1a623aef2,1a31965718b90a49] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:32:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,b307193069d49087,d92c1d5c9515d2c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,b307193069d49087,d92c1d5c9515d2c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:53.418 INFO 2981023 --- [ocp-schedules-13,d089a4c6e4729279,4e020c9316f46abf] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:32:53.418 INFO 2981023 --- [ocp-schedules-13,d089a4c6e4729279,4e020c9316f46abf] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:32:53.488 INFO 2981023 --- [ocp-async-1,ccbe2ffc1c9f0c25,9f0e56ecc3965986] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:54.449 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:54.450 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:54.540 INFO 2981023 --- [ocp-schedules-20,3e9da63e22c69250,d5f68b84d8f0e528] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:32:55.488 INFO 2981023 --- [ocp-async-3,477f38b83da16734,b8b4a42fd5be1dc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:55.693 INFO 2981023 --- [ocp-schedules-3,7df0657d3b14a17d,58e3a586fd561062] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:32:55.712 WARN 2981023 --- [ocp-schedules-3,7df0657d3b14a17d,58e3a586fd561062] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:32:55.943 INFO 2981023 --- [ocp-schedules-10,c38e055cdef67ed5,1cfd2c2daeb87991] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:32:55.943 INFO 2981023 --- [ocp-schedules-10,c38e055cdef67ed5,1cfd2c2daeb87991] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:32:55.944 INFO 2981023 --- [ocp-schedules-10,c38e055cdef67ed5,1cfd2c2daeb87991] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:32:55.944 INFO 2981023 --- [ocp-schedules-10,c38e055cdef67ed5,1cfd2c2daeb87991] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:32:55.944 INFO 2981023 --- [ocp-schedules-10,c38e055cdef67ed5,1cfd2c2daeb87991] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:32:55.944 INFO 2981023 --- [ocp-schedules-10,c38e055cdef67ed5,1cfd2c2daeb87991] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:32:56.058 INFO 2981023 --- [ocp-schedules-14,a112649cb709acdd,611866b274e55b2f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:32:56.066 INFO 2981023 --- [ocp-schedules-14,a112649cb709acdd,611866b274e55b2f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:32:56.066 INFO 2981023 --- [ocp-schedules-14,a112649cb709acdd,611866b274e55b2f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:32:56.187 INFO 2981023 --- [ocp-schedules-12,57ed5033c35db787,668a84cc8990c06b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:32:56.193 INFO 2981023 --- [ocp-schedules-12,57ed5033c35db787,668a84cc8990c06b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:32:56.193 INFO 2981023 --- [ocp-schedules-12,57ed5033c35db787,668a84cc8990c06b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:32:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,de9105d835ead738,49177517e959ae18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,de9105d835ead738,49177517e959ae18] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:32:56.352886+08:00, lastProcessTime=2024-09-11T18:32:46.391937+08:00 2024-09-11 18:32:56.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:32:56.352886+08:00 2024-09-11 18:32:57.450 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:32:57.452 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:32:57.488 INFO 2981023 --- [ocp-async-2,35d7fc0af7d84224,299cbe092f5fd4f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:32:57.801 INFO 2981023 --- [ocp-async-10,ad8eaccdc5f2314c,19debc95eea43966] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:32:59.228 INFO 2981023 --- [ocp-schedules-2,9fc9eb004f69d4a5,1f744a04e6be1449] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:32:59.228 INFO 2981023 --- [ocp-schedules-2,9fc9eb004f69d4a5,1f744a04e6be1449] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:32:59.259 INFO 2981023 --- [Collect all tenants major compactions2,9fc9eb004f69d4a5,a89fe5c5ad1062bb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:32:59.259 INFO 2981023 --- [Collect all tenants major compactions3,9fc9eb004f69d4a5,28f8a0fc4b718897] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:32:59.262 INFO 2981023 --- [Collect all tenants major compactions6,9fc9eb004f69d4a5,2373c02f4829ced9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:32:59.265 INFO 2981023 --- [Collect all tenants major compactions4,9fc9eb004f69d4a5,51b37a22078bccdc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:32:59.267 INFO 2981023 --- [Collect all tenants major compactions5,9fc9eb004f69d4a5,cd39096570878aca] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:32:59.269 INFO 2981023 --- [Collect all tenants major compactions1,9fc9eb004f69d4a5,38f509d6f07395b3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:32:59.283 INFO 2981023 --- [Collect all tenants major compactions4,9fc9eb004f69d4a5,51b37a22078bccdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:59.284 INFO 2981023 --- [Collect all tenants major compactions5,9fc9eb004f69d4a5,cd39096570878aca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:59.284 INFO 2981023 --- [Collect all tenants major compactions1,9fc9eb004f69d4a5,38f509d6f07395b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:59.284 INFO 2981023 --- [Collect all tenants major compactions3,9fc9eb004f69d4a5,28f8a0fc4b718897] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:59.284 INFO 2981023 --- [Collect all tenants major compactions2,9fc9eb004f69d4a5,a89fe5c5ad1062bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:59.284 INFO 2981023 --- [Collect all tenants major compactions4,9fc9eb004f69d4a5,51b37a22078bccdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:32:59.285 INFO 2981023 --- [Collect all tenants major compactions1,9fc9eb004f69d4a5,38f509d6f07395b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:32:59.285 INFO 2981023 --- [Collect all tenants major compactions5,9fc9eb004f69d4a5,cd39096570878aca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:32:59.285 INFO 2981023 --- [Collect all tenants major compactions2,9fc9eb004f69d4a5,a89fe5c5ad1062bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:32:59.286 INFO 2981023 --- [Collect all tenants major compactions3,9fc9eb004f69d4a5,28f8a0fc4b718897] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:32:59.288 INFO 2981023 --- [Collect all tenants major compactions1,9fc9eb004f69d4a5,38f509d6f07395b3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:32:59.288 INFO 2981023 --- [Collect all tenants major compactions1,9fc9eb004f69d4a5,38f509d6f07395b3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:32:59.288 INFO 2981023 --- [Collect all tenants major compactions3,9fc9eb004f69d4a5,28f8a0fc4b718897] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:32:59.288 INFO 2981023 --- [Collect all tenants major compactions3,9fc9eb004f69d4a5,28f8a0fc4b718897] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:32:59.289 INFO 2981023 --- [Collect all tenants major compactions5,9fc9eb004f69d4a5,cd39096570878aca] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:32:59.289 INFO 2981023 --- [Collect all tenants major compactions5,9fc9eb004f69d4a5,cd39096570878aca] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:32:59.289 INFO 2981023 --- [Collect all tenants major compactions2,9fc9eb004f69d4a5,a89fe5c5ad1062bb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:32:59.289 INFO 2981023 --- [Collect all tenants major compactions2,9fc9eb004f69d4a5,a89fe5c5ad1062bb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:32:59.289 INFO 2981023 --- [Collect all tenants major compactions4,9fc9eb004f69d4a5,51b37a22078bccdc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:32:59.289 INFO 2981023 --- [Collect all tenants major compactions4,9fc9eb004f69d4a5,51b37a22078bccdc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:32:59.290 INFO 2981023 --- [Collect all tenants major compactions6,9fc9eb004f69d4a5,2373c02f4829ced9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:32:59.291 INFO 2981023 --- [Collect all tenants major compactions6,9fc9eb004f69d4a5,2373c02f4829ced9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:32:59.293 INFO 2981023 --- [Collect all tenants major compactions6,9fc9eb004f69d4a5,2373c02f4829ced9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:32:59.293 INFO 2981023 --- [Collect all tenants major compactions6,9fc9eb004f69d4a5,2373c02f4829ced9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:32:59.294 INFO 2981023 --- [ocp-schedules-2,9fc9eb004f69d4a5,1f744a04e6be1449] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:32:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,2e4c5c528fd9d50f,211b39de7667985f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050780, interval=60 2024-09-11 18:32:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,2e4c5c528fd9d50f,211b39de7667985f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:32:59.334 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.337 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=17199 2024-09-11 18:32:59.345 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.352 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.357 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.363 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.366 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.369 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.372 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.375 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.375 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.376 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.376 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.377 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.380 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.382 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.384 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.386 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.387 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.388 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.388 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.389 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.390 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.390 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.391 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.392 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:26:59.238, end=2024-09-11T18:31:59.33} 2024-09-11 18:32:59.392 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 18:32:59.488 INFO 2981023 --- [ocp-async-1,cf52ea99e1566a9f,e6723abddbad06d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:00.452 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:00.453 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:01.488 INFO 2981023 --- [ocp-async-3,dc5087b1a0ed64a4,23452b650a62a7a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:01.791 INFO 2981023 --- [ocp-schedules-13,d70b1fde02ae3a3d,c017b13f1aa3e62d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:33:01.791+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:33:01.799 INFO 2981023 --- [ocp-schedules-13,d70b1fde02ae3a3d,c017b13f1aa3e62d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:33:01.942 INFO 2981023 --- [ocp-schedules-13,d70b1fde02ae3a3d,c017b13f1aa3e62d] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 40 to SqlAggregateData 2024-09-11 18:33:01.944 INFO 2981023 --- [ocp-schedules-13,d70b1fde02ae3a3d,c017b13f1aa3e62d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:33:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,f2821b9b025d5332,80b094e845a2d2cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,f2821b9b025d5332,80b094e845a2d2cd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:02.771 INFO 2981023 --- [ocp-schedules-8,6662507eb4afda29,0cb14e8d8adc9a9d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:33:02.816 WARN 2981023 --- [scanner-13688521-1,6662507eb4afda29,dd73bbf309c5026d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:33:02.936 INFO 2981023 --- [ocp-schedules-30,c6133e6f7caca544,97b7992d02f4c242] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:33:03.453 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:03.454 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:03.488 INFO 2981023 --- [ocp-async-6,3bc12173e80561e8,2bd7791b3c3c8995] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:04.767 INFO 2981023 --- [ocp-schedules-10,aff060f5f2a95dcd,7468b2ce7910cc7a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:33:04.768 INFO 2981023 --- [ocp-schedules-10,aff060f5f2a95dcd,7468b2ce7910cc7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:33:04.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:33:04.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.800 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:33:04.801 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:33:04.801 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.803 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.803 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:33:04.804 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.807 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.807 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.808 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.808 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.813 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.814 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:33:04.818 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.818 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.818 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:33:04.819 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.820 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:33:04.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.825 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:33:04.825 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.829 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.829 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.839 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.840 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:33:04.840 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.841 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.841 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:33:04.841 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:33:04.842 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.843 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:33:04.843 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:33:04.847 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:33:04.851 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:33:04.855 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.856 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:33:04.856 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.857 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:33:04.857 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.858 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:33:04.858 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.859 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:33:04.859 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.860 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:33:04.861 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.861 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:33:04.862 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.863 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:33:04.864 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.864 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:33:04.865 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.866 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:33:04.866 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:33:04.867 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.867 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:04.868 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.868 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:33:04.869 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.870 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.881 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:04.882 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.883 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:33:04.887 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:33:04.890 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.890 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.890 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:33:04.890 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.891 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:33:04.897 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:33:04.899 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:33:04.899 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:33:04.899 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.900 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.904 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:33:04.905 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:33:04.905 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:33:04.905 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:33:04.906 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:33:04.906 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:33:04.906 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:33:04.913 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.917 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:04.918 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:33:04.919 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6FB1D704-0-0] 2024-09-11 18:33:04.919 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:33:04.920 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.920 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:33:04.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.933 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:33:04.948 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.948 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:33:04.949 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:33:04.949 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE6FB1D70E-0-0] 2024-09-11 18:33:04.949 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:33:04.967 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.983 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:33:04.984 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.984 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:33:04.984 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.985 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:33:04.986 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.986 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:33:04.987 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.987 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:33:04.987 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:04.988 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:33:04.988 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:33:04.990 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:33:05.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:05.004 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:05.004 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:33:05.014 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:33:05.015 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:33:05.021 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:05.023 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:33:05.032 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:33:05.033 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:33:05.033 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:33:05.033 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:33:05.034 INFO 2981023 --- [ocp-schedules-10,aff060f5f2a95dcd,7468b2ce7910cc7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:33:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,30bbfc947c6c992c,b20caed150ee2934] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,30bbfc947c6c992c,b20caed150ee2934] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:05.488 INFO 2981023 --- [ocp-async-7,ae2b12ecc2867392,40dc9523d54957fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:33:06.337460+08:00, lastProcessTime=2024-09-11T18:32:56.352886+08:00 2024-09-11 18:33:06.454 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:06.454 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:06.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:33:06.337460+08:00 2024-09-11 18:33:07.488 INFO 2981023 --- [ocp-async-9,e1d4fc84221ff484,f57d134c8135f921] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:07.801 INFO 2981023 --- [ocp-async-6,cea8e63362831495,063cdfcdf145aed5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:33:08.312 INFO 2981023 --- [ocp-schedules-21,c67f7c0721f930f3,ea01a9badfd4a566] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:33:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,4c0f0b593ec746e7,15a032960c93f4c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:08.315 INFO 2981023 --- [prometheus-minute-schedule-1,4c0f0b593ec746e7,15a032960c93f4c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:08.390 INFO 2981023 --- [ocp-schedules-23,c8ae715389602e41,9910bd2c87de564a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:33:08.390+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:33:08.391 INFO 2981023 --- [ocp-schedules-23,c8ae715389602e41,9910bd2c87de564a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:33:08.392 INFO 2981023 --- [ocp-schedules-23,c8ae715389602e41,9910bd2c87de564a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:33:08.393 INFO 2981023 --- [ocp-schedules-23,c8ae715389602e41,9910bd2c87de564a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:33:09.454 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:09.455 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:09.488 INFO 2981023 --- [ocp-async-2,70facdeee7360c3e,3fc7bdf98752ef85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:10.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:33:10.003 INFO 2981023 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:33:10.914 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:33:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:33:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,91f6cc42ea36c9e6,9a80464b60d603fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,91f6cc42ea36c9e6,9a80464b60d603fb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:11.489 INFO 2981023 --- [ocp-async-10,14e2fdda9b56b5f6,6fc5100e04fb15e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:12.455 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:12.455 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:12.476 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050720, nodeCount=1000, affectRows=1000 2024-09-11 18:33:12.476 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050720, nodeCount=1000, affectRows=1000 2024-09-11 18:33:12.478 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050720, nodeCount=1000, validCount=1000 2024-09-11 18:33:12.478 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050720, nodeCount=999, validCount=999 2024-09-11 18:33:12.485 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050720, nodeCount=49, affectRows=49 2024-09-11 18:33:12.485 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050720, nodeCount=47, validCount=47 2024-09-11 18:33:12.939 INFO 2981023 --- [ocp-schedules-2,e5a884bfcb715e48,2038ccc079068083] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:33:12.958 INFO 2981023 --- [ocp-schedules-21,0d6ea8966f353e3e,dc962133f157d1aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:33:13.030 INFO 2981023 --- [alarm-detect-12,0d6ea8966f353e3e,9b032a29414c3d61] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646646, data is empty. 2024-09-11 18:33:13.030 WARN 2981023 --- [scanner-13690270-1,0d6ea8966f353e3e,c22bc079e0147cf1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:33:13.034 INFO 2981023 --- [alarm-detect-12,0d6ea8966f353e3e,9b032a29414c3d61] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646686, data is empty. 2024-09-11 18:33:13.038 INFO 2981023 --- [alarm-detect-12,0d6ea8966f353e3e,9b032a29414c3d61] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646686, data is empty. 2024-09-11 18:33:13.053 INFO 2981023 --- [alarm-detect-16,0d6ea8966f353e3e,4b4ce10c8cf9e693] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646646, data is empty. 2024-09-11 18:33:13.054 INFO 2981023 --- [alarm-detect-12,0d6ea8966f353e3e,eba7bee83e21f44e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646646, data is empty. 2024-09-11 18:33:13.057 INFO 2981023 --- [alarm-detect-12,0d6ea8966f353e3e,eba7bee83e21f44e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646686, data is empty. 2024-09-11 18:33:13.058 INFO 2981023 --- [alarm-detect-16,0d6ea8966f353e3e,4b4ce10c8cf9e693] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646686, data is empty. 2024-09-11 18:33:13.063 INFO 2981023 --- [alarm-detect-12,0d6ea8966f353e3e,eba7bee83e21f44e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646686, data is empty. 2024-09-11 18:33:13.065 INFO 2981023 --- [alarm-detect-16,0d6ea8966f353e3e,4b4ce10c8cf9e693] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646686, data is empty. 2024-09-11 18:33:13.187 INFO 2981023 --- [alarm-detect-6,0d6ea8966f353e3e,91047d73429c1190] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646646, data is empty. 2024-09-11 18:33:13.194 INFO 2981023 --- [alarm-detect-6,0d6ea8966f353e3e,91047d73429c1190] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646686, data is empty. 2024-09-11 18:33:13.199 INFO 2981023 --- [alarm-detect-6,0d6ea8966f353e3e,91047d73429c1190] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646686, data is empty. 2024-09-11 18:33:13.489 INFO 2981023 --- [ocp-async-5,1bbc2c4d5d25bdb4,5f174032691f4e3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:14.046 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:29:13.665, end=2024-09-11T18:31:14.043} 2024-09-11 18:33:14.048 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClu 2024-09-11 18:33:14.049 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:33:14.049 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:29:13.665, end=2024-09-11T18:31:14.043},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:33:14.050 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:33:14.050 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:33:14.051 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:33:14.051 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:29:13.665, end=2024-09-11T18:31:14.043},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:33:14.051 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:33:14.051 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:33:14.052 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:33:14.053 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:29:13.665, end=2024-09-11T18:31:14.043},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:33:14.053 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:33:14.053 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:33:14.062 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:33:14.080 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394080000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.080Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.083 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:33:14.083 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.088 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.088 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.090 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.090 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.094 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.094 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.095 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:33:14.095 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.08, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.096 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:33:14.096 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:29:13.665, end=2024-09-11T18:31:14.043},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:33:14.096 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:33:14.096 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:33:14.097 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:33:14.099 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:29:13.665, end=2024-09-11T18:31:14.043},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:33:14.099 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:33:14.099 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:33:14.142 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394142000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=8A1B851989150B802AEEACA99152D145, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.142Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.146 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394146000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.146Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.146 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394146000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.146Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.147 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394147000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.147Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.151 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394151000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.151Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.153 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394153000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.153Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.155 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394155000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.155Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.155 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 31 2024-09-11 18:33:14.155 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.156 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 71 2024-09-11 18:33:14.156 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.157 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394157000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.157Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.166 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.166 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.169 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.169 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.172 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.172 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.176 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:33:14.176 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.182 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:33:14.182 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.183 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.183 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.187 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.187 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.188 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.188 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.190 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.190 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.191 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.191 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.193 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.193 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.194 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.194 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.200 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.200 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.200 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.200 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=8A1B851989150B802AEEACA99152D145, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.201 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.201 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.202 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.202 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.203 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.203 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.203 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.203 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.203 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.204 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.204 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.204 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.205 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.205 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.208 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.208 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.208 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.208 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.209 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.209 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.213 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.213 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.214 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.142, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=8A1B851989150B802AEEACA99152D145, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.215 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.215 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.215 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.215 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.218 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.146, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.219 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.220 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.222 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.222 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.222 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.222 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.224 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.224 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.225 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.225 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.227 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.147, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.227 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.230 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.230 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.232 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.153, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.233 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.146, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.234 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.236 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.151, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.240 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.155, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.240 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.157, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.254 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394254000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.254Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.261 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394261000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.261Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.266 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:33:14.266 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.266 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394266000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.266Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.271 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394271000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1ED31157F7096F47941EA2233AA4101C, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.271Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.278 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.278 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.279 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394279000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.279Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.280 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394280000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.280Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.281 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394281000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=91740952E2573F82234F7235A70D4910, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.281Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.283 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.283 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.291 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.291 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.291 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964394291000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:33:14.291Z, endTime=2024-09-11T10:31:14.043Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:33:14.296 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.296 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.299 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.299 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.300 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.300 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.302 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.302 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.303 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.303 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.305 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.305 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.308 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.308 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.314 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.314 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.314 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.314 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.315 INFO 2981023 --- [prometheus-minute-schedule-1,395cd8928bd78ee0,1ac1020c0d1ec95c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:14.315 INFO 2981023 --- [prometheus-hour-schedule-1,c77909070ccb5ae9,5600824eab3e373b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:33:14.316 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.316 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.317 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.317 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.319 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.319 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.319 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.319 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.319 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.320 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.320 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.320 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:33:14.320 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:33:14.322 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.322 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.323 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.323 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.323 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.323 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.323 INFO 2981023 --- [prometheus-minute-schedule-1,395cd8928bd78ee0,1ac1020c0d1ec95c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:14.327 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.332 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.332 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.333 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.254, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.335 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.335 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.336 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.341 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.341 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.341 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.341 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.341 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.261, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.344 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.344 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.344 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:33:14.344 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:33:14.346 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.347 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.266, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.348 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.348 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=91740952E2573F82234F7235A70D4910, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.350 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.351 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.351 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.352 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.271, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=1ED31157F7096F47941EA2233AA4101C, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.353 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.354 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.354 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.354 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:33:14.354 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050674043000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:33:14.355 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.356 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:33:14.356 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.281, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=91740952E2573F82234F7235A70D4910, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.357 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.279, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.359 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.28, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.359 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:33:14.291, end=2024-09-11T18:31:14.043}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:33:14.360 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:33:14.362 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:29:13.665, end=2024-09-11T18:31:14.043},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:33:14.362 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:33:14.362 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:33:14.372 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:33:14.372 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:33:14.612 INFO 2981023 --- [ocp-schedules-28,c1d2b01ca636dcce,ad9c9faa8871af1c] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:33:15.455 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:15.456 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:15.489 INFO 2981023 --- [ocp-async-8,a1d18155edd2b287,bdf2ac607f548fdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:15.567 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050720, nodeCount=480, affectRows=480 2024-09-11 18:33:16.014 INFO 2981023 --- [ocp-schedules-9,a6268a99f7338780,ad709a5fcbd74b59] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:33:16.014 INFO 2981023 --- [host-check-clockdiff1,a6268a99f7338780,050d03fbdf2f405b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:33:16.014 INFO 2981023 --- [host-check-clockdiff2,a6268a99f7338780,5eb3d55689303cd0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:33:16.015 INFO 2981023 --- [host-check-clockdiff3,a6268a99f7338780,452caba33a20284e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:33:16.015 INFO 2981023 --- [host-check-clockdiff4,a6268a99f7338780,7d4eaff3990fc5de] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:33:16.025 INFO 2981023 --- [ocp-schedules-26,ef7493e834be674f,2416d155c26d5ebd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:33:16.027 INFO 2981023 --- [ocp-schedules-10,cd91a678f4446de2,15daaba1a78f03c0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:33:16.027 INFO 2981023 --- [host-check-agent-status1,cd91a678f4446de2,8ba7e0fc4699cc8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:33:16.027 INFO 2981023 --- [host-check-agent-status2,cd91a678f4446de2,ded08cd4d8d488d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:33:16.029 INFO 2981023 --- [host-check-clockdiff3,a6268a99f7338780,452caba33a20284e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050796 0 0 , error= 2024-09-11 18:33:16.029 INFO 2981023 --- [host-check-agent-status4,cd91a678f4446de2,df32321f9908b53b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:33:16.030 INFO 2981023 --- [host-check-agent-status3,cd91a678f4446de2,d360911fca4b0d4f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:33:16.031 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:33:16.045 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:33:16.045 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:33:16.048 INFO 2981023 --- [host-check-clockdiff4,a6268a99f7338780,7d4eaff3990fc5de] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050796 0 0 , error= 2024-09-11 18:33:16.046 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:33:16.064 INFO 2981023 --- [host-check-agent-status2,cd91a678f4446de2,ded08cd4d8d488d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:33:16.067 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:33:16.068 INFO 2981023 --- [host-check-agent-status4,cd91a678f4446de2,df32321f9908b53b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:33:16.073 INFO 2981023 --- [host-check-agent-status2,cd91a678f4446de2,ded08cd4d8d488d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:33:16.073 INFO 2981023 --- [host-check-agent-status1,cd91a678f4446de2,8ba7e0fc4699cc8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:33:16.074 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:33:16.077 INFO 2981023 --- [host-check-agent-status1,cd91a678f4446de2,8ba7e0fc4699cc8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:33:16.080 INFO 2981023 --- [host-check-agent-status3,cd91a678f4446de2,d360911fca4b0d4f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:33:16.084 INFO 2981023 --- [host-check-agent-status4,cd91a678f4446de2,df32321f9908b53b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:33:16.093 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:33:16.094 INFO 2981023 --- [host-check-agent-status3,cd91a678f4446de2,d360911fca4b0d4f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:33:16.094 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:33:16.189 INFO 2981023 --- [ocp-schedules-16,f14234bb4cf3183d,6ebf69bcf8e8fbdf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:33:16.189 INFO 2981023 --- [ocp-schedules-1,2dabb55f846fa4bd,09ae98bd2ac35c96] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:33:16.189 INFO 2981023 --- [ocp-schedules-28,97af152d9ea95d8c,539de88303e8f861] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:33:16.189 INFO 2981023 --- [ocp-schedules-3,8406f38560f818ab,084c3ae326f93506] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:33:16.260 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:33:16.261 INFO 2981023 --- [ocp-schedules-28,97af152d9ea95d8c,539de88303e8f861] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:33:16.262 INFO 2981023 --- [ocp-schedules-1,2dabb55f846fa4bd,09ae98bd2ac35c96] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:33:16.263 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:33:16.264 INFO 2981023 --- [ocp-schedules-5,ffeb51f774f860b2,71f6785aaa4cf619] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:33:16.282 INFO 2981023 --- [ocp-schedules-16,f14234bb4cf3183d,6ebf69bcf8e8fbdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:33:16.287 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:33:16.292 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:33:16.300 INFO 2981023 --- [ocp-schedules-5,ffeb51f774f860b2,71f6785aaa4cf619] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:33:16.300 INFO 2981023 --- [ocp-schedules-5,ffeb51f774f860b2,71f6785aaa4cf619] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:33:16.306 INFO 2981023 --- [ocp-schedules-1,2dabb55f846fa4bd,09ae98bd2ac35c96] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:33:16.307 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050723377000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:33:16.366 INFO 2981023 --- [ocp-schedules-23,88a3fe098da88089,2a13b3aef3b923e3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:33:16.385 INFO 2981023 --- [host-check-agent-status2,cd91a678f4446de2,ded08cd4d8d488d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:33:16.391 INFO 2981023 --- [host-check-agent-status2,cd91a678f4446de2,ded08cd4d8d488d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:33:16.391 INFO 2981023 --- [host-check-agent-status2,cd91a678f4446de2,ded08cd4d8d488d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:33:16.391 INFO 2981023 --- [host-check-agent-status2,cd91a678f4446de2,ded08cd4d8d488d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:33:16.391 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:33:16.388617+08:00, lastProcessTime=2024-09-11T18:33:06.337460+08:00 2024-09-11 18:33:16.395 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:33:16.397 INFO 2981023 --- [host-check-agent-status1,cd91a678f4446de2,8ba7e0fc4699cc8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:33:16.397 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:33:16.397589589+08:00 2024-09-11 18:33:16.397 INFO 2981023 --- [host-check-agent-status3,cd91a678f4446de2,d360911fca4b0d4f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:33:16.399 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:33:16.400 INFO 2981023 --- [host-check-agent-status1,cd91a678f4446de2,8ba7e0fc4699cc8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:33:16.400 INFO 2981023 --- [host-check-agent-status1,cd91a678f4446de2,8ba7e0fc4699cc8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:33:16.400 INFO 2981023 --- [host-check-agent-status1,cd91a678f4446de2,8ba7e0fc4699cc8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:33:16.401 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:33:16.397+08:00 2024-09-11 18:33:16.402 INFO 2981023 --- [host-check-agent-status3,cd91a678f4446de2,d360911fca4b0d4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:33:16.402 INFO 2981023 --- [host-check-agent-status3,cd91a678f4446de2,d360911fca4b0d4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:33:16.402 INFO 2981023 --- [host-check-agent-status3,cd91a678f4446de2,d360911fca4b0d4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:33:16.402 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:33:16.401795405+08:00 2024-09-11 18:33:16.404 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:33:16.404 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:33:16.402+08:00 2024-09-11 18:33:16.405 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:33:16.409627794+08:00 2024-09-11 18:33:16.405 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:33:16.405+08:00 2024-09-11 18:33:16.406 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:33:16.397+08:00 and status AVAILABLE 2024-09-11 18:33:16.406 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:33:16.402+08:00 and status AVAILABLE 2024-09-11 18:33:16.410 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:33:16.410 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:33:16.405+08:00 and status AVAILABLE 2024-09-11 18:33:16.411 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:33:16.410498064+08:00 2024-09-11 18:33:16.412 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.412 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:33:16.411+08:00 2024-09-11 18:33:16.412 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:33:16.413 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:33:16.413 INFO 2981023 --- [host-check-agent-status4,cd91a678f4446de2,df32321f9908b53b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:33:16.414 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:33:16.411+08:00 and status AVAILABLE 2024-09-11 18:33:16.416 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.417 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.417 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:33:16.418 INFO 2981023 --- [host-check-agent-status4,cd91a678f4446de2,df32321f9908b53b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:33:16.418 INFO 2981023 --- [host-check-agent-status4,cd91a678f4446de2,df32321f9908b53b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:33:16.418 INFO 2981023 --- [host-check-agent-status4,cd91a678f4446de2,df32321f9908b53b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:33:16.418 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:16.419 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:33:16.419 INFO 2981023 --- [ocp-schedules-16,f14234bb4cf3183d,6ebf69bcf8e8fbdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:33:16.419 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:33:16.419 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.420 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:16.424 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:33:16.429 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:33:16.431 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:33:16.460 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.461 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:16.469 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:33:16.471 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:33:16.472 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:33:16.473 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:33:16.476 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:33:16.477 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:33:16.478 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:33:16.479 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:33:16.486 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.487 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:33:16.491 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:33:16.517 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:33:16.520 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:33:16.522 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:33:16.523 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:33:16.523 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050723377000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:33:16.524 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:33:16.541 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:33:16.548 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:33:16.550 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:33:16.560 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:16.560 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.561 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:16.561 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:33:16.562 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:33:16.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:33:16.566 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:33:16.566 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:33:16.570 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:33:16.571 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:33:16.580 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:33:16.580 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:33:16.594 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:33:16.601 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:33:16.605 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:33:16.614 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:33:16.616 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:16.616 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.617 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:33:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:33:16.619 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:33:16.620 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:33:16.627 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:33:16.628 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:33:16.631 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:33:16.635 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:33:16.639 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:33:16.640 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:33:16.640 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:33:16.656 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:33:16.658 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:33:16.663 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:33:16.666 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:33:16.668 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:16.669 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.670 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:33:16.671 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:33:16.681 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:33:16.692 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:16.692 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:33:16.693 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.693 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:16.694 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:33:16.694 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.696 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:33:16.697 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:33:16.700 INFO 2981023 --- [host-check-host-status3,ef7493e834be674f,d326c474e70b97ef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:33:16.704 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:33:16.707 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:33:16.708 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:33:16.708 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:33:16.710 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:33:16.720 INFO 2981023 --- [host-check-host-status2,ef7493e834be674f,4f2b7f03c0a85f24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:33:16.721 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:33:16.728 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:33:16.729 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:33:16.730 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:33:16.732 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:33:16.737 INFO 2981023 --- [host-check-host-status1,ef7493e834be674f,252d2c1b81e4863f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:33:16.739 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:33:16.740 INFO 2981023 --- [host-check-host-status4,ef7493e834be674f,1382c237b8122d3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:33:16.744 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:33:16.745 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:33:16.755 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:33:16.767 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:33:16.767 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:16.768 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.769 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:33:16.775 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:33:16.775 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:33:16.794 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:33:16.388617+08:00 2024-09-11 18:33:16.803 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:33:16.809 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:16.809 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:16.809 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:33:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:33:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:33:16.815 INFO 2981023 --- [ocp-schedules-28,97af152d9ea95d8c,539de88303e8f861] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:33:16.968 INFO 2981023 --- [host-check-clockdiff2,a6268a99f7338780,5eb3d55689303cd0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050796 5 5 , error= 2024-09-11 18:33:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,790d7b697beacfc3,fcae23a2f1ffdd52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,790d7b697beacfc3,fcae23a2f1ffdd52] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:17.489 INFO 2981023 --- [ocp-async-4,f4f9696d90a35163,7414b7cb8efb8ddc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:17.733 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050720, nodeCount=480, validCount=480 2024-09-11 18:33:17.801 INFO 2981023 --- [ocp-async-5,4ec5c1e0e6ed10e7,b1ced76da89152e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:33:17.833 INFO 2981023 --- [host-check-clockdiff1,a6268a99f7338780,050d03fbdf2f405b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050797 0 0 , error= 2024-09-11 18:33:18.456 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:18.456 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:19.489 INFO 2981023 --- [ocp-async-9,a2e108a3069bad0a,fa1b25f22c59baba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,5a4db207de25d5be,9145d08cdeeb4b57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,5a4db207de25d5be,9145d08cdeeb4b57] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:21.456 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:21.458 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:21.489 INFO 2981023 --- [ocp-async-6,583f5a3a352c0915,5bc0e36d30914c14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:22.942 INFO 2981023 --- [ocp-schedules-5,eb5b92dac814f2ef,0aaad88000eee2bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:33:23.241 INFO 2981023 --- [ocp-schedules-2,92d9220eb582992a,e3d5eeae777f03f5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:33:23.278 WARN 2981023 --- [scanner-13690557-1,92d9220eb582992a,7f81e96a93b25036] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:33:23.314 INFO 2981023 --- [prometheus-minute-schedule-1,37bddb05d3c9bfd0,68af0d02b0b60434] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:23.319 INFO 2981023 --- [prometheus-minute-schedule-1,37bddb05d3c9bfd0,68af0d02b0b60434] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:23.342 INFO 2981023 --- [alarm-detect-14,92d9220eb582992a,278305170c60329c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050765 ~ 1726050793 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050780, value=2.467)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=2.467, values={1726050780=2.467}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:33:23.489 INFO 2981023 --- [ocp-async-7,92d04b2b79a27e23,216334f17ff67715] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:24.458 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:24.459 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:24.542 INFO 2981023 --- [ocp-schedules-1,1b738d4fcadaf877,2f960660f8eb6f48] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:33:24.653 INFO 2981023 --- [ocp-schedules-7,4330b2affc0806f7,17e4b90954ffea18] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 18:33:24.653 INFO 2981023 --- [ocp-schedules-7,4330b2affc0806f7,17e4b90954ffea18] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 18:33:24.684 INFO 2981023 --- [pool-620-thread-2,4330b2affc0806f7,a0e96fceea4775b5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:33:24.684 INFO 2981023 --- [pool-620-thread-1,4330b2affc0806f7,460daf7b244dcaa1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:33:24.689 INFO 2981023 --- [pool-620-thread-3,4330b2affc0806f7,af449327764290f0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:33:24.717 INFO 2981023 --- [pool-620-thread-2,4330b2affc0806f7,a0e96fceea4775b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:24.719 INFO 2981023 --- [pool-620-thread-1,4330b2affc0806f7,460daf7b244dcaa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:24.719 INFO 2981023 --- [pool-620-thread-2,4330b2affc0806f7,a0e96fceea4775b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726050504000000, 1726050804000000] 2024-09-11 18:33:24.720 INFO 2981023 --- [pool-620-thread-3,4330b2affc0806f7,af449327764290f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:24.721 INFO 2981023 --- [pool-620-thread-1,4330b2affc0806f7,460daf7b244dcaa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726050504000000, 1726050804000000] 2024-09-11 18:33:24.724 INFO 2981023 --- [pool-620-thread-3,4330b2affc0806f7,af449327764290f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726050504000000, 1726050804000000] 2024-09-11 18:33:24.728 INFO 2981023 --- [pool-620-thread-3,4330b2affc0806f7,af449327764290f0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:33:24.807 INFO 2981023 --- [pool-620-thread-1,4330b2affc0806f7,460daf7b244dcaa1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:33:24.864 INFO 2981023 --- [pool-620-thread-2,4330b2affc0806f7,a0e96fceea4775b5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:33:24.865 INFO 2981023 --- [ocp-schedules-7,4330b2affc0806f7,17e4b90954ffea18] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 18:33:25.489 INFO 2981023 --- [ocp-async-1,f00bbc226d46c325,b04a59fd2c5c001c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,1c75246fa18f978c,9c416734726fb3f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,1c75246fa18f978c,9c416734726fb3f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:33:26.352540+08:00, lastProcessTime=2024-09-11T18:33:16.388617+08:00 2024-09-11 18:33:26.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:33:26.352540+08:00 2024-09-11 18:33:27.459 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:27.461 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:27.490 INFO 2981023 --- [ocp-async-3,dba8fb55da48566e,728093a09d18f3c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:27.801 INFO 2981023 --- [ocp-async-7,5a040b06b2ffb824,710454333672ea5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:33:28.804 INFO 2981023 --- [ocp-schedules-25,49f05124af0e7895,fb40f680b89f2828] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:33:28.804 INFO 2981023 --- [ocp-schedules-25,49f05124af0e7895,fb40f680b89f2828] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:33:28.804 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:33:28.804 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:33:28.805 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:33:28.828 INFO 2981023 --- [ocp-schedules-25,49f05124af0e7895,fb40f680b89f2828] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:33:28.829 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:33:28.832 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:33:28.832 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:33:28.833 INFO 2981023 --- [ocp-schedules-25,49f05124af0e7895,fb40f680b89f2828] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:33:28.833 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:33:28.833 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:33:28.835 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:33:28.983 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:33:28.984 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:33:28.985 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:33:29.001 INFO 2981023 --- [ocp-schedules-25,49f05124af0e7895,fb40f680b89f2828] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:33:29.003 INFO 2981023 --- [ocp-schedules-25,49f05124af0e7895,fb40f680b89f2828] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:33:29.192 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.195 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=171999 2024-09-11 18:33:29.202 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.202 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:33:29.206 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.211 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.215 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.219 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.241 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 36 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.317 INFO 2981023 --- [prometheus-minute-schedule-1,25e891edb4236514,b7ffa1218533dac1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:29.322 INFO 2981023 --- [prometheus-minute-schedule-1,25e891edb4236514,b7ffa1218533dac1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:29.326 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 8 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.359 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 197 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.375 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 2 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.387 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.387 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:33:29.389 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.392 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 7 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.412 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.415 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.415 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:33:29.417 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.420 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 3 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.427 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.428 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.429 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.429 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.430 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.431 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.431 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.432 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.433 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.433 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.434 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:27:28.951, end=2024-09-11T18:32:29.188} 2024-09-11 18:33:29.434 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 6 dubious sqls 2024-09-11 18:33:29.490 INFO 2981023 --- [ocp-async-4,83b71d8afbe2eebb,be88e91fa41f84e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:30.461 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:30.461 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:31.490 INFO 2981023 --- [ocp-async-5,4cdb9f2fc9edcae0,587f38a13ee3f169] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,4a14ad0abd9c480f,e9a2a70dd28ffbb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,4a14ad0abd9c480f,e9a2a70dd28ffbb3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:32.945 INFO 2981023 --- [ocp-schedules-15,864b1e7337504b21,93832cbb4193a10e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:33:33.453 INFO 2981023 --- [ocp-schedules-1,ea30efc71f3bbeae,44b688e21009a7b3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:33:33.461 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:33.462 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:33.490 INFO 2981023 --- [ocp-async-8,199964c274a7b666,6c916f88a657c01a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:33.507 WARN 2981023 --- [scanner-13690787-1,ea30efc71f3bbeae,6634dc1482ac954b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:33:33.530 INFO 2981023 --- [alarm-detect-9,ea30efc71f3bbeae,e5738a0960ece93c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646686, data is empty. 2024-09-11 18:33:33.532 INFO 2981023 --- [alarm-detect-9,ea30efc71f3bbeae,e5738a0960ece93c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646686, data is empty. 2024-09-11 18:33:33.535 INFO 2981023 --- [alarm-detect-9,ea30efc71f3bbeae,e5738a0960ece93c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646686, data is empty. 2024-09-11 18:33:33.567 INFO 2981023 --- [alarm-detect-17,ea30efc71f3bbeae,fd3ece4f713db18e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646686, data is empty. 2024-09-11 18:33:33.569 INFO 2981023 --- [alarm-detect-10,ea30efc71f3bbeae,a18c7e0e2733a01f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646686, data is empty. 2024-09-11 18:33:33.573 INFO 2981023 --- [alarm-detect-10,ea30efc71f3bbeae,a18c7e0e2733a01f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646686, data is empty. 2024-09-11 18:33:33.574 INFO 2981023 --- [alarm-detect-17,ea30efc71f3bbeae,fd3ece4f713db18e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646686, data is empty. 2024-09-11 18:33:33.580 INFO 2981023 --- [alarm-detect-10,ea30efc71f3bbeae,a18c7e0e2733a01f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646686, data is empty. 2024-09-11 18:33:33.580 INFO 2981023 --- [alarm-detect-17,ea30efc71f3bbeae,fd3ece4f713db18e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646686, data is empty. 2024-09-11 18:33:33.644 INFO 2981023 --- [alarm-detect-6,ea30efc71f3bbeae,f826a281708f071a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050780 ~ 1726050803 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050795, value=1.921)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.921, values={1726050795=1.921}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:33:33.704 INFO 2981023 --- [alarm-detect-7,ea30efc71f3bbeae,e9fe4ce133d422e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646686, data is empty. 2024-09-11 18:33:33.709 INFO 2981023 --- [alarm-detect-7,ea30efc71f3bbeae,e9fe4ce133d422e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646686, data is empty. 2024-09-11 18:33:33.712 INFO 2981023 --- [alarm-detect-7,ea30efc71f3bbeae,e9fe4ce133d422e6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646686, data is empty. 2024-09-11 18:33:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,56cab0f6bed36184,d46edf4395e238d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,56cab0f6bed36184,d46edf4395e238d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:35.490 INFO 2981023 --- [ocp-async-2,6c645fd994f5e179,7189f64848737e99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:36.304 INFO 2981023 --- [ocp-schedules-7,0245105900dc366e,642584bf274673e6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:33:36.312 INFO 2981023 --- [ocp-schedules-7,0245105900dc366e,642584bf274673e6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:33:36.312 INFO 2981023 --- [ocp-schedules-7,0245105900dc366e,642584bf274673e6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:33:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:33:36.337274+08:00, lastProcessTime=2024-09-11T18:33:26.352540+08:00 2024-09-11 18:33:36.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:33:36.337274+08:00 2024-09-11 18:33:36.462 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:36.462 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:37.314 INFO 2981023 --- [prometheus-cache-monitor1,e357dbef3abea53f,cb423ddeb1134b57] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:33:37.315 INFO 2981023 --- [prometheus-cache-monitor1,e357dbef3abea53f,cb423ddeb1134b57] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:33:37.490 INFO 2981023 --- [ocp-async-10,9b9f6bd465ff3b8d,a3b5bd96574ed69a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:37.801 INFO 2981023 --- [ocp-async-8,bfca2ffcf6ab272e,b0612183f092b57d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:33:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,7db310301f2bf680,44e8f4bf3f694502] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,7db310301f2bf680,44e8f4bf3f694502] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:39.462 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:39.463 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:39.490 INFO 2981023 --- [ocp-async-3,cc0bacd21d0ed828,5dbef49d3f2c3342] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:40.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:33:40.003 INFO 2981023 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:33:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,96ea67871049f97c,816a1300bbe2517e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,96ea67871049f97c,816a1300bbe2517e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:41.490 INFO 2981023 --- [ocp-async-7,b2aa7850d5e20a31,3b968a28df5d35ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:42.463 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:42.463 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:42.947 INFO 2981023 --- [ocp-schedules-28,28f604cb04f37d06,f84817f70c2b0b4c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:33:43.490 INFO 2981023 --- [ocp-async-1,007a22cf452c9361,e88de59523361b82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:43.750 INFO 2981023 --- [ocp-schedules-24,f12a72671225bc05,851d712dd6774444] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:33:43.791 WARN 2981023 --- [scanner-13691051-1,f12a72671225bc05,950d7e2b0924fa81] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:33:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,e762581d77e2af72,9e2fb51b822db904] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,e762581d77e2af72,9e2fb51b822db904] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:45.101 INFO 2981023 --- [prometheus-second-schedule-1,5865c24a788b1355,b9cd142127ead4c5] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:33:45.463 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:45.464 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:45.490 INFO 2981023 --- [ocp-async-9,4459e3abe648263f,1d34002421be2f5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:46.024 INFO 2981023 --- [ocp-schedules-29,598fc08c0e8e72d4,7e59214d479e8934] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:33:46.024 INFO 2981023 --- [host-check-clockdiff1,598fc08c0e8e72d4,2a207bac40fd80ce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:33:46.025 INFO 2981023 --- [ocp-schedules-8,ce6f34ece185995b,4ef4fe889f565d2d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:33:46.040 INFO 2981023 --- [host-check-clockdiff2,598fc08c0e8e72d4,67747c2dbcf2855c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:33:46.041 INFO 2981023 --- [ocp-schedules-3,d6b3082e605b334d,bb83f30886760332] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:33:46.041 INFO 2981023 --- [host-check-agent-status1,d6b3082e605b334d,2c85cde48afdb479] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:33:46.042 INFO 2981023 --- [host-check-agent-status2,d6b3082e605b334d,03724378429abed0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:33:46.042 INFO 2981023 --- [host-check-agent-status3,d6b3082e605b334d,23e7d0b3908d3acf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:33:46.042 INFO 2981023 --- [host-check-agent-status4,d6b3082e605b334d,8f5c2d8cbbc4604a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:33:46.043 INFO 2981023 --- [host-check-clockdiff3,598fc08c0e8e72d4,9c147c96e19f8c40] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:33:46.044 INFO 2981023 --- [host-check-clockdiff4,598fc08c0e8e72d4,a7f8fccba7c6d843] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:33:46.049 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:33:46.049 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:33:46.049 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:33:46.049 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:33:46.051 INFO 2981023 --- [host-check-clockdiff2,598fc08c0e8e72d4,67747c2dbcf2855c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050826 5 5 , error= 2024-09-11 18:33:46.051 INFO 2981023 --- [host-check-clockdiff1,598fc08c0e8e72d4,2a207bac40fd80ce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050826 0 0 , error= 2024-09-11 18:33:46.052 INFO 2981023 --- [host-check-agent-status1,d6b3082e605b334d,2c85cde48afdb479] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:33:46.053 INFO 2981023 --- [host-check-agent-status1,d6b3082e605b334d,2c85cde48afdb479] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:33:46.053 INFO 2981023 --- [host-check-agent-status4,d6b3082e605b334d,8f5c2d8cbbc4604a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:33:46.054 INFO 2981023 --- [host-check-agent-status4,d6b3082e605b334d,8f5c2d8cbbc4604a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:33:46.055 INFO 2981023 --- [host-check-agent-status2,d6b3082e605b334d,03724378429abed0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:33:46.055 INFO 2981023 --- [host-check-agent-status3,d6b3082e605b334d,23e7d0b3908d3acf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:33:46.055 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:33:46.058 INFO 2981023 --- [host-check-clockdiff3,598fc08c0e8e72d4,9c147c96e19f8c40] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050826 0 0 , error= 2024-09-11 18:33:46.059 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:33:46.060 INFO 2981023 --- [host-check-agent-status2,d6b3082e605b334d,03724378429abed0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:33:46.061 INFO 2981023 --- [host-check-agent-status3,d6b3082e605b334d,23e7d0b3908d3acf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:33:46.063 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:33:46.066 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:33:46.066 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:33:46.067 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:33:46.068 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:33:46.071 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:33:46.087 INFO 2981023 --- [host-check-agent-status4,d6b3082e605b334d,8f5c2d8cbbc4604a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:33:46.088 INFO 2981023 --- [host-check-agent-status1,d6b3082e605b334d,2c85cde48afdb479] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:33:46.088 INFO 2981023 --- [host-check-agent-status3,d6b3082e605b334d,23e7d0b3908d3acf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:33:46.092 INFO 2981023 --- [host-check-agent-status1,d6b3082e605b334d,2c85cde48afdb479] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:33:46.092 INFO 2981023 --- [host-check-agent-status1,d6b3082e605b334d,2c85cde48afdb479] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:33:46.092 INFO 2981023 --- [host-check-agent-status1,d6b3082e605b334d,2c85cde48afdb479] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:33:46.092 INFO 2981023 --- [host-check-agent-status3,d6b3082e605b334d,23e7d0b3908d3acf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:33:46.092 INFO 2981023 --- [host-check-agent-status3,d6b3082e605b334d,23e7d0b3908d3acf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:33:46.092 INFO 2981023 --- [host-check-agent-status3,d6b3082e605b334d,23e7d0b3908d3acf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:33:46.098 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:33:46.099 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:33:46.099140167+08:00 2024-09-11 18:33:46.100 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:33:46.101 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:33:46.099+08:00 2024-09-11 18:33:46.103 INFO 2981023 --- [host-check-agent-status4,d6b3082e605b334d,8f5c2d8cbbc4604a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:33:46.103 INFO 2981023 --- [host-check-agent-status4,d6b3082e605b334d,8f5c2d8cbbc4604a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:33:46.103 INFO 2981023 --- [host-check-agent-status4,d6b3082e605b334d,8f5c2d8cbbc4604a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:33:46.103 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:33:46.099+08:00 and status AVAILABLE 2024-09-11 18:33:46.103 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:33:46.103350367+08:00 2024-09-11 18:33:46.107 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:33:46.104+08:00 2024-09-11 18:33:46.108 INFO 2981023 --- [host-check-agent-status2,d6b3082e605b334d,03724378429abed0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:33:46.108 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:33:46.109 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:33:46.104+08:00 and status AVAILABLE 2024-09-11 18:33:46.109 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:33:46.109432251+08:00 2024-09-11 18:33:46.110 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:33:46.109+08:00 2024-09-11 18:33:46.112 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:33:46.109+08:00 and status AVAILABLE 2024-09-11 18:33:46.112 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:33:46.118 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:33:46.123 INFO 2981023 --- [host-check-agent-status2,d6b3082e605b334d,03724378429abed0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:33:46.123 INFO 2981023 --- [host-check-agent-status2,d6b3082e605b334d,03724378429abed0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:33:46.123 INFO 2981023 --- [host-check-agent-status2,d6b3082e605b334d,03724378429abed0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:33:46.126 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:33:46.140 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:33:46.140 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:33:46.140 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:33:46.141 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:33:46.142 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:33:46.147 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:33:46.148 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:33:46.153799572+08:00 2024-09-11 18:33:46.149 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:33:46.164 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:33:46.164 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:33:46.148+08:00 2024-09-11 18:33:46.173 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:33:46.148+08:00 and status AVAILABLE 2024-09-11 18:33:46.173 INFO 2981023 --- [ocp-schedules-24,48341469633d31e1,254c594ee2b73472] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:33:46.190 INFO 2981023 --- [ocp-schedules-10,2c545400c07d9200,66860653011d9d5a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:33:46.190 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:33:46.195 INFO 2981023 --- [ocp-schedules-2,83e85b1cc78afe34,7bad507a98bfa713] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:33:46.195 INFO 2981023 --- [ocp-schedules-6,0a10fc471e22e91a,947142c572a30b44] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:33:46.195 INFO 2981023 --- [ocp-schedules-5,37d1074ef18c9f37,295e560b55a7b5e2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:33:46.196 INFO 2981023 --- [ocp-schedules-24,48341469633d31e1,254c594ee2b73472] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:33:46.200 INFO 2981023 --- [ocp-schedules-10,2c545400c07d9200,66860653011d9d5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:33:46.201 INFO 2981023 --- [ocp-schedules-6,0a10fc471e22e91a,947142c572a30b44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:33:46.205 INFO 2981023 --- [ocp-schedules-2,83e85b1cc78afe34,7bad507a98bfa713] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:33:46.206 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:33:46.210 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:33:46.214 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:33:46.214 INFO 2981023 --- [ocp-schedules-1,fa3aa1f2a002542b,d772d0c49a4eb532] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:33:46.221 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:33:46.221 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:33:46.222 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:33:46.233 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:33:46.234 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:33:46.245 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.245 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:33:46.249 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:33:46.250 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050723377000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:33:46.253 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:33:46.253 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:33:46.267 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:33:46.276 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.276 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:33:46.277 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:33:46.281 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:33:46.282 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.282 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:33:46.283 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:33:46.283 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.286 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:33:46.300 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:33:46.301 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:33:46.308 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:33:46.309 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:33:46.314 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:33:46.321 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:33:46.321 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:33:46.321 INFO 2981023 --- [ocp-schedules-10,2c545400c07d9200,66860653011d9d5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:33:46.322 INFO 2981023 --- [ocp-schedules-2,83e85b1cc78afe34,7bad507a98bfa713] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:33:46.335 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.335 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:46.345 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:33:46.348 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:33:46.356 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.356 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:46.360 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:33:46.363 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:33:46.362084+08:00, lastProcessTime=2024-09-11T18:33:36.337274+08:00 2024-09-11 18:33:46.364 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:33:46.386 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.388 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:33:46.389 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:33:46.389 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:33:46.390 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.390 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:33:46.393 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.395 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.396 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:33:46.396 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:33:46.396 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.400 INFO 2981023 --- [host-check-host-status1,ce6f34ece185995b,46b7048301bbc585] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:33:46.405 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:33:46.408 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:33:46.410 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:33:46.417 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.418 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:46.420 INFO 2981023 --- [host-check-host-status4,ce6f34ece185995b,95dea1e8d6c46dab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:33:46.424 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:33:46.435 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:33:46.438 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.439 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:33:46.441 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.442 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:46.443 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:33:46.445 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.447 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:33:46.448 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:33:46.453 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:33:46.456 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.460 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:33:46.460 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:33:46.470 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.471 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.471 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:33:46.472 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:33:46.472 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.473 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:33:46.474 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:33:46.474 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050723377000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:33:46.474 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:33:46.475 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:33:46.475 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.479 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:33:46.481 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:33:46.485 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:33:46.486 INFO 2981023 --- [host-check-host-status2,ce6f34ece185995b,9987fa9654979430] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:33:46.489 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.490 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:33:46.491 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:33:46.496 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:33:46.500 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:33:46.501 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:33:46.502 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:33:46.512 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:33:46.527 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:33:46.529 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.529 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.530 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.530 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:33:46.530 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:33:46.531 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:33:46.531 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.535 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:33:46.545 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:33:46.549 INFO 2981023 --- [host-check-host-status3,ce6f34ece185995b,4e05e33eecae7f02] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:33:46.564 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:33:46.568 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.568 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.569 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:33:46.569 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:33:46.570 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.573 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:33:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:33:46.585 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:33:46.585 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.585 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:33:46.586 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.586 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:33:46.588 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:33:46.588 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.597 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:33:46.605 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:33:46.610 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.610 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.611 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:33:46.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:33:46.614 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:33:46.617 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:33:46.619 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:33:46.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.620 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.620 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:33:46.628 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:33:46.628 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:33:46.641 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:33:46.644 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:33:46.656 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:33:46.362084+08:00 2024-09-11 18:33:46.662 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:33:46.668 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:33:46.671 INFO 2981023 --- [ocp-schedules-17,b8965a872ee7a83f,85787f220ab012f2] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:33:46.673 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:33:46.678 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:33:46.679 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:46.679 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:33:46.684 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:33:46.684 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:33:46.685 INFO 2981023 --- [ocp-schedules-6,0a10fc471e22e91a,947142c572a30b44] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:33:46.849 INFO 2981023 --- [ocp-schedules-9,dae9283416fab5ed,0e229194daedaf83] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:33:46.849 INFO 2981023 --- [ocp-schedules-9,dae9283416fab5ed,0e229194daedaf83] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:33:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,055b439cbe1491da,287dd5d15d74ed0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,055b439cbe1491da,287dd5d15d74ed0d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:47.490 INFO 2981023 --- [ocp-async-2,af1dff9883e6e006,0a7cf1c9a74a3ed2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:47.801 INFO 2981023 --- [ocp-async-8,bb514caa9b327813,3763147d3532219f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:33:48.237 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:33:48.237 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:33:48.239 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:33:48.239 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:33:48.239 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:33:48.239 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:33:48.242 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:33:48.242 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:33:48.242 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:33:48.242 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:33:48.264 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:33:48.268 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:33:48.268 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:33:48.268 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:33:48.268 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:33:48.268 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:33:48.272 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:33:48.464 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:48.464 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:48.541 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:33:48.785 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.543730857S 2024-09-11 18:33:48.785 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:33:48.832 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:33:48.832 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:33:48.848 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:33:48.882 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:33:48.882 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:33:48.882 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:33:48.882 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:33:48.882 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:33:48.882 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:33:48.887 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:33:48.972 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] o.s.boot.SpringApplication : Started application in 0.732 seconds (JVM running for 86628.844) 2024-09-11 18:33:48.972 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.73267485S 2024-09-11 18:33:49.195 INFO 2981023 --- [ocp-schedules-12,0d527bf4d84c0076,be51b2af24b85d8f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:33:49.217 INFO 2981023 --- [ocp-schedules-14,2f6b3f8db9d465f9,79d86967869366e7] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:33:49.235 WARN 2981023 --- [ocp-schedules-14,2f6b3f8db9d465f9,79d86967869366e7] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:33:49.491 INFO 2981023 --- [ocp-async-1,f6887dd095bb4cfb,70260950e61afcd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:33:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:33:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,551270e2ea8999ac,0d6820725b4890f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,551270e2ea8999ac,0d6820725b4890f8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:50.390 INFO 2981023 --- [ocp-schedules-28,275008ead6d7f0e4,1003dc5e1eeb292a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:33:51.465 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:51.466 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:51.491 INFO 2981023 --- [ocp-async-5,9ffe9a72acd3e9f5,31984d6a9c197378] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:51.657 INFO 2981023 --- [ocp-schedules-22,a21c1375bbb3cf86,44b1c0e4e2d28c99] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:33:51.658 INFO 2981023 --- [ocp-schedules-22,a21c1375bbb3cf86,44b1c0e4e2d28c99] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:33:51.659 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:33:51.659 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:33:51.659 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:33:51.659 INFO 2981023 --- [ocp-schedules-22,a21c1375bbb3cf86,44b1c0e4e2d28c99] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:33:52.000 INFO 2981023 --- [host-check-clockdiff4,598fc08c0e8e72d4,a7f8fccba7c6d843] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050831 0 0 , error= 2024-09-11 18:33:52.950 INFO 2981023 --- [ocp-schedules-3,8da4a65aba5ee2a2,17970f48feb4ac67] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:33:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,a1064998fc01d831,50ead526b3edf2e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,a1064998fc01d831,50ead526b3edf2e4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:53.424 INFO 2981023 --- [ocp-schedules-20,241fe3082c585d05,582e53b2d5890f49] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:33:53.424 INFO 2981023 --- [ocp-schedules-20,241fe3082c585d05,582e53b2d5890f49] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:33:53.491 INFO 2981023 --- [ocp-async-7,4c9ae1ed34c9c31c,9969a47477db0287] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:53.957 INFO 2981023 --- [ocp-schedules-25,dc0a4f0f65ae2ce4,ab460c149abbbff1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:33:54.007 WARN 2981023 --- [scanner-13691333-1,dc0a4f0f65ae2ce4,fb3c0975575a1023] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:33:54.042 INFO 2981023 --- [alarm-detect-12,dc0a4f0f65ae2ce4,f315205db18e57ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646686, data is empty. 2024-09-11 18:33:54.048 INFO 2981023 --- [alarm-detect-12,dc0a4f0f65ae2ce4,f315205db18e57ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646686, data is empty. 2024-09-11 18:33:54.053 INFO 2981023 --- [alarm-detect-12,dc0a4f0f65ae2ce4,f315205db18e57ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646686, data is empty. 2024-09-11 18:33:54.062 INFO 2981023 --- [alarm-detect-3,dc0a4f0f65ae2ce4,476fd06c44f7fdfc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646686, data is empty. 2024-09-11 18:33:54.065 INFO 2981023 --- [alarm-detect-15,dc0a4f0f65ae2ce4,5a11a43a440b97cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646686, data is empty. 2024-09-11 18:33:54.066 INFO 2981023 --- [alarm-detect-3,dc0a4f0f65ae2ce4,476fd06c44f7fdfc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646686, data is empty. 2024-09-11 18:33:54.068 INFO 2981023 --- [alarm-detect-15,dc0a4f0f65ae2ce4,5a11a43a440b97cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646686, data is empty. 2024-09-11 18:33:54.071 INFO 2981023 --- [alarm-detect-3,dc0a4f0f65ae2ce4,476fd06c44f7fdfc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646686, data is empty. 2024-09-11 18:33:54.072 INFO 2981023 --- [alarm-detect-15,dc0a4f0f65ae2ce4,5a11a43a440b97cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646686, data is empty. 2024-09-11 18:33:54.104 INFO 2981023 --- [alarm-detect-14,dc0a4f0f65ae2ce4,dd62f70cc20aa615] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050795 ~ 1726050824 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050810, value=1.54)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.54, values={1726050810=1.54}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:33:54.215 INFO 2981023 --- [alarm-detect-17,dc0a4f0f65ae2ce4,af3826ff86abdd6f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646686, data is empty. 2024-09-11 18:33:54.219 INFO 2981023 --- [alarm-detect-17,dc0a4f0f65ae2ce4,af3826ff86abdd6f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646686, data is empty. 2024-09-11 18:33:54.224 INFO 2981023 --- [alarm-detect-17,dc0a4f0f65ae2ce4,af3826ff86abdd6f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646686, data is empty. 2024-09-11 18:33:54.466 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:54.467 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:54.544 INFO 2981023 --- [ocp-schedules-24,c1be064c34aa0607,263ab91489ebe632] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:33:55.029 INFO 2981023 --- [ocp-schedules-23,35cb8ed1856f1752,60dd54b3a3d73bfb] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T18:33:55.029+08:00, tenantSize=6 2024-09-11 18:33:55.033 INFO 2981023 --- [ocp-schedules-23,35cb8ed1856f1752,60dd54b3a3d73bfb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726050835029000 2024-09-11 18:33:55.033 INFO 2981023 --- [ocp-schedules-23,35cb8ed1856f1752,60dd54b3a3d73bfb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726050535000000 2024-09-11 18:33:55.064 INFO 2981023 --- [ocp-schedules-23,35cb8ed1856f1752,60dd54b3a3d73bfb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 18:33:55.193 INFO 2981023 --- [ocp-schedules-21,91299ac119e17935,eadd59442c4b91f4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:33:55.193 INFO 2981023 --- [ocp-schedules-21,91299ac119e17935,eadd59442c4b91f4] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:33:55.199 INFO 2981023 --- [ocp-schedules-21,91299ac119e17935,eadd59442c4b91f4] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:33:55.199 INFO 2981023 --- [ocp-schedules-21,91299ac119e17935,eadd59442c4b91f4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:33:55.199 INFO 2981023 --- [ocp-schedules-21,91299ac119e17935,eadd59442c4b91f4] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:33:55.210 INFO 2981023 --- [ocp-schedules-21,91299ac119e17935,eadd59442c4b91f4] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:33:55.210 INFO 2981023 --- [ocp-schedules-21,91299ac119e17935,eadd59442c4b91f4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:33:55.210 INFO 2981023 --- [ocp-schedules-21,91299ac119e17935,eadd59442c4b91f4] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:33:55.221 INFO 2981023 --- [ocp-schedules-21,91299ac119e17935,eadd59442c4b91f4] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:33:55.491 INFO 2981023 --- [ocp-async-10,118ffaaed57b3b91,697a09eec2e3515b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:55.770 INFO 2981023 --- [ocp-schedules-12,3032f879da657b3f,a2f193f871605021] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:33:55.789 WARN 2981023 --- [ocp-schedules-12,3032f879da657b3f,a2f193f871605021] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:33:55.945 INFO 2981023 --- [ocp-schedules-3,1d34016630e868b1,e3243e60f95e8065] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:33:55.945 INFO 2981023 --- [ocp-schedules-3,1d34016630e868b1,e3243e60f95e8065] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:33:55.950 INFO 2981023 --- [ocp-schedules-3,1d34016630e868b1,e3243e60f95e8065] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:33:55.950 INFO 2981023 --- [ocp-schedules-3,1d34016630e868b1,e3243e60f95e8065] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:33:55.954 INFO 2981023 --- [ocp-schedules-3,1d34016630e868b1,e3243e60f95e8065] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:33:55.954 INFO 2981023 --- [ocp-schedules-3,1d34016630e868b1,e3243e60f95e8065] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:33:56.193 INFO 2981023 --- [ocp-schedules-14,12c4f0325d4eee6f,bfec69a2e920a1dd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:33:56.199 INFO 2981023 --- [ocp-schedules-14,12c4f0325d4eee6f,bfec69a2e920a1dd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:33:56.199 INFO 2981023 --- [ocp-schedules-14,12c4f0325d4eee6f,bfec69a2e920a1dd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:33:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,24c3f5b6cbae50ac,6ebdd097e4ae2c1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:56.315 INFO 2981023 --- [ocp-schedules-20,12ac475aeeac34a5,410c68575d61e0e5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:33:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,24c3f5b6cbae50ac,6ebdd097e4ae2c1b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:56.333 INFO 2981023 --- [ocp-schedules-20,12ac475aeeac34a5,410c68575d61e0e5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:33:56.333 INFO 2981023 --- [ocp-schedules-20,12ac475aeeac34a5,410c68575d61e0e5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:33:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:33:56.352353+08:00, lastProcessTime=2024-09-11T18:33:46.362084+08:00 2024-09-11 18:33:56.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:33:56.352353+08:00 2024-09-11 18:33:57.467 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:33:57.468 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:33:57.491 INFO 2981023 --- [ocp-async-3,d2ac7431000f03b0,cc951c007ef1feae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:33:57.801 INFO 2981023 --- [ocp-async-2,c02f7c220ecddc76,fb3cb6a526f47d95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:33:59.294 INFO 2981023 --- [ocp-schedules-30,d029702d05872d9f,de9062c54280b341] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:33:59.294 INFO 2981023 --- [ocp-schedules-30,d029702d05872d9f,de9062c54280b341] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:33:59.310 INFO 2981023 --- [Collect all tenants major compactions1,d029702d05872d9f,da24f05cdf1507ba] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:33:59.311 INFO 2981023 --- [Collect all tenants major compactions2,d029702d05872d9f,55386fe6fe251024] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:33:59.311 INFO 2981023 --- [Collect all tenants major compactions6,d029702d05872d9f,175b08bed984417a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:33:59.312 INFO 2981023 --- [Collect all tenants major compactions3,d029702d05872d9f,a77904343987f903] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:33:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,20ceacf4625b8ebf,438e901388cc0ce2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050840, interval=60 2024-09-11 18:33:59.321 INFO 2981023 --- [Collect all tenants major compactions5,d029702d05872d9f,758af15f421bc5b6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:33:59.323 INFO 2981023 --- [Collect all tenants major compactions4,d029702d05872d9f,a12837b6202d1f04] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:33:59.337 INFO 2981023 --- [prometheus-minute-schedule-1,20ceacf4625b8ebf,438e901388cc0ce2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:33:59.343 INFO 2981023 --- [Collect all tenants major compactions5,d029702d05872d9f,758af15f421bc5b6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:59.345 INFO 2981023 --- [Collect all tenants major compactions5,d029702d05872d9f,758af15f421bc5b6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:33:59.347 INFO 2981023 --- [Collect all tenants major compactions2,d029702d05872d9f,55386fe6fe251024] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:59.347 INFO 2981023 --- [Collect all tenants major compactions1,d029702d05872d9f,da24f05cdf1507ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:59.348 INFO 2981023 --- [Collect all tenants major compactions1,d029702d05872d9f,da24f05cdf1507ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:33:59.350 INFO 2981023 --- [Collect all tenants major compactions2,d029702d05872d9f,55386fe6fe251024] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:33:59.350 INFO 2981023 --- [Collect all tenants major compactions3,d029702d05872d9f,a77904343987f903] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:59.351 INFO 2981023 --- [Collect all tenants major compactions1,d029702d05872d9f,da24f05cdf1507ba] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:33:59.352 INFO 2981023 --- [Collect all tenants major compactions5,d029702d05872d9f,758af15f421bc5b6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:33:59.355 INFO 2981023 --- [Collect all tenants major compactions2,d029702d05872d9f,55386fe6fe251024] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:33:59.355 INFO 2981023 --- [Collect all tenants major compactions3,d029702d05872d9f,a77904343987f903] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:33:59.356 INFO 2981023 --- [Collect all tenants major compactions1,d029702d05872d9f,da24f05cdf1507ba] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:33:59.357 INFO 2981023 --- [Collect all tenants major compactions6,d029702d05872d9f,175b08bed984417a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:59.357 INFO 2981023 --- [Collect all tenants major compactions3,d029702d05872d9f,a77904343987f903] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:33:59.357 INFO 2981023 --- [Collect all tenants major compactions3,d029702d05872d9f,a77904343987f903] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:33:59.357 INFO 2981023 --- [Collect all tenants major compactions4,d029702d05872d9f,a12837b6202d1f04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:33:59.358 INFO 2981023 --- [Collect all tenants major compactions6,d029702d05872d9f,175b08bed984417a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:33:59.358 INFO 2981023 --- [Collect all tenants major compactions5,d029702d05872d9f,758af15f421bc5b6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:33:59.358 INFO 2981023 --- [Collect all tenants major compactions4,d029702d05872d9f,a12837b6202d1f04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:33:59.359 INFO 2981023 --- [Collect all tenants major compactions2,d029702d05872d9f,55386fe6fe251024] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:33:59.360 INFO 2981023 --- [Collect all tenants major compactions6,d029702d05872d9f,175b08bed984417a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:33:59.360 INFO 2981023 --- [Collect all tenants major compactions6,d029702d05872d9f,175b08bed984417a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:33:59.360 INFO 2981023 --- [Collect all tenants major compactions4,d029702d05872d9f,a12837b6202d1f04] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:33:59.360 INFO 2981023 --- [Collect all tenants major compactions4,d029702d05872d9f,a12837b6202d1f04] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:33:59.361 INFO 2981023 --- [ocp-schedules-30,d029702d05872d9f,de9062c54280b341] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:33:59.491 INFO 2981023 --- [ocp-async-7,cde40eedcb45d190,98fd9f905be4d610] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:00.468 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:00.468 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:01.491 INFO 2981023 --- [ocp-async-10,bd6ef4507d380246,f65d623e52b86b2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:01.958 INFO 2981023 --- [ocp-schedules-22,7bcb25d91ed0b13b,f9ef869d21d674ac] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:34:01.958+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:34:01.959 INFO 2981023 --- [ocp-schedules-22,7bcb25d91ed0b13b,f9ef869d21d674ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:34:02.116 INFO 2981023 --- [ocp-schedules-22,7bcb25d91ed0b13b,f9ef869d21d674ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 46 to SqlAggregateData 2024-09-11 18:34:02.117 INFO 2981023 --- [ocp-schedules-22,7bcb25d91ed0b13b,f9ef869d21d674ac] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:34:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,889f88219dc45580,26848a854ca7abba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,889f88219dc45580,26848a854ca7abba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:02.353 INFO 2981023 --- [ocp-schedules-30,dd75a8e899285158,9d4be6bf36532610] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:34:02.354 INFO 2981023 --- [ocp-schedules-30,dd75a8e899285158,9d4be6bf36532610] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:34:02.953 INFO 2981023 --- [ocp-schedules-27,666417b8a3e5e714,212628d613c0f238] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:34:03.386 INFO 2981023 --- [ocp-schedules-23,d4e120cdd4a352cc,03cd1fda7371fca3] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:34:03.387 INFO 2981023 --- [ocp-schedules-23,d4e120cdd4a352cc,03cd1fda7371fca3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:34:03.409 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:03.410 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:34:03.417 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:03.419 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:34:03.420 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050843420000} 2024-09-11 18:34:03.425 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:34:03.429 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000241, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:34:03.429 INFO 2981023 --- [ocp-schedules-23,d4e120cdd4a352cc,03cd1fda7371fca3] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:34:03.468 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:03.469 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:03.491 INFO 2981023 --- [ocp-async-5,a501f395f3974581,df815a60dbbcf6c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:04.170 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:34:04.170 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:34:04.170 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:34:04.173 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000241, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:34:04.175 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:34:04.276 INFO 2981023 --- [ocp-schedules-3,7818a74fa38b9d6a,43f1e80f313f9ae4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:34:04.376 WARN 2981023 --- [scanner-13691664-1,7818a74fa38b9d6a,a9275e24d2d971af] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:34:04.490 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000241 2024-09-11 18:34:05.034 INFO 2981023 --- [ocp-schedules-19,4f388186b4090337,88ed177faf457371] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:34:05.035 INFO 2981023 --- [ocp-schedules-19,4f388186b4090337,88ed177faf457371] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:34:05.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:34:05.066 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.068 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:34:05.068 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.069 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:34:05.071 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:34:05.075 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.075 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.075 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.076 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.080 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.080 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.085 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:34:05.086 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.087 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.092 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.092 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:34:05.098 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:34:05.098 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.098 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:34:05.101 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.101 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.104 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.105 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:34:05.112 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.112 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:34:05.113 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.113 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:34:05.114 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.114 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.114 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:34:05.115 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:34:05.116 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.116 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:34:05.117 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.117 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:34:05.118 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.118 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:34:05.119 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.119 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:34:05.119 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.120 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:34:05.120 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.121 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:34:05.121 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:34:05.121 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.122 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:34:05.122 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.123 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:34:05.123 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.124 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:34:05.124 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:34:05.126 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:34:05.136 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:34:05.141 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:05.141 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:05.142 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.142 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.142 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:34:05.142 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.147 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:34:05.150 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:34:05.150 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.150 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:34:05.154 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:34:05.155 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:34:05.156 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:34:05.156 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:34:05.156 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:34:05.156 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:34:05.156 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.157 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.161 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:34:05.162 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:34:05.162 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:34:05.162 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:34:05.165 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.168 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.186 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.187 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.196 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.197 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.197 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:34:05.197 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE70C1D6E5-0-0] 2024-09-11 18:34:05.197 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:34:05.198 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.198 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:34:05.208 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.208 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:34:05.220 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.221 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:34:05.221 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:34:05.222 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE70C1D6EF-0-0] 2024-09-11 18:34:05.222 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:34:05.228 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.228 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:34:05.229 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.229 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:34:05.229 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.230 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:34:05.231 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.231 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:34:05.232 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.232 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:34:05.233 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.233 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:34:05.234 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:34:05.236 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:34:05.247 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:05.248 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.248 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:34:05.258 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:34:05.260 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:34:05.261 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.261 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:34:05.276 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:34:05.278 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:34:05.278 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:34:05.278 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:34:05.278 INFO 2981023 --- [ocp-schedules-19,4f388186b4090337,88ed177faf457371] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:34:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f9cbfe4b9d1d6f89,da7877d948190688] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,f9cbfe4b9d1d6f89,da7877d948190688] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:05.492 INFO 2981023 --- [ocp-async-4,b1a4efa1b62094ba,8938b9d79fc16f14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:34:05.971 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:05.971 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:34:05.971 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:34:05.986 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:34:05.991 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:34:05.996 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:34:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:34:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:34:06.000 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:06.000 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:34:06.001 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:06.002 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:06.070 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:31:41, start=2024-09-11T18:27:41, end=2024-09-11T18:30:11, updates=293, granule=120000ms, delay=240000ms 2024-09-11 18:34:06.079 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:31:41, target=2024-09-11T18:30:11, granule=120000ms, delay=90000ms) 2024-09-11 18:34:06.079 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:30:11, target=2024-09-11T18:20:51, granule=600000ms, delay=560000ms) 2024-09-11 18:34:06.079 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=108ms, period=120000ms 2024-09-11 18:34:06.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:34:06.337504+08:00, lastProcessTime=2024-09-11T18:33:56.352353+08:00 2024-09-11 18:34:06.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:34:06.337504+08:00 2024-09-11 18:34:06.469 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:06.469 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:07.492 INFO 2981023 --- [ocp-async-9,face2d9a5f7ab5cb,a8ac63d00fb5263a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:07.801 INFO 2981023 --- [ocp-async-5,37809f1f011edad5,016d36cd82fb4227] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:34:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,7856acd477a55cba,a74b4a7740e25141] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,7856acd477a55cba,a74b4a7740e25141] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:08.335 INFO 2981023 --- [ocp-schedules-22,d383704bfd59b1ad,cb3a4d2d1715d7b7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:34:08.407 INFO 2981023 --- [ocp-schedules-25,deba327eecd664a5,243c12d9f7b64497] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:34:08.407+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:34:08.408 INFO 2981023 --- [ocp-schedules-25,deba327eecd664a5,243c12d9f7b64497] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:34:08.409 INFO 2981023 --- [ocp-schedules-25,deba327eecd664a5,243c12d9f7b64497] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:34:08.409 INFO 2981023 --- [ocp-schedules-25,deba327eecd664a5,243c12d9f7b64497] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:34:09.469 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:09.470 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:09.492 INFO 2981023 --- [ocp-async-3,6be1ed8e8b33a135,73c68975a26d0217] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:10.002 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:34:10.002 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:34:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:34:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:34:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,7cae5b7b5f443a40,d68c387857bf00eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,7cae5b7b5f443a40,d68c387857bf00eb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:11.492 INFO 2981023 --- [ocp-async-2,0f30185e5079d186,b99a188982324321] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:12.470 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:12.470 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:12.586 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050780, nodeCount=1000, affectRows=1000 2024-09-11 18:34:12.589 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050780, nodeCount=1000, validCount=1000 2024-09-11 18:34:12.596 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050780, nodeCount=49, affectRows=49 2024-09-11 18:34:12.596 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050780, nodeCount=47, validCount=47 2024-09-11 18:34:12.602 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050780, nodeCount=1000, affectRows=1000 2024-09-11 18:34:12.604 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050780, nodeCount=999, validCount=999 2024-09-11 18:34:12.774 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050780, nodeCount=480, affectRows=480 2024-09-11 18:34:12.956 INFO 2981023 --- [ocp-schedules-5,9b431ecd0d1fa7b4,3191b044e4642a23] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:34:13.492 INFO 2981023 --- [ocp-async-1,fe2451d5c9e57cfd,1294171a01d7dc51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,242c199667677bd1,eeb59488c424074a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,5e7e975b9dacc4e7,d1cfc566207f5275] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:34:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,242c199667677bd1,eeb59488c424074a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:14.490 INFO 2981023 --- [ocp-schedules-7,ae35763b45524cd1,3d6b32ad87378756] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:34:14.499 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050780, nodeCount=480, validCount=480 2024-09-11 18:34:14.516 WARN 2981023 --- [scanner-13693351-1,ae35763b45524cd1,b1bb8d78e6000c64] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:34:14.564 INFO 2981023 --- [alarm-detect-13,ae35763b45524cd1,e04548e4fea619cd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646686, data is empty. 2024-09-11 18:34:14.568 INFO 2981023 --- [alarm-detect-13,ae35763b45524cd1,e04548e4fea619cd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646732, data is empty. 2024-09-11 18:34:14.571 INFO 2981023 --- [alarm-detect-10,ae35763b45524cd1,3c0bdbc5b58b6174] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646686, data is empty. 2024-09-11 18:34:14.575 INFO 2981023 --- [alarm-detect-13,ae35763b45524cd1,e04548e4fea619cd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646732, data is empty. 2024-09-11 18:34:14.576 INFO 2981023 --- [alarm-detect-12,ae35763b45524cd1,7e328ec48aa8a81e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646686, data is empty. 2024-09-11 18:34:14.577 INFO 2981023 --- [alarm-detect-10,ae35763b45524cd1,3c0bdbc5b58b6174] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646732, data is empty. 2024-09-11 18:34:14.579 INFO 2981023 --- [alarm-detect-12,ae35763b45524cd1,7e328ec48aa8a81e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646732, data is empty. 2024-09-11 18:34:14.586 INFO 2981023 --- [alarm-detect-10,ae35763b45524cd1,3c0bdbc5b58b6174] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646732, data is empty. 2024-09-11 18:34:14.588 INFO 2981023 --- [alarm-detect-12,ae35763b45524cd1,7e328ec48aa8a81e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646732, data is empty. 2024-09-11 18:34:14.711 INFO 2981023 --- [alarm-detect-7,ae35763b45524cd1,888020927bd87abe] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646686, data is empty. 2024-09-11 18:34:14.716 INFO 2981023 --- [alarm-detect-7,ae35763b45524cd1,888020927bd87abe] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646732, data is empty. 2024-09-11 18:34:14.721 INFO 2981023 --- [alarm-detect-7,ae35763b45524cd1,888020927bd87abe] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646732, data is empty. 2024-09-11 18:34:15.077 INFO 2981023 --- [ocp-schedules-3,4164d8a1d9215147,f65c272ae58b3f17] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:34:15.471 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:15.471 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:15.492 INFO 2981023 --- [ocp-async-8,0752a6da60daebe4,84bfd4f66baa7902] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:15.998 INFO 2981023 --- [ocp-schedules-17,a1f0be7901ce2442,6dc546364040e1a0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:34:15.998 INFO 2981023 --- [host-check-clockdiff1,a1f0be7901ce2442,d0eaf8cd089fa541] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:34:15.998 INFO 2981023 --- [host-check-clockdiff2,a1f0be7901ce2442,11d975d3abba1ae7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:34:15.998 INFO 2981023 --- [host-check-clockdiff4,a1f0be7901ce2442,d00822f8b2b92bf4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:34:15.999 INFO 2981023 --- [host-check-clockdiff3,a1f0be7901ce2442,7eba08a2f59fb8a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:34:16.004 INFO 2981023 --- [ocp-schedules-20,7e0e979c47b543d4,52b1d37754c47f8a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:34:16.006 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:34:16.006 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:34:16.006 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:34:16.006 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:34:16.016 INFO 2981023 --- [ocp-schedules-22,9dc22fd1beb052d3,9c3259f5f4cec1f6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:34:16.017 INFO 2981023 --- [host-check-agent-status1,9dc22fd1beb052d3,1096d7c1b2090f9a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:34:16.017 INFO 2981023 --- [host-check-agent-status2,9dc22fd1beb052d3,983a5b61189968ce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:34:16.017 INFO 2981023 --- [host-check-agent-status3,9dc22fd1beb052d3,08c94e4460a52b73] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:34:16.017 INFO 2981023 --- [host-check-agent-status4,9dc22fd1beb052d3,4304d79e28ff4048] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:34:16.036 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:34:16.039 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:34:16.041 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:16.043 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:16.044 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:16.045 INFO 2981023 --- [host-check-agent-status3,9dc22fd1beb052d3,08c94e4460a52b73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:16.045 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:16.046 INFO 2981023 --- [host-check-agent-status3,9dc22fd1beb052d3,08c94e4460a52b73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:16.046 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:34:16.048 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:34:16.051 INFO 2981023 --- [host-check-agent-status1,9dc22fd1beb052d3,1096d7c1b2090f9a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:16.057 INFO 2981023 --- [host-check-agent-status2,9dc22fd1beb052d3,983a5b61189968ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:34:16.066 INFO 2981023 --- [host-check-agent-status1,9dc22fd1beb052d3,1096d7c1b2090f9a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:16.066 INFO 2981023 --- [host-check-agent-status2,9dc22fd1beb052d3,983a5b61189968ce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:34:16.068 INFO 2981023 --- [host-check-agent-status4,9dc22fd1beb052d3,4304d79e28ff4048] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:34:16.180 INFO 2981023 --- [host-check-agent-status4,9dc22fd1beb052d3,4304d79e28ff4048] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:34:16.193 INFO 2981023 --- [ocp-schedules-7,7b30f52a130700f5,aee4e0614fc2843e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:34:16.193 INFO 2981023 --- [ocp-schedules-10,727e5c40b91cc792,b7191fcee7a39ed2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:34:16.200 INFO 2981023 --- [ocp-schedules-14,49315df8c3ab8f53,378784b587bc5a5a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:34:16.202 INFO 2981023 --- [ocp-schedules-2,f4e28e3283e913cd,ede8a58e3c6967e0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:34:16.202 INFO 2981023 --- [ocp-schedules-10,727e5c40b91cc792,b7191fcee7a39ed2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:34:16.208 INFO 2981023 --- [ocp-schedules-14,49315df8c3ab8f53,378784b587bc5a5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:34:16.210 INFO 2981023 --- [ocp-schedules-7,7b30f52a130700f5,aee4e0614fc2843e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:34:16.243 INFO 2981023 --- [ocp-schedules-10,727e5c40b91cc792,b7191fcee7a39ed2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:34:16.256 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050843420000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:34:16.278 INFO 2981023 --- [ocp-schedules-16,0c625485ec2c63e5,8d522c98b6d0f132] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:34:16.294 INFO 2981023 --- [host-check-agent-status3,9dc22fd1beb052d3,08c94e4460a52b73] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:34:16.296 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:34:16.300 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:34:16.299530859+08:00 2024-09-11 18:34:16.301 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:34:16.300+08:00 2024-09-11 18:34:16.302 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:34:16.303 INFO 2981023 --- [host-check-agent-status4,9dc22fd1beb052d3,4304d79e28ff4048] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:34:16.303 INFO 2981023 --- [host-check-agent-status2,9dc22fd1beb052d3,983a5b61189968ce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:34:16.303 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:34:16.304 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:34:16.304226652+08:00 2024-09-11 18:34:16.305 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:34:16.310571095+08:00 2024-09-11 18:34:16.305 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:34:16.304+08:00 2024-09-11 18:34:16.306 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:34:16.300+08:00 and status AVAILABLE 2024-09-11 18:34:16.306 INFO 2981023 --- [host-check-agent-status2,9dc22fd1beb052d3,983a5b61189968ce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:34:16.306 INFO 2981023 --- [host-check-agent-status2,9dc22fd1beb052d3,983a5b61189968ce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:34:16.306 INFO 2981023 --- [host-check-agent-status2,9dc22fd1beb052d3,983a5b61189968ce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:34:16.307 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:34:16.304+08:00 and status AVAILABLE 2024-09-11 18:34:16.308 INFO 2981023 --- [host-check-agent-status3,9dc22fd1beb052d3,08c94e4460a52b73] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:34:16.308 INFO 2981023 --- [host-check-agent-status3,9dc22fd1beb052d3,08c94e4460a52b73] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:34:16.308 INFO 2981023 --- [host-check-agent-status3,9dc22fd1beb052d3,08c94e4460a52b73] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:34:16.314 INFO 2981023 --- [host-check-agent-status4,9dc22fd1beb052d3,4304d79e28ff4048] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:34:16.314 INFO 2981023 --- [host-check-agent-status4,9dc22fd1beb052d3,4304d79e28ff4048] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:34:16.314 INFO 2981023 --- [host-check-agent-status4,9dc22fd1beb052d3,4304d79e28ff4048] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:34:16.315 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:34:16.305+08:00 2024-09-11 18:34:16.316 INFO 2981023 --- [host-check-agent-status1,9dc22fd1beb052d3,1096d7c1b2090f9a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:34:16.317 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.317 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:34:16.305+08:00 and status AVAILABLE 2024-09-11 18:34:16.317 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:34:16.318 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:34:16.319 INFO 2981023 --- [host-check-agent-status1,9dc22fd1beb052d3,1096d7c1b2090f9a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:34:16.319 INFO 2981023 --- [host-check-agent-status1,9dc22fd1beb052d3,1096d7c1b2090f9a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:34:16.319 INFO 2981023 --- [host-check-agent-status1,9dc22fd1beb052d3,1096d7c1b2090f9a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:34:16.320 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:34:16.321 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:34:16.320820750+08:00 2024-09-11 18:34:16.322 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:34:16.321+08:00 2024-09-11 18:34:16.324 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:34:16.321+08:00 and status AVAILABLE 2024-09-11 18:34:16.330 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.331 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.331 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:34:16.332 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:34:16.332 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.332 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:16.332 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:16.332 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:34:16.333 INFO 2981023 --- [ocp-schedules-7,7b30f52a130700f5,aee4e0614fc2843e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:34:16.335 INFO 2981023 --- [ocp-schedules-19,b37607cb536f1659,dd939ba1413d75f7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:34:16.343 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:34:16.363 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:34:16.362609+08:00, lastProcessTime=2024-09-11T18:34:06.337504+08:00 2024-09-11 18:34:16.364 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:34:16.366 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.367 INFO 2981023 --- [ocp-schedules-19,b37607cb536f1659,dd939ba1413d75f7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:34:16.367 INFO 2981023 --- [ocp-schedules-19,b37607cb536f1659,dd939ba1413d75f7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:34:16.367 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:16.368 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:34:16.370 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:16.371 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:16.372 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:16.373 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:16.377 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:34:16.378 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:34:16.378 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:34:16.381 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:34:16.400 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:34:16.401 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:34:16.402 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:16.404 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.404 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:34:16.410 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:34:16.412 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:34:16.427 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:34:16.428 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:34:16.433 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:34:16.441 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:34:16.441 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050843420000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:34:16.441 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:34:16.443 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:34:16.448 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:34:16.448 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:16.451 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:34:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:16.452 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:34:16.452 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.455 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:34:16.456 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:34:16.459 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:34:16.474 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:34:16.474 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:34:16.487 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:34:16.489 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:34:16.491 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:34:16.492 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:34:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:34:16.512 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:34:16.516 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:34:16.519 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:34:16.519 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:34:16.520 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:16.522 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.524 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:34:16.524 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:34:16.525 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:34:16.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:34:16.529 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:34:16.530 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:34:16.530 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:16.532 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.532 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:34:16.533 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:34:16.537 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:16.538 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:16.554 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:34:16.558 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:34:16.560 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:34:16.563 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:34:16.568 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:34:16.578 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:34:16.587 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:34:16.588 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:16.589 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.589 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:16.592 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:34:16.592 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.595 INFO 2981023 --- [host-check-host-status2,7e0e979c47b543d4,d42d144c489fb557] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:34:16.595 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:34:16.596 INFO 2981023 --- [host-check-host-status4,7e0e979c47b543d4,0d0bdbb1df491e6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:34:16.596 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:34:16.596 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:34:16.604 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:34:16.610 INFO 2981023 --- [host-check-host-status1,7e0e979c47b543d4,45b1ff57cb4d54c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:34:16.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:34:16.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:34:16.624 INFO 2981023 --- [host-check-host-status3,7e0e979c47b543d4,0a048e52414d4e6d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:34:16.646 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:34:16.656 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:16.656 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.656 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:34:16.657 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:34:16.657 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:34:16.661 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:34:16.664 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:16.664 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:16.668 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.668 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:34:16.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:34:16.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:34:16.679 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:34:16.680 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:34:16.362609+08:00 2024-09-11 18:34:16.709 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:34:16.715 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:16.716 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:16.716 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:34:16.721 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:34:16.721 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:34:16.722 INFO 2981023 --- [ocp-schedules-14,49315df8c3ab8f53,378784b587bc5a5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:34:16.946 INFO 2981023 --- [host-check-clockdiff3,a1f0be7901ce2442,7eba08a2f59fb8a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050856 0 0 , error= 2024-09-11 18:34:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,6339de36e60f3af9,608ecd8aacc06e93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,6339de36e60f3af9,608ecd8aacc06e93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:17.499 INFO 2981023 --- [ocp-async-3,db9be4491efac155,9170b17887b6abb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:17.802 INFO 2981023 --- [ocp-async-9,320b5d55e533d859,82e5e7a19651f4a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:34:18.471 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:18.472 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:19.157 INFO 2981023 --- [host-check-clockdiff2,a1f0be7901ce2442,11d975d3abba1ae7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050859 5 5 , error= 2024-09-11 18:34:19.492 INFO 2981023 --- [ocp-async-7,9ff2f41f81acbba0,1943e1e0e07d5c76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,e06e4c0597c1d82e,3df6361046154483] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,e06e4c0597c1d82e,3df6361046154483] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:20.508 INFO 2981023 --- [host-check-clockdiff1,a1f0be7901ce2442,d0eaf8cd089fa541] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050860 0 0 , error= 2024-09-11 18:34:21.472 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:21.474 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:21.493 INFO 2981023 --- [ocp-async-10,a26dc93b1ab1bf50,4dba68303d1dce70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:21.573 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:34:21.573 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000009 2024-09-11 18:34:21.573 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000010 2024-09-11 18:34:21.573 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2000007 2024-09-11 18:34:21.573 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2024-09-11 18:34:21.618 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:34:21.619 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:34:21.624 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:34:21.640 WARN 2981023 --- [ForkJoinPool.commonPool-worker-2,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:34:21.642 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:34:21.648 WARN 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:34:21.657 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:34:21.657 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:34:21.657 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.5 2024-09-11 18:34:21.657 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2024-09-11 18:34:21.660 WARN 2981023 --- [ForkJoinPool.commonPool-worker-3,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:34:21.663 WARN 2981023 --- [ForkJoinPool.commonPool-worker-1,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:34:21.669 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:21.682 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:34:21.682 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:34:21.683 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.7 2024-09-11 18:34:21.683 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000010 agent sql collect property 2024-09-11 18:34:21.683 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:34:21.683 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:34:21.683 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.10 2024-09-11 18:34:21.684 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000009 agent sql collect property 2024-09-11 18:34:21.685 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:34:21.685 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:34:21.685 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.31.35.6 2024-09-11 18:34:21.685 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2000007 agent sql collect property 2024-09-11 18:34:21.687 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:21.692 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2000009 to set sql collect property inactive 2024-09-11 18:34:21.693 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:34:21.693 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:21.695 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:21.696 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:34:21.699 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:34:21.701 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:34:21.718 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1718689631), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.5), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (ob.data.path=/data/oceanbase), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=rtdob), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"in 2024-09-11 18:34:21.721 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994437), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.31.35.10), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=linkprdocp), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"include 2024-09-11 18:34:21.721 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (monagent.ob.err.code.prefix=errcode=-), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.cgroup.status=active), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.run.user=admin), (ob.logcleaner.enabled=true), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.pipeline.awr.collect.disabled=true), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.ob.rpc.port=2882), (monagent.monitordb.user=root@ocp_monitor), (monagent.host.ip=10.31.35.7), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"includeKeywordsRelation":"Or"," 2024-09-11 18:34:21.729 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.run.path=/data/rtdob/oceanbase), (ob.install.path=/data/rtdob/oceanbase), (monagent.ob.err.code.prefix=errcode=-), (ob.data.path=/data/oceanbase), (monagent.cluster.id=1000006), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.cgroup.status=active), (monagent.ob.monitor.password=xxx (ob.log.path=/oceanbase/redo), (ob.run.user=admin), (ob.logcleaner.enabled=true), (monagent.monitordb.host=10.31.35.10), (ob.sync.status=VALID), (monagent.ob.cluster.id=1719994443), (monagent.pipeline.awr.collect.disabled=true), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.cluster.name=linkrtd), (ob.logtailer.enabled=true), (monagent.ob.rpc.port=2882), (monagent.monitordb.user=root@ocp_monitor), (monagent.host.ip=10.31.35.6), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local server.*ret=-4018"],"includeKeywordsRelation":"Or"," 2024-09-11 18:34:21.735 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000009, do nothing 2024-09-11 18:34:21.735 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2024-09-11 18:34:21.739 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000010, do nothing 2024-09-11 18:34:21.743 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2000007, do nothing 2024-09-11 18:34:21.744 INFO 2981023 --- [ocp-schedules-12,5d3b7ba38e41b7b4,59d6221f92b0ceff] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2024-09-11 18:34:21.961 INFO 2981023 --- [host-check-clockdiff4,a1f0be7901ce2442,d00822f8b2b92bf4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050861 0 0 , error= 2024-09-11 18:34:22.959 INFO 2981023 --- [ocp-schedules-5,0f28537f97d89b79,38129a8241b25a60] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:34:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,1e171459f6c6a572,66d2c77c46086654] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,1e171459f6c6a572,66d2c77c46086654] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:23.493 INFO 2981023 --- [ocp-async-4,5cd622a232b973f3,defad8a79627c0cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:24.474 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:24.475 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:24.548 INFO 2981023 --- [ocp-schedules-19,69ac1e89b1efe291,35caba59eab2f3fa] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:34:24.773 INFO 2981023 --- [ocp-schedules-21,e5d944a1a872f956,330dd11b147e9c1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:34:24.827 WARN 2981023 --- [scanner-13693661-1,e5d944a1a872f956,603d44a60f6f56b7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:34:24.898 INFO 2981023 --- [alarm-detect-16,e5d944a1a872f956,8ad17d6e3f15786f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1726050825 ~ 1726050854 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5}}, data=[ValueData(timestamp=1726050840, value=1.612)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.612, values={1726050840=1.612}, valueCount=1, labels={app=HOST, host=10.31.35.5, host_name=vm-eas-prd-rtd-mid03, ob_cluster_id=1718689631, ob_cluster_name=rtdob, obzone=zone1, svr_ip=10.31.35.5})]}, matched=true, isContainValidValue=null, message=null, level=3) 2024-09-11 18:34:25.493 INFO 2981023 --- [ocp-async-2,87319dc24bca41f0,67222b0391852112] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,a742009317d6ac0f,1475573115237f6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,a742009317d6ac0f,1475573115237f6a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:34:26.352770+08:00, lastProcessTime=2024-09-11T18:34:16.362609+08:00 2024-09-11 18:34:26.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:34:26.352770+08:00 2024-09-11 18:34:27.475 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:27.477 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:27.493 INFO 2981023 --- [ocp-async-9,7a6dc3dabd1954e5,907782698f4c232f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:27.802 INFO 2981023 --- [ocp-async-4,39d7745217f653e8,44a591a462b68f2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:34:28.533 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 18:34:28.533 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 18:34:28.533 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.534 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:34:28.535 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.535 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.535 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.536 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 18:34:28.548 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.548 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.548 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.549 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 18:34:28.557 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.557 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.558 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.558 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 18:34:28.567 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.567 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.568 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.568 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 18:34:28.577 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.577 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.578 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.578 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 18:34:28.582 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.582 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.582 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.583 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 18:34:28.585 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.586 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.586 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.586 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 18:34:28.590 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.590 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.590 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.591 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 18:34:28.595 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.595 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.596 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.596 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 18:34:28.599 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.600 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.600 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.600 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 18:34:28.604 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.604 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.604 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.605 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 18:34:28.610 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.611 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.611 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.611 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 18:34:28.620 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.620 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.621 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.621 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 18:34:28.624 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.624 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.625 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.625 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 18:34:28.628 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.629 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.629 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.629 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 18:34:28.633 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.633 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.633 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.634 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 18:34:28.636 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.636 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.637 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.637 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 18:34:28.640 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.640 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.641 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.641 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 18:34:28.644 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.644 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.644 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.645 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 18:34:28.647 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.648 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.648 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.648 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 18:34:28.651 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.651 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.652 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.652 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 18:34:28.655 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.655 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.656 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.656 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 18:34:28.658 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.659 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.659 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.659 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 18:34:28.662 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.662 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.663 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.663 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 18:34:28.666 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.666 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.666 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.667 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 18:34:28.669 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.669 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.670 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.670 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 18:34:28.673 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.673 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.674 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.674 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 18:34:28.677 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.677 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.677 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.677 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 18:34:28.680 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.680 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.681 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.681 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 18:34:28.684 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.684 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.684 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.685 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 18:34:28.687 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.688 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.688 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.688 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 18:34:28.691 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.691 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.692 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.692 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 18:34:28.695 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.695 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.696 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.696 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 18:34:28.699 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.699 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.699 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.700 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 18:34:28.705 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.705 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.705 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.706 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 18:34:28.710 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.710 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.710 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.711 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 18:34:28.714 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.714 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.715 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.715 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 18:34:28.718 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.718 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.719 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.719 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 18:34:28.721 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.721 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.722 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.722 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 18:34:28.724 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.725 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.725 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.725 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 18:34:28.727 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.728 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.728 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.728 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 18:34:28.731 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.731 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.731 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.732 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 18:34:28.738 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.738 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.738 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.739 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 18:34:28.754 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.754 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.754 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.755 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 18:34:28.759 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.759 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.759 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.760 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 18:34:28.763 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.764 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.764 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.764 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 18:34:28.768 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.768 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.769 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.769 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 18:34:28.771 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.772 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.772 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.772 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 18:34:28.775 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.775 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.775 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.776 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 18:34:28.778 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 18:34:28.778 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 18:34:28.779 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.779 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:34:28.781 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.781 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.782 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.782 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 18:34:28.785 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.786 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:34:28.786 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:28.786 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 18:34:28.788 INFO 2981023 --- [ocp-schedules-16,dafcdc60828db69f,59ddecee9c0796a6] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 18:34:29.006 INFO 2981023 --- [ocp-schedules-28,5144d157e3aaa5e5,152351a62dd771bc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:34:29.006 INFO 2981023 --- [ocp-schedules-28,5144d157e3aaa5e5,152351a62dd771bc] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:34:29.006 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:34:29.007 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:34:29.007 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:34:29.027 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:29.032 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:34:29.032 INFO 2981023 --- [ocp-schedules-28,5144d157e3aaa5e5,152351a62dd771bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:29.032 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:29.033 INFO 2981023 --- [ocp-schedules-28,5144d157e3aaa5e5,152351a62dd771bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:29.033 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:34:29.034 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:34:29.034 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:34:29.059 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:34:29.060 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:34:29.061 INFO 2981023 --- [ocp-schedules-28,5144d157e3aaa5e5,152351a62dd771bc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:34:29.061 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:34:29.065 INFO 2981023 --- [ocp-schedules-28,5144d157e3aaa5e5,152351a62dd771bc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:34:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,788a8a5681906e79,3571173479125fbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,788a8a5681906e79,3571173479125fbc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:29.493 INFO 2981023 --- [ocp-async-6,011da95c7ffa237a,6314649556feb838] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:30.477 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:30.477 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:31.493 INFO 2981023 --- [ocp-async-1,9605e41c8dae69e3,de867e204bffc541] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,d6dbfbb1b266a05b,15b96a9d0d4b17fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,d6dbfbb1b266a05b,15b96a9d0d4b17fa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:32.962 INFO 2981023 --- [ocp-schedules-27,398ef33d7d061ad1,b31792d607799e16] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:34:33.477 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:33.478 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:33.493 INFO 2981023 --- [ocp-async-5,234f1b5c6c12320e,0d0ebb4f584e7a77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:34.979 INFO 2981023 --- [ocp-schedules-30,1192ca33f0270294,a8ba6e288761fd05] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:34:35.021 WARN 2981023 --- [scanner-13693902-1,1192ca33f0270294,79859e81a519d855] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:34:35.033 INFO 2981023 --- [alarm-detect-13,1192ca33f0270294,9c4c9aef9716eb6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646732, data is empty. 2024-09-11 18:34:35.036 INFO 2981023 --- [alarm-detect-13,1192ca33f0270294,9c4c9aef9716eb6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646732, data is empty. 2024-09-11 18:34:35.043 INFO 2981023 --- [alarm-detect-13,1192ca33f0270294,9c4c9aef9716eb6d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646732, data is empty. 2024-09-11 18:34:35.072 INFO 2981023 --- [alarm-detect-2,1192ca33f0270294,c3995f4def22cad2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646732, data is empty. 2024-09-11 18:34:35.078 INFO 2981023 --- [alarm-detect-4,1192ca33f0270294,5d47e36b80364908] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646732, data is empty. 2024-09-11 18:34:35.080 INFO 2981023 --- [alarm-detect-2,1192ca33f0270294,c3995f4def22cad2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646732, data is empty. 2024-09-11 18:34:35.082 INFO 2981023 --- [alarm-detect-4,1192ca33f0270294,5d47e36b80364908] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646732, data is empty. 2024-09-11 18:34:35.085 INFO 2981023 --- [alarm-detect-4,1192ca33f0270294,5d47e36b80364908] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646732, data is empty. 2024-09-11 18:34:35.085 INFO 2981023 --- [alarm-detect-2,1192ca33f0270294,c3995f4def22cad2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646732, data is empty. 2024-09-11 18:34:35.131 INFO 2981023 --- [alarm-detect-8,1192ca33f0270294,8f9ae4c99ce55ad5] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.31.35.10-2, alarm_template_id=0:host=10.31.35.10-3, alarm_template_id=0:host=10.31.35.5-3, alarm_template_id=0:host=10.31.35.6-2, alarm_template_id=0:host=10.31.35.5-2, alarm_template_id=0:host=10.31.35.7-3, alarm_template_id=0:host=10.31.35.7-2, alarm_template_id=0:host=10.31.35.6-3] 2024-09-11 18:34:35.221 INFO 2981023 --- [alarm-detect-9,1192ca33f0270294,a1981d142db2d02b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646732, data is empty. 2024-09-11 18:34:35.228 INFO 2981023 --- [alarm-detect-9,1192ca33f0270294,a1981d142db2d02b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646732, data is empty. 2024-09-11 18:34:35.234 INFO 2981023 --- [alarm-detect-9,1192ca33f0270294,a1981d142db2d02b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646732, data is empty. 2024-09-11 18:34:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,b59cbe52ea78242d,2d849ac598c691fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,b59cbe52ea78242d,2d849ac598c691fc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:35.493 INFO 2981023 --- [ocp-async-8,a90acbc0bbe7b209,8222a953043826fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:34:36.338292+08:00, lastProcessTime=2024-09-11T18:34:26.352770+08:00 2024-09-11 18:34:36.370 INFO 2981023 --- [ocp-schedules-12,ade9ff9035733655,d18bc003a87e164d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:34:36.380 INFO 2981023 --- [ocp-schedules-12,ade9ff9035733655,d18bc003a87e164d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:34:36.380 INFO 2981023 --- [ocp-schedules-12,ade9ff9035733655,d18bc003a87e164d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:34:36.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:34:36.467 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:34:36.338292+08:00 2024-09-11 18:34:36.478 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:36.478 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:37.314 INFO 2981023 --- [prometheus-cache-monitor1,2ee9e208996c24a2,3aec68ce45d309cf] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:34:37.315 INFO 2981023 --- [prometheus-cache-monitor1,2ee9e208996c24a2,3aec68ce45d309cf] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:34:37.493 INFO 2981023 --- [ocp-async-3,a8b33269c9440489,c14fcdec1df069b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:37.802 INFO 2981023 --- [ocp-async-5,4c3323f16874c35e,a9b5536660daef09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:34:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,d56dd98412a2d628,f8c2d9a9e14547a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,d56dd98412a2d628,f8c2d9a9e14547a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:39.478 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:39.479 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:39.494 INFO 2981023 --- [ocp-async-9,723ff81ddc71541e,ac3513479612ae7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:40.003 INFO 2981023 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:34:40.003 INFO 2981023 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:34:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,d9c1efdf1d6aa999,cec473cac3c0d9c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,d9c1efdf1d6aa999,cec473cac3c0d9c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:41.494 INFO 2981023 --- [ocp-async-4,c9587fffdc4e66bc,6129ec4052c95e89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:42.479 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:42.479 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:42.964 INFO 2981023 --- [ocp-schedules-5,11e22070e332f5a6,dce62b9abd6ecb3b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:34:43.494 INFO 2981023 --- [ocp-async-2,3f8f034f7cd6f9bb,f41a9768456e8c9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,544f64ce568c4d83,db0ec6e1e3ad3bf1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,544f64ce568c4d83,db0ec6e1e3ad3bf1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:45.101 INFO 2981023 --- [prometheus-second-schedule-1,1444e1b6da21ef48,c3659f1ff2cb4168] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:34:45.287 INFO 2981023 --- [ocp-schedules-8,d03aba79b3deaee7,e794b174f6601200] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:34:45.341 WARN 2981023 --- [scanner-13694214-1,d03aba79b3deaee7,8f1f47c537866a67] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:34:45.479 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:45.480 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:45.494 INFO 2981023 --- [ocp-async-7,616e3f984259585d,2fbae9ec4eccd3eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:46.018 INFO 2981023 --- [ocp-schedules-16,35ec6c4e38718d96,a659ffe433ef0692] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:34:46.019 INFO 2981023 --- [ocp-schedules-10,d70815cc69661318,a127a1dbad2c89c3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:34:46.019 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:34:46.019 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:34:46.019 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:34:46.020 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:34:46.021 INFO 2981023 --- [host-check-clockdiff1,d70815cc69661318,7c00c0eb60115dc7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:34:46.022 INFO 2981023 --- [host-check-clockdiff3,d70815cc69661318,35cf6e95d2b88501] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:34:46.024 INFO 2981023 --- [host-check-clockdiff2,d70815cc69661318,a701282205ccc44f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:34:46.025 INFO 2981023 --- [host-check-clockdiff4,d70815cc69661318,55924ab3d49e7bc4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:34:46.028 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:46.029 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:34:46.032 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:46.033 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:34:46.033 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:34:46.034 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:34:46.036 INFO 2981023 --- [host-check-clockdiff3,d70815cc69661318,35cf6e95d2b88501] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050886 0 0 , error= 2024-09-11 18:34:46.037 INFO 2981023 --- [ocp-schedules-18,468533fecbad3c3e,96ef6084a9b9b080] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:34:46.037 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:46.037 INFO 2981023 --- [host-check-agent-status2,468533fecbad3c3e,56f693bc67ea31b0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:34:46.037 INFO 2981023 --- [host-check-agent-status1,468533fecbad3c3e,72355b9d40267366] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:34:46.037 INFO 2981023 --- [host-check-agent-status3,468533fecbad3c3e,66cf19497d29a1d4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:34:46.038 INFO 2981023 --- [host-check-agent-status4,468533fecbad3c3e,713eccb7f74c452d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:34:46.040 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:46.045 INFO 2981023 --- [host-check-agent-status2,468533fecbad3c3e,56f693bc67ea31b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:34:46.047 INFO 2981023 --- [host-check-agent-status3,468533fecbad3c3e,66cf19497d29a1d4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:46.047 INFO 2981023 --- [host-check-agent-status1,468533fecbad3c3e,72355b9d40267366] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:46.048 INFO 2981023 --- [host-check-agent-status2,468533fecbad3c3e,56f693bc67ea31b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:34:46.048 INFO 2981023 --- [host-check-agent-status1,468533fecbad3c3e,72355b9d40267366] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:46.050 INFO 2981023 --- [host-check-agent-status3,468533fecbad3c3e,66cf19497d29a1d4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:46.050 INFO 2981023 --- [host-check-agent-status4,468533fecbad3c3e,713eccb7f74c452d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:34:46.051 INFO 2981023 --- [host-check-agent-status4,468533fecbad3c3e,713eccb7f74c452d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:34:46.065 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:34:46.065 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:34:46.066 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:34:46.067 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:34:46.067234806+08:00 2024-09-11 18:34:46.068 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:34:46.073197120+08:00 2024-09-11 18:34:46.068 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:34:46.068029982+08:00 2024-09-11 18:34:46.068 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:34:46.068+08:00 2024-09-11 18:34:46.069 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:34:46.068+08:00 2024-09-11 18:34:46.069 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:34:46.068+08:00 2024-09-11 18:34:46.070 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:34:46.071 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:34:46.068+08:00 and status AVAILABLE 2024-09-11 18:34:46.072 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:34:46.072146786+08:00 2024-09-11 18:34:46.073 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:34:46.072+08:00 2024-09-11 18:34:46.075 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:34:46.072+08:00 and status AVAILABLE 2024-09-11 18:34:46.077 INFO 2981023 --- [host-check-agent-status4,468533fecbad3c3e,713eccb7f74c452d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:34:46.080 INFO 2981023 --- [host-check-agent-status4,468533fecbad3c3e,713eccb7f74c452d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:34:46.080 INFO 2981023 --- [host-check-agent-status4,468533fecbad3c3e,713eccb7f74c452d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:34:46.080 INFO 2981023 --- [host-check-agent-status4,468533fecbad3c3e,713eccb7f74c452d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:34:46.081 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:34:46.068+08:00 and status AVAILABLE 2024-09-11 18:34:46.081 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:34:46.068+08:00 and status AVAILABLE 2024-09-11 18:34:46.084 INFO 2981023 --- [host-check-agent-status1,468533fecbad3c3e,72355b9d40267366] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:34:46.087 INFO 2981023 --- [host-check-agent-status1,468533fecbad3c3e,72355b9d40267366] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:34:46.087 INFO 2981023 --- [host-check-agent-status1,468533fecbad3c3e,72355b9d40267366] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:34:46.087 INFO 2981023 --- [host-check-agent-status1,468533fecbad3c3e,72355b9d40267366] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:34:46.090 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:34:46.108 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:46.109 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:46.112 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:34:46.112 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:34:46.119 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:46.120 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:46.122 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:34:46.122 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:34:46.124 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:34:46.125 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:34:46.127 INFO 2981023 --- [host-check-agent-status2,468533fecbad3c3e,56f693bc67ea31b0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:34:46.129 INFO 2981023 --- [host-check-agent-status3,468533fecbad3c3e,66cf19497d29a1d4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:34:46.130 INFO 2981023 --- [host-check-agent-status2,468533fecbad3c3e,56f693bc67ea31b0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:34:46.130 INFO 2981023 --- [host-check-agent-status2,468533fecbad3c3e,56f693bc67ea31b0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:34:46.130 INFO 2981023 --- [host-check-agent-status2,468533fecbad3c3e,56f693bc67ea31b0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:34:46.138 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:34:46.140 INFO 2981023 --- [host-check-agent-status3,468533fecbad3c3e,66cf19497d29a1d4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:34:46.140 INFO 2981023 --- [host-check-agent-status3,468533fecbad3c3e,66cf19497d29a1d4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:34:46.140 INFO 2981023 --- [host-check-agent-status3,468533fecbad3c3e,66cf19497d29a1d4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:34:46.143 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:34:46.150 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:34:46.159 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:34:46.162 INFO 2981023 --- [ocp-schedules-30,d6c522dd79388188,bb588242e6d04268] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:34:46.165 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:34:46.171 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:34:46.172 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:34:46.179 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:34:46.187 INFO 2981023 --- [ocp-schedules-3,9bf050ee18991769,b549eac90bcd61f0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:34:46.187 INFO 2981023 --- [ocp-schedules-11,71faf1d40ce4070c,3227172aea31fcdb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:34:46.187 INFO 2981023 --- [ocp-schedules-9,9c90511668c77773,e9a6f70535551145] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:34:46.188 INFO 2981023 --- [ocp-schedules-24,c1d9bac6878f910e,2e12a3af8d96f945] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:34:46.191 INFO 2981023 --- [ocp-schedules-3,9bf050ee18991769,b549eac90bcd61f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:34:46.192 INFO 2981023 --- [ocp-schedules-30,d6c522dd79388188,bb588242e6d04268] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:34:46.192 INFO 2981023 --- [ocp-schedules-9,9c90511668c77773,e9a6f70535551145] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:34:46.195 INFO 2981023 --- [ocp-schedules-11,71faf1d40ce4070c,3227172aea31fcdb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:34:46.200 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:34:46.205 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050843420000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:34:46.227 INFO 2981023 --- [ocp-schedules-21,33dba46cadbdc695,cc3b09087e3dd14f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:34:46.234 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:34:46.242 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:34:46.250 INFO 2981023 --- [ocp-schedules-11,71faf1d40ce4070c,3227172aea31fcdb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:34:46.264 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:34:46.267 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:34:46.278 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:34:46.284 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:34:46.292 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:34:46.300 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:34:46.307 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:34:46.312 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.313 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:34:46.314 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:34:46.316 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.316 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:34:46.318 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:34:46.322 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:34:46.331 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:34:46.331 INFO 2981023 --- [ocp-schedules-3,9bf050ee18991769,b549eac90bcd61f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:34:46.331 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.332 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:46.333 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:34:46.334 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.334 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:34:46.335 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:34:46.335 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:34:46.340 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.354 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:34:46.355 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:34:46.356 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:34:46.355440+08:00, lastProcessTime=2024-09-11T18:34:36.338292+08:00 2024-09-11 18:34:46.364 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.366 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:46.368 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.368 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:46.381 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:34:46.383 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:34:46.417 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:34:46.419 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:34:46.423 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:34:46.428 INFO 2981023 --- [host-check-host-status1,35ec6c4e38718d96,7b62bf903e531fea] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:34:46.441 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:34:46.444 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:34:46.444 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050843420000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:34:46.450 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.451 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.451 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:34:46.451 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:34:46.452 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.453 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:34:46.460 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:34:46.465 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.466 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:34:46.469 INFO 2981023 --- [host-check-host-status2,35ec6c4e38718d96,12895f4d51ca7b94] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:34:46.478 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:34:46.480 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:34:46.484 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:34:46.491 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:34:46.500 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:34:46.504 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.505 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.506 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:46.506 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:34:46.507 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.508 INFO 2981023 --- [host-check-host-status4,35ec6c4e38718d96,35ccab38183a28bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:34:46.510 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:34:46.511 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:34:46.513 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:34:46.515 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.520 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.521 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.521 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:34:46.524 INFO 2981023 --- [host-check-host-status3,35ec6c4e38718d96,c42c6b2d5dd5c234] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:34:46.524 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.524 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.525 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:34:46.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:34:46.525 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:34:46.526 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:34:46.529 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.530 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:34:46.531 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:34:46.531 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.537 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:34:46.544 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.544 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.544 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:34:46.545 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:34:46.545 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:34:46.545 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.548 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:34:46.548 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:34:46.554 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.555 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.555 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:34:46.556 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:34:46.556 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:34:46.556 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:34:46.556 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:34:46.560 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:34:46.565 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:34:46.566 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:34:46.570 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:34:46.574 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.575 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.575 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:34:46.576 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:34:46.576 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.581 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.581 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.582 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:34:46.582 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:34:46.584 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:34:46.586 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:34:46.589 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:34:46.590 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:34:46.592 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:34:46.611 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:34:46.611 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:34:46.628 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.629 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.629 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:34:46.630 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:34:46.630 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.642 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:34:46.650 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.651 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.651 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:34:46.658 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:34:46.658 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:34:46.669 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:34:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:34:46.676 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:46.677 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:34:46.680 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:34:46.681 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:34:46.355440+08:00 2024-09-11 18:34:46.683 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:34:46.683 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:34:46.684 INFO 2981023 --- [ocp-schedules-9,9c90511668c77773,e9a6f70535551145] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:34:46.694 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:34:46.700 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:34:46.703 INFO 2981023 --- [ocp-schedules-4,c53aab1304886b79,97a3b5e8ac21709b] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:34:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,4da6c6b3d813f500,ee0e33593128a737] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,4da6c6b3d813f500,ee0e33593128a737] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:47.494 INFO 2981023 --- [ocp-async-4,e0337afc646472ae,db7771ab744f770c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:47.802 INFO 2981023 --- [ocp-async-7,06cb93adf38a11dc,b0dbd7b842caf839] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:34:47.813 INFO 2981023 --- [host-check-clockdiff1,d70815cc69661318,7c00c0eb60115dc7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050887 0 0 , error= 2024-09-11 18:34:48.480 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:48.481 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:48.551 INFO 2981023 --- [host-check-clockdiff2,d70815cc69661318,a701282205ccc44f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050888 5 5 , error= 2024-09-11 18:34:49.373 INFO 2981023 --- [ocp-schedules-12,3297ca2bae35416e,8d853bd58a0216b5] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:34:49.392 WARN 2981023 --- [ocp-schedules-12,3297ca2bae35416e,8d853bd58a0216b5] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:34:49.494 INFO 2981023 --- [ocp-async-5,5de72c2511e6b81e,fbed749813270faf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:34:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:34:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,afc04f963208533e,2a570a4322c3c716] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,afc04f963208533e,2a570a4322c3c716] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:50.392 INFO 2981023 --- [ocp-schedules-24,2dc77143a8537575,712f1b259cb9ef9f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:34:51.481 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:51.482 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:51.494 INFO 2981023 --- [ocp-async-9,e15d3746951fcaad,b0994bbf9d656d21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:51.659 INFO 2981023 --- [ocp-schedules-25,720653f97e52769e,694df9fc2c0c5d65] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:34:51.660 INFO 2981023 --- [ocp-schedules-25,720653f97e52769e,694df9fc2c0c5d65] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:34:51.660 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:34:51.660 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:34:51.660 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:34:51.661 INFO 2981023 --- [ocp-schedules-25,720653f97e52769e,694df9fc2c0c5d65] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:34:51.989 INFO 2981023 --- [host-check-clockdiff4,d70815cc69661318,55924ab3d49e7bc4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050891 0 0 , error= 2024-09-11 18:34:52.967 INFO 2981023 --- [ocp-schedules-5,d037dc00825342c6,649e7387c51849c6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:34:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,98fcf2915dd6e8fb,a33c5a78c0d110b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,98fcf2915dd6e8fb,a33c5a78c0d110b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:53.430 INFO 2981023 --- [ocp-schedules-6,8ab994c84b66a8f0,64833ed216253848] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:34:53.430 INFO 2981023 --- [ocp-schedules-6,8ab994c84b66a8f0,64833ed216253848] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:34:53.494 INFO 2981023 --- [ocp-async-6,0f3db5500650f158,ee3cd532d1bdbc98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:54.482 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:54.482 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:54.550 INFO 2981023 --- [ocp-schedules-22,5584688a20c35bdb,63f0c37a2d235dba] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:34:55.495 INFO 2981023 --- [ocp-async-2,e2291fc6a749017e,43d56bb3c6deb096] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:55.575 INFO 2981023 --- [ocp-schedules-5,4c140ed1e0b2073d,8145533645f000c0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:34:55.664 WARN 2981023 --- [scanner-13694431-1,4c140ed1e0b2073d,18267beff877e787] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:34:55.687 INFO 2981023 --- [alarm-detect-15,4c140ed1e0b2073d,501679dfab0d9750] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646732, data is empty. 2024-09-11 18:34:55.691 INFO 2981023 --- [alarm-detect-15,4c140ed1e0b2073d,501679dfab0d9750] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646732, data is empty. 2024-09-11 18:34:55.695 INFO 2981023 --- [alarm-detect-15,4c140ed1e0b2073d,501679dfab0d9750] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646732, data is empty. 2024-09-11 18:34:55.726 INFO 2981023 --- [alarm-detect-12,4c140ed1e0b2073d,40330a47e337a1e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646732, data is empty. 2024-09-11 18:34:55.727 INFO 2981023 --- [alarm-detect-2,4c140ed1e0b2073d,8f11536d0f0bb4b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646732, data is empty. 2024-09-11 18:34:55.731 INFO 2981023 --- [alarm-detect-12,4c140ed1e0b2073d,40330a47e337a1e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646732, data is empty. 2024-09-11 18:34:55.731 INFO 2981023 --- [alarm-detect-2,4c140ed1e0b2073d,8f11536d0f0bb4b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646732, data is empty. 2024-09-11 18:34:55.734 INFO 2981023 --- [alarm-detect-12,4c140ed1e0b2073d,40330a47e337a1e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646732, data is empty. 2024-09-11 18:34:55.737 INFO 2981023 --- [alarm-detect-2,4c140ed1e0b2073d,8f11536d0f0bb4b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646732, data is empty. 2024-09-11 18:34:55.825 INFO 2981023 --- [ocp-schedules-7,3efa6e88b1ade4f6,2fa50a916cbdb783] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:34:55.852 WARN 2981023 --- [ocp-schedules-7,3efa6e88b1ade4f6,2fa50a916cbdb783] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:34:55.865 INFO 2981023 --- [alarm-detect-15,4c140ed1e0b2073d,3399b79dfc87fb66] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646732, data is empty. 2024-09-11 18:34:55.868 INFO 2981023 --- [alarm-detect-15,4c140ed1e0b2073d,3399b79dfc87fb66] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646732, data is empty. 2024-09-11 18:34:55.870 INFO 2981023 --- [alarm-detect-15,4c140ed1e0b2073d,3399b79dfc87fb66] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646732, data is empty. 2024-09-11 18:34:55.958 INFO 2981023 --- [ocp-schedules-22,10f1ffcdb8ed90fd,94b5e31bc9a8598f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:34:55.958 INFO 2981023 --- [ocp-schedules-22,10f1ffcdb8ed90fd,94b5e31bc9a8598f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:34:55.958 INFO 2981023 --- [ocp-schedules-22,10f1ffcdb8ed90fd,94b5e31bc9a8598f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:34:55.958 INFO 2981023 --- [ocp-schedules-22,10f1ffcdb8ed90fd,94b5e31bc9a8598f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:34:55.959 INFO 2981023 --- [ocp-schedules-22,10f1ffcdb8ed90fd,94b5e31bc9a8598f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:34:55.959 INFO 2981023 --- [ocp-schedules-22,10f1ffcdb8ed90fd,94b5e31bc9a8598f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:34:56.199 INFO 2981023 --- [ocp-schedules-6,e8c3401f13279940,19ef2f8c5237217b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:34:56.206 INFO 2981023 --- [ocp-schedules-6,e8c3401f13279940,19ef2f8c5237217b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:34:56.206 INFO 2981023 --- [ocp-schedules-6,e8c3401f13279940,19ef2f8c5237217b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:34:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,51ebc7f1621ff85c,1d498ec9141c5b37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,51ebc7f1621ff85c,1d498ec9141c5b37] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:56.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:34:56.338235+08:00, lastProcessTime=2024-09-11T18:34:46.355440+08:00 2024-09-11 18:34:56.381 INFO 2981023 --- [ocp-schedules-13,f6ae0348f7e41a1d,7fff239654ebbdc9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:34:56.387 INFO 2981023 --- [ocp-schedules-13,f6ae0348f7e41a1d,7fff239654ebbdc9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:34:56.387 INFO 2981023 --- [ocp-schedules-13,f6ae0348f7e41a1d,7fff239654ebbdc9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:34:56.465 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:34:56.338235+08:00 2024-09-11 18:34:57.483 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:34:57.483 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:34:57.495 INFO 2981023 --- [ocp-async-10,e73173a6a27c4a50,200e04d376d9e951] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:34:57.802 INFO 2981023 --- [ocp-async-3,c9284582afb050f8,1d5261b0f5c0f504] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:34:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,2344263f9f6ab31a,ba536acf1ae9e6a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050900, interval=60 2024-09-11 18:34:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,2344263f9f6ab31a,ba536acf1ae9e6a7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:34:59.361 INFO 2981023 --- [ocp-schedules-16,93be6e75b2eb219a,6b3e9fa58b8ae18e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:34:59.361 INFO 2981023 --- [ocp-schedules-16,93be6e75b2eb219a,6b3e9fa58b8ae18e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:34:59.377 INFO 2981023 --- [Collect all tenants major compactions1,93be6e75b2eb219a,fe8ebdcd0591337a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:34:59.378 INFO 2981023 --- [Collect all tenants major compactions5,93be6e75b2eb219a,fc91730acb902485] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:34:59.378 INFO 2981023 --- [Collect all tenants major compactions3,93be6e75b2eb219a,c236b379351bda2e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:34:59.379 INFO 2981023 --- [Collect all tenants major compactions6,93be6e75b2eb219a,1abe900601712660] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:34:59.380 INFO 2981023 --- [Collect all tenants major compactions4,93be6e75b2eb219a,b21703754d90f3a7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:34:59.380 INFO 2981023 --- [Collect all tenants major compactions2,93be6e75b2eb219a,9cf289fd18f5178e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:34:59.390 INFO 2981023 --- [Collect all tenants major compactions1,93be6e75b2eb219a,fe8ebdcd0591337a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:59.391 INFO 2981023 --- [Collect all tenants major compactions1,93be6e75b2eb219a,fe8ebdcd0591337a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:34:59.391 INFO 2981023 --- [Collect all tenants major compactions3,93be6e75b2eb219a,c236b379351bda2e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:59.393 INFO 2981023 --- [Collect all tenants major compactions5,93be6e75b2eb219a,fc91730acb902485] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:59.393 INFO 2981023 --- [Collect all tenants major compactions2,93be6e75b2eb219a,9cf289fd18f5178e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:59.393 INFO 2981023 --- [Collect all tenants major compactions3,93be6e75b2eb219a,c236b379351bda2e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:34:59.393 INFO 2981023 --- [Collect all tenants major compactions6,93be6e75b2eb219a,1abe900601712660] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:59.394 INFO 2981023 --- [Collect all tenants major compactions4,93be6e75b2eb219a,b21703754d90f3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:34:59.394 INFO 2981023 --- [Collect all tenants major compactions5,93be6e75b2eb219a,fc91730acb902485] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:34:59.395 INFO 2981023 --- [Collect all tenants major compactions6,93be6e75b2eb219a,1abe900601712660] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:34:59.395 INFO 2981023 --- [Collect all tenants major compactions4,93be6e75b2eb219a,b21703754d90f3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:34:59.395 INFO 2981023 --- [Collect all tenants major compactions2,93be6e75b2eb219a,9cf289fd18f5178e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:34:59.396 INFO 2981023 --- [Collect all tenants major compactions1,93be6e75b2eb219a,fe8ebdcd0591337a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:34:59.396 INFO 2981023 --- [Collect all tenants major compactions1,93be6e75b2eb219a,fe8ebdcd0591337a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:34:59.396 INFO 2981023 --- [Collect all tenants major compactions3,93be6e75b2eb219a,c236b379351bda2e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:34:59.397 INFO 2981023 --- [Collect all tenants major compactions3,93be6e75b2eb219a,c236b379351bda2e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:34:59.397 INFO 2981023 --- [Collect all tenants major compactions6,93be6e75b2eb219a,1abe900601712660] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:34:59.397 INFO 2981023 --- [Collect all tenants major compactions6,93be6e75b2eb219a,1abe900601712660] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:34:59.397 INFO 2981023 --- [Collect all tenants major compactions4,93be6e75b2eb219a,b21703754d90f3a7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:34:59.397 INFO 2981023 --- [Collect all tenants major compactions4,93be6e75b2eb219a,b21703754d90f3a7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:34:59.397 INFO 2981023 --- [Collect all tenants major compactions5,93be6e75b2eb219a,fc91730acb902485] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:34:59.397 INFO 2981023 --- [Collect all tenants major compactions5,93be6e75b2eb219a,fc91730acb902485] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:34:59.397 INFO 2981023 --- [Collect all tenants major compactions2,93be6e75b2eb219a,9cf289fd18f5178e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:34:59.397 INFO 2981023 --- [Collect all tenants major compactions2,93be6e75b2eb219a,9cf289fd18f5178e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:34:59.398 INFO 2981023 --- [ocp-schedules-16,93be6e75b2eb219a,6b3e9fa58b8ae18e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:34:59.495 INFO 2981023 --- [ocp-async-6,0e2ee70ef597470c,9f518b739a27d660] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:00.483 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:00.484 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:00.778 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:30:01.0, nextRun=2024-09-11 18:35:00.0, currentDate=Wed Sep 11 18:35:00 HKT 2024 2024-09-11 18:35:00.778 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:30:01.0, nextRunTime=2024-09-11 18:35:00.0} 2024-09-11 18:35:00.781 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 18:35:00.781 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:35:00.781 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:35:00.781+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:35:00.793 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:30:01.0, nextRun=2024-09-11 18:35:00.0, currentDate=Wed Sep 11 18:35:00 HKT 2024 2024-09-11 18:35:00.793 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:30:01.0, nextRunTime=2024-09-11 18:35:00.0} 2024-09-11 18:35:00.806 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 18:35:00.812 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 18:35:00.813 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 18:35:00.813 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:35:00.813 INFO 2981023 --- [task-coordinator1,ddd72be6e49ae268,58ad04b34fdff4bf] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T18:35:00.813+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:35:01.420 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,3541e6024ef7cd43] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048256, operation:EXECUTE, state: RUNNING 2024-09-11 18:35:01.431 INFO 2981023 --- [subtask-coordinator1,ddd72be6e49ae268,3541e6024ef7cd43] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048257, operation:EXECUTE, state: RUNNING 2024-09-11 18:35:01.444 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048256, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048226, task_operation=execute, latest_execution_start_time=2024-09-11T18:35:01.412+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048256}, listMap={}}, executor=10.31.35.10 2024-09-11 18:35:01.461 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048257, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048227, task_operation=execute, latest_execution_start_time=2024-09-11T18:35:01.430+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048257}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 18:35:01.462 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 18:35:01.463 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:35:01.483 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 18:35:01.485 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 18:35:01.495 INFO 2981023 --- [ocp-async-7,b06fb84d271bf4a2,6144c7813635c05b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:01.496 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 18:35:01.498 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 18:35:01.498 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 18:35:01.512 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:35:01.514 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 18:35:01.522 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 18:35:01.534 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,2e8d0f2ce239530e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048257, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:35:01.573 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048227, status:SUCCESS 2024-09-11 18:35:01.580 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048227, status:SUCCESS 2024-09-11 18:35:01.591 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:35:01.608 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:35:01.643 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:35:01.652 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 18:35:01.653 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 18:35:01.658 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:35:01.760 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:35:01.811 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:35:01.830 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:35:01.852 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:35:01.895 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:35:01.906 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:35:01.924 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:35:01.932 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 18:35:01.932 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3395.459579986], b:[-5.807309276841022E12] 2024-09-11 18:35:01.937 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:35:01.998 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:35:02.030 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:35:02.041 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:35:02.047 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:35:02.059 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:35:02.065 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:35:02.071 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:35:02.077 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 18:35:02.078 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:35:02.093 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:35:02.106 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:35:02.117 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,aed8da5e6ea46d6f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048256, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:35:02.130 INFO 2981023 --- [ocp-schedules-6,58ad516c093a0e40,7500c70b1d0fe5cd] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:35:02.130+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:35:02.133 INFO 2981023 --- [ocp-schedules-6,58ad516c093a0e40,7500c70b1d0fe5cd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:35:02.171 INFO 2981023 --- [ocp-schedules-6,58ad516c093a0e40,7500c70b1d0fe5cd] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 42 to SqlAggregateData 2024-09-11 18:35:02.172 INFO 2981023 --- [ocp-schedules-6,58ad516c093a0e40,7500c70b1d0fe5cd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:35:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,d59bd59228b313e5,c3727fe1d1f4d85c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,d59bd59228b313e5,c3727fe1d1f4d85c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:02.603 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048226, status:SUCCESS 2024-09-11 18:35:02.606 INFO 2981023 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048226, status:SUCCESS 2024-09-11 18:35:02.970 INFO 2981023 --- [ocp-schedules-29,5b2c3345a5e02d20,eb27f5cf05e19380] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:35:03.484 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:03.485 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:03.495 INFO 2981023 --- [ocp-async-1,aa3b449174f6f600,22c3ffb2a88b7b9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:05.278 INFO 2981023 --- [ocp-schedules-20,5aff9a33dd3b7748,1cad867b587b7a90] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:35:05.282 INFO 2981023 --- [ocp-schedules-20,5aff9a33dd3b7748,1cad867b587b7a90] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:35:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,ebe9ca38f831892a,ce6bfec734a972b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:05.319 INFO 2981023 --- [prometheus-minute-schedule-1,ebe9ca38f831892a,ce6bfec734a972b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:05.328 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:35:05.329 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.329 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:35:05.329 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.332 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:35:05.333 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:35:05.336 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.336 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:35:05.338 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.340 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:35:05.344 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.344 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.344 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.346 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.348 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.348 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.350 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.350 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.357 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.358 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:35:05.359 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.360 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.365 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.365 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:35:05.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.367 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.368 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.368 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:35:05.369 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.369 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:35:05.370 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.370 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:35:05.370 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.371 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:35:05.371 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:35:05.374 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:35:05.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.378 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:35:05.388 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.389 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:35:05.389 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.389 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:35:05.390 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.390 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:35:05.391 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.392 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:05.392 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:35:05.392 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.393 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:35:05.393 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.393 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.393 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.394 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:35:05.394 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.394 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:35:05.395 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.395 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:35:05.396 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.396 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.396 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:35:05.396 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:35:05.398 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:35:05.405 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:35:05.408 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:35:05.408 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:35:05.408 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:35:05.410 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:05.411 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.412 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:35:05.413 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.413 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:35:05.414 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE7201D6DD-0-0] 2024-09-11 18:35:05.414 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:35:05.414 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.414 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:35:05.416 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:35:05.417 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:35:05.418 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.418 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:35:05.423 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:35:05.424 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:35:05.424 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.425 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.429 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:35:05.430 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:35:05.430 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:35:05.430 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:35:05.438 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.439 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:35:05.465 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.465 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:35:05.466 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:35:05.466 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE7201D6E7-0-0] 2024-09-11 18:35:05.466 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:35:05.482 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.482 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:35:05.483 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.483 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:35:05.484 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.484 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:35:05.485 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.485 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:35:05.486 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.486 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:35:05.487 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.487 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:35:05.488 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:35:05.489 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:35:05.495 INFO 2981023 --- [ocp-async-8,282ef8a8c70b9510,0bed2eee0788c2df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:05.500 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:05.501 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.501 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:35:05.512 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:35:05.513 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:35:05.524 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:05.525 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:35:05.534 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:35:05.536 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:35:05.536 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:35:05.536 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:35:05.536 INFO 2981023 --- [ocp-schedules-20,5aff9a33dd3b7748,1cad867b587b7a90] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:35:05.924 INFO 2981023 --- [ocp-schedules-25,73c7e99e5043e451,cdf03429608a0c90] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:35:05.954 WARN 2981023 --- [scanner-13694917-1,73c7e99e5043e451,5d827d6b49b3212b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:35:06.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:35:06.354561+08:00, lastProcessTime=2024-09-11T18:34:56.338235+08:00 2024-09-11 18:35:06.485 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:06.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:35:06.354561+08:00 2024-09-11 18:35:06.485 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:07.495 INFO 2981023 --- [ocp-async-6,380d3ec4d49d72fb,9d28159f5fcbf661] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:07.802 INFO 2981023 --- [ocp-async-9,5f1e79172a101a26,11738755b60ddd2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:35:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,f5a7e399c1e03c84,6418f3b931f15371] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,f5a7e399c1e03c84,6418f3b931f15371] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:08.351 INFO 2981023 --- [ocp-schedules-14,7fc737c564a376b0,78a942e65e14cd43] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:35:08.422 INFO 2981023 --- [ocp-schedules-10,88475e76e8ecb7f4,dec3d5483e3fd3b9] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:35:08.422+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:35:08.423 INFO 2981023 --- [ocp-schedules-10,88475e76e8ecb7f4,dec3d5483e3fd3b9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:35:08.424 INFO 2981023 --- [ocp-schedules-10,88475e76e8ecb7f4,dec3d5483e3fd3b9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:35:08.425 INFO 2981023 --- [ocp-schedules-10,88475e76e8ecb7f4,dec3d5483e3fd3b9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:35:09.486 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:09.486 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:09.495 INFO 2981023 --- [ocp-async-10,5dc07cad1544c06c,f5055161a025b85b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:10.003 INFO 2981023 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:35:10.003 INFO 2981023 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:35:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:35:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:35:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,e014096a0142b0da,248faf097df8d9a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,e014096a0142b0da,248faf097df8d9a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:11.496 INFO 2981023 --- [ocp-async-3,4fced6efbe41e9fd,fa6af21c3943ad54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:12.486 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:12.487 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:12.641 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050840, nodeCount=1000, affectRows=1000 2024-09-11 18:35:12.644 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050840, nodeCount=1000, validCount=1000 2024-09-11 18:35:12.644 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050840, nodeCount=1000, affectRows=1000 2024-09-11 18:35:12.647 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050840, nodeCount=999, validCount=999 2024-09-11 18:35:12.651 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050840, nodeCount=48, affectRows=48 2024-09-11 18:35:12.652 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050840, nodeCount=46, validCount=46 2024-09-11 18:35:12.973 INFO 2981023 --- [ocp-schedules-2,f27f92649e652f0e,26e15432ef3c1bc4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:35:13.496 INFO 2981023 --- [ocp-async-2,c9d82263ab2e4a23,b2925f23defc0906] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,0c738ef664cce511,109dd24b76f86095] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,20aed9732b8729a2,c81a07724b8b2f61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:35:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,0c738ef664cce511,109dd24b76f86095] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:14.397 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:31:14.043, end=2024-09-11T18:33:14.392} 2024-09-11 18:35:14.401 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obCluste 2024-09-11 18:35:14.402 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:35:14.404 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:14.043, end=2024-09-11T18:33:14.392},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:35:14.404 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:35:14.404 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:35:14.451 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514451000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.451Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.452 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514452000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.452Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.453 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514453000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.453Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.454 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514454000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.454Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.456 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514456000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.456Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.459 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514459000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.459Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.461 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 71 2024-09-11 18:35:14.461 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.463 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514463000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.463Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.474 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.474 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.476 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.476 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.478 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.478 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.478 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514478000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.478Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.483 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:35:14.483 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.483 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:35:14.483 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.486 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.486 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.487 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.487 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.489 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.489 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.494 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.494 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.496 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.496 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.497 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.497 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.497 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.497 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.497 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.497 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.503 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.503 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.507 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:35:14.507 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.508 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.508 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.508 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.508 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.514 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.514 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.514 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.514 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.516 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.516 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.518 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.518 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.522 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.522 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.522 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.522 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.524 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.524 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.524 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.525 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.526 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.526 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.526 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.527 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.527 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.528 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.528 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.530 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.531 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.531 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.531 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.531 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.531 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.533 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.533 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.533 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.456, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.534 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.452, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.535 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.451, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.535 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.536 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.454, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.538 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.541 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.541 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.541 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.459, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.544 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.545 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.453, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.548 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.463, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.552 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.478, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.569 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514569000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.569Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.570 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514570000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=563D19C678935F4895C494E06A5CA13F, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.570Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.574 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514574000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.574Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.575 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514575000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.575Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.578 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514578000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.578Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.580 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514580000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.580Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.583 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514583000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.583Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.585 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.585 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.589 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.589 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.591 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.591 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=563D19C678935F4895C494E06A5CA13F, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.595 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.595 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.597 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.597 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.598 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.598 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.604 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.604 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.607 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:35:14.607 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.607 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.607 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.607 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.607 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.609 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.609 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.609 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:35:14.609 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.612 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.612 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.619 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.619 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.619 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.619 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.619 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.619 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=563D19C678935F4895C494E06A5CA13F, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.624 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.624 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.626 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.626 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.626 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.626 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.627 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.627 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.627 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.628 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.628 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.629 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.629 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.629 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.630 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.630 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.631 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.631 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.632 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.636 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.575, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.637 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.57, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=563D19C678935F4895C494E06A5CA13F, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.640 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.640 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.642 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.642 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.642 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.643 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.574, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.643 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.643 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.643 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.643 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.644 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.645 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.645 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.647 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.578, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.651 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.569, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.651 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.583, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.652 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.58, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.652 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:35:14.653 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:14.043, end=2024-09-11T18:33:14.392},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:35:14.653 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:35:14.653 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:35:14.658 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:35:14.660 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 2 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:14.043, end=2024-09-11T18:33:14.392},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:35:14.664 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:35:14.664 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:35:14.678 INFO 2981023 --- [plan_change_detector11,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:35:14.689 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050840, nodeCount=480, affectRows=480 2024-09-11 18:35:14.703 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514703000, endTimeUs=1726050794392000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=19BFFEFE537DC72F4980E8E655B3E19B, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T10:35:14.703Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000029, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000010, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 18:35:14.703 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514703000, endTimeUs=1726050794392000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=E9E2014C8CE705871C555597A6A32456, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T10:35:14.703Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000029, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000010, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 18:35:14.704 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514704000, endTimeUs=1726050794392000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=58CF457EBE63FE043C17A1F621D296E5, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T10:35:14.704Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000029, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000010, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 18:35:14.706 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:35:14.706 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.706 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 18:35:14.706 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.706 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 18:35:14.706 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.715 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.715 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=19BFFEFE537DC72F4980E8E655B3E19B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.716 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.716 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=58CF457EBE63FE043C17A1F621D296E5, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.717 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.717 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.718 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.718 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.718 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.718 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.719 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.719 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.727 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.727 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=19BFFEFE537DC72F4980E8E655B3E19B, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.728 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.728 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.728 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.729 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.729 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=58CF457EBE63FE043C17A1F621D296E5, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.730 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:35:14.730 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.703, end=2024-09-11T18:33:14.392}, clusterId=1000006, tenantId=1000006, sqlId=E9E2014C8CE705871C555597A6A32456, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.730 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:35:14.733 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.703, end=2024-09-11T18:33:14.392}, clusterId=1000006, tenantId=1000006, sqlId=19BFFEFE537DC72F4980E8E655B3E19B, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.734 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.704, end=2024-09-11T18:33:14.392}, clusterId=1000006, tenantId=1000006, sqlId=58CF457EBE63FE043C17A1F621D296E5, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.734 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:35:14.736 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:14.043, end=2024-09-11T18:33:14.392},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:35:14.736 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:35:14.736 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:35:14.755 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:35:14.765 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964514765000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:35:14.765Z, endTime=2024-09-11T10:33:14.392Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:35:14.767 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:35:14.767 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:35:14.773 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.774 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.775 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:35:14.775 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:35:14.781 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:35:14.781 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050794392000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:35:14.782 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:35:14.782 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:35:14.765, end=2024-09-11T18:33:14.392}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:35:14.782 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:35:14.783 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:14.043, end=2024-09-11T18:33:14.392},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:35:14.783 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:35:14.783 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:35:14.784 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:35:14.785 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:14.043, end=2024-09-11T18:33:14.392},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:35:14.785 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:35:14.785 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:35:14.787 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:35:15.487 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:15.487 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:15.496 INFO 2981023 --- [ocp-async-5,f81757af7ef9f4d3,be8fb823ba95428f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:15.565 INFO 2981023 --- [ocp-schedules-18,7a5fe6cb2b3b3f3e,a505eb44886e3e40] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:35:16.026 INFO 2981023 --- [ocp-schedules-6,dbbc42d29ecefaac,552c25aa979242aa] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:35:16.026 INFO 2981023 --- [host-check-clockdiff1,dbbc42d29ecefaac,f297011eb07a67cc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:35:16.026 INFO 2981023 --- [host-check-clockdiff2,dbbc42d29ecefaac,12c5878524c3ecbf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:35:16.031 INFO 2981023 --- [host-check-clockdiff3,dbbc42d29ecefaac,3f4366d25b233ffa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:35:16.033 INFO 2981023 --- [host-check-clockdiff4,dbbc42d29ecefaac,54af23374659a1d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:35:16.035 INFO 2981023 --- [ocp-schedules-12,3f4aaa378cac5753,2d189e030b4d66ad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:35:16.036 INFO 2981023 --- [ocp-schedules-3,c052ce442f3d9f3b,ea64a476da172b94] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:35:16.038 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:35:16.038 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:35:16.038 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:35:16.039 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:35:16.041 INFO 2981023 --- [host-check-agent-status2,c052ce442f3d9f3b,728d89bc355f60cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:35:16.041 INFO 2981023 --- [host-check-agent-status1,c052ce442f3d9f3b,296e94fe3914ceca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:35:16.042 INFO 2981023 --- [host-check-agent-status3,c052ce442f3d9f3b,38e3a23329247f4a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:35:16.042 INFO 2981023 --- [host-check-agent-status4,c052ce442f3d9f3b,e443d097acaf73e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:35:16.046 INFO 2981023 --- [host-check-clockdiff2,dbbc42d29ecefaac,12c5878524c3ecbf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050916 5 5 , error= 2024-09-11 18:35:16.048 INFO 2981023 --- [host-check-clockdiff3,dbbc42d29ecefaac,3f4366d25b233ffa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050916 0 0 , error= 2024-09-11 18:35:16.125 INFO 2981023 --- [ocp-schedules-8,ad330b3ac2d65798,26497f8ed92bea4d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:35:16.156 INFO 2981023 --- [host-check-agent-status2,c052ce442f3d9f3b,728d89bc355f60cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:35:16.156 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:35:16.161 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:35:16.162 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:35:16.162 INFO 2981023 --- [host-check-agent-status2,c052ce442f3d9f3b,728d89bc355f60cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:35:16.174 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:35:16.175 INFO 2981023 --- [host-check-agent-status4,c052ce442f3d9f3b,e443d097acaf73e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:35:16.176 INFO 2981023 --- [host-check-agent-status3,c052ce442f3d9f3b,38e3a23329247f4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:35:16.183 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:35:16.184 INFO 2981023 --- [host-check-agent-status1,c052ce442f3d9f3b,296e94fe3914ceca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:35:16.187 INFO 2981023 --- [host-check-agent-status1,c052ce442f3d9f3b,296e94fe3914ceca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:35:16.191 INFO 2981023 --- [host-check-agent-status4,c052ce442f3d9f3b,e443d097acaf73e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:35:16.192 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:35:16.192 INFO 2981023 --- [ocp-schedules-16,2e02037912ebc28a,535118e89c906d93] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:35:16.193 INFO 2981023 --- [ocp-schedules-29,6dbd494b55fcfb6d,995452e78057b7aa] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:35:16.193 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:35:16.193 INFO 2981023 --- [host-check-agent-status3,c052ce442f3d9f3b,38e3a23329247f4a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:35:16.205 INFO 2981023 --- [ocp-schedules-20,f6487a0797a86294,57d16f0cfb0ffe9f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:35:16.205 INFO 2981023 --- [ocp-schedules-11,cf7918f8f0d1bd59,cc1714aad48835d3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:35:16.213 INFO 2981023 --- [ocp-schedules-29,6dbd494b55fcfb6d,995452e78057b7aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:35:16.219 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:35:16.228 INFO 2981023 --- [ocp-schedules-16,2e02037912ebc28a,535118e89c906d93] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:35:16.282 INFO 2981023 --- [ocp-schedules-20,f6487a0797a86294,57d16f0cfb0ffe9f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:35:16.288 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050843420000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:35:16.289 WARN 2981023 --- [scanner-13696607-1,ad330b3ac2d65798,376679cb77f5f2e5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:35:16.293 INFO 2981023 --- [ocp-schedules-29,6dbd494b55fcfb6d,995452e78057b7aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:35:16.456 INFO 2981023 --- [ocp-schedules-5,e23b4e315a8b1490,a2bf38a070b5cbb1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:35:16.465 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:35:16.467 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:35:16.467467916+08:00 2024-09-11 18:35:16.469 INFO 2981023 --- [ocp-schedules-7,1fc717692799a600,4b3cab65fbd56af2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:35:16.470 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:35:16.467+08:00 2024-09-11 18:35:16.475 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:35:16.467+08:00 and status AVAILABLE 2024-09-11 18:35:16.479 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.479 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:35:16.480 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:35:16.498 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:35:16.499 INFO 2981023 --- [host-check-agent-status3,c052ce442f3d9f3b,38e3a23329247f4a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:35:16.503 INFO 2981023 --- [ocp-schedules-5,e23b4e315a8b1490,a2bf38a070b5cbb1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:35:16.503 INFO 2981023 --- [ocp-schedules-5,e23b4e315a8b1490,a2bf38a070b5cbb1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:35:16.507 INFO 2981023 --- [host-check-agent-status3,c052ce442f3d9f3b,38e3a23329247f4a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:35:16.507 INFO 2981023 --- [host-check-agent-status3,c052ce442f3d9f3b,38e3a23329247f4a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:35:16.507 INFO 2981023 --- [host-check-agent-status3,c052ce442f3d9f3b,38e3a23329247f4a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:35:16.512 INFO 2981023 --- [host-check-agent-status4,c052ce442f3d9f3b,e443d097acaf73e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:35:16.514 INFO 2981023 --- [host-check-agent-status2,c052ce442f3d9f3b,728d89bc355f60cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:35:16.516 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:35:16.517 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:35:16.517179447+08:00 2024-09-11 18:35:16.517 INFO 2981023 --- [host-check-agent-status4,c052ce442f3d9f3b,e443d097acaf73e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:35:16.517 INFO 2981023 --- [host-check-agent-status4,c052ce442f3d9f3b,e443d097acaf73e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:35:16.517 INFO 2981023 --- [host-check-agent-status4,c052ce442f3d9f3b,e443d097acaf73e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:35:16.518 INFO 2981023 --- [host-check-agent-status2,c052ce442f3d9f3b,728d89bc355f60cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:35:16.518 INFO 2981023 --- [host-check-agent-status2,c052ce442f3d9f3b,728d89bc355f60cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:35:16.518 INFO 2981023 --- [host-check-agent-status2,c052ce442f3d9f3b,728d89bc355f60cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:35:16.554 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.555 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:35:16.556 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:35:16.554451+08:00, lastProcessTime=2024-09-11T18:35:06.354561+08:00 2024-09-11 18:35:16.556 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:35:16.557 INFO 2981023 --- [ocp-schedules-16,2e02037912ebc28a,535118e89c906d93] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:35:16.557 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:35:16.517+08:00 2024-09-11 18:35:16.560 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:35:16.517+08:00 and status AVAILABLE 2024-09-11 18:35:16.581 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:35:16.583 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:35:16.582656224+08:00 2024-09-11 18:35:16.584 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:35:16.586 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:35:16.583+08:00 2024-09-11 18:35:16.591 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:35:16.583+08:00 and status AVAILABLE 2024-09-11 18:35:16.599 INFO 2981023 --- [host-check-agent-status1,c052ce442f3d9f3b,296e94fe3914ceca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:35:16.600 INFO 2981023 --- [alarm-detect-13,ad330b3ac2d65798,24bc2e662799f5c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646732, data is empty. 2024-09-11 18:35:16.602 INFO 2981023 --- [host-check-agent-status1,c052ce442f3d9f3b,296e94fe3914ceca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:35:16.602 INFO 2981023 --- [host-check-agent-status1,c052ce442f3d9f3b,296e94fe3914ceca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:35:16.602 INFO 2981023 --- [host-check-agent-status1,c052ce442f3d9f3b,296e94fe3914ceca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:35:16.604 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:35:16.605 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:35:16.606 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:35:16.607 INFO 2981023 --- [alarm-detect-13,ad330b3ac2d65798,24bc2e662799f5c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646774, data is empty. 2024-09-11 18:35:16.607 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:35:16.610 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:35:16.612 INFO 2981023 --- [alarm-detect-8,ad330b3ac2d65798,72957da7a5909b29] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646732, data is empty. 2024-09-11 18:35:16.620 INFO 2981023 --- [alarm-detect-20,ad330b3ac2d65798,f93f9bd3891aff8b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646732, data is empty. 2024-09-11 18:35:16.626 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.626 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:16.628 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:35:16.628 INFO 2981023 --- [alarm-detect-13,ad330b3ac2d65798,24bc2e662799f5c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646774, data is empty. 2024-09-11 18:35:16.637 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:35:16.640329046+08:00 2024-09-11 18:35:16.638 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:35:16.637+08:00 2024-09-11 18:35:16.639 INFO 2981023 --- [alarm-detect-8,ad330b3ac2d65798,72957da7a5909b29] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646774, data is empty. 2024-09-11 18:35:16.639 INFO 2981023 --- [alarm-detect-20,ad330b3ac2d65798,f93f9bd3891aff8b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646774, data is empty. 2024-09-11 18:35:16.640 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:35:16.637+08:00 and status AVAILABLE 2024-09-11 18:35:16.649 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:35:16.649 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:35:16.651 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:35:16.651 INFO 2981023 --- [alarm-detect-8,ad330b3ac2d65798,72957da7a5909b29] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646774, data is empty. 2024-09-11 18:35:16.656 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.656 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:16.662 INFO 2981023 --- [alarm-detect-20,ad330b3ac2d65798,f93f9bd3891aff8b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646774, data is empty. 2024-09-11 18:35:16.665 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:35:16.669 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.669 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:16.670 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:35:16.678 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:35:16.686 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:35:16.695 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:35:16.705 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:35:16.726 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:35:16.745 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:35:16.746 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:35:16.752 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:16.754 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.755 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:35:16.758 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:35:16.765 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:35:16.770 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:35:16.770 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050843420000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:35:16.784 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:35:16.787 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:35:16.803 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:35:16.815 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:16.817 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.817 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:16.819 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:35:16.819 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.822 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:35:16.824 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:35:16.838 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:35:16.842 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:35:16.845 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:35:16.849 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:35:16.849 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:35:16.863 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:35:16.870 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:35:16.876 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:35:16.877 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:35:16.878 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:16.879 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.880 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:35:16.881 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:35:16.883 INFO 2981023 --- [alarm-detect-20,ad330b3ac2d65798,3b26d31bcd28bd6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646732, data is empty. 2024-09-11 18:35:16.888 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:35:16.890 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:35:16.890 INFO 2981023 --- [alarm-detect-20,ad330b3ac2d65798,3b26d31bcd28bd6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646774, data is empty. 2024-09-11 18:35:16.891 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:16.892 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.894 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:35:16.896 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:35:16.897 INFO 2981023 --- [alarm-detect-20,ad330b3ac2d65798,3b26d31bcd28bd6e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646774, data is empty. 2024-09-11 18:35:16.902 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:35:16.905 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:35:16.910 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:35:16.914 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:35:16.918 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:35:16.931 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:16.932 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.932 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:16.933 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:35:16.934 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:16.934 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:35:16.935 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:35:16.937 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:35:16.937 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:35:16.940 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:35:16.946 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:35:16.956 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:35:16.959 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:35:16.959 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:35:16.960 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:35:16.961 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:35:16.974 INFO 2981023 --- [host-check-host-status4,3f4aaa378cac5753,a20c57db3ae4fa2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:35:16.974 INFO 2981023 --- [host-check-host-status1,3f4aaa378cac5753,787175ed3e25a8bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:35:16.980 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:35:16.989 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:35:16.995 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:35:17.004 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:17.004 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:17.005 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:35:17.006 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:35:17.008 INFO 2981023 --- [host-check-host-status3,3f4aaa378cac5753,8bdc213b9009546b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:35:17.010 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:35:17.012 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:35:17.020 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:35:17.020 INFO 2981023 --- [host-check-host-status2,3f4aaa378cac5753,25cd6bfb491f6724] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:35:17.021 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:35:17.037 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:35:17.043 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:35:17.045 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:17.045 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:17.046 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:35:17.052 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:35:17.052 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:35:17.072 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:35:17.078 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:17.079 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:17.079 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:35:17.086 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:35:17.086 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:35:17.086 INFO 2981023 --- [ocp-schedules-20,f6487a0797a86294,57d16f0cfb0ffe9f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:35:17.103 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:35:16.554451+08:00 2024-09-11 18:35:17.160 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050840, nodeCount=480, validCount=480 2024-09-11 18:35:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,673732324858311d,a1682c98531968ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,673732324858311d,a1682c98531968ea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:17.496 INFO 2981023 --- [ocp-async-10,7325f840f3b291b7,55d5c2451a3926a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:17.802 INFO 2981023 --- [ocp-async-2,ca4ec3dadb76e57b,6202e7d2736be919] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:35:18.487 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:18.488 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:19.496 INFO 2981023 --- [ocp-async-7,6b15a1624a49f4a7,2c110bf949ad73f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,1137d1a94a3b2e1e,41ecaef6d7dec81e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,1137d1a94a3b2e1e,41ecaef6d7dec81e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:20.839 INFO 2981023 --- [host-check-clockdiff1,dbbc42d29ecefaac,f297011eb07a67cc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050920 0 0 , error= 2024-09-11 18:35:21.488 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:21.489 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:21.496 INFO 2981023 --- [ocp-async-1,c0bcae30b9051049,8a4498d491e80fb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:22.005 INFO 2981023 --- [host-check-clockdiff4,dbbc42d29ecefaac,54af23374659a1d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050922 0 0 , error= 2024-09-11 18:35:22.976 INFO 2981023 --- [ocp-schedules-25,af2e1cbdd21e204a,08209a59037cf266] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:35:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,e08b69b70f313ac8,9f2641ce55e11aa2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,e08b69b70f313ac8,9f2641ce55e11aa2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:23.496 INFO 2981023 --- [ocp-async-4,6d82463a19fe9d56,2ce9a40c78e21445] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:24.489 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:24.490 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:24.553 INFO 2981023 --- [ocp-schedules-22,35fc50a885cc539c,4b483a6ed42686d9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:35:25.496 INFO 2981023 --- [ocp-async-8,664adaa21ff1e04d,4cea48118df5470f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,2eeb94f6efeb7cb7,bd8997593f07fdda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,2eeb94f6efeb7cb7,bd8997593f07fdda] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:35:26.337598+08:00, lastProcessTime=2024-09-11T18:35:16.554451+08:00 2024-09-11 18:35:26.463 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:35:26.337598+08:00 2024-09-11 18:35:26.999 INFO 2981023 --- [ocp-schedules-19,003d9613e6a934fe,b3df425afbdaa550] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:35:27.031 WARN 2981023 --- [scanner-13696920-1,003d9613e6a934fe,c4233210bb591673] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:35:27.490 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:27.490 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:27.496 INFO 2981023 --- [ocp-async-6,1ecf8e07c97fe58c,5a74724cae2e5b9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:27.802 INFO 2981023 --- [ocp-async-4,68b19f23959de0be,2fb0d712b74cef01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:35:29.069 INFO 2981023 --- [ocp-schedules-10,76a1eb2730c8ac0a,ed62b3e36aee63ec] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:35:29.069 INFO 2981023 --- [ocp-schedules-10,76a1eb2730c8ac0a,ed62b3e36aee63ec] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:35:29.069 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:35:29.069 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:35:29.069 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:35:29.072 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:35:29.073 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:35:29.073 INFO 2981023 --- [ocp-schedules-10,76a1eb2730c8ac0a,ed62b3e36aee63ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:35:29.074 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:35:29.074 INFO 2981023 --- [ocp-schedules-10,76a1eb2730c8ac0a,ed62b3e36aee63ec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:35:29.074 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:35:29.075 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:35:29.076 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:35:29.090 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:35:29.091 INFO 2981023 --- [ocp-schedules-10,76a1eb2730c8ac0a,ed62b3e36aee63ec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:35:29.091 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:35:29.092 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:35:29.095 INFO 2981023 --- [ocp-schedules-10,76a1eb2730c8ac0a,ed62b3e36aee63ec] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:35:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,ef0e883a2dd5c9dc,6fdf2654b2fd4c99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,ef0e883a2dd5c9dc,6fdf2654b2fd4c99] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:29.496 INFO 2981023 --- [ocp-async-10,d1205e463501d525,2c0acecdb350f405] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:30.490 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:30.491 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:31.497 INFO 2981023 --- [ocp-async-2,665759fb32601ab8,a90af74895578791] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,7a28213d3349eae8,36081230c8f0e0d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,7a28213d3349eae8,36081230c8f0e0d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:32.979 INFO 2981023 --- [ocp-schedules-20,a6fdf3b9d6b509d6,3138b27b332f5148] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:35:33.491 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:33.491 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:33.497 INFO 2981023 --- [ocp-async-5,28cf8ab501d62d22,f90fb6dc94cb0306] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,b71addfa99e7be44,ae43af3be8ca2ce5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,b71addfa99e7be44,ae43af3be8ca2ce5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:35.497 INFO 2981023 --- [ocp-async-9,c15ebf96dc91b974,2638c9cfe03897be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:35:36.337704+08:00, lastProcessTime=2024-09-11T18:35:26.337598+08:00 2024-09-11 18:35:36.461 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:35:36.337704+08:00 2024-09-11 18:35:36.492 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:36.492 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:36.505 INFO 2981023 --- [ocp-schedules-26,7b211cfc69fa4035,b2459f27b63ce958] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:35:36.510 INFO 2981023 --- [ocp-schedules-26,7b211cfc69fa4035,b2459f27b63ce958] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:35:36.510 INFO 2981023 --- [ocp-schedules-26,7b211cfc69fa4035,b2459f27b63ce958] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:35:37.193 INFO 2981023 --- [ocp-schedules-21,93a04064f251a852,d356460e8d76dbfe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:35:37.204 WARN 2981023 --- [scanner-13697142-1,93a04064f251a852,3cc869a0708822d5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:35:37.250 INFO 2981023 --- [alarm-detect-2,93a04064f251a852,91ecd53605cc204c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646774, data is empty. 2024-09-11 18:35:37.259 INFO 2981023 --- [alarm-detect-2,93a04064f251a852,91ecd53605cc204c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646774, data is empty. 2024-09-11 18:35:37.263 INFO 2981023 --- [alarm-detect-2,93a04064f251a852,91ecd53605cc204c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646774, data is empty. 2024-09-11 18:35:37.278 INFO 2981023 --- [alarm-detect-20,93a04064f251a852,4f803a0670b6ee1e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646774, data is empty. 2024-09-11 18:35:37.279 INFO 2981023 --- [alarm-detect-3,93a04064f251a852,34e44d5561e5711e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646774, data is empty. 2024-09-11 18:35:37.281 INFO 2981023 --- [alarm-detect-20,93a04064f251a852,4f803a0670b6ee1e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646774, data is empty. 2024-09-11 18:35:37.285 INFO 2981023 --- [alarm-detect-20,93a04064f251a852,4f803a0670b6ee1e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646774, data is empty. 2024-09-11 18:35:37.285 INFO 2981023 --- [alarm-detect-3,93a04064f251a852,34e44d5561e5711e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646774, data is empty. 2024-09-11 18:35:37.293 INFO 2981023 --- [alarm-detect-3,93a04064f251a852,34e44d5561e5711e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646774, data is empty. 2024-09-11 18:35:37.315 INFO 2981023 --- [prometheus-cache-monitor1,a9c62f7de10b9716,452dd360d3bb7693] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:35:37.315 INFO 2981023 --- [prometheus-cache-monitor1,a9c62f7de10b9716,452dd360d3bb7693] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:35:37.412 INFO 2981023 --- [alarm-detect-5,93a04064f251a852,e170e956591fcae6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646774, data is empty. 2024-09-11 18:35:37.418 INFO 2981023 --- [alarm-detect-5,93a04064f251a852,e170e956591fcae6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646774, data is empty. 2024-09-11 18:35:37.421 INFO 2981023 --- [alarm-detect-5,93a04064f251a852,e170e956591fcae6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646774, data is empty. 2024-09-11 18:35:37.497 INFO 2981023 --- [ocp-async-3,ace6cfcd5c7ce03d,65466769566b7927] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:37.803 INFO 2981023 --- [ocp-async-5,52a6b123188e61e1,ba7301934ce84962] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:35:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,db249024b3f84a71,e3af6839ae5e4084] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,db249024b3f84a71,e3af6839ae5e4084] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:39.492 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:39.493 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:39.497 INFO 2981023 --- [ocp-async-6,182de6b364a89b19,a93d217ade70a5fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:40.003 INFO 2981023 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:35:40.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:35:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,e32a072456a22f24,2e5b3eab63ed099e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,e32a072456a22f24,2e5b3eab63ed099e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:41.497 INFO 2981023 --- [ocp-async-4,0d584db8cbc799f3,a00376430feacc1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:42.493 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:42.493 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:42.981 INFO 2981023 --- [ocp-schedules-19,1e29a4907b87ec55,8e802cb0f061443b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:35:43.497 INFO 2981023 --- [ocp-async-8,d013980d3b347475,f5dcefb42a170ad6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,b89e0bceb45f7199,84c4836abbf5e50b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,b89e0bceb45f7199,84c4836abbf5e50b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:45.101 INFO 2981023 --- [prometheus-second-schedule-1,3f829489d3d71430,e8b949c9b8844096] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:35:45.493 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:45.494 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:45.497 INFO 2981023 --- [ocp-async-7,e6ec2c2aa55554a0,815a3cef99458d55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:46.016 INFO 2981023 --- [ocp-schedules-16,2531405f2b2e961d,465b6327f9adb103] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:35:46.017 INFO 2981023 --- [host-check-clockdiff1,2531405f2b2e961d,889b078261a7d858] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:35:46.017 INFO 2981023 --- [host-check-clockdiff2,2531405f2b2e961d,346f26a7308fe694] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:35:46.019 INFO 2981023 --- [host-check-clockdiff3,2531405f2b2e961d,43be5ec4800d93f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:35:46.023 INFO 2981023 --- [host-check-clockdiff4,2531405f2b2e961d,5a106d0c7ab80afd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:35:46.025 INFO 2981023 --- [ocp-schedules-24,7549ce9dc1cd1bdf,4c64106e348140a6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:35:46.029 INFO 2981023 --- [ocp-schedules-4,7d69d48020837ffe,6d78d5e2cbddb1de] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:35:46.029 INFO 2981023 --- [host-check-agent-status1,7d69d48020837ffe,81952918fe774c12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:35:46.030 INFO 2981023 --- [host-check-agent-status2,7d69d48020837ffe,1374bb1044498cf9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:35:46.030 INFO 2981023 --- [host-check-agent-status4,7d69d48020837ffe,b468d6f30db9af33] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:35:46.030 INFO 2981023 --- [host-check-agent-status3,7d69d48020837ffe,e49f48a9b7718d5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:35:46.031 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:35:46.032 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:35:46.032 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:35:46.041 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:35:46.043 INFO 2981023 --- [host-check-clockdiff3,2531405f2b2e961d,43be5ec4800d93f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050946 0 0 , error= 2024-09-11 18:35:46.049 INFO 2981023 --- [host-check-clockdiff2,2531405f2b2e961d,346f26a7308fe694] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050946 5 5 , error= 2024-09-11 18:35:46.068 INFO 2981023 --- [host-check-agent-status3,7d69d48020837ffe,e49f48a9b7718d5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:35:46.068 INFO 2981023 --- [host-check-agent-status1,7d69d48020837ffe,81952918fe774c12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:35:46.071 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:35:46.074 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:35:46.076 INFO 2981023 --- [host-check-agent-status4,7d69d48020837ffe,b468d6f30db9af33] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:35:46.076 INFO 2981023 --- [host-check-agent-status1,7d69d48020837ffe,81952918fe774c12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:35:46.077 INFO 2981023 --- [host-check-agent-status4,7d69d48020837ffe,b468d6f30db9af33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:35:46.081 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:35:46.082 INFO 2981023 --- [host-check-agent-status3,7d69d48020837ffe,e49f48a9b7718d5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:35:46.082 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:35:46.096 INFO 2981023 --- [host-check-agent-status2,7d69d48020837ffe,1374bb1044498cf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:35:46.113 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:35:46.114 INFO 2981023 --- [host-check-agent-status2,7d69d48020837ffe,1374bb1044498cf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:35:46.178 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:35:46.184 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:35:46.187 INFO 2981023 --- [ocp-schedules-26,42b014ca3aafd533,0dbb72ae398728b4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:35:46.187 INFO 2981023 --- [ocp-schedules-20,e926366205171059,c381da91fc38c40b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:35:46.187 INFO 2981023 --- [ocp-schedules-15,030479d25f3765db,f187780c331df3c0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:35:46.187 INFO 2981023 --- [ocp-schedules-10,66a5c8c3cafc0ee9,cd14bb4b47ae1e4f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:35:46.272 INFO 2981023 --- [ocp-schedules-15,030479d25f3765db,f187780c331df3c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:35:46.273 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:35:46.275 INFO 2981023 --- [ocp-schedules-28,6229bcc75581e0a6,e6ec21a5959fc1cf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:35:46.282 INFO 2981023 --- [ocp-schedules-26,42b014ca3aafd533,0dbb72ae398728b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:35:46.282 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:35:46.295 INFO 2981023 --- [ocp-schedules-20,e926366205171059,c381da91fc38c40b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:35:46.343 INFO 2981023 --- [ocp-schedules-20,e926366205171059,c381da91fc38c40b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:35:46.343 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050843420000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:35:46.368 INFO 2981023 --- [ocp-schedules-28,6229bcc75581e0a6,e6ec21a5959fc1cf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:35:46.382 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:35:46.379616+08:00, lastProcessTime=2024-09-11T18:35:36.337704+08:00 2024-09-11 18:35:46.397 INFO 2981023 --- [ocp-schedules-14,746aa29a29a4e19e,7ab0518d6e4e5c65] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:35:46.458 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:35:46.487 INFO 2981023 --- [host-check-agent-status4,7d69d48020837ffe,b468d6f30db9af33] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:35:46.487 INFO 2981023 --- [host-check-agent-status2,7d69d48020837ffe,1374bb1044498cf9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:35:46.489 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:35:46.490 INFO 2981023 --- [host-check-agent-status3,7d69d48020837ffe,e49f48a9b7718d5d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:35:46.492 INFO 2981023 --- [host-check-agent-status2,7d69d48020837ffe,1374bb1044498cf9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:35:46.492 INFO 2981023 --- [host-check-agent-status2,7d69d48020837ffe,1374bb1044498cf9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:35:46.492 INFO 2981023 --- [host-check-agent-status2,7d69d48020837ffe,1374bb1044498cf9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:35:46.492 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:35:46.493 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:35:46.497625248+08:00 2024-09-11 18:35:46.497 INFO 2981023 --- [host-check-agent-status4,7d69d48020837ffe,b468d6f30db9af33] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:35:46.497 INFO 2981023 --- [host-check-agent-status4,7d69d48020837ffe,b468d6f30db9af33] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:35:46.497 INFO 2981023 --- [host-check-agent-status4,7d69d48020837ffe,b468d6f30db9af33] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:35:46.497 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:35:46.493+08:00 2024-09-11 18:35:46.497 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:35:46.498 INFO 2981023 --- [host-check-agent-status3,7d69d48020837ffe,e49f48a9b7718d5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:35:46.498 INFO 2981023 --- [host-check-agent-status3,7d69d48020837ffe,e49f48a9b7718d5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:35:46.498 INFO 2981023 --- [host-check-agent-status3,7d69d48020837ffe,e49f48a9b7718d5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:35:46.498 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:35:46.498709642+08:00 2024-09-11 18:35:46.499 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:35:46.499328149+08:00 2024-09-11 18:35:46.499 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:35:46.498+08:00 2024-09-11 18:35:46.499 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:35:46.493+08:00 and status AVAILABLE 2024-09-11 18:35:46.501 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:35:46.502 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:35:46.499+08:00 2024-09-11 18:35:46.502 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:35:46.498+08:00 and status AVAILABLE 2024-09-11 18:35:46.503 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:35:46.502627882+08:00 2024-09-11 18:35:46.504 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:35:46.503+08:00 2024-09-11 18:35:46.505 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:35:46.499+08:00 and status AVAILABLE 2024-09-11 18:35:46.506 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:35:46.503+08:00 and status AVAILABLE 2024-09-11 18:35:46.554 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:35:46.554 INFO 2981023 --- [host-check-agent-status1,7d69d48020837ffe,81952918fe774c12] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:35:46.555 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:35:46.557 INFO 2981023 --- [host-check-agent-status1,7d69d48020837ffe,81952918fe774c12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:35:46.557 INFO 2981023 --- [host-check-agent-status1,7d69d48020837ffe,81952918fe774c12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:35:46.557 INFO 2981023 --- [host-check-agent-status1,7d69d48020837ffe,81952918fe774c12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:35:46.565 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:35:46.570 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.571 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:35:46.576 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.577 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.577 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:35:46.578 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:35:46.578 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:35:46.579 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.585 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:35:46.594 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:35:46.595 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:35:46.596 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:35:46.596 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.597 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:35:46.597 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:35:46.598 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:35:46.598 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:35:46.598 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.598 INFO 2981023 --- [ocp-schedules-26,42b014ca3aafd533,0dbb72ae398728b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:35:46.599 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:46.602 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:35:46.603 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:35:46.606 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:35:46.607 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.608 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:46.609 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:35:46.657 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.658 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:46.666 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:35:46.668 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:35:46.670 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:35:46.675 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:35:46.686 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:35:46.696 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:35:46.700 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:35:46.712 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.713 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:35:46.714 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:35:46.714 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.714 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:35:46.714 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:35:46.717 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:35:46.721 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.722 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.723 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:35:46.723 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:35:46.723 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.742 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:35:46.754 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:35:46.763 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:35:46.773 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:35:46.773 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050843420000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:35:46.777 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.785 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.785 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.786 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:35:46.787 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:35:46.787 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.787 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.788 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:35:46.788 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.788 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:46.789 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:35:46.789 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:35:46.789 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.790 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.793 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:35:46.793 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:35:46.794 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:35:46.795 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:35:46.795 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:35:46.796 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:35:46.799 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:35:46.814 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:35:46.814 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:35:46.823 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:35:46.831 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:35:46.833 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:35:46.837 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.838 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:35:46.838 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.839 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:35:46.839 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:35:46.841 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:35:46.843 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:35:46.846 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:35:46.850 INFO 2981023 --- [ocp-schedules-5,6991753c23755776,6f430bfa69a06a2f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:35:46.850 INFO 2981023 --- [ocp-schedules-5,6991753c23755776,6f430bfa69a06a2f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:35:46.860 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:35:46.861 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.861 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.862 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:35:46.863 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:35:46.868 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:35:46.870 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:35:46.875 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:35:46.876 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.876 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.876 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.876 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.877 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:35:46.877 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:35:46.877 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:35:46.877 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:35:46.877 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.878 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:35:46.878 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.881 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:35:46.886 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:35:46.890 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:35:46.891 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:35:46.891 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:35:46.895 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:35:46.898 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:35:46.899 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:35:46.899 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:35:46.908 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:35:46.911 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:35:46.912 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:35:46.918 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:35:46.920 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:35:46.924 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:35:46.928 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.931 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.932 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:35:46.933 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:35:46.937 INFO 2981023 --- [host-check-host-status1,7549ce9dc1cd1bdf,d28eea7006ceabc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:35:46.938 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:35:46.943 INFO 2981023 --- [host-check-host-status4,7549ce9dc1cd1bdf,e84c25350ddcf428] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:35:46.944 INFO 2981023 --- [host-check-host-status2,7549ce9dc1cd1bdf,9fda5ee2a4316953] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:35:46.949 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:46.950 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:35:46.950 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.951 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:35:46.951 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:35:46.952 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:46.964 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:35:46.965 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:35:46.971 INFO 2981023 --- [host-check-host-status3,7549ce9dc1cd1bdf,e050e8f92e338cb8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:35:46.990 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:35:46.997 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:35:47.004 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:47.004 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:47.005 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:35:47.009 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:35:47.011 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:35:47.011 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:35:47.025 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:35:47.026 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:35:46.379616+08:00 2024-09-11 18:35:47.027 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:35:47.035 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:35:47.035 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:47.036 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:35:47.037 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:35:47.042 INFO 2981023 --- [ocp-schedules-2,12d47928ef02dd01,ab32029eda2db809] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:35:47.043 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:35:47.043 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:35:47.044 INFO 2981023 --- [ocp-schedules-15,030479d25f3765db,f187780c331df3c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:35:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,6bc31f95e3c3e179,88e5cd619d0d4e43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,6bc31f95e3c3e179,88e5cd619d0d4e43] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:47.470 INFO 2981023 --- [ocp-schedules-28,3f7431ac5e6b490c,4362f489ce8a0afe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:35:47.497 INFO 2981023 --- [ocp-async-4,a5a358e79258eeb6,7fc79af4d4d4dc3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:47.535 WARN 2981023 --- [scanner-13697491-1,3f7431ac5e6b490c,fa1e55042a5ec9a6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:35:47.555 INFO 2981023 --- [alarm-detect-12,3f7431ac5e6b490c,79c560b0f33cae65] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007111] will be resolved. 2024-09-11 18:35:47.555 INFO 2981023 --- [alarm-detect-12,3f7431ac5e6b490c,79c560b0f33cae65] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007111] will be resolved. 2024-09-11 18:35:47.570 INFO 2981023 --- [alarm-detect-20,3f7431ac5e6b490c,709ab3ccdd84f82f] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007112] will be resolved. 2024-09-11 18:35:47.570 INFO 2981023 --- [alarm-detect-20,3f7431ac5e6b490c,709ab3ccdd84f82f] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007112] will be resolved. 2024-09-11 18:35:47.606 INFO 2981023 --- [alarm-detect-9,3f7431ac5e6b490c,e62510bd1880897b] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007113] will be resolved. 2024-09-11 18:35:47.606 INFO 2981023 --- [alarm-detect-9,3f7431ac5e6b490c,e62510bd1880897b] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007113] will be resolved. 2024-09-11 18:35:47.612 INFO 2981023 --- [alarm-detect-12,3f7431ac5e6b490c,79c560b0f33cae65] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:35:47.618 INFO 2981023 --- [alarm-detect-20,3f7431ac5e6b490c,709ab3ccdd84f82f] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:35:47.653 INFO 2981023 --- [alarm-detect-9,3f7431ac5e6b490c,e62510bd1880897b] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:35:47.685 INFO 2981023 --- [alarm-detect-14,3f7431ac5e6b490c,cd620c7ea5f62131] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007114] will be resolved. 2024-09-11 18:35:47.685 INFO 2981023 --- [alarm-detect-14,3f7431ac5e6b490c,cd620c7ea5f62131] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007114] will be resolved. 2024-09-11 18:35:47.709 INFO 2981023 --- [alarm-detect-14,3f7431ac5e6b490c,cd620c7ea5f62131] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:35:47.803 INFO 2981023 --- [ocp-async-6,6990c0e5ca7d3214,4000b0ae1d92144f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:35:48.494 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:48.495 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:49.498 INFO 2981023 --- [ocp-async-7,307f6d734d43e8c1,e6cd88e98acb008a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:49.529 INFO 2981023 --- [ocp-schedules-12,43d73aacfd5b2988,617b29e3357ce071] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:35:49.547 WARN 2981023 --- [ocp-schedules-12,43d73aacfd5b2988,617b29e3357ce071] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:35:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:35:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:35:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,b6d4fd76bb0ffbb1,53115be557916dfc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,b6d4fd76bb0ffbb1,53115be557916dfc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:50.385 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.390 INFO 2981023 --- [ocp-schedules-28,16fb50b14d6fb284,c9926251e6e6935a] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:35:50.394 INFO 2981023 --- [ocp-schedules-15,92cca9af742ae6ab,a99ba0a15963b797] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:35:50.527 INFO 2981023 --- [host-check-clockdiff1,2531405f2b2e961d,889b078261a7d858] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050950 0 0 , error= 2024-09-11 18:35:51.495 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:51.497 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:51.498 INFO 2981023 --- [ocp-async-1,8d3c811bc424ac55,a1f77c2b6f4d49b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:51.661 INFO 2981023 --- [ocp-schedules-17,2d534f048fc9eec2,0457761da0079b32] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:35:51.662 INFO 2981023 --- [ocp-schedules-17,2d534f048fc9eec2,0457761da0079b32] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:35:51.662 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:35:51.662 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:35:51.662 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:35:51.662 INFO 2981023 --- [ocp-schedules-17,2d534f048fc9eec2,0457761da0079b32] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:35:51.940 INFO 2981023 --- [host-check-clockdiff4,2531405f2b2e961d,5a106d0c7ab80afd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050951 0 0 , error= 2024-09-11 18:35:52.984 INFO 2981023 --- [ocp-schedules-24,2c86d06cd60ca6db,34a783223530561b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:35:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,6f84bdcd93096989,bfa31d52e54826d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,6f84bdcd93096989,bfa31d52e54826d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:53.433 INFO 2981023 --- [ocp-schedules-11,498f56ccf321b17d,3264081937aa0b6a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:35:53.433 INFO 2981023 --- [ocp-schedules-11,498f56ccf321b17d,3264081937aa0b6a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:35:53.498 INFO 2981023 --- [ocp-async-9,fea96127a89d4239,742571b0a2e0e42f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:54.497 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:54.498 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:54.555 INFO 2981023 --- [ocp-schedules-23,9a9a02885b2ef105,c7dca085b97425cc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:35:55.498 INFO 2981023 --- [ocp-async-3,bdfedc8371011aee,11feec2b7f6b0db3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:55.895 INFO 2981023 --- [ocp-schedules-12,c20429745a081605,02bbb423cb6e52c8] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:35:55.913 WARN 2981023 --- [ocp-schedules-12,c20429745a081605,02bbb423cb6e52c8] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:35:55.959 INFO 2981023 --- [ocp-schedules-2,cbafdc7f61e394f0,79f8c7539d9c2bdd] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:35:55.959 INFO 2981023 --- [ocp-schedules-2,cbafdc7f61e394f0,79f8c7539d9c2bdd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:35:55.960 INFO 2981023 --- [ocp-schedules-2,cbafdc7f61e394f0,79f8c7539d9c2bdd] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:35:55.960 INFO 2981023 --- [ocp-schedules-2,cbafdc7f61e394f0,79f8c7539d9c2bdd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:35:55.960 INFO 2981023 --- [ocp-schedules-2,cbafdc7f61e394f0,79f8c7539d9c2bdd] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:35:55.960 INFO 2981023 --- [ocp-schedules-2,cbafdc7f61e394f0,79f8c7539d9c2bdd] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:35:56.206 INFO 2981023 --- [ocp-schedules-11,e7f49603dc2029c3,9c63a314aea5ef99] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:35:56.212 INFO 2981023 --- [ocp-schedules-11,e7f49603dc2029c3,9c63a314aea5ef99] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:35:56.212 INFO 2981023 --- [ocp-schedules-11,e7f49603dc2029c3,9c63a314aea5ef99] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:35:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,904929d4c586bd4a,61e20b723e1e5e07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,904929d4c586bd4a,61e20b723e1e5e07] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:56.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:35:56.353013+08:00, lastProcessTime=2024-09-11T18:35:46.379616+08:00 2024-09-11 18:35:56.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=4, resultsCount=4 2024-09-11 18:35:56.495 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:35:56.353013+08:00 2024-09-11 18:35:56.512 INFO 2981023 --- [ocp-schedules-10,855baa74acfa1e1c,43b3e76ed2541c3d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:35:56.520 INFO 2981023 --- [ocp-schedules-10,855baa74acfa1e1c,43b3e76ed2541c3d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:35:56.520 INFO 2981023 --- [ocp-schedules-10,855baa74acfa1e1c,43b3e76ed2541c3d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:35:57.498 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:35:57.498 INFO 2981023 --- [ocp-async-5,8b0ec7b4c56b8c12,082ec7ac0a37b787] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:35:57.498 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:35:57.723 INFO 2981023 --- [ocp-schedules-23,cd2af25e8faf4675,fc480e6d4e6f85ea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:35:57.748 WARN 2981023 --- [scanner-13697699-1,cd2af25e8faf4675,e31cc3f2cf5f647d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:35:57.792 INFO 2981023 --- [alarm-detect-1,cd2af25e8faf4675,8e4fdebcb4949efd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646774, data is empty. 2024-09-11 18:35:57.796 INFO 2981023 --- [alarm-detect-1,cd2af25e8faf4675,8e4fdebcb4949efd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646774, data is empty. 2024-09-11 18:35:57.800 INFO 2981023 --- [alarm-detect-1,cd2af25e8faf4675,8e4fdebcb4949efd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646774, data is empty. 2024-09-11 18:35:57.803 INFO 2981023 --- [ocp-async-9,810f1bb0305c9f04,41d68327ad03b2d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:35:57.829 INFO 2981023 --- [alarm-detect-20,cd2af25e8faf4675,a698ba048bbb1fd1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646774, data is empty. 2024-09-11 18:35:57.833 INFO 2981023 --- [alarm-detect-13,cd2af25e8faf4675,500cf8b061df1b53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646774, data is empty. 2024-09-11 18:35:57.838 INFO 2981023 --- [alarm-detect-20,cd2af25e8faf4675,a698ba048bbb1fd1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646774, data is empty. 2024-09-11 18:35:57.841 INFO 2981023 --- [alarm-detect-13,cd2af25e8faf4675,500cf8b061df1b53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646774, data is empty. 2024-09-11 18:35:57.844 INFO 2981023 --- [alarm-detect-20,cd2af25e8faf4675,a698ba048bbb1fd1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646774, data is empty. 2024-09-11 18:35:57.846 INFO 2981023 --- [alarm-detect-13,cd2af25e8faf4675,500cf8b061df1b53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646774, data is empty. 2024-09-11 18:35:57.942 INFO 2981023 --- [alarm-detect-7,cd2af25e8faf4675,30b472ac75b038be] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646774, data is empty. 2024-09-11 18:35:57.944 INFO 2981023 --- [alarm-detect-7,cd2af25e8faf4675,30b472ac75b038be] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646774, data is empty. 2024-09-11 18:35:57.947 INFO 2981023 --- [alarm-detect-7,cd2af25e8faf4675,30b472ac75b038be] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646774, data is empty. 2024-09-11 18:35:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,c89076e98b63f6e1,ca61f14011109568] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726050960, interval=60 2024-09-11 18:35:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,c89076e98b63f6e1,ca61f14011109568] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:35:59.398 INFO 2981023 --- [ocp-schedules-22,1d210013f1147859,73d2d3baac634bb8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:35:59.398 INFO 2981023 --- [ocp-schedules-22,1d210013f1147859,73d2d3baac634bb8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:35:59.430 INFO 2981023 --- [Collect all tenants major compactions1,1d210013f1147859,79c13fd68962929c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:35:59.431 INFO 2981023 --- [Collect all tenants major compactions3,1d210013f1147859,0304bb053cc0d2e2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:35:59.435 INFO 2981023 --- [Collect all tenants major compactions5,1d210013f1147859,cf2840b11d989d65] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:35:59.437 INFO 2981023 --- [Collect all tenants major compactions4,1d210013f1147859,c3c6e3f8cd0cb7e0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:35:59.441 INFO 2981023 --- [Collect all tenants major compactions2,1d210013f1147859,800685ba8b107c21] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:35:59.442 INFO 2981023 --- [Collect all tenants major compactions6,1d210013f1147859,2098f11003e03475] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:35:59.451 INFO 2981023 --- [Collect all tenants major compactions4,1d210013f1147859,c3c6e3f8cd0cb7e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:59.452 INFO 2981023 --- [Collect all tenants major compactions5,1d210013f1147859,cf2840b11d989d65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:59.452 INFO 2981023 --- [Collect all tenants major compactions1,1d210013f1147859,79c13fd68962929c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:59.453 INFO 2981023 --- [Collect all tenants major compactions5,1d210013f1147859,cf2840b11d989d65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:35:59.453 INFO 2981023 --- [Collect all tenants major compactions1,1d210013f1147859,79c13fd68962929c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:35:59.453 INFO 2981023 --- [Collect all tenants major compactions4,1d210013f1147859,c3c6e3f8cd0cb7e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:35:59.455 INFO 2981023 --- [Collect all tenants major compactions3,1d210013f1147859,0304bb053cc0d2e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:59.456 INFO 2981023 --- [Collect all tenants major compactions3,1d210013f1147859,0304bb053cc0d2e2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:35:59.456 INFO 2981023 --- [Collect all tenants major compactions5,1d210013f1147859,cf2840b11d989d65] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:35:59.456 INFO 2981023 --- [Collect all tenants major compactions5,1d210013f1147859,cf2840b11d989d65] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:35:59.457 INFO 2981023 --- [Collect all tenants major compactions4,1d210013f1147859,c3c6e3f8cd0cb7e0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:35:59.457 INFO 2981023 --- [Collect all tenants major compactions4,1d210013f1147859,c3c6e3f8cd0cb7e0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:35:59.457 INFO 2981023 --- [Collect all tenants major compactions1,1d210013f1147859,79c13fd68962929c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:35:59.457 INFO 2981023 --- [Collect all tenants major compactions1,1d210013f1147859,79c13fd68962929c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:35:59.458 INFO 2981023 --- [Collect all tenants major compactions3,1d210013f1147859,0304bb053cc0d2e2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:35:59.458 INFO 2981023 --- [Collect all tenants major compactions3,1d210013f1147859,0304bb053cc0d2e2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:35:59.459 INFO 2981023 --- [Collect all tenants major compactions2,1d210013f1147859,800685ba8b107c21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:59.460 INFO 2981023 --- [Collect all tenants major compactions2,1d210013f1147859,800685ba8b107c21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:35:59.461 INFO 2981023 --- [Collect all tenants major compactions6,1d210013f1147859,2098f11003e03475] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:35:59.462 INFO 2981023 --- [Collect all tenants major compactions6,1d210013f1147859,2098f11003e03475] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:35:59.462 INFO 2981023 --- [Collect all tenants major compactions2,1d210013f1147859,800685ba8b107c21] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:35:59.462 INFO 2981023 --- [Collect all tenants major compactions2,1d210013f1147859,800685ba8b107c21] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:35:59.464 INFO 2981023 --- [Collect all tenants major compactions6,1d210013f1147859,2098f11003e03475] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:35:59.464 INFO 2981023 --- [Collect all tenants major compactions6,1d210013f1147859,2098f11003e03475] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:35:59.464 INFO 2981023 --- [ocp-schedules-22,1d210013f1147859,73d2d3baac634bb8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:35:59.498 INFO 2981023 --- [ocp-async-4,f1e8e32c791df30c,cb9b27d8244d296a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:00.498 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:00.499 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:01.498 INFO 2981023 --- [ocp-async-8,7e4a5b8fc5ce9f71,40bb35a99e974da5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:02.185 INFO 2981023 --- [ocp-schedules-13,b184885b28eb2cd5,5233869727343f34] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:36:02.185+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:36:02.187 INFO 2981023 --- [ocp-schedules-13,b184885b28eb2cd5,5233869727343f34] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:36:02.231 INFO 2981023 --- [ocp-schedules-13,b184885b28eb2cd5,5233869727343f34] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 41 to SqlAggregateData 2024-09-11 18:36:02.232 INFO 2981023 --- [ocp-schedules-13,b184885b28eb2cd5,5233869727343f34] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:36:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,5f28b57cb7e9aa8b,f17fcecc39f09503] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,5f28b57cb7e9aa8b,f17fcecc39f09503] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:02.986 INFO 2981023 --- [ocp-schedules-26,3ba9bc2be19b8c15,647f9850e6eb562e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:36:03.429 INFO 2981023 --- [ocp-schedules-30,d554e5d16377ed12,ddbf9562118779f2] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:36:03.430 INFO 2981023 --- [ocp-schedules-30,d554e5d16377ed12,ddbf9562118779f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:03.451 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:03.451 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:36:03.456 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:03.458 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:36:03.458 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050963458000} 2024-09-11 18:36:03.470 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:36:03.474 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000242, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:36:03.474 INFO 2981023 --- [ocp-schedules-30,d554e5d16377ed12,ddbf9562118779f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:36:03.481 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:36:03.481 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:36:03.481 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:36:03.484 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000242, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:36:03.487 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:36:03.498 INFO 2981023 --- [ocp-async-1,60dbbbd71fb13bca,f1ed9e1a9c8e963d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:03.499 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:03.499 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:03.512 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000242 2024-09-11 18:36:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,1ddbd1b2752110b3,41f4a7995509c9a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,1ddbd1b2752110b3,41f4a7995509c9a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:05.499 INFO 2981023 --- [ocp-async-10,4643c578e920e95e,4778b53bff96d17d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:05.536 INFO 2981023 --- [ocp-schedules-27,ae2e0f36f15a345b,b529b4f1564baa48] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:36:05.537 INFO 2981023 --- [ocp-schedules-27,ae2e0f36f15a345b,b529b4f1564baa48] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:05.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:36:05.582 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.586 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:36:05.590 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:36:05.590 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.591 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.591 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.593 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:36:05.598 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.598 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.598 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.598 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.599 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:36:05.599 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.602 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:36:05.604 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.604 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:36:05.606 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.606 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.610 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.611 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.611 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.612 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:36:05.618 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.618 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:36:05.619 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.620 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.621 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.622 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.622 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:36:05.622 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE7321D6DC-0-0] 2024-09-11 18:36:05.622 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:36:05.623 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.623 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:36:05.625 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.625 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:36:05.633 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.634 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.634 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.634 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:36:05.639 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:36:05.640 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.641 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:36:05.641 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.642 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:36:05.642 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.642 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.643 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:36:05.643 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:36:05.643 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.643 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.644 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:36:05.644 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.644 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:36:05.645 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:36:05.645 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.645 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.646 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:36:05.646 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:36:05.646 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.646 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:36:05.647 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:36:05.647 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.647 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.647 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:36:05.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:36:05.648 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:36:05.648 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE7321D6E6-0-0] 2024-09-11 18:36:05.648 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:36:05.648 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:36:05.650 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:36:05.651 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:36:05.654 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.655 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:36:05.656 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.657 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:36:05.657 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.658 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:36:05.659 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.659 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:36:05.660 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:36:05.661 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.662 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:36:05.662 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:36:05.663 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:05.664 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:36:05.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.665 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:36:05.667 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:05.669 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.670 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.671 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:36:05.675 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:36:05.675 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:05.675 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.675 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:36:05.677 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.678 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:36:05.680 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:36:05.683 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:36:05.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:36:05.684 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:36:05.684 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:36:05.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.685 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:36:05.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.689 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:36:05.691 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:36:05.691 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:36:05.691 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.691 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:36:05.692 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:36:05.692 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:36:05.692 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:36:05.699 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:36:05.700 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:36:05.700 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:36:05.700 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:36:05.700 INFO 2981023 --- [ocp-schedules-27,ae2e0f36f15a345b,b529b4f1564baa48] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:36:05.967 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:36:05.982 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:36:05.987 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:36:05.992 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:36:05.992 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:36:05.992 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:36:05.997 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.997 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:36:05.998 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:05.999 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:36:06.049 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:33:41, start=2024-09-11T18:30:11, end=2024-09-11T18:32:41, updates=305, granule=120000ms, delay=210000ms 2024-09-11 18:36:06.058 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:33:41, target=2024-09-11T18:32:41, granule=120000ms, delay=60000ms) 2024-09-11 18:36:06.058 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:32:41, target=2024-09-11T18:20:51, granule=600000ms, delay=710000ms) 2024-09-11 18:36:06.058 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=91ms, period=120000ms 2024-09-11 18:36:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:36:06.352953+08:00, lastProcessTime=2024-09-11T18:35:56.353013+08:00 2024-09-11 18:36:06.512 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:06.512 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:36:06.352953+08:00 2024-09-11 18:36:06.514 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:07.499 INFO 2981023 --- [ocp-async-4,ddcb118a1ffdb6dd,935bb9572b301432] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:07.803 INFO 2981023 --- [ocp-async-5,d274ced69ecc1007,6e0cb5a10963633b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:36:08.002 INFO 2981023 --- [ocp-schedules-28,ee4b4194154689c1,453edf5de3f954ec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:36:08.048 WARN 2981023 --- [scanner-13698013-1,ee4b4194154689c1,54329306b6db9041] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:36:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,6249382a82c9233d,6e44d39ff608f7fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,6249382a82c9233d,6e44d39ff608f7fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:08.364 INFO 2981023 --- [ocp-schedules-3,ba40e076452c8386,cae919bd8cf95515] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:36:08.438 INFO 2981023 --- [ocp-schedules-2,6e88a08b9d80dbe1,f73b7d315628f0ed] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:36:08.438+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:36:08.439 INFO 2981023 --- [ocp-schedules-2,6e88a08b9d80dbe1,f73b7d315628f0ed] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:36:08.440 INFO 2981023 --- [ocp-schedules-2,6e88a08b9d80dbe1,f73b7d315628f0ed] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:36:08.441 INFO 2981023 --- [ocp-schedules-2,6e88a08b9d80dbe1,f73b7d315628f0ed] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:36:09.499 INFO 2981023 --- [ocp-async-2,3c479d957a006e3a,db67e34708d1c213] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:09.514 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:09.514 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:10.003 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:36:10.003 INFO 2981023 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:36:10.915 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:36:10.918 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:36:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,bdd0d52203abdd24,1348dd51e5b6c8ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:11.321 INFO 2981023 --- [prometheus-minute-schedule-1,bdd0d52203abdd24,1348dd51e5b6c8ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:11.499 INFO 2981023 --- [ocp-async-9,779da6f88943fc84,7aa8ba84838e4f46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:12.187 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050900, nodeCount=480, affectRows=480 2024-09-11 18:36:12.514 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:12.516 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:12.708 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050900, nodeCount=1000, affectRows=1000 2024-09-11 18:36:12.711 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050900, nodeCount=999, validCount=999 2024-09-11 18:36:12.720 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050900, nodeCount=48, affectRows=48 2024-09-11 18:36:12.721 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050900, nodeCount=46, validCount=46 2024-09-11 18:36:12.721 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050900, nodeCount=1000, affectRows=1000 2024-09-11 18:36:12.723 INFO 2981023 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050900, nodeCount=1000, validCount=1000 2024-09-11 18:36:12.989 INFO 2981023 --- [ocp-schedules-1,7575653b86730535,fcc32a7de69a507e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:36:13.499 INFO 2981023 --- [ocp-async-3,e6af94f1cc5d8955,181d3d993f2b8f2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:14.185 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050900, nodeCount=480, validCount=480 2024-09-11 18:36:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,6d24895e08772447,b2f1e94bd6c8c5b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,5837e90a37c51aea,4678d619586777e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:36:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,6d24895e08772447,b2f1e94bd6c8c5b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:15.499 INFO 2981023 --- [ocp-async-8,b43929dab728e9fd,e47ad2692f2fcc03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:15.516 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:15.517 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:16.017 INFO 2981023 --- [ocp-schedules-18,909264ebc6f8b59c,38e32822ae145307] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:36:16.018 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:36:16.018 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:36:16.018 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:36:16.018 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:36:16.029 INFO 2981023 --- [ocp-schedules-24,a06fcc0f6ad67ae3,c99edcd3dfdce19b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:36:16.029 INFO 2981023 --- [host-check-clockdiff1,a06fcc0f6ad67ae3,b734ff98b7cd193e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:36:16.030 INFO 2981023 --- [host-check-clockdiff2,a06fcc0f6ad67ae3,5da4504327eed47e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:36:16.030 INFO 2981023 --- [host-check-clockdiff3,a06fcc0f6ad67ae3,16abed3ee7f4fa74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:36:16.033 INFO 2981023 --- [host-check-clockdiff4,a06fcc0f6ad67ae3,cf39644deeebc326] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:36:16.042 INFO 2981023 --- [host-check-clockdiff2,a06fcc0f6ad67ae3,5da4504327eed47e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050976 5 5 , error= 2024-09-11 18:36:16.050 INFO 2981023 --- [host-check-clockdiff3,a06fcc0f6ad67ae3,16abed3ee7f4fa74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050976 0 0 , error= 2024-09-11 18:36:16.053 INFO 2981023 --- [ocp-schedules-29,6ade9272e5437459,f6c7b42d85eca95f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:36:16.055 INFO 2981023 --- [host-check-agent-status1,6ade9272e5437459,ab8b1e2a275ede01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:36:16.055 INFO 2981023 --- [host-check-agent-status2,6ade9272e5437459,bc87c07a6f9b795a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:36:16.056 INFO 2981023 --- [host-check-agent-status3,6ade9272e5437459,9d24d9f2de5d98fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:36:16.056 INFO 2981023 --- [host-check-agent-status4,6ade9272e5437459,7fffd0cfe951219b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:36:16.059 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:36:16.064 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:36:16.067 INFO 2981023 --- [host-check-agent-status1,6ade9272e5437459,ab8b1e2a275ede01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:36:16.068 INFO 2981023 --- [host-check-agent-status2,6ade9272e5437459,bc87c07a6f9b795a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:36:16.068 INFO 2981023 --- [host-check-agent-status1,6ade9272e5437459,ab8b1e2a275ede01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:36:16.081 INFO 2981023 --- [host-check-agent-status2,6ade9272e5437459,bc87c07a6f9b795a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:36:16.082 INFO 2981023 --- [host-check-agent-status4,6ade9272e5437459,7fffd0cfe951219b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:36:16.119 INFO 2981023 --- [host-check-agent-status4,6ade9272e5437459,7fffd0cfe951219b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:36:16.146 INFO 2981023 --- [ocp-schedules-27,d4de0a5eba2ff9af,bac0d678a6277015] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:36:16.176 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:36:16.197 INFO 2981023 --- [ocp-schedules-14,1f9250facd92c5be,a1674042df0f2c6b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:36:16.197 INFO 2981023 --- [ocp-schedules-5,391449c642fe0067,120036c91f2acd9c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:36:16.197 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:36:16.198 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:36:16.206 INFO 2981023 --- [ocp-schedules-19,6a7d91f9b66836db,7374ddde347fa103] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:36:16.206 INFO 2981023 --- [ocp-schedules-21,bc8a530bd5138a11,ab17211c2f7b625b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:36:16.207 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:36:16.208 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:36:16.210 INFO 2981023 --- [ocp-schedules-14,1f9250facd92c5be,a1674042df0f2c6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:16.232 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050963458000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:36:16.260 INFO 2981023 --- [ocp-schedules-5,391449c642fe0067,120036c91f2acd9c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:16.261 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:36:16.279 INFO 2981023 --- [host-check-agent-status3,6ade9272e5437459,9d24d9f2de5d98fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:36:16.283 INFO 2981023 --- [host-check-agent-status3,6ade9272e5437459,9d24d9f2de5d98fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:36:16.291 INFO 2981023 --- [ocp-schedules-21,bc8a530bd5138a11,ab17211c2f7b625b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:16.293 INFO 2981023 --- [ocp-schedules-5,391449c642fe0067,120036c91f2acd9c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:36:16.328 INFO 2981023 --- [host-check-agent-status1,6ade9272e5437459,ab8b1e2a275ede01] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:36:16.332 INFO 2981023 --- [host-check-agent-status1,6ade9272e5437459,ab8b1e2a275ede01] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:36:16.332 INFO 2981023 --- [host-check-agent-status1,6ade9272e5437459,ab8b1e2a275ede01] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:36:16.332 INFO 2981023 --- [host-check-agent-status1,6ade9272e5437459,ab8b1e2a275ede01] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:36:16.359 INFO 2981023 --- [ocp-schedules-20,6169a847507b8b92,36faeb2919118156] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:36:16.359 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:36:16.361 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:36:16.361644518+08:00 2024-09-11 18:36:16.366 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:36:16.361+08:00 2024-09-11 18:36:16.370 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:36:16.361+08:00 and status AVAILABLE 2024-09-11 18:36:16.371 INFO 2981023 --- [host-check-agent-status4,6ade9272e5437459,7fffd0cfe951219b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:36:16.376 INFO 2981023 --- [host-check-agent-status4,6ade9272e5437459,7fffd0cfe951219b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:36:16.376 INFO 2981023 --- [host-check-agent-status4,6ade9272e5437459,7fffd0cfe951219b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:36:16.376 INFO 2981023 --- [host-check-agent-status4,6ade9272e5437459,7fffd0cfe951219b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:36:16.384 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.385 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:36:16.387 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:36:16.388 INFO 2981023 --- [host-check-agent-status2,6ade9272e5437459,bc87c07a6f9b795a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:36:16.389 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:36:16.389 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:36:16.389416389+08:00 2024-09-11 18:36:16.391 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:36:16.389+08:00 2024-09-11 18:36:16.393 INFO 2981023 --- [host-check-agent-status3,6ade9272e5437459,9d24d9f2de5d98fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:36:16.394 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:36:16.389+08:00 and status AVAILABLE 2024-09-11 18:36:16.401 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:36:16.400219+08:00, lastProcessTime=2024-09-11T18:36:06.352953+08:00 2024-09-11 18:36:16.403 INFO 2981023 --- [host-check-agent-status2,6ade9272e5437459,bc87c07a6f9b795a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:36:16.403 INFO 2981023 --- [host-check-agent-status2,6ade9272e5437459,bc87c07a6f9b795a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:36:16.403 INFO 2981023 --- [host-check-agent-status2,6ade9272e5437459,bc87c07a6f9b795a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:36:16.405 INFO 2981023 --- [host-check-agent-status3,6ade9272e5437459,9d24d9f2de5d98fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:36:16.405 INFO 2981023 --- [host-check-agent-status3,6ade9272e5437459,9d24d9f2de5d98fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:36:16.405 INFO 2981023 --- [host-check-agent-status3,6ade9272e5437459,9d24d9f2de5d98fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:36:16.406 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:36:16.408 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:36:16.414 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.415 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:16.425 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.432 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:36:16.432 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:36:16.433 INFO 2981023 --- [ocp-schedules-14,1f9250facd92c5be,a1674042df0f2c6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:36:16.455 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:36:16.456 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:36:16.461176844+08:00 2024-09-11 18:36:16.458 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:36:16.456+08:00 2024-09-11 18:36:16.460 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:36:16.462 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:36:16.461446522+08:00 2024-09-11 18:36:16.462 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:36:16.462+08:00 2024-09-11 18:36:16.462 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:36:16.463 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.465 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:16.465 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:36:16.466 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:36:16.456+08:00 and status AVAILABLE 2024-09-11 18:36:16.466 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:36:16.462+08:00 and status AVAILABLE 2024-09-11 18:36:16.467 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:36:16.468 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:36:16.480 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:36:16.480 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:36:16.490 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:36:16.491 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:36:16.495 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:36:16.496 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:36:16.499 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:36:16.499 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.500 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:16.502 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:36:16.506 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:16.507 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.508 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:36:16.523 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:36:16.530 INFO 2981023 --- [ocp-schedules-23,bf4302654d663b60,6386da0cf259e650] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:36:16.539 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:36:16.539 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:36:16.545 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:36:16.554 INFO 2981023 --- [ocp-schedules-23,bf4302654d663b60,6386da0cf259e650] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:36:16.554 INFO 2981023 --- [ocp-schedules-23,bf4302654d663b60,6386da0cf259e650] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:36:16.560 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:36:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:36:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050963458000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:36:16.566 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:36:16.570 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:16.571 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.572 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:16.572 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:36:16.572 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.574 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:36:16.575 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:36:16.576 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:36:16.583 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:36:16.588 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:36:16.589 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:36:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:36:16.592 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:36:16.601 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:36:16.601 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:36:16.615 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:36:16.615 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:36:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:36:16.623 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:36:16.636 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:16.637 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.638 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:36:16.638 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:36:16.641 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:36:16.644 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:36:16.644 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:36:16.648 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:36:16.656 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:36:16.656 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:36:16.657 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:36:16.657 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:16.658 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.659 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:36:16.659 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:36:16.664 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:36:16.666 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:36:16.676 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:36:16.678 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:36:16.686 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:36:16.687 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:16.688 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.689 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:16.690 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:36:16.690 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.693 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:36:16.693 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:36:16.698 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:36:16.701 INFO 2981023 --- [host-check-host-status1,909264ebc6f8b59c,0734bed042e549de] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:36:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:36:16.707 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:36:16.708 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:36:16.719 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:36:16.727 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:36:16.735 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:16.736 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:36:16.736 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.736 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:36:16.737 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:36:16.738 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:36:16.740 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:36:16.742 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:36:16.748 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:36:16.749 INFO 2981023 --- [host-check-host-status2,909264ebc6f8b59c,e1a4e591fe840c01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:36:16.750 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:36:16.751 INFO 2981023 --- [host-check-host-status3,909264ebc6f8b59c,2785d9145175b3ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:36:16.752 INFO 2981023 --- [host-check-host-status4,909264ebc6f8b59c,0d8c30641238436d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:36:16.767 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:36:16.773 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:36:16.776 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:16.776 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.777 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:36:16.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:36:16.784 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:36:16.803 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:36:16.813 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:16.814 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:16.814 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:36:16.821 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:36:16.821 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:36:16.822 INFO 2981023 --- [ocp-schedules-21,bc8a530bd5138a11,ab17211c2f7b625b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:36:16.831 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:36:16.400219+08:00 2024-09-11 18:36:16.982 INFO 2981023 --- [host-check-clockdiff1,a06fcc0f6ad67ae3,b734ff98b7cd193e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050976 0 0 , error= 2024-09-11 18:36:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,a80647896da65133,68b361536694b7cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:17.319 INFO 2981023 --- [prometheus-minute-schedule-1,a80647896da65133,68b361536694b7cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:17.499 INFO 2981023 --- [ocp-async-1,7c58e80df80255f8,d085ae41e698fd77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:17.803 INFO 2981023 --- [ocp-async-3,c5809574683d43e3,2ef61ff35581e7ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:36:18.192 INFO 2981023 --- [ocp-schedules-30,5d593cfcb6065eb4,b27d9000b4f3db71] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:36:18.240 WARN 2981023 --- [scanner-13699779-1,5d593cfcb6065eb4,45477cfb8ac60cda] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:36:18.257 INFO 2981023 --- [alarm-detect-19,5d593cfcb6065eb4,e532b92b5524c1b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646774, data is empty. 2024-09-11 18:36:18.265 INFO 2981023 --- [alarm-detect-19,5d593cfcb6065eb4,e532b92b5524c1b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646815, data is empty. 2024-09-11 18:36:18.269 INFO 2981023 --- [alarm-detect-19,5d593cfcb6065eb4,e532b92b5524c1b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646815, data is empty. 2024-09-11 18:36:18.276 INFO 2981023 --- [alarm-detect-19,5d593cfcb6065eb4,9b45e9f17a827812] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646774, data is empty. 2024-09-11 18:36:18.282 INFO 2981023 --- [alarm-detect-19,5d593cfcb6065eb4,9b45e9f17a827812] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646815, data is empty. 2024-09-11 18:36:18.283 INFO 2981023 --- [alarm-detect-11,5d593cfcb6065eb4,09acefa3755d7e70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646774, data is empty. 2024-09-11 18:36:18.285 INFO 2981023 --- [alarm-detect-19,5d593cfcb6065eb4,9b45e9f17a827812] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646815, data is empty. 2024-09-11 18:36:18.288 INFO 2981023 --- [alarm-detect-11,5d593cfcb6065eb4,09acefa3755d7e70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646815, data is empty. 2024-09-11 18:36:18.292 INFO 2981023 --- [alarm-detect-11,5d593cfcb6065eb4,09acefa3755d7e70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646815, data is empty. 2024-09-11 18:36:18.391 INFO 2981023 --- [alarm-detect-10,5d593cfcb6065eb4,5801bc56b733ddad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646774, data is empty. 2024-09-11 18:36:18.394 INFO 2981023 --- [alarm-detect-10,5d593cfcb6065eb4,5801bc56b733ddad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646815, data is empty. 2024-09-11 18:36:18.397 INFO 2981023 --- [alarm-detect-10,5d593cfcb6065eb4,5801bc56b733ddad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646815, data is empty. 2024-09-11 18:36:18.517 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:18.517 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:19.499 INFO 2981023 --- [ocp-async-7,2e36d0744b967892,331fb111d16c10d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,cf0047ee315cc1aa,3854c8b8bc84efd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,cf0047ee315cc1aa,3854c8b8bc84efd6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:21.500 INFO 2981023 --- [ocp-async-5,5561bcaa04b6f42e,0989b5de64a992b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:21.517 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:21.518 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:22.021 INFO 2981023 --- [host-check-clockdiff4,a06fcc0f6ad67ae3,cf39644deeebc326] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726050982 0 0 , error= 2024-09-11 18:36:22.992 INFO 2981023 --- [ocp-schedules-3,707dd3bf94b0de6e,5550e3fe5d8be027] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:36:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,161e2665d73804b5,6d6088d51f865e27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,161e2665d73804b5,6d6088d51f865e27] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:23.500 INFO 2981023 --- [ocp-async-6,014071592e48a19a,57f5799094401b67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:24.518 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:24.519 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:24.558 INFO 2981023 --- [ocp-schedules-25,98bd60055a4359cb,6c0b1d9505300d2c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:36:25.500 INFO 2981023 --- [ocp-async-10,e7184b7334cd4cd0,7778c3440d8d27b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,73fb7dcdde943518,823e96ff999de4cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:26.317 INFO 2981023 --- [prometheus-minute-schedule-1,73fb7dcdde943518,823e96ff999de4cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:36:26.351538+08:00, lastProcessTime=2024-09-11T18:36:16.400219+08:00 2024-09-11 18:36:26.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:36:26.351538+08:00 2024-09-11 18:36:27.500 INFO 2981023 --- [ocp-async-4,6061cd2098f5a708,e8e2c3af73d3fef4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:27.519 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:27.522 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:27.803 INFO 2981023 --- [ocp-async-6,f19e1605785b72eb,010cd424ede2b95b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:36:28.457 INFO 2981023 --- [ocp-schedules-19,0ab00b2bce74246d,d7b45817040951fd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:36:28.500 WARN 2981023 --- [scanner-13700068-1,0ab00b2bce74246d,230b4004d2e1d5f7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:36:29.098 INFO 2981023 --- [ocp-schedules-10,944d67a858933ed5,ddce3ba6c0273e66] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:36:29.098 INFO 2981023 --- [ocp-schedules-10,944d67a858933ed5,ddce3ba6c0273e66] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:36:29.098 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:36:29.098 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:36:29.098 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:36:29.124 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:36:29.124 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:36:29.129 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:36:29.129 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:36:29.135 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:36:29.136 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:36:29.176 INFO 2981023 --- [ocp-schedules-10,944d67a858933ed5,ddce3ba6c0273e66] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:36:29.183 INFO 2981023 --- [ocp-schedules-10,944d67a858933ed5,ddce3ba6c0273e66] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:36:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,bcfc0ac872227765,b7625f45fcedd7be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:29.357 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:36:29.358 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:36:29.359 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:36:29.361 INFO 2981023 --- [prometheus-minute-schedule-1,bcfc0ac872227765,b7625f45fcedd7be] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:29.381 INFO 2981023 --- [ocp-schedules-10,944d67a858933ed5,ddce3ba6c0273e66] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:36:29.382 INFO 2981023 --- [ocp-schedules-10,944d67a858933ed5,ddce3ba6c0273e66] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:36:29.500 INFO 2981023 --- [ocp-async-1,61b4ada7e49c5b8b,ac7ee0f949a09c5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:30.523 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:30.523 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:31.500 INFO 2981023 --- [ocp-async-3,f7776e113d778461,55a52e3e22e0359d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,a3a074f0adb5ccf8,8beeec46af87d728] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,a3a074f0adb5ccf8,8beeec46af87d728] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:32.995 INFO 2981023 --- [ocp-schedules-29,bb6a6a00b594c01a,823134d994317691] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:36:33.500 INFO 2981023 --- [ocp-async-8,ac34f6d7444718e6,662b78df43b91767] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:33.523 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:33.523 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,8ff24167055a58a1,aba8979a9b5017d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,8ff24167055a58a1,aba8979a9b5017d1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:35.500 INFO 2981023 --- [ocp-async-2,5b6a6b9901f5e38d,b241b2ed9837af6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:36:36.336807+08:00, lastProcessTime=2024-09-11T18:36:26.351538+08:00 2024-09-11 18:36:36.459 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:36:36.336807+08:00 2024-09-11 18:36:36.523 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:36.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:36.557 INFO 2981023 --- [ocp-schedules-12,920e51faee7eee91,64b00aeae9215fdc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:36:36.567 INFO 2981023 --- [ocp-schedules-12,920e51faee7eee91,64b00aeae9215fdc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:36:36.568 INFO 2981023 --- [ocp-schedules-12,920e51faee7eee91,64b00aeae9215fdc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:36:37.314 INFO 2981023 --- [prometheus-cache-monitor1,3b7f0d098267c212,d71bfe8e87b58a3d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8194,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:36:37.315 INFO 2981023 --- [prometheus-cache-monitor1,3b7f0d098267c212,d71bfe8e87b58a3d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:36:37.501 INFO 2981023 --- [ocp-async-9,bad951832a4efbd7,2204f99f331404da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:37.803 INFO 2981023 --- [ocp-async-8,d3928831f85efdfe,c83f1b118efc43b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:36:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,de2390eb18cfb52f,b4362715ff592e02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,de2390eb18cfb52f,b4362715ff592e02] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:38.674 INFO 2981023 --- [ocp-schedules-4,dd3553b6e6d1f475,3b2a4e05b5b702e5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:36:38.716 WARN 2981023 --- [scanner-13700272-1,dd3553b6e6d1f475,2a82c9b216accd4a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:36:38.746 INFO 2981023 --- [alarm-detect-17,dd3553b6e6d1f475,f140c4fc484bb3e2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646815, data is empty. 2024-09-11 18:36:38.752 INFO 2981023 --- [alarm-detect-17,dd3553b6e6d1f475,f140c4fc484bb3e2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646815, data is empty. 2024-09-11 18:36:38.757 INFO 2981023 --- [alarm-detect-17,dd3553b6e6d1f475,f140c4fc484bb3e2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646815, data is empty. 2024-09-11 18:36:38.768 INFO 2981023 --- [alarm-detect-2,dd3553b6e6d1f475,e06045a4da3fd2ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646815, data is empty. 2024-09-11 18:36:38.771 INFO 2981023 --- [alarm-detect-11,dd3553b6e6d1f475,8060e0e7fa6c6312] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646815, data is empty. 2024-09-11 18:36:38.775 INFO 2981023 --- [alarm-detect-11,dd3553b6e6d1f475,8060e0e7fa6c6312] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646815, data is empty. 2024-09-11 18:36:38.777 INFO 2981023 --- [alarm-detect-2,dd3553b6e6d1f475,e06045a4da3fd2ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646815, data is empty. 2024-09-11 18:36:38.784 INFO 2981023 --- [alarm-detect-11,dd3553b6e6d1f475,8060e0e7fa6c6312] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646815, data is empty. 2024-09-11 18:36:38.788 INFO 2981023 --- [alarm-detect-2,dd3553b6e6d1f475,e06045a4da3fd2ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646815, data is empty. 2024-09-11 18:36:38.883 INFO 2981023 --- [alarm-detect-19,dd3553b6e6d1f475,bf0c76f50050538b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646815, data is empty. 2024-09-11 18:36:38.887 INFO 2981023 --- [alarm-detect-19,dd3553b6e6d1f475,bf0c76f50050538b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646815, data is empty. 2024-09-11 18:36:38.890 INFO 2981023 --- [alarm-detect-19,dd3553b6e6d1f475,bf0c76f50050538b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646815, data is empty. 2024-09-11 18:36:39.501 INFO 2981023 --- [ocp-async-4,4e60e0e59a6ccbbf,1fd34a72cab6cf78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:39.524 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:39.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:40.004 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:36:40.004 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:36:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,21efeb42704a382a,4d45d133348f7391] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,21efeb42704a382a,4d45d133348f7391] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:41.501 INFO 2981023 --- [ocp-async-6,c31a884dd135e5f7,0a67667888e8b24c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:42.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:42.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:42.997 INFO 2981023 --- [ocp-schedules-28,00760de10e545667,2822b8e8e1305373] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:36:43.501 INFO 2981023 --- [ocp-async-10,2a3350ed26ce44ab,7ac9ec1f642e0f6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,6a296e95cea4100d,fd4e1f6fab00f62e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,6a296e95cea4100d,fd4e1f6fab00f62e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:45.101 INFO 2981023 --- [prometheus-second-schedule-1,e24622a7d69a65ba,adb33182ce0ad15d] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:36:45.501 INFO 2981023 --- [ocp-async-7,78ce31e569eff581,8f9ded3fce144de7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:45.525 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:45.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:46.014 INFO 2981023 --- [ocp-schedules-1,793207864b904cc0,e8e02d9be3255c55] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:36:46.015 INFO 2981023 --- [host-check-clockdiff1,793207864b904cc0,7c4ab2a2fee43328] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:36:46.015 INFO 2981023 --- [host-check-clockdiff2,793207864b904cc0,a8413e7834152878] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:36:46.015 INFO 2981023 --- [host-check-clockdiff3,793207864b904cc0,aa0e728ad9d3d220] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:36:46.018 INFO 2981023 --- [host-check-clockdiff4,793207864b904cc0,a4d660413caffe4e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:36:46.027 INFO 2981023 --- [host-check-clockdiff3,793207864b904cc0,aa0e728ad9d3d220] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051006 0 0 , error= 2024-09-11 18:36:46.029 INFO 2981023 --- [ocp-schedules-3,3bd8cccd594e7a2e,1a7ffbba13cd55fb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:36:46.029 INFO 2981023 --- [host-check-agent-status1,3bd8cccd594e7a2e,48d3c874b23c3df2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:36:46.035 INFO 2981023 --- [host-check-agent-status3,3bd8cccd594e7a2e,d912c1d0315cf80b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:36:46.035 INFO 2981023 --- [host-check-agent-status2,3bd8cccd594e7a2e,bc070c06e88c360a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:36:46.035 INFO 2981023 --- [ocp-schedules-8,fdd4dc456b38f169,46d49987fef3bc66] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:36:46.035 INFO 2981023 --- [host-check-agent-status4,3bd8cccd594e7a2e,ad65abde072b68c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:36:46.036 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:36:46.036 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:36:46.037 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:36:46.038 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:36:46.041 INFO 2981023 --- [host-check-agent-status2,3bd8cccd594e7a2e,bc070c06e88c360a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:36:46.042 INFO 2981023 --- [host-check-agent-status3,3bd8cccd594e7a2e,d912c1d0315cf80b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:36:46.043 INFO 2981023 --- [host-check-agent-status4,3bd8cccd594e7a2e,ad65abde072b68c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:36:46.044 INFO 2981023 --- [host-check-agent-status2,3bd8cccd594e7a2e,bc070c06e88c360a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:36:46.044 INFO 2981023 --- [host-check-agent-status3,3bd8cccd594e7a2e,d912c1d0315cf80b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:36:46.044 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:36:46.045 INFO 2981023 --- [host-check-agent-status1,3bd8cccd594e7a2e,48d3c874b23c3df2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:36:46.045 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:36:46.045 INFO 2981023 --- [host-check-agent-status4,3bd8cccd594e7a2e,ad65abde072b68c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:36:46.046 INFO 2981023 --- [host-check-agent-status1,3bd8cccd594e7a2e,48d3c874b23c3df2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:36:46.046 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:36:46.047 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:36:46.048 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:36:46.050 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:36:46.051 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:36:46.051 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:36:46.070 INFO 2981023 --- [host-check-agent-status3,3bd8cccd594e7a2e,d912c1d0315cf80b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:36:46.074 INFO 2981023 --- [host-check-agent-status3,3bd8cccd594e7a2e,d912c1d0315cf80b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:36:46.074 INFO 2981023 --- [host-check-agent-status3,3bd8cccd594e7a2e,d912c1d0315cf80b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:36:46.074 INFO 2981023 --- [host-check-agent-status3,3bd8cccd594e7a2e,d912c1d0315cf80b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:36:46.075 INFO 2981023 --- [host-check-agent-status4,3bd8cccd594e7a2e,ad65abde072b68c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:36:46.077 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:36:46.077 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:36:46.077 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:36:46.077234624+08:00 2024-09-11 18:36:46.077 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:36:46.077+08:00 2024-09-11 18:36:46.078 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:36:46.083793641+08:00 2024-09-11 18:36:46.079 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:36:46.079+08:00 2024-09-11 18:36:46.079 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:36:46.077+08:00 and status AVAILABLE 2024-09-11 18:36:46.080 INFO 2981023 --- [host-check-agent-status4,3bd8cccd594e7a2e,ad65abde072b68c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:36:46.080 INFO 2981023 --- [host-check-agent-status4,3bd8cccd594e7a2e,ad65abde072b68c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:36:46.080 INFO 2981023 --- [host-check-agent-status4,3bd8cccd594e7a2e,ad65abde072b68c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:36:46.081 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:36:46.079+08:00 and status AVAILABLE 2024-09-11 18:36:46.081 INFO 2981023 --- [host-check-agent-status2,3bd8cccd594e7a2e,bc070c06e88c360a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:36:46.088 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:36:46.090 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:36:46.089451809+08:00 2024-09-11 18:36:46.090 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:36:46.090 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:36:46.090+08:00 2024-09-11 18:36:46.093 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:36:46.090+08:00 and status AVAILABLE 2024-09-11 18:36:46.094 INFO 2981023 --- [host-check-agent-status2,3bd8cccd594e7a2e,bc070c06e88c360a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:36:46.094 INFO 2981023 --- [host-check-agent-status2,3bd8cccd594e7a2e,bc070c06e88c360a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:36:46.094 INFO 2981023 --- [host-check-agent-status2,3bd8cccd594e7a2e,bc070c06e88c360a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:36:46.096 INFO 2981023 --- [host-check-agent-status1,3bd8cccd594e7a2e,48d3c874b23c3df2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:36:46.097 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:36:46.101 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:36:46.110 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:36:46.110 INFO 2981023 --- [host-check-agent-status1,3bd8cccd594e7a2e,48d3c874b23c3df2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:36:46.110 INFO 2981023 --- [host-check-agent-status1,3bd8cccd594e7a2e,48d3c874b23c3df2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:36:46.110 INFO 2981023 --- [host-check-agent-status1,3bd8cccd594e7a2e,48d3c874b23c3df2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:36:46.114 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:36:46.115 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:36:46.117 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:36:46.119 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:36:46.111737035+08:00 2024-09-11 18:36:46.120 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:36:46.124 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:36:46.126 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:36:46.129 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:36:46.119+08:00 2024-09-11 18:36:46.138 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:36:46.119+08:00 and status AVAILABLE 2024-09-11 18:36:46.143 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:36:46.147 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:36:46.149 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:36:46.160 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:36:46.168 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:36:46.170 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:36:46.178 INFO 2981023 --- [ocp-schedules-17,d38bbe4e79a0a609,f75d0b02b0d2591b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:36:46.180 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:36:46.182 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:36:46.187 INFO 2981023 --- [ocp-schedules-27,03b46a32a23dd26e,8dcdec4092b1b482] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:36:46.187 INFO 2981023 --- [ocp-schedules-26,26c4dc09625dc706,d00cac6a2f9aece1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:36:46.189 INFO 2981023 --- [ocp-schedules-27,03b46a32a23dd26e,8dcdec4092b1b482] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:46.190 INFO 2981023 --- [ocp-schedules-23,32fed61679394918,fa5f1db23a280a6a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:36:46.191 INFO 2981023 --- [ocp-schedules-9,8dafabbe2f2b0b44,083c48ddc03db09a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:36:46.193 INFO 2981023 --- [ocp-schedules-23,32fed61679394918,fa5f1db23a280a6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:46.199 INFO 2981023 --- [ocp-schedules-17,d38bbe4e79a0a609,f75d0b02b0d2591b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:36:46.199 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:36:46.200 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:36:46.205 INFO 2981023 --- [ocp-schedules-26,26c4dc09625dc706,d00cac6a2f9aece1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:46.207 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050963458000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:36:46.211 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:36:46.213 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:36:46.235 INFO 2981023 --- [ocp-schedules-26,26c4dc09625dc706,d00cac6a2f9aece1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:36:46.238 INFO 2981023 --- [ocp-schedules-15,553d0d4f3e54c68c,003348c65ca45c3d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:36:46.276 INFO 2981023 --- [ocp-schedules-28,af81820b2c93afd7,62f935d90f1202cb] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-09-11 18:36:46.276 INFO 2981023 --- [ocp-schedules-28,af81820b2c93afd7,62f935d90f1202cb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-09-11 18:36:46.282 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:36:46.291 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:36:46.292 INFO 2981023 --- [ocp-schedules-10,72c3166f290844f8,9a12f96518143a98] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-09-11 18:36:46.300 INFO 2981023 --- [ocp-schedules-10,72c3166f290844f8,9a12f96518143a98] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:46.318 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:36:46.320 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:36:46.328 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:36:46.343 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:36:46.342946+08:00, lastProcessTime=2024-09-11T18:36:36.336807+08:00 2024-09-11 18:36:46.345 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:36:46.346 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.346 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:46.354 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.355 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:36:46.356 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:36:46.359 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:36:46.360 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.360 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:36:46.361 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:36:46.361 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.362 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:36:46.370 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:36:46.374 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.374 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:36:46.376 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.377 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:46.378 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.379 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:36:46.380 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:36:46.380 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:36:46.380 INFO 2981023 --- [ocp-schedules-27,03b46a32a23dd26e,8dcdec4092b1b482] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:36:46.381 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,965550438c64ee0a,ba752be10219ea9e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&ObClusterId=1719994437&version=2, client=10.31.35.10, traceId=965550438c64ee0a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=linkprdocp,1719994437,false,2,] 2024-09-11 18:36:46.381 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,965550438c64ee0a,ba752be10219ea9e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=965550438c64ee0a, duration=1 ms] 2024-09-11 18:36:46.387 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.387 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.387 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-09-11 18:36:46.387 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:46.388 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.388 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:36:46.390 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:36:46.397 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0f4e747234c24cce,f52ce6e355d14ab6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&ObClusterId=1718689631&version=2, client=10.31.35.10, traceId=0f4e747234c24cce, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=rtdob,1718689631,false,2,] 2024-09-11 18:36:46.397 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,0f4e747234c24cce,f52ce6e355d14ab6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.31.35.10, traceId=0f4e747234c24cce, duration=1 ms] 2024-09-11 18:36:46.399 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.399 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.401 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:36:46.402 INFO 2981023 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.10, 2882] 2024-09-11 18:36:46.405 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:36:46.409 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.412 INFO 2981023 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.5, 2882] 2024-09-11 18:36:46.414 INFO 2981023 --- [ocp-schedules-10,72c3166f290844f8,9a12f96518143a98] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, tryCount=3, successCount=3 2024-09-11 18:36:46.415 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:36:46.429 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:36:46.431 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:36:46.439 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:36:46.442 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.453 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.453 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:36:46.455 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:36:46.466 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:36:46.468 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.469 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.469 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:36:46.469 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:36:46.470 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.470 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:36:46.497 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:36:46.498 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:36:46.500 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:36:46.508 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:36:46.509 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:36:46.509 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050963458000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:36:46.520 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:36:46.531 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:36:46.533 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:36:46.536 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:36:46.554 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.554 INFO 2981023 --- [host-check-host-status4,fdd4dc456b38f169,b36ce80bc4578682] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:36:46.555 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.555 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:46.555 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:36:46.556 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.559 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:36:46.559 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:36:46.562 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:36:46.564 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.564 INFO 2981023 --- [pool-10-thread-6,a540b1321c54c2b8,8b8eb4a3bacb38de] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-09-11 18:36:46.564 INFO 2981023 --- [host-check-host-status3,fdd4dc456b38f169,e3b3d7bd06ea56b7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:36:46.564 INFO 2981023 --- [pool-10-thread-7,a540b1321c54c2b8,a0c28da5507a14db] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-09-11 18:36:46.565 INFO 2981023 --- [pool-10-thread-7,a540b1321c54c2b8,a0c28da5507a14db] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-09-11 18:36:46.570 INFO 2981023 --- [host-check-host-status2,fdd4dc456b38f169,fbe7f04598df5b67] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:36:46.580 INFO 2981023 --- [host-check-host-status1,fdd4dc456b38f169,b569c094cbc4e533] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:36:46.580 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.581 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.582 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:36:46.583 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:36:46.583 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.583 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:36:46.584 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:36:46.584 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.592 INFO 2981023 --- [pool-10-thread-6,a540b1321c54c2b8,8b8eb4a3bacb38de] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-09-11 18:36:46.592 INFO 2981023 --- [pool-10-thread-6,a540b1321c54c2b8,8b8eb4a3bacb38de] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-09-11 18:36:46.592 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:36:46.595 INFO 2981023 --- [pool-10-thread-8,a540b1321c54c2b8,2ade23a5e2f1f406] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-09-11 18:36:46.595 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.595 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.595 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:36:46.598 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:36:46.598 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:36:46.604 INFO 2981023 --- [pool-10-thread-5,a540b1321c54c2b8,77047218a320e647] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-09-11 18:36:46.609 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkrtd 2024-09-11 18:36:46.609 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkrtd 2024-09-11 18:36:46.615 INFO 2981023 --- [ForkJoinPool-6-worker-14,a540b1321c54c2b8,015eef266fb0b074] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of linkprdocp 2024-09-11 18:36:46.615 INFO 2981023 --- [ForkJoinPool-6-worker-14,a540b1321c54c2b8,015eef266fb0b074] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of linkprdocp 2024-09-11 18:36:46.626 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of rtdob 2024-09-11 18:36:46.626 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of rtdob 2024-09-11 18:36:46.633 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:36:46.675 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:36:46.693 WARN 2981023 --- [ForkJoinPool-6-worker-14,a540b1321c54c2b8,015eef266fb0b074] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :linkprdocp com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=linkprdocp,ocp_meta,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:164) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:141) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$88427643.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) 2024-09-11 18:36:46.703 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.704 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.704 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:36:46.705 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:36:46.705 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.710 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.711 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.712 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:36:46.712 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.713 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:36:46.713 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.713 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:36:46.713 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:36:46.714 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:36:46.717 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.717 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:36:46.718 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:36:46.721 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:36:46.722 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:36:46.722 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.724 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.724 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:36:46.725 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:36:46.725 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.726 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:36:46.727 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:36:46.728 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of linkrtd 2024-09-11 18:36:46.758 INFO 2981023 --- [ForkJoinPool-6-worker-2,a540b1321c54c2b8,39c72085fca2fb69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.759 INFO 2981023 --- [ForkJoinPool-6-worker-2,a540b1321c54c2b8,39c72085fca2fb69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:36:46.769 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:36:46.770 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:36:46.783 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:36:46.784 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.785 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:36:46.786 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.787 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:36:46.792 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:36:46.797 INFO 2981023 --- [ForkJoinPool-6-worker-2,a540b1321c54c2b8,39c72085fca2fb69] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:36:46.798 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.799 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state ='ACTIVE' 2024-09-11 18:36:46.799 INFO 2981023 --- [ForkJoinPool-6-worker-2,a540b1321c54c2b8,39c72085fca2fb69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.800 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.800 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-09-11 18:36:46.801 INFO 2981023 --- [ForkJoinPool-6-worker-2,a540b1321c54c2b8,39c72085fca2fb69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:36:46.801 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of rtdob 2024-09-11 18:36:46.802 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:36:46.802 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:36:46.802 INFO 2981023 --- [ForkJoinPool-6-worker-2,a540b1321c54c2b8,39c72085fca2fb69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.803 INFO 2981023 --- [ForkJoinPool-6-worker-2,a540b1321c54c2b8,39c72085fca2fb69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:36:46.812 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=root 2024-09-11 18:36:46.824 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.825 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:36:46, 2024-09-11 18:36:46] 2024-09-11 18:36:46.827 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of linkrtd 2024-09-11 18:36:46.828 INFO 2981023 --- [ForkJoinPool-6-worker-0,a540b1321c54c2b8,693259d6b76d97c0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of linkrtd 2024-09-11 18:36:46.861 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:36:46.861 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.862 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.862 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:36:46.863 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:36:46.863 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.886 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.891 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-09-11 18:36:46.892 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:36:46.907 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=linkreit, dbUser=root 2024-09-11 18:36:46.913 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.915 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-09-11 18:36:46.919 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.926 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:36:46, 2024-09-11 18:36:46] 2024-09-11 18:36:46.926 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:36:46.949 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.949 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.950 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:36:46.951 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:36:46.958 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:36:46.958 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:36:46.959 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.961 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-09-11 18:36:46.962 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=root 2024-09-11 18:36:46.962 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.963 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.964 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:36:46.964 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-09-04 18:36:46, 2024-09-11 18:36:46] 2024-09-11 18:36:46.965 INFO 2981023 --- [ForkJoinPool-6-worker-7,a540b1321c54c2b8,8a3cca67ed92b728] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-09-11 18:36:46.966 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:36:46.967 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of rtdob 2024-09-11 18:36:46.967 INFO 2981023 --- [ForkJoinPool-6-worker-5,a540b1321c54c2b8,0605dcd14322e9be] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of rtdob 2024-09-11 18:36:46.968 INFO 2981023 --- [pool-10-thread-8,a540b1321c54c2b8,2ade23a5e2f1f406] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-09-11 18:36:46.970 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:36:46.976 INFO 2981023 --- [pool-10-thread-5,a540b1321c54c2b8,77047218a320e647] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-09-11 18:36:46.990 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:36:46.990 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:46.992 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:36:46.342946+08:00 2024-09-11 18:36:46.993 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:36:46.997 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:46.999 INFO 2981023 --- [ocp-schedules-11,94f4734fc4bdaf7f,ad6fa02eaefb10b4] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:36:47.001 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-09-11 18:36:47.002 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:36:47.003 INFO 2981023 --- [pool-10-thread-7,a540b1321c54c2b8,a0c28da5507a14db] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-09-11 18:36:47.006 INFO 2981023 --- [pool-10-thread-7,a540b1321c54c2b8,a0c28da5507a14db] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-09-11 18:36:47.008 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:47.008 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:47.009 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:36:47.014 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:36:47.014 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:36:47.026 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:36:47.031 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:36:47.031 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:47.032 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:36:47.037 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:36:47.037 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:36:47.037 INFO 2981023 --- [ocp-schedules-23,32fed61679394918,fa5f1db23a280a6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:36:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,a296c3e1bc2f284f,7852c16a2a3e6ead] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:47.315 INFO 2981023 --- [prometheus-minute-schedule-1,a296c3e1bc2f284f,7852c16a2a3e6ead] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:47.501 INFO 2981023 --- [ocp-async-6,8631af4fed4eeefd,6a834b7861cf9f5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:47.803 INFO 2981023 --- [ocp-async-7,1dd9b0bfb0adafd9,ca2448ec246ec8ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:36:47.812 INFO 2981023 --- [host-check-clockdiff2,793207864b904cc0,a8413e7834152878] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051007 5 5 , error= 2024-09-11 18:36:48.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:48.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:48.549 INFO 2981023 --- [host-check-clockdiff1,793207864b904cc0,7c4ab2a2fee43328] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051008 0 0 , error= 2024-09-11 18:36:48.937 INFO 2981023 --- [ocp-schedules-2,431a0947b894fe11,e542db8ced657b85] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:36:48.990 WARN 2981023 --- [scanner-13700596-1,431a0947b894fe11,a15cc40efd93b853] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:36:49.195 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:36:49.195 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:36:49.198 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:36:49.198 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:36:49.198 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:36:49.198 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:36:49.200 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:36:49.200 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:36:49.200 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:36:49.200 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:36:49.222 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:36:49.225 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:36:49.225 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:36:49.225 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:36:49.225 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:36:49.225 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:36:49.229 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:36:49.463 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:36:49.501 INFO 2981023 --- [ocp-async-8,665cb91b1777da10,1f1425e56cff49ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:49.709 INFO 2981023 --- [ocp-schedules-15,c02f77199217c53e,b7282ceae8c37b08] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:36:49.732 WARN 2981023 --- [ocp-schedules-15,c02f77199217c53e,b7282ceae8c37b08] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:36:49.776 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.576615612S 2024-09-11 18:36:49.777 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:36:49.823 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:36:49.824 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:36:49.839 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:36:49.869 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:36:49.869 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:36:49.869 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:36:49.869 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:36:49.869 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:36:49.869 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:36:49.874 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:36:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:36:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:36:49.955 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] o.s.boot.SpringApplication : Started application in 0.756 seconds (JVM running for 86809.827) 2024-09-11 18:36:49.955 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.756947953S 2024-09-11 18:36:50.150 INFO 2981023 --- [ocp-schedules-4,e90816b7fc0f8f86,bf5332d284e5f21d] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:36:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,d4180471ec12ef7b,602aede968eef11f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:50.316 INFO 2981023 --- [prometheus-minute-schedule-1,d4180471ec12ef7b,602aede968eef11f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:50.398 INFO 2981023 --- [ocp-schedules-13,fd0923bbe98b69dd,9189fa7f4ec243c9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:36:51.501 INFO 2981023 --- [ocp-async-3,a756af6e4371ddd3,9b37e59cd2d3e7cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:51.526 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:51.527 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:51.662 INFO 2981023 --- [ocp-schedules-28,3967ab9e231caf5f,60de7ee669b9e20d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:36:51.663 INFO 2981023 --- [ocp-schedules-28,3967ab9e231caf5f,60de7ee669b9e20d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:36:51.664 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:36:51.664 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:36:51.664 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:36:51.664 INFO 2981023 --- [ocp-schedules-28,3967ab9e231caf5f,60de7ee669b9e20d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:36:52.022 INFO 2981023 --- [host-check-clockdiff4,793207864b904cc0,a4d660413caffe4e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051012 0 0 , error= 2024-09-11 18:36:53.000 INFO 2981023 --- [ocp-schedules-10,9d49e4bab02476a3,18c975d347b30a7c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:36:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,e7988439f51554e0,1e51575786e40e67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,e7988439f51554e0,1e51575786e40e67] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:53.440 INFO 2981023 --- [ocp-schedules-26,4636ec28faca5bcf,feb6f1a63e9cc319] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:36:53.440 INFO 2981023 --- [ocp-schedules-26,4636ec28faca5bcf,feb6f1a63e9cc319] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:36:53.501 INFO 2981023 --- [ocp-async-2,569514916d0783fa,b962c0c3db9f69bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:54.528 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:54.528 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:54.560 INFO 2981023 --- [ocp-schedules-25,70bd10e9d84f57f2,9e5c2911ab2c7519] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:36:55.221 INFO 2981023 --- [ocp-schedules-29,e4bc5cc5ef39a8cd,98c71aa9fa6b6768] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:36:55.221 INFO 2981023 --- [ocp-schedules-29,e4bc5cc5ef39a8cd,98c71aa9fa6b6768] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:36:55.226 INFO 2981023 --- [ocp-schedules-29,e4bc5cc5ef39a8cd,98c71aa9fa6b6768] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:36:55.226 INFO 2981023 --- [ocp-schedules-29,e4bc5cc5ef39a8cd,98c71aa9fa6b6768] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:36:55.226 INFO 2981023 --- [ocp-schedules-29,e4bc5cc5ef39a8cd,98c71aa9fa6b6768] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:36:55.236 INFO 2981023 --- [ocp-schedules-29,e4bc5cc5ef39a8cd,98c71aa9fa6b6768] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:36:55.236 INFO 2981023 --- [ocp-schedules-29,e4bc5cc5ef39a8cd,98c71aa9fa6b6768] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:36:55.236 INFO 2981023 --- [ocp-schedules-29,e4bc5cc5ef39a8cd,98c71aa9fa6b6768] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:36:55.247 INFO 2981023 --- [ocp-schedules-29,e4bc5cc5ef39a8cd,98c71aa9fa6b6768] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:36:55.502 INFO 2981023 --- [ocp-async-9,a13d93193e776cb8,8590dc271124d198] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:55.951 INFO 2981023 --- [ocp-schedules-22,5570db78bead87e3,af9d84bf3d30e011] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:36:55.961 INFO 2981023 --- [ocp-schedules-17,ed2639cd1b35ab14,23ce7fa84cae8a6e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:36:55.961 INFO 2981023 --- [ocp-schedules-17,ed2639cd1b35ab14,23ce7fa84cae8a6e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:36:55.969 INFO 2981023 --- [ocp-schedules-17,ed2639cd1b35ab14,23ce7fa84cae8a6e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:36:55.969 INFO 2981023 --- [ocp-schedules-17,ed2639cd1b35ab14,23ce7fa84cae8a6e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:36:55.976 WARN 2981023 --- [ocp-schedules-22,5570db78bead87e3,af9d84bf3d30e011] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:36:55.978 INFO 2981023 --- [ocp-schedules-17,ed2639cd1b35ab14,23ce7fa84cae8a6e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:36:55.978 INFO 2981023 --- [ocp-schedules-17,ed2639cd1b35ab14,23ce7fa84cae8a6e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:36:56.212 INFO 2981023 --- [ocp-schedules-13,7281bab9c5b162fd,e44eb25e992a9e4e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:36:56.219 INFO 2981023 --- [ocp-schedules-13,7281bab9c5b162fd,e44eb25e992a9e4e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:36:56.219 INFO 2981023 --- [ocp-schedules-13,7281bab9c5b162fd,e44eb25e992a9e4e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:36:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,c5205cb63cc5fce0,f3000ecab095432e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,c5205cb63cc5fce0,f3000ecab095432e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:36:56.352540+08:00, lastProcessTime=2024-09-11T18:36:46.342946+08:00 2024-09-11 18:36:56.505 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:36:56.352540+08:00 2024-09-11 18:36:56.570 INFO 2981023 --- [ocp-schedules-30,f31815ddfa4444e9,fe2e390495663437] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:36:56.579 INFO 2981023 --- [ocp-schedules-30,f31815ddfa4444e9,fe2e390495663437] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:36:56.579 INFO 2981023 --- [ocp-schedules-30,f31815ddfa4444e9,fe2e390495663437] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:36:57.502 INFO 2981023 --- [ocp-async-4,35acd8567301b3e9,256b81c6cfb0cd8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:57.528 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:36:57.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:36:57.804 INFO 2981023 --- [ocp-async-2,df5934cff0d40925,6a380edd4e8a1169] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:36:59.120 INFO 2981023 --- [ocp-schedules-15,50fa2dac9aad3c62,5110e1b8fe1f608d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:36:59.179 WARN 2981023 --- [scanner-13700785-1,50fa2dac9aad3c62,887e28ceb2eb0749] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:36:59.192 INFO 2981023 --- [alarm-detect-2,50fa2dac9aad3c62,f2bb9608247a6209] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646815, data is empty. 2024-09-11 18:36:59.197 INFO 2981023 --- [alarm-detect-2,50fa2dac9aad3c62,f2bb9608247a6209] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646815, data is empty. 2024-09-11 18:36:59.201 INFO 2981023 --- [alarm-detect-2,50fa2dac9aad3c62,f2bb9608247a6209] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646815, data is empty. 2024-09-11 18:36:59.216 INFO 2981023 --- [alarm-detect-17,50fa2dac9aad3c62,b07d8523e61c8159] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646815, data is empty. 2024-09-11 18:36:59.220 INFO 2981023 --- [alarm-detect-17,50fa2dac9aad3c62,b07d8523e61c8159] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646815, data is empty. 2024-09-11 18:36:59.226 INFO 2981023 --- [alarm-detect-17,50fa2dac9aad3c62,b07d8523e61c8159] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646815, data is empty. 2024-09-11 18:36:59.228 INFO 2981023 --- [alarm-detect-7,50fa2dac9aad3c62,209bbc44da05abfa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646815, data is empty. 2024-09-11 18:36:59.234 INFO 2981023 --- [alarm-detect-7,50fa2dac9aad3c62,209bbc44da05abfa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646815, data is empty. 2024-09-11 18:36:59.238 INFO 2981023 --- [alarm-detect-7,50fa2dac9aad3c62,209bbc44da05abfa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646815, data is empty. 2024-09-11 18:36:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,721d575d0e110ea7,e2d70824f7b10207] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051020, interval=60 2024-09-11 18:36:59.321 INFO 2981023 --- [prometheus-minute-schedule-1,721d575d0e110ea7,e2d70824f7b10207] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:36:59.342 INFO 2981023 --- [alarm-detect-9,50fa2dac9aad3c62,084b6d54f3430b53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646815, data is empty. 2024-09-11 18:36:59.347 INFO 2981023 --- [alarm-detect-9,50fa2dac9aad3c62,084b6d54f3430b53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646815, data is empty. 2024-09-11 18:36:59.350 INFO 2981023 --- [alarm-detect-9,50fa2dac9aad3c62,084b6d54f3430b53] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646815, data is empty. 2024-09-11 18:36:59.465 INFO 2981023 --- [ocp-schedules-10,2faa4af42a884a04,4df69cc0733683b3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:36:59.465 INFO 2981023 --- [ocp-schedules-10,2faa4af42a884a04,4df69cc0733683b3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:36:59.498 INFO 2981023 --- [Collect all tenants major compactions1,2faa4af42a884a04,7d65752c8581b8c0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:36:59.499 INFO 2981023 --- [Collect all tenants major compactions4,2faa4af42a884a04,2c73af0b2b47690b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:36:59.501 INFO 2981023 --- [Collect all tenants major compactions2,2faa4af42a884a04,9e5b90a2c28ad7e9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:36:59.502 INFO 2981023 --- [ocp-async-6,6138d15291d93e08,8c6460a325644e2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:36:59.502 INFO 2981023 --- [Collect all tenants major compactions5,2faa4af42a884a04,f3543cbb3eaa03e6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:36:59.504 INFO 2981023 --- [Collect all tenants major compactions3,2faa4af42a884a04,8816b89fef3563b6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:36:59.505 INFO 2981023 --- [Collect all tenants major compactions6,2faa4af42a884a04,767cb5a98bd75548] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:36:59.534 INFO 2981023 --- [Collect all tenants major compactions2,2faa4af42a884a04,9e5b90a2c28ad7e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:59.535 INFO 2981023 --- [Collect all tenants major compactions2,2faa4af42a884a04,9e5b90a2c28ad7e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:36:59.535 INFO 2981023 --- [Collect all tenants major compactions3,2faa4af42a884a04,8816b89fef3563b6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:59.536 INFO 2981023 --- [Collect all tenants major compactions3,2faa4af42a884a04,8816b89fef3563b6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:36:59.537 INFO 2981023 --- [Collect all tenants major compactions4,2faa4af42a884a04,2c73af0b2b47690b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:59.538 INFO 2981023 --- [Collect all tenants major compactions4,2faa4af42a884a04,2c73af0b2b47690b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:36:59.538 INFO 2981023 --- [Collect all tenants major compactions5,2faa4af42a884a04,f3543cbb3eaa03e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:59.540 INFO 2981023 --- [Collect all tenants major compactions3,2faa4af42a884a04,8816b89fef3563b6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:36:59.540 INFO 2981023 --- [Collect all tenants major compactions1,2faa4af42a884a04,7d65752c8581b8c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:59.541 INFO 2981023 --- [Collect all tenants major compactions4,2faa4af42a884a04,2c73af0b2b47690b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:36:59.541 INFO 2981023 --- [Collect all tenants major compactions2,2faa4af42a884a04,9e5b90a2c28ad7e9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:36:59.542 INFO 2981023 --- [Collect all tenants major compactions6,2faa4af42a884a04,767cb5a98bd75548] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:36:59.542 INFO 2981023 --- [Collect all tenants major compactions3,2faa4af42a884a04,8816b89fef3563b6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:36:59.543 INFO 2981023 --- [Collect all tenants major compactions6,2faa4af42a884a04,767cb5a98bd75548] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:36:59.544 INFO 2981023 --- [Collect all tenants major compactions1,2faa4af42a884a04,7d65752c8581b8c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:36:59.544 INFO 2981023 --- [Collect all tenants major compactions5,2faa4af42a884a04,f3543cbb3eaa03e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:36:59.545 INFO 2981023 --- [Collect all tenants major compactions1,2faa4af42a884a04,7d65752c8581b8c0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:36:59.545 INFO 2981023 --- [Collect all tenants major compactions1,2faa4af42a884a04,7d65752c8581b8c0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:36:59.545 INFO 2981023 --- [Collect all tenants major compactions6,2faa4af42a884a04,767cb5a98bd75548] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:36:59.545 INFO 2981023 --- [Collect all tenants major compactions6,2faa4af42a884a04,767cb5a98bd75548] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:36:59.545 INFO 2981023 --- [Collect all tenants major compactions5,2faa4af42a884a04,f3543cbb3eaa03e6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:36:59.545 INFO 2981023 --- [Collect all tenants major compactions5,2faa4af42a884a04,f3543cbb3eaa03e6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:36:59.546 INFO 2981023 --- [Collect all tenants major compactions2,2faa4af42a884a04,9e5b90a2c28ad7e9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:36:59.547 INFO 2981023 --- [Collect all tenants major compactions4,2faa4af42a884a04,2c73af0b2b47690b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:36:59.547 INFO 2981023 --- [ocp-schedules-10,2faa4af42a884a04,4df69cc0733683b3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:37:00.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:00.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:01.502 INFO 2981023 --- [ocp-async-10,dbf4a1a5b98ee8d9,1817d93dc38ea6dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:02.246 INFO 2981023 --- [ocp-schedules-30,b39652c6b8390155,3e08805be4dd431e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:37:02.246+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:37:02.250 INFO 2981023 --- [ocp-schedules-30,b39652c6b8390155,3e08805be4dd431e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:37:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,0bc87046df8739ba,d74971688fb30e33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,0bc87046df8739ba,d74971688fb30e33] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:02.451 INFO 2981023 --- [ocp-schedules-30,b39652c6b8390155,3e08805be4dd431e] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 63 to SqlAggregateData 2024-09-11 18:37:02.453 INFO 2981023 --- [ocp-schedules-30,b39652c6b8390155,3e08805be4dd431e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:37:03.002 INFO 2981023 --- [ocp-schedules-20,2a0a04ad64b9cfb4,5ffb7c01ef346e0f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:37:03.502 INFO 2981023 --- [ocp-async-3,3f7567a9fa58147d,15614c2825f437a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:03.531 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:03.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,0af25197da107177,9277fcb5b499f8a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,0af25197da107177,9277fcb5b499f8a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:05.502 INFO 2981023 --- [ocp-async-1,e903ff968f108662,4fdf0a457ab65264] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:05.700 INFO 2981023 --- [ocp-schedules-27,a2f43482ec6a1bcf,3033265a0b1c601e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:37:05.702 INFO 2981023 --- [ocp-schedules-27,a2f43482ec6a1bcf,3033265a0b1c601e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:37:05.735 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:37:05.735 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.737 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:37:05.741 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:37:05.742 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:37:05.742 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.744 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.744 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.745 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:37:05.746 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.749 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.749 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.749 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.750 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.755 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:37:05.755 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.756 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:37:05.761 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.761 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.762 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.762 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:37:05.768 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.768 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:37:05.774 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.775 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:37:05.776 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.776 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.788 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.789 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:37:05.789 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.789 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:37:05.790 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.790 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:37:05.791 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:37:05.791 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.791 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.792 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:37:05.792 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:37:05.792 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.792 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.792 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:37:05.793 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:37:05.793 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.793 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:37:05.794 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.794 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:37:05.795 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.795 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:37:05.796 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.796 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:37:05.796 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:37:05.797 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:37:05.798 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.799 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:37:05.799 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:37:05.810 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:05.813 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.813 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:37:05.813 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:05.815 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.815 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.840 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:37:05.843 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:37:05.844 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.844 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:37:05.847 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:37:05.849 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:37:05.849 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:37:05.849 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:37:05.850 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:37:05.851 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:37:05.851 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.851 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.855 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:37:05.856 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:37:05.856 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:37:05.856 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:37:05.893 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.908 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.937 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.942 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:05.945 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:37:05.950 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE7461D6DF-0-0] 2024-09-11 18:37:05.950 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:37:05.956 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.962 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:37:05.976 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.979 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:37:05.993 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:05.994 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:37:05.994 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:37:05.995 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE7461D6E9-0-0] 2024-09-11 18:37:05.995 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221574525) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:37:06.002 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:06.003 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:37:06.003 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:06.004 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:37:06.004 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:06.005 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:37:06.005 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:06.006 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:37:06.006 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:06.007 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:37:06.007 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:06.008 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:37:06.008 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:37:06.010 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:37:06.020 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:06.028 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:06.034 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:37:06.046 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:37:06.048 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:37:06.048 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:06.048 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:37:06.057 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:37:06.058 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:37:06.058 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:37:06.058 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:37:06.058 INFO 2981023 --- [ocp-schedules-27,a2f43482ec6a1bcf,3033265a0b1c601e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:37:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:37:06.337504+08:00, lastProcessTime=2024-09-11T18:36:56.352540+08:00 2024-09-11 18:37:06.465 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:37:06.337504+08:00 2024-09-11 18:37:06.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:06.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:07.502 INFO 2981023 --- [ocp-async-6,6978f0b432785fe8,13502d90a2e590f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:07.804 INFO 2981023 --- [ocp-async-4,decf1b1b5a27f5b1,e5eb4fac2f7aeeef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:37:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,b6331c9634726f5f,a6c90df643408dbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,b6331c9634726f5f,a6c90df643408dbc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:08.379 INFO 2981023 --- [ocp-schedules-13,018f98ccec23372e,32f3615d53900b0f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:37:08.454 INFO 2981023 --- [ocp-schedules-18,e9411f15a8069f8b,67a558c06b493c9b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:37:08.454+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:37:08.455 INFO 2981023 --- [ocp-schedules-18,e9411f15a8069f8b,67a558c06b493c9b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:37:08.456 INFO 2981023 --- [ocp-schedules-18,e9411f15a8069f8b,67a558c06b493c9b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:37:08.457 INFO 2981023 --- [ocp-schedules-18,e9411f15a8069f8b,67a558c06b493c9b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:37:09.392 INFO 2981023 --- [ocp-schedules-3,54a2cb16cc591a75,9642cb7afb409c1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:37:09.428 WARN 2981023 --- [scanner-13701147-1,54a2cb16cc591a75,4d6e8b979cb4f1da] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:37:09.503 INFO 2981023 --- [ocp-async-5,377fb163b82cce9a,f8114c97c40e0394] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:09.507 INFO 2981023 --- [alarm-detect-8,54a2cb16cc591a75,2ac08723170b1ead] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [6007110] will be resolved. 2024-09-11 18:37:09.507 INFO 2981023 --- [alarm-detect-8,54a2cb16cc591a75,2ac08723170b1ead] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [6007110] will be resolved. 2024-09-11 18:37:09.532 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:09.533 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:09.552 INFO 2981023 --- [alarm-detect-8,54a2cb16cc591a75,2ac08723170b1ead] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2024-09-11 18:37:10.002 INFO 2981023 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:37:10.002 INFO 2981023 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:37:10.918 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:37:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:37:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,d68d27bb958d3832,70e563bea3968eef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,d68d27bb958d3832,70e563bea3968eef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:11.503 INFO 2981023 --- [ocp-async-2,7a8b6fd7e5f26def,7cc94094c730002f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:12.533 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:12.534 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:12.878 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050960, nodeCount=1000, affectRows=1000 2024-09-11 18:37:12.878 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050960, nodeCount=1000, affectRows=1000 2024-09-11 18:37:12.881 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050960, nodeCount=1000, validCount=1000 2024-09-11 18:37:12.881 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050960, nodeCount=999, validCount=999 2024-09-11 18:37:12.887 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726050960, nodeCount=48, affectRows=48 2024-09-11 18:37:12.888 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726050960, nodeCount=46, validCount=46 2024-09-11 18:37:13.005 INFO 2981023 --- [ocp-schedules-14,ad6601c2dd27ee72,14531391c06c7f95] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:37:13.503 INFO 2981023 --- [ocp-async-9,67b7e7523e2dce3f,6edce8f842db106b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:13.807 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-09-11 18:37:13.816 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:13.819 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:13.871 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:37:13.872 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:37:13.883 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:13.884 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:13.894 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:37:13.895 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:37:13.906 INFO 2981023 --- [ocp-schedules-7,7cbdf3d37d70be55,a88e2213541267c5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2024-09-11 18:37:14.306 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726050960, nodeCount=480, affectRows=480 2024-09-11 18:37:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,d17c4776946a84ae,ff71d5b444df0d91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,95b49be1c297f431,31099ca386f3f204] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:37:14.317 INFO 2981023 --- [prometheus-minute-schedule-1,d17c4776946a84ae,ff71d5b444df0d91] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:14.822 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:33:14.392, end=2024-09-11T18:35:14.817} 2024-09-11 18:37:14.829 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, 2024-09-11 18:37:14.829 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:37:14.834 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:33:14.392, end=2024-09-11T18:35:14.817},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:37:14.834 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:37:14.834 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:37:14.908 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964634908000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:14.908Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:14.908 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964634908000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:14.908Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:14.909 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964634909000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:14.909Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:14.916 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964634916000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:14.916Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:14.916 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964634916000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:14.916Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:14.917 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964634917000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:14.917Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:14.921 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964634921000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:14.921Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:14.926 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964634926000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:14.926Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:14.973 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:14.973 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:14.973 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:14.973 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:14.983 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:14.983 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:14.985 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:14.985 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:14.989 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:37:14.989 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:14.990 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:37:14.990 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:14.990 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:14.990 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:14.998 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:37:14.998 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.000 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.000 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.001 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.001 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.001 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.001 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.002 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.002 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.005 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 71 2024-09-11 18:37:15.005 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.006 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.006 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.009 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.009 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.010 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.010 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.010 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.010 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.011 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.011 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.012 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.012 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.015 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.015 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.017 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.017 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.019 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.019 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.020 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.020 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.022 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.022 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.022 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.022 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.023 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.023 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.023 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.025 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.025 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.026 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.026 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.028 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.028 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.029 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.029 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.029 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.033 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.033 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.035 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.035 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.036 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.038 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.039 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.039 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.039 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:14.908, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.039 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:14.908, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.041 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:14.917, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.041 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.041 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:14.916, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.042 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.044 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:14.909, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.048 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:14.921, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.049 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:14.916, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.049 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:14.926, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.077 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635076000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.076Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.081 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635081000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.081Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.084 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635084000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.084Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.084 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635084000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3043450C260575FA1FB0F432BD07FEB0, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.084Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.085 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635085000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.085Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.088 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635088000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=6060DE14985EEB942B04DAAC306FE98D, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.088Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.089 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635089000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=9562F1BE552A476225AECD175FEB3062, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.089Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.089 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635089000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.089Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.091 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.091 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.093 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.093 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.097 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.097 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.098 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:37:15.098 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.107 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:37:15.107 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.111 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.113 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.113 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.114 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.114 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.117 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.117 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.121 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.121 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.126 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.126 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.126 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.126 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.127 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.127 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.130 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.130 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.131 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.131 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.131 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.131 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.132 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.132 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.132 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.132 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.139 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.139 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3043450C260575FA1FB0F432BD07FEB0, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.142 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.142 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.142 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.142 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9562F1BE552A476225AECD175FEB3062, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.143 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.143 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.143 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.143 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.144 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.144 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.144 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.145 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.145 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.145 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.148 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.149 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.149 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.150 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.150 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=6060DE14985EEB942B04DAAC306FE98D, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.150 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.150 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.153 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.153 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3043450C260575FA1FB0F432BD07FEB0, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.154 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.154 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=9562F1BE552A476225AECD175FEB3062, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.154 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.154 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.155 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.157 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.158 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.084, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=CF8C6B84F99F07EE735FF2B370FF12AC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.158 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.158 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.158 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.081, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.160 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.076, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=4881DE41C929AB4A190CFCBC0FCF5082, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.165 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.085, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.166 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.089, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=9562F1BE552A476225AECD175FEB3062, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.168 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.088, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=6060DE14985EEB942B04DAAC306FE98D, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.168 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.168 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.170 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.171 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.084, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=3043450C260575FA1FB0F432BD07FEB0, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.175 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.089, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=1D5FD2A699B4B7B444ED527DC05DCB2A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.186 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635186000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.186Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.186 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635186000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.186Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.192 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635192000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=4B198558E9701B7EC26260935540DC23, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.192Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.194 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.194 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.197 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.197 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.197 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635197000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.197Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.205 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.205 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.206 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.206 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.206 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.206 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.209 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 261 2024-09-11 18:37:15.209 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.210 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.210 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.210 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.210 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.213 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.213 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.215 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.215 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.217 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.217 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.218 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.218 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.218 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.218 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.219 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.220 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.220 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.220 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.223 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.186, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.224 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.186, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.224 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.224 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=4B198558E9701B7EC26260935540DC23, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.225 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.228 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.228 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.228 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.192, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=4B198558E9701B7EC26260935540DC23, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.229 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:37:15.231 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.197, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.232 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:37:15.234 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:33:14.392, end=2024-09-11T18:35:14.817},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:37:15.234 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:37:15.234 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:37:15.244 INFO 2981023 --- [plan_change_detector15,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:37:15.244 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:37:15.245 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:33:14.392, end=2024-09-11T18:35:14.817},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:37:15.245 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:37:15.245 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:37:15.246 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:37:15.247 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:33:14.392, end=2024-09-11T18:35:14.817},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:37:15.247 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:37:15.247 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:37:15.256 INFO 2981023 --- [plan_change_detector20,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:37:15.271 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964635271000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:37:15.271Z, endTime=2024-09-11T10:35:14.817Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:37:15.275 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:37:15.275 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:37:15.282 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.282 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.284 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:37:15.284 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:37:15.289 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:37:15.289 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726050914817000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:37:15.291 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:37:15.291 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:37:15.271, end=2024-09-11T18:35:14.817}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:37:15.292 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:37:15.292 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:33:14.392, end=2024-09-11T18:35:14.817},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:37:15.292 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:37:15.292 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:37:15.293 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:37:15.294 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:33:14.392, end=2024-09-11T18:35:14.817},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:37:15.294 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:37:15.294 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:37:15.295 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:37:15.503 INFO 2981023 --- [ocp-async-10,1b920a6e6ad1fbed,2ecc931cc4f42212] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:15.534 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:15.534 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:16.023 INFO 2981023 --- [ocp-schedules-12,546bba47d6f98ae7,ae536d1ca8bbf22f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:37:16.023 INFO 2981023 --- [ocp-schedules-9,905813a3e07a312a,8c3c930737a91b46] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:37:16.024 INFO 2981023 --- [host-check-clockdiff1,546bba47d6f98ae7,77d2d0b15556b9a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:37:16.024 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:37:16.024 INFO 2981023 --- [host-check-clockdiff2,546bba47d6f98ae7,d3580474e3df1952] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:37:16.024 INFO 2981023 --- [host-check-clockdiff3,546bba47d6f98ae7,5cf6a5f3506c61fa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:37:16.029 INFO 2981023 --- [host-check-clockdiff4,546bba47d6f98ae7,33509c994d2a6fdf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:37:16.031 INFO 2981023 --- [ocp-schedules-3,e58ce4e005b34588,62501b4076076f6a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:37:16.033 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:37:16.034 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:37:16.034 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:37:16.048 INFO 2981023 --- [host-check-agent-status1,e58ce4e005b34588,b1d9bf3f3625c799] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:37:16.048 INFO 2981023 --- [host-check-agent-status2,e58ce4e005b34588,4475962704513e65] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:37:16.050 INFO 2981023 --- [host-check-agent-status3,e58ce4e005b34588,8da730143c56720d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:37:16.050 INFO 2981023 --- [host-check-agent-status4,e58ce4e005b34588,5fcd7d773ab629b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:37:16.062 INFO 2981023 --- [host-check-agent-status3,e58ce4e005b34588,8da730143c56720d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:16.065 INFO 2981023 --- [host-check-agent-status3,e58ce4e005b34588,8da730143c56720d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:16.065 INFO 2981023 --- [host-check-clockdiff1,546bba47d6f98ae7,77d2d0b15556b9a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051036 0 0 , error= 2024-09-11 18:37:16.071 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:16.073 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:16.075 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:37:16.077 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:37:16.077 INFO 2981023 --- [host-check-agent-status1,e58ce4e005b34588,b1d9bf3f3625c799] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:16.078 INFO 2981023 --- [host-check-agent-status4,e58ce4e005b34588,5fcd7d773ab629b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:37:16.079 INFO 2981023 --- [host-check-agent-status1,e58ce4e005b34588,b1d9bf3f3625c799] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:16.080 INFO 2981023 --- [host-check-agent-status4,e58ce4e005b34588,5fcd7d773ab629b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:37:16.081 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:16.081 INFO 2981023 --- [host-check-clockdiff3,546bba47d6f98ae7,5cf6a5f3506c61fa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051036 0 0 , error= 2024-09-11 18:37:16.083 INFO 2981023 --- [host-check-clockdiff2,546bba47d6f98ae7,d3580474e3df1952] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051036 5 5 , error= 2024-09-11 18:37:16.084 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:16.085 INFO 2981023 --- [host-check-agent-status2,e58ce4e005b34588,4475962704513e65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:37:16.087 INFO 2981023 --- [host-check-agent-status2,e58ce4e005b34588,4475962704513e65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:37:16.104 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:37:16.108 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:37:16.113 INFO 2981023 --- [host-check-agent-status3,e58ce4e005b34588,8da730143c56720d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:37:16.126 INFO 2981023 --- [host-check-agent-status3,e58ce4e005b34588,8da730143c56720d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:37:16.126 INFO 2981023 --- [host-check-agent-status3,e58ce4e005b34588,8da730143c56720d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:37:16.126 INFO 2981023 --- [host-check-agent-status3,e58ce4e005b34588,8da730143c56720d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:37:16.128 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:37:16.128 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:37:16.128396383+08:00 2024-09-11 18:37:16.129 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:37:16.129 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:37:16.128+08:00 2024-09-11 18:37:16.130 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:37:16.135931400+08:00 2024-09-11 18:37:16.134 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:37:16.134 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:37:16.128+08:00 and status AVAILABLE 2024-09-11 18:37:16.135 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:37:16.130+08:00 2024-09-11 18:37:16.139 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:37:16.138559618+08:00 2024-09-11 18:37:16.139 INFO 2981023 --- [host-check-agent-status1,e58ce4e005b34588,b1d9bf3f3625c799] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:37:16.140 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:37:16.139+08:00 2024-09-11 18:37:16.142 INFO 2981023 --- [host-check-agent-status1,e58ce4e005b34588,b1d9bf3f3625c799] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:37:16.142 INFO 2981023 --- [host-check-agent-status1,e58ce4e005b34588,b1d9bf3f3625c799] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:37:16.142 INFO 2981023 --- [host-check-agent-status1,e58ce4e005b34588,b1d9bf3f3625c799] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:37:16.142 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:37:16.139+08:00 and status AVAILABLE 2024-09-11 18:37:16.142 INFO 2981023 --- [host-check-agent-status4,e58ce4e005b34588,5fcd7d773ab629b9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:37:16.143 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:37:16.130+08:00 and status AVAILABLE 2024-09-11 18:37:16.147 INFO 2981023 --- [host-check-agent-status4,e58ce4e005b34588,5fcd7d773ab629b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:37:16.147 INFO 2981023 --- [host-check-agent-status4,e58ce4e005b34588,5fcd7d773ab629b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:37:16.147 INFO 2981023 --- [host-check-agent-status4,e58ce4e005b34588,5fcd7d773ab629b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:37:16.150 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:37:16.153 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:37:16.170 INFO 2981023 --- [host-check-agent-status2,e58ce4e005b34588,4475962704513e65] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:37:16.172 INFO 2981023 --- [host-check-agent-status2,e58ce4e005b34588,4475962704513e65] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:37:16.172 INFO 2981023 --- [host-check-agent-status2,e58ce4e005b34588,4475962704513e65] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:37:16.172 INFO 2981023 --- [host-check-agent-status2,e58ce4e005b34588,4475962704513e65] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:37:16.173 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:37:16.189 INFO 2981023 --- [ocp-schedules-27,e28ab932e7919e77,ffcf552767b489b8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:37:16.189 INFO 2981023 --- [ocp-schedules-21,69e01a2b8aed24b7,05037388520ab22d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:37:16.190 INFO 2981023 --- [ocp-schedules-14,1d3d8b87d19b8255,1bd5e0081297b74d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:37:16.190 INFO 2981023 --- [ocp-schedules-23,326c10f10919678b,eda9a88d029cc9d3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:37:16.195 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:37:16.198 INFO 2981023 --- [ocp-schedules-27,e28ab932e7919e77,ffcf552767b489b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:37:16.199 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:37:16.196418065+08:00 2024-09-11 18:37:16.200 INFO 2981023 --- [ocp-schedules-21,69e01a2b8aed24b7,05037388520ab22d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:37:16.202 INFO 2981023 --- [ocp-schedules-14,1d3d8b87d19b8255,1bd5e0081297b74d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:37:16.202 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:37:16.199+08:00 2024-09-11 18:37:16.204 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:37:16.199+08:00 and status AVAILABLE 2024-09-11 18:37:16.205 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:16.206 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:16.208 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:37:16.209 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:37:16.212 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:16.214 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:16.214 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050963458000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:37:16.223 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:37:16.229 INFO 2981023 --- [ocp-schedules-17,2bed0fc7025b2458,09e1ee04cf6285e3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:37:16.235 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:37:16.244 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:37:16.246 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:37:16.251 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:37:16.252 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.253 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:37:16.255 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:37:16.256 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:37:16.272 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:37:16.274 INFO 2981023 --- [ocp-schedules-21,69e01a2b8aed24b7,05037388520ab22d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:37:16.279 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:37:16.280 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.280 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:16.285 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:37:16.300 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:37:16.301 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:37:16.303 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.304 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.305 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:37:16.305 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:16.305 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:37:16.306 INFO 2981023 --- [ocp-schedules-27,e28ab932e7919e77,ffcf552767b489b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:37:16.307 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:37:16.308 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:37:16.311 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:37:16.322 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.323 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:16.330 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:37:16.343 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:37:16.358 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:37:16.379 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:37:16.381 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:37:16.381 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:37:16.380959+08:00, lastProcessTime=2024-09-11T18:37:06.337504+08:00 2024-09-11 18:37:16.394 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:37:16.399 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:16.400 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:37:16.402 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.403 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:37:16.405 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:37:16.410 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:37:16.411 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:37:16.419 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:37:16.419 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050963458000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:37:16.425 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:37:16.427 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:37:16.436 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:37:16.437 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:37:16.441 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:37:16.441 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:16.444 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.445 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:16.445 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:37:16.449 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.451 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:37:16.452 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:37:16.458 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:37:16.464 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:37:16.469 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:37:16.469 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:37:16.478 INFO 2981023 --- [host-check-host-status2,905813a3e07a312a,f90b2cae57504f4c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:37:16.479 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:37:16.482 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:37:16.483 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:37:16.488 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:37:16.493 INFO 2981023 --- [host-check-host-status3,905813a3e07a312a,10718f7e237eaf58] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:37:16.493 INFO 2981023 --- [host-check-host-status1,905813a3e07a312a,26f815e255e6f176] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:37:16.499 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:16.500 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:37:16.500 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.501 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:37:16.501 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.501 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:37:16.502 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:37:16.506 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:37:16.509 INFO 2981023 --- [host-check-host-status4,905813a3e07a312a,097d56631fa455c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:37:16.510 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:16.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:16.535 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:16.536 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.536 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:16.537 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:37:16.537 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.539 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:37:16.540 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:37:16.541 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:37:16.548 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:37:16.549 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:37:16.559 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:37:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:16.571 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.572 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:37:16.573 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:37:16.577 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:37:16.584 INFO 2981023 --- [ocp-schedules-28,d6d553ae1341c915,fa510d72e8ec9b4c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:37:16.584 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:16.586 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:16.599 INFO 2981023 --- [ocp-schedules-28,d6d553ae1341c915,fa510d72e8ec9b4c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:37:16.599 INFO 2981023 --- [ocp-schedules-28,d6d553ae1341c915,fa510d72e8ec9b4c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:37:16.607 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:37:16.613 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:37:16.619 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:16.623 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.623 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:37:16.632 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:37:16.632 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:37:16.639 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:37:16.645 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : aggregate done, channelId=1, alarmsCount=1, resultsCount=1 2024-09-11 18:37:16.646 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:16.646 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:16.647 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:37:16.650 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:37:16.380959+08:00 2024-09-11 18:37:16.652 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:37:16.652 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:37:16.653 INFO 2981023 --- [ocp-schedules-14,1d3d8b87d19b8255,1bd5e0081297b74d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:37:16.669 INFO 2981023 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726050960, nodeCount=480, validCount=480 2024-09-11 18:37:17.058 INFO 2981023 --- [ocp-schedules-24,7be51dc4a03b6f91,ceef1d4f17be1c1b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:37:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,51a0cc80d018ec66,c557ed69fe7b5856] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,51a0cc80d018ec66,c557ed69fe7b5856] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:17.503 INFO 2981023 --- [ocp-async-5,2133a6eb63225528,79fb87ce926cc72b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:17.804 INFO 2981023 --- [ocp-async-9,d816104faca386dc,075b2613e99202c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:37:18.534 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:18.535 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:19.503 INFO 2981023 --- [ocp-async-6,740672cf65611a86,6fb4afe578bb202f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:19.579 INFO 2981023 --- [ocp-schedules-4,a15dabf4e0a914ed,8af0a7b27490312b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:37:19.628 WARN 2981023 --- [scanner-13702857-1,a15dabf4e0a914ed,46e63d99eb0d6a55] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:37:19.650 INFO 2981023 --- [alarm-detect-14,a15dabf4e0a914ed,2df0d30601673f68] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646815, data is empty. 2024-09-11 18:37:19.653 INFO 2981023 --- [alarm-detect-14,a15dabf4e0a914ed,2df0d30601673f68] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646878, data is empty. 2024-09-11 18:37:19.659 INFO 2981023 --- [alarm-detect-14,a15dabf4e0a914ed,2df0d30601673f68] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646878, data is empty. 2024-09-11 18:37:19.670 INFO 2981023 --- [alarm-detect-16,a15dabf4e0a914ed,8231a89485e781e8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646815, data is empty. 2024-09-11 18:37:19.675 INFO 2981023 --- [alarm-detect-16,a15dabf4e0a914ed,8231a89485e781e8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646878, data is empty. 2024-09-11 18:37:19.677 INFO 2981023 --- [alarm-detect-14,a15dabf4e0a914ed,d0ecfa694e5b50c4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646815, data is empty. 2024-09-11 18:37:19.686 INFO 2981023 --- [alarm-detect-16,a15dabf4e0a914ed,8231a89485e781e8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646878, data is empty. 2024-09-11 18:37:19.686 INFO 2981023 --- [alarm-detect-14,a15dabf4e0a914ed,d0ecfa694e5b50c4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646878, data is empty. 2024-09-11 18:37:19.697 INFO 2981023 --- [alarm-detect-14,a15dabf4e0a914ed,d0ecfa694e5b50c4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646878, data is empty. 2024-09-11 18:37:19.820 INFO 2981023 --- [alarm-detect-15,a15dabf4e0a914ed,5ea6713cd5d410ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646815, data is empty. 2024-09-11 18:37:19.823 INFO 2981023 --- [alarm-detect-15,a15dabf4e0a914ed,5ea6713cd5d410ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646878, data is empty. 2024-09-11 18:37:19.826 INFO 2981023 --- [alarm-detect-15,a15dabf4e0a914ed,5ea6713cd5d410ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646878, data is empty. 2024-09-11 18:37:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,155736ec0ce8d367,9ceba00b645fef5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,155736ec0ce8d367,9ceba00b645fef5b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:21.503 INFO 2981023 --- [ocp-async-4,a03c3ae2ccf5e918,45d7afc673e59a61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:21.535 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:21.536 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:21.950 INFO 2981023 --- [host-check-clockdiff4,546bba47d6f98ae7,33509c994d2a6fdf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051041 0 0 , error= 2024-09-11 18:37:23.008 INFO 2981023 --- [ocp-schedules-16,6ea5c865cbe1fe1b,b6a70354bc7aa0ad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:37:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,ff0f56b930f5d28f,6eb8d04c95e2f390] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,ff0f56b930f5d28f,6eb8d04c95e2f390] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:23.503 INFO 2981023 --- [ocp-async-1,7a73de893fd8ea9d,2a8ab01f1df5c094] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:24.536 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:24.537 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:24.562 INFO 2981023 --- [ocp-schedules-16,63a0c3ef02db0889,9e9f40deb5f2e9fc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:37:25.504 INFO 2981023 --- [ocp-async-7,b619881c930c812f,1937f3c98aec261f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,ea9e8efdbeb4bd05,9c88e4145f806470] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,ea9e8efdbeb4bd05,9c88e4145f806470] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:26.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:37:26.353624+08:00, lastProcessTime=2024-09-11T18:37:16.380959+08:00 2024-09-11 18:37:26.491 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:37:26.353624+08:00 2024-09-11 18:37:27.504 INFO 2981023 --- [ocp-async-8,e49d12d163ae59bc,09bb4a4a82e569ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:27.537 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:27.538 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:27.804 INFO 2981023 --- [ocp-async-1,31604cc175c3cfdf,7cb7508c474dfdbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:37:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,c177b4634c491335,521cfbcceb747b28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,c177b4634c491335,521cfbcceb747b28] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:29.385 INFO 2981023 --- [ocp-schedules-1,3ac5a760d4dd1be4,7144c9b4173d3148] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:37:29.385 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:37:29.386 INFO 2981023 --- [ocp-schedules-1,3ac5a760d4dd1be4,7144c9b4173d3148] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:37:29.386 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:37:29.386 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:37:29.411 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:37:29.411 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:37:29.414 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:37:29.415 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:37:29.417 INFO 2981023 --- [ocp-schedules-1,3ac5a760d4dd1be4,7144c9b4173d3148] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:29.418 INFO 2981023 --- [ocp-schedules-1,3ac5a760d4dd1be4,7144c9b4173d3148] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:29.472 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:29.504 INFO 2981023 --- [ocp-async-5,9e09345d0457444f,d3dc59911d9dea5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:29.565 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:29.591 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:37:29.592 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:37:29.592 INFO 2981023 --- [ocp-schedules-1,3ac5a760d4dd1be4,7144c9b4173d3148] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:37:29.594 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:37:29.597 INFO 2981023 --- [ocp-schedules-1,3ac5a760d4dd1be4,7144c9b4173d3148] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:37:29.865 INFO 2981023 --- [ocp-schedules-23,d50aa2c382985d18,33162369daa02cfd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:37:29.904 WARN 2981023 --- [scanner-13703169-1,d50aa2c382985d18,6c688b05719c6f02] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:37:30.538 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:30.539 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:31.504 INFO 2981023 --- [ocp-async-9,7911048f8e0018cb,b60cfd99553cce19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,c0603066cd9d3c62,59bda13d7161e67c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:32.317 INFO 2981023 --- [prometheus-minute-schedule-1,c0603066cd9d3c62,59bda13d7161e67c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:33.011 INFO 2981023 --- [ocp-schedules-28,ed338ec9e93fc118,899fd694982245a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:37:33.504 INFO 2981023 --- [ocp-async-6,1293e981836ae2d2,4e40b1ee5c288e52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:33.539 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:33.540 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,47c8184a0cf4f705,12930e69da2f0e6c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,47c8184a0cf4f705,12930e69da2f0e6c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:35.504 INFO 2981023 --- [ocp-async-4,bc425b1aab8d4c3b,e8bb56347b5bda09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:36.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:37:36.351141+08:00, lastProcessTime=2024-09-11T18:37:26.353624+08:00 2024-09-11 18:37:36.473 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:37:36.351141+08:00 2024-09-11 18:37:36.540 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:36.540 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:36.602 INFO 2981023 --- [ocp-schedules-10,c57b2d2b68ee793c,f8f5a7c1f90724e1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:37:36.610 INFO 2981023 --- [ocp-schedules-10,c57b2d2b68ee793c,f8f5a7c1f90724e1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:37:36.610 INFO 2981023 --- [ocp-schedules-10,c57b2d2b68ee793c,f8f5a7c1f90724e1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:37:37.314 INFO 2981023 --- [prometheus-cache-monitor1,90f86a4dcac082b2,51a3520aed7aef95] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:37:37.315 INFO 2981023 --- [prometheus-cache-monitor1,90f86a4dcac082b2,51a3520aed7aef95] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:37:37.504 INFO 2981023 --- [ocp-async-2,ad066b0bc85bdac3,f462514f619c5d4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:37.804 INFO 2981023 --- [ocp-async-8,b2b91f91889cba00,2653ed4a982374b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:37:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,f4f7341a5ac2042e,ac3ba6ac57830ebd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,f4f7341a5ac2042e,ac3ba6ac57830ebd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:39.504 INFO 2981023 --- [ocp-async-10,1a65a0d6253c8c12,cdb783007fe1601e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:39.540 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:39.541 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:40.003 INFO 2981023 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:37:40.003 INFO 2981023 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:37:40.070 INFO 2981023 --- [ocp-schedules-30,75ab843b3fb33208,28b26fca09c5ff28] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:37:40.118 WARN 2981023 --- [scanner-13703398-1,75ab843b3fb33208,624c3e0b8f5cf009] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:37:40.147 INFO 2981023 --- [alarm-detect-15,75ab843b3fb33208,137602d5a120aa8a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646878, data is empty. 2024-09-11 18:37:40.153 INFO 2981023 --- [alarm-detect-15,75ab843b3fb33208,137602d5a120aa8a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646878, data is empty. 2024-09-11 18:37:40.159 INFO 2981023 --- [alarm-detect-15,75ab843b3fb33208,137602d5a120aa8a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646878, data is empty. 2024-09-11 18:37:40.165 INFO 2981023 --- [alarm-detect-11,75ab843b3fb33208,b53f7bcd7fa994dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646878, data is empty. 2024-09-11 18:37:40.166 INFO 2981023 --- [alarm-detect-2,75ab843b3fb33208,10b3b2d16fc3fdd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646878, data is empty. 2024-09-11 18:37:40.170 INFO 2981023 --- [alarm-detect-2,75ab843b3fb33208,10b3b2d16fc3fdd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646878, data is empty. 2024-09-11 18:37:40.178 INFO 2981023 --- [alarm-detect-2,75ab843b3fb33208,10b3b2d16fc3fdd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646878, data is empty. 2024-09-11 18:37:40.182 INFO 2981023 --- [alarm-detect-11,75ab843b3fb33208,b53f7bcd7fa994dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646878, data is empty. 2024-09-11 18:37:40.185 INFO 2981023 --- [alarm-detect-11,75ab843b3fb33208,b53f7bcd7fa994dd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646878, data is empty. 2024-09-11 18:37:40.327 INFO 2981023 --- [alarm-detect-15,75ab843b3fb33208,74a7907f652c3122] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646878, data is empty. 2024-09-11 18:37:40.331 INFO 2981023 --- [alarm-detect-15,75ab843b3fb33208,74a7907f652c3122] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646878, data is empty. 2024-09-11 18:37:40.336 INFO 2981023 --- [alarm-detect-15,75ab843b3fb33208,74a7907f652c3122] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646878, data is empty. 2024-09-11 18:37:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,cffc1fbb3cf54ce8,376335d3ee5e6847] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:41.317 INFO 2981023 --- [prometheus-minute-schedule-1,cffc1fbb3cf54ce8,376335d3ee5e6847] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:41.504 INFO 2981023 --- [ocp-async-3,9c9043621c2f2f2c,f74b667aa6cbbd6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:42.541 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:42.542 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:43.013 INFO 2981023 --- [ocp-schedules-17,2ff000532721ed82,935ffa8370791995] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:37:43.505 INFO 2981023 --- [ocp-async-7,1cf0fa17bfb5a03e,6c9d8d1a864328dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,8f6295fe40aab748,1b4fe3410b2d5d11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:44.315 INFO 2981023 --- [prometheus-minute-schedule-1,8f6295fe40aab748,1b4fe3410b2d5d11] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:45.101 INFO 2981023 --- [prometheus-second-schedule-1,ca5a99a8fdbf6b29,16a9b570e8aba79c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:37:45.505 INFO 2981023 --- [ocp-async-9,45e914f690c7f02c,1abfd3eb215a795f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:45.542 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:45.543 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:46.015 INFO 2981023 --- [ocp-schedules-13,8351fbfcc69ffbf7,5f5e92b08a2993ad] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:37:46.015 INFO 2981023 --- [host-check-clockdiff1,8351fbfcc69ffbf7,b803d4bdbb48b9a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:37:46.016 INFO 2981023 --- [host-check-clockdiff2,8351fbfcc69ffbf7,60e54f80b0e9b8f6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:37:46.017 INFO 2981023 --- [host-check-clockdiff3,8351fbfcc69ffbf7,58d922096cdd8484] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:37:46.017 INFO 2981023 --- [host-check-clockdiff4,8351fbfcc69ffbf7,d771e21257f95528] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:37:46.031 INFO 2981023 --- [ocp-schedules-29,b0e644597005af25,7d9a46fe20b16d22] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:37:46.031 INFO 2981023 --- [ocp-schedules-18,8b1d14346c73f54f,6fb4c353f42f7e87] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:37:46.032 INFO 2981023 --- [host-check-agent-status2,b0e644597005af25,9abaf1624b1a1946] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:37:46.032 INFO 2981023 --- [host-check-agent-status3,b0e644597005af25,41a20365ad15c785] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:37:46.032 INFO 2981023 --- [host-check-agent-status1,b0e644597005af25,02a343673800bd45] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:37:46.033 INFO 2981023 --- [host-check-agent-status4,b0e644597005af25,2ae80a2906bf8fa6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:37:46.033 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:37:46.033 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:37:46.036 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:37:46.036 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:37:46.050 INFO 2981023 --- [host-check-clockdiff2,8351fbfcc69ffbf7,60e54f80b0e9b8f6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051066 6 6 , error= 2024-09-11 18:37:46.052 INFO 2981023 --- [host-check-clockdiff3,8351fbfcc69ffbf7,58d922096cdd8484] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051066 0 0 , error= 2024-09-11 18:37:46.064 INFO 2981023 --- [host-check-agent-status3,b0e644597005af25,41a20365ad15c785] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:46.067 INFO 2981023 --- [host-check-agent-status4,b0e644597005af25,2ae80a2906bf8fa6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:37:46.068 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:46.070 INFO 2981023 --- [host-check-agent-status3,b0e644597005af25,41a20365ad15c785] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:46.070 INFO 2981023 --- [host-check-agent-status4,b0e644597005af25,2ae80a2906bf8fa6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:37:46.073 INFO 2981023 --- [host-check-agent-status1,b0e644597005af25,02a343673800bd45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:46.074 INFO 2981023 --- [host-check-agent-status1,b0e644597005af25,02a343673800bd45] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:46.083 INFO 2981023 --- [host-check-agent-status2,b0e644597005af25,9abaf1624b1a1946] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:37:46.085 INFO 2981023 --- [host-check-agent-status2,b0e644597005af25,9abaf1624b1a1946] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:37:46.087 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:37:46.088 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:37:46.090 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:46.104 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:46.115 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:37:46.115 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:46.170 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:37:46.197 INFO 2981023 --- [ocp-schedules-16,92a90024b636465e,bda2d3f593364ae9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:37:46.202 INFO 2981023 --- [ocp-schedules-24,bccbaa2f2d3c26da,e682ac14fda91f5b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:37:46.207 INFO 2981023 --- [ocp-schedules-7,2db030d13573cbc7,b64b8baad1a65093] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:37:46.207 INFO 2981023 --- [ocp-schedules-28,72253d4aa2ed95b6,6fe643005bff2d82] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:37:46.271 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:37:46.280 INFO 2981023 --- [ocp-schedules-16,92a90024b636465e,bda2d3f593364ae9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:37:46.281 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050963458000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:37:46.284 INFO 2981023 --- [ocp-schedules-7,2db030d13573cbc7,b64b8baad1a65093] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:37:46.286 INFO 2981023 --- [ocp-schedules-24,bccbaa2f2d3c26da,e682ac14fda91f5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:37:46.288 INFO 2981023 --- [ocp-schedules-24,bccbaa2f2d3c26da,e682ac14fda91f5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:37:46.289 INFO 2981023 --- [ocp-schedules-6,fb2ca1a9f43cf3d5,67ed899d27dbf3b6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:37:46.392 INFO 2981023 --- [ocp-schedules-27,660e7bfaa5916990,083fdb7710128605] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:37:46.461 INFO 2981023 --- [ocp-schedules-6,fb2ca1a9f43cf3d5,67ed899d27dbf3b6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:37:46.472 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:37:46.487 INFO 2981023 --- [host-check-agent-status2,b0e644597005af25,9abaf1624b1a1946] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:37:46.491 INFO 2981023 --- [host-check-agent-status2,b0e644597005af25,9abaf1624b1a1946] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:37:46.491 INFO 2981023 --- [host-check-agent-status2,b0e644597005af25,9abaf1624b1a1946] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:37:46.491 INFO 2981023 --- [host-check-agent-status2,b0e644597005af25,9abaf1624b1a1946] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:37:46.556 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:37:46.555444+08:00, lastProcessTime=2024-09-11T18:37:36.351141+08:00 2024-09-11 18:37:46.556 INFO 2981023 --- [host-check-agent-status1,b0e644597005af25,02a343673800bd45] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:37:46.559 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:37:46.559 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:37:46.564741329+08:00 2024-09-11 18:37:46.560 INFO 2981023 --- [host-check-agent-status3,b0e644597005af25,41a20365ad15c785] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:37:46.560 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:37:46.561 INFO 2981023 --- [host-check-agent-status1,b0e644597005af25,02a343673800bd45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:37:46.561 INFO 2981023 --- [host-check-agent-status1,b0e644597005af25,02a343673800bd45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:37:46.561 INFO 2981023 --- [host-check-agent-status1,b0e644597005af25,02a343673800bd45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:37:46.561 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:37:46.561 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:37:46.561503683+08:00 2024-09-11 18:37:46.561 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:37:46.559+08:00 2024-09-11 18:37:46.563 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:37:46.562653364+08:00 2024-09-11 18:37:46.564 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:37:46.563+08:00 2024-09-11 18:37:46.564 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:37:46.561+08:00 2024-09-11 18:37:46.565 INFO 2981023 --- [host-check-agent-status4,b0e644597005af25,2ae80a2906bf8fa6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:37:46.565 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:37:46.567 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:37:46.561+08:00 and status AVAILABLE 2024-09-11 18:37:46.567 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:37:46.559+08:00 and status AVAILABLE 2024-09-11 18:37:46.567 INFO 2981023 --- [host-check-agent-status3,b0e644597005af25,41a20365ad15c785] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:37:46.567 INFO 2981023 --- [host-check-agent-status3,b0e644597005af25,41a20365ad15c785] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:37:46.567 INFO 2981023 --- [host-check-agent-status3,b0e644597005af25,41a20365ad15c785] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:37:46.567 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:37:46.567709669+08:00 2024-09-11 18:37:46.569 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:37:46.567+08:00 2024-09-11 18:37:46.570 INFO 2981023 --- [host-check-agent-status4,b0e644597005af25,2ae80a2906bf8fa6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:37:46.570 INFO 2981023 --- [host-check-agent-status4,b0e644597005af25,2ae80a2906bf8fa6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:37:46.570 INFO 2981023 --- [host-check-agent-status4,b0e644597005af25,2ae80a2906bf8fa6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:37:46.571 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:37:46.567+08:00 and status AVAILABLE 2024-09-11 18:37:46.574 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:37:46.563+08:00 and status AVAILABLE 2024-09-11 18:37:46.575 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.575 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:37:46.576 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:37:46.576 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:37:46.582 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:37:46.584 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:37:46.586 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:37:46.606 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.606 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.606 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:46.608 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:37:46.608 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:37:46.609 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:37:46.610 INFO 2981023 --- [ocp-schedules-16,92a90024b636465e,bda2d3f593364ae9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:37:46.610 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:37:46.611 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:37:46.611 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.611 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:46.654 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:46.655 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:37:46.656 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:46.657 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:46.658 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:46.669 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.670 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:37:46.671 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:37:46.672 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.674 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.675 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:46.676 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:37:46.685 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:37:46.686 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:37:46.691 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:37:46.699 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:37:46.704 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:37:46.714 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:37:46.724 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:37:46.727 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:37:46.732 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:37:46.739 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.739 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.740 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:37:46.754 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:37:46.766 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:37:46.768 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.771 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.771 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:37:46.772 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:37:46.772 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.774 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:37:46.774 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:37:46.774 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050963458000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:37:46.795 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:37:46.797 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:37:46.797 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.798 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.798 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:46.799 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:37:46.799 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.802 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:37:46.802 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:37:46.803 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:37:46.804 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:37:46.808 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:37:46.812 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:37:46.812 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:37:46.815 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:37:46.828 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.828 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:37:46.830 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:37:46.833 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:37:46.850 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.850 INFO 2981023 --- [ocp-schedules-15,c9214c8b7c3b0a96,3baf8945c14cd9cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2049,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:37:46.850 INFO 2981023 --- [ocp-schedules-15,c9214c8b7c3b0a96,3baf8945c14cd9cc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:37:46.850 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.851 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.851 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.851 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:37:46.851 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:37:46.852 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:37:46.852 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:37:46.855 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:37:46.856 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:37:46.857 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:37:46.861 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:37:46.862 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:37:46.863 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.863 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:37:46.864 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:37:46.869 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:37:46.869 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:37:46.869 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.875 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:37:46.876 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:37:46.882 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:37:46.885 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:37:46.889 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:37:46.890 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:37:46.894 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:37:46.906 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:37:46.907 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:37:46.909 INFO 2981023 --- [host-check-host-status4,8b1d14346c73f54f,89df9b13e70b8dd7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:37:46.918 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:37:46.921 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:37:46.934 INFO 2981023 --- [host-check-host-status2,8b1d14346c73f54f,7b5bf9ec5aa5a7bd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:37:46.936 INFO 2981023 --- [host-check-host-status3,8b1d14346c73f54f,08b99a2e8d96a115] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:37:46.937 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.938 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.938 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:37:46.940 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.940 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.943 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:37:46.944 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:37:46.944 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.948 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:37:46.959 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:37:46.970 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:37:46.975 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:37:46.975 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.976 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.976 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:37:46.977 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:37:46.978 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.980 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:37:46.980 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:37:46.981 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:37:46.981 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:46.982 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:46.982 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:37:46.990 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:37:46.990 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:37:46.992 INFO 2981023 --- [host-check-host-status1,8b1d14346c73f54f,6635278e43e10504] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:37:46.994 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:37:46.994 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:37:47.003 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:37:47.004 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:47.005 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:47.005 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:37:47.006 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:37:47.006 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:47.008 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:37:46.555444+08:00 2024-09-11 18:37:47.009 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:47.010 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:47.010 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:37:47.011 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:37:47.014 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:37:47.017 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:37:47.017 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:37:47.029 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:37:47.050 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:37:47.062 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:37:47.067 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:37:47.068 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:37:47.068 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:47.069 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:37:47.074 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:37:47.075 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:37:47.075 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:37:47.075 INFO 2981023 --- [ocp-schedules-7,2db030d13573cbc7,b64b8baad1a65093] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:37:47.077 INFO 2981023 --- [ocp-schedules-8,a7bc8f59772e3d6c,8a9e518432591860] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:37:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,7168cb866b29ae4b,e852d323c45c94b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,7168cb866b29ae4b,e852d323c45c94b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:47.505 INFO 2981023 --- [ocp-async-1,47fa807d820f7000,5bb5f573affa6cab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:47.804 INFO 2981023 --- [ocp-async-6,1775d46d5756c4f0,a001fb35817d4f78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:37:47.812 INFO 2981023 --- [host-check-clockdiff1,8351fbfcc69ffbf7,b803d4bdbb48b9a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051067 0 0 , error= 2024-09-11 18:37:48.543 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:48.544 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:49.505 INFO 2981023 --- [ocp-async-7,5bf11a27792a1bd9,1e46130e68c7c5ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:49.888 INFO 2981023 --- [ocp-schedules-10,da4f1f9e47a9e2e9,f0cff7bc46988879] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:37:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:37:49.907 WARN 2981023 --- [ocp-schedules-10,da4f1f9e47a9e2e9,f0cff7bc46988879] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:37:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:37:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,12346f20a96c546e,beb82408853fd098] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,12346f20a96c546e,beb82408853fd098] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:50.376 INFO 2981023 --- [ocp-schedules-14,ce403949d64282d3,8dec63cd6912e61f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:37:50.407 INFO 2981023 --- [ocp-schedules-22,a9108417a0c782bc,a50fdd6ce9035e94] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:37:50.434 WARN 2981023 --- [scanner-13703714-1,ce403949d64282d3,12cc48a8adb17c87] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:37:51.505 INFO 2981023 --- [ocp-async-9,9dc3f62ef1a820cb,76165410f5ea2f67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:51.544 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:51.546 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:51.664 INFO 2981023 --- [ocp-schedules-28,21ac19f6226d16e9,5479767248687ecf] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:37:51.665 INFO 2981023 --- [ocp-schedules-28,21ac19f6226d16e9,5479767248687ecf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:37:51.665 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:37:51.665 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:37:51.665 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:37:51.665 INFO 2981023 --- [ocp-schedules-28,21ac19f6226d16e9,5479767248687ecf] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:37:52.008 INFO 2981023 --- [host-check-clockdiff4,8351fbfcc69ffbf7,d771e21257f95528] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051072 0 0 , error= 2024-09-11 18:37:53.016 INFO 2981023 --- [ocp-schedules-2,c7074be5ed5d9a55,678ee4b94a3075eb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:37:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,ff35abfb35e04235,04c718db14e2d5ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,ff35abfb35e04235,04c718db14e2d5ea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:53.446 INFO 2981023 --- [ocp-schedules-29,6ce899b927ce2ccd,879623cdfb5e2cb7] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:37:53.446 INFO 2981023 --- [ocp-schedules-29,6ce899b927ce2ccd,879623cdfb5e2cb7] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:37:53.505 INFO 2981023 --- [ocp-async-3,c004292178663485,7d9e15347ad8c029] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:54.546 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:54.546 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:54.564 INFO 2981023 --- [ocp-schedules-26,4c54dbc8f154b5da,6f2850eddf260a5f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:37:55.505 INFO 2981023 --- [ocp-async-2,eeb737920934efd7,4b718d0d65ac54b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:55.983 INFO 2981023 --- [ocp-schedules-14,029dd0a2360529b9,4602e5dcd4888b4d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:37:55.983 INFO 2981023 --- [ocp-schedules-14,029dd0a2360529b9,4602e5dcd4888b4d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:37:55.983 INFO 2981023 --- [ocp-schedules-14,029dd0a2360529b9,4602e5dcd4888b4d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:37:55.983 INFO 2981023 --- [ocp-schedules-14,029dd0a2360529b9,4602e5dcd4888b4d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:37:55.984 INFO 2981023 --- [ocp-schedules-14,029dd0a2360529b9,4602e5dcd4888b4d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:37:55.984 INFO 2981023 --- [ocp-schedules-14,029dd0a2360529b9,4602e5dcd4888b4d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:37:56.018 INFO 2981023 --- [ocp-schedules-27,64e390e970fec081,2ce6afd7fc6a3327] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:37:56.037 WARN 2981023 --- [ocp-schedules-27,64e390e970fec081,2ce6afd7fc6a3327] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:37:56.219 INFO 2981023 --- [ocp-schedules-17,ad4b46a0a539c417,67c926c7a3de050d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:37:56.226 INFO 2981023 --- [ocp-schedules-17,ad4b46a0a539c417,67c926c7a3de050d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:37:56.226 INFO 2981023 --- [ocp-schedules-17,ad4b46a0a539c417,67c926c7a3de050d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:37:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,665439e043481060,a0a682b8b694f626] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,665439e043481060,a0a682b8b694f626] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:56.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:37:56.354309+08:00, lastProcessTime=2024-09-11T18:37:46.555444+08:00 2024-09-11 18:37:56.507 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:37:56.354309+08:00 2024-09-11 18:37:56.613 INFO 2981023 --- [ocp-schedules-22,a31a8514199b6165,bc2de629fc3213c3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:37:56.621 INFO 2981023 --- [ocp-schedules-22,a31a8514199b6165,bc2de629fc3213c3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:37:56.621 INFO 2981023 --- [ocp-schedules-22,a31a8514199b6165,bc2de629fc3213c3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:37:57.506 INFO 2981023 --- [ocp-async-4,a52ee231a733b874,45916fe77b6963c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:57.546 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:37:57.548 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:37:57.804 INFO 2981023 --- [ocp-async-1,10a93717bf15b4f3,9ca38f6283ba7228] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:37:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,2d82520075a7ef14,e5c172472a7ccc00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051080, interval=60 2024-09-11 18:37:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,2d82520075a7ef14,e5c172472a7ccc00] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:37:59.405 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.408 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=4, name=sys, clusterName 2024-09-11 18:37:59.417 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.425 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.428 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.437 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.438 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.439 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.439 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.440 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.441 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.441 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.442 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.443 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.446 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.449 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.451 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.454 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.457 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.460 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.463 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.465 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.466 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.467 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.467 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.468 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:31:59.33, end=2024-09-11T18:36:59.401} 2024-09-11 18:37:59.468 INFO 2981023 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2024-09-11 18:37:59.506 INFO 2981023 --- [ocp-async-3,3bee1fd66308e80b,8849a9d0f5e995ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:37:59.547 INFO 2981023 --- [ocp-schedules-2,5318fbde85ff7914,dba86d6c2fab0b5f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:37:59.547 INFO 2981023 --- [ocp-schedules-2,5318fbde85ff7914,dba86d6c2fab0b5f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:37:59.579 INFO 2981023 --- [Collect all tenants major compactions1,5318fbde85ff7914,02bc52bed24ff03c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:37:59.579 INFO 2981023 --- [Collect all tenants major compactions3,5318fbde85ff7914,cfa415ce2afa0887] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:37:59.580 INFO 2981023 --- [Collect all tenants major compactions4,5318fbde85ff7914,7ca0fb8d9cb8febc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:37:59.580 INFO 2981023 --- [Collect all tenants major compactions2,5318fbde85ff7914,ce9181ec82ed838a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:37:59.580 INFO 2981023 --- [Collect all tenants major compactions5,5318fbde85ff7914,f213c01446260b36] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:37:59.589 INFO 2981023 --- [Collect all tenants major compactions6,5318fbde85ff7914,b4fb989f14b82f4f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:37:59.600 INFO 2981023 --- [Collect all tenants major compactions1,5318fbde85ff7914,02bc52bed24ff03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:59.601 INFO 2981023 --- [Collect all tenants major compactions3,5318fbde85ff7914,cfa415ce2afa0887] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:59.601 INFO 2981023 --- [Collect all tenants major compactions1,5318fbde85ff7914,02bc52bed24ff03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:37:59.601 INFO 2981023 --- [Collect all tenants major compactions4,5318fbde85ff7914,7ca0fb8d9cb8febc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:59.602 INFO 2981023 --- [Collect all tenants major compactions3,5318fbde85ff7914,cfa415ce2afa0887] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:37:59.602 INFO 2981023 --- [Collect all tenants major compactions2,5318fbde85ff7914,ce9181ec82ed838a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:59.603 INFO 2981023 --- [Collect all tenants major compactions5,5318fbde85ff7914,f213c01446260b36] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:59.604 INFO 2981023 --- [Collect all tenants major compactions4,5318fbde85ff7914,7ca0fb8d9cb8febc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:37:59.604 INFO 2981023 --- [Collect all tenants major compactions5,5318fbde85ff7914,f213c01446260b36] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:37:59.604 INFO 2981023 --- [Collect all tenants major compactions6,5318fbde85ff7914,b4fb989f14b82f4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:37:59.605 INFO 2981023 --- [Collect all tenants major compactions3,5318fbde85ff7914,cfa415ce2afa0887] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:37:59.605 INFO 2981023 --- [Collect all tenants major compactions3,5318fbde85ff7914,cfa415ce2afa0887] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:37:59.605 INFO 2981023 --- [Collect all tenants major compactions1,5318fbde85ff7914,02bc52bed24ff03c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:37:59.605 INFO 2981023 --- [Collect all tenants major compactions1,5318fbde85ff7914,02bc52bed24ff03c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:37:59.605 INFO 2981023 --- [Collect all tenants major compactions2,5318fbde85ff7914,ce9181ec82ed838a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:37:59.606 INFO 2981023 --- [Collect all tenants major compactions6,5318fbde85ff7914,b4fb989f14b82f4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:37:59.606 INFO 2981023 --- [Collect all tenants major compactions5,5318fbde85ff7914,f213c01446260b36] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:37:59.606 INFO 2981023 --- [Collect all tenants major compactions5,5318fbde85ff7914,f213c01446260b36] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:37:59.607 INFO 2981023 --- [Collect all tenants major compactions4,5318fbde85ff7914,7ca0fb8d9cb8febc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:37:59.607 INFO 2981023 --- [Collect all tenants major compactions4,5318fbde85ff7914,7ca0fb8d9cb8febc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:37:59.610 INFO 2981023 --- [Collect all tenants major compactions6,5318fbde85ff7914,b4fb989f14b82f4f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:37:59.610 INFO 2981023 --- [Collect all tenants major compactions6,5318fbde85ff7914,b4fb989f14b82f4f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:37:59.611 INFO 2981023 --- [Collect all tenants major compactions2,5318fbde85ff7914,ce9181ec82ed838a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:37:59.611 INFO 2981023 --- [Collect all tenants major compactions2,5318fbde85ff7914,ce9181ec82ed838a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:37:59.611 INFO 2981023 --- [ocp-schedules-2,5318fbde85ff7914,dba86d6c2fab0b5f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:38:00.548 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:00.549 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:00.569 INFO 2981023 --- [ocp-schedules-24,8ffc2247afb8c1b6,fe8f6d3858ce533d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:38:00.655 WARN 2981023 --- [scanner-13703925-1,8ffc2247afb8c1b6,0b7a1690e3905007] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:38:00.671 INFO 2981023 --- [alarm-detect-7,8ffc2247afb8c1b6,ce0314272e74b7a2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646878, data is empty. 2024-09-11 18:38:00.674 INFO 2981023 --- [alarm-detect-7,8ffc2247afb8c1b6,ce0314272e74b7a2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646878, data is empty. 2024-09-11 18:38:00.679 INFO 2981023 --- [alarm-detect-7,8ffc2247afb8c1b6,ce0314272e74b7a2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646878, data is empty. 2024-09-11 18:38:00.699 INFO 2981023 --- [alarm-detect-9,8ffc2247afb8c1b6,c5c73bc6a3fe4958] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646878, data is empty. 2024-09-11 18:38:00.701 INFO 2981023 --- [alarm-detect-6,8ffc2247afb8c1b6,26cdc0d4d0c40175] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646878, data is empty. 2024-09-11 18:38:00.705 INFO 2981023 --- [alarm-detect-9,8ffc2247afb8c1b6,c5c73bc6a3fe4958] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646878, data is empty. 2024-09-11 18:38:00.710 INFO 2981023 --- [alarm-detect-6,8ffc2247afb8c1b6,26cdc0d4d0c40175] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646878, data is empty. 2024-09-11 18:38:00.711 INFO 2981023 --- [alarm-detect-9,8ffc2247afb8c1b6,c5c73bc6a3fe4958] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646878, data is empty. 2024-09-11 18:38:00.716 INFO 2981023 --- [alarm-detect-6,8ffc2247afb8c1b6,26cdc0d4d0c40175] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646878, data is empty. 2024-09-11 18:38:00.813 INFO 2981023 --- [alarm-detect-9,8ffc2247afb8c1b6,147e5b99c6a8fdad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646878, data is empty. 2024-09-11 18:38:00.817 INFO 2981023 --- [alarm-detect-9,8ffc2247afb8c1b6,147e5b99c6a8fdad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646878, data is empty. 2024-09-11 18:38:00.823 INFO 2981023 --- [alarm-detect-9,8ffc2247afb8c1b6,147e5b99c6a8fdad] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646878, data is empty. 2024-09-11 18:38:01.506 INFO 2981023 --- [ocp-async-2,e539e2d8545b828d,02e6e83413d1a935] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,f58b52152be0404b,428709c3c0ab0af2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,f58b52152be0404b,428709c3c0ab0af2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:02.375 INFO 2981023 --- [ocp-schedules-21,ff107c171a503ae6,2baf2e023dce17bb] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:38:02.375 INFO 2981023 --- [ocp-schedules-21,ff107c171a503ae6,2baf2e023dce17bb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2024-09-11 18:38:02.466 INFO 2981023 --- [ocp-schedules-2,77e9a2bafe5caa38,26b2eb603c8678ca] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:38:02.466+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:38:02.469 INFO 2981023 --- [ocp-schedules-2,77e9a2bafe5caa38,26b2eb603c8678ca] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:38:02.612 INFO 2981023 --- [ocp-schedules-2,77e9a2bafe5caa38,26b2eb603c8678ca] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 46 to SqlAggregateData 2024-09-11 18:38:02.613 INFO 2981023 --- [ocp-schedules-2,77e9a2bafe5caa38,26b2eb603c8678ca] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:38:03.019 INFO 2981023 --- [ocp-schedules-16,b93f5e34d3461636,1c9e0302854ba7cf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:38:03.475 INFO 2981023 --- [ocp-schedules-10,1d5e81d92a080181,a569341e4cd0c5b1] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:38:03.476 INFO 2981023 --- [ocp-schedules-10,1d5e81d92a080181,a569341e4cd0c5b1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:38:03.495 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:03.496 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:38:03.500 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:03.503 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:38:03.504 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051083504000} 2024-09-11 18:38:03.506 INFO 2981023 --- [ocp-async-9,1f463f103a493969,7b170656b40427a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:03.509 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:38:03.514 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000243, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:38:03.515 INFO 2981023 --- [ocp-schedules-10,1d5e81d92a080181,a569341e4cd0c5b1] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:38:03.549 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:03.549 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:03.675 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:38:03.675 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:38:03.675 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:38:03.679 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000243, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:38:03.680 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:38:04.530 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000243 2024-09-11 18:38:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,f90183a444aeade1,836b7a75fd253af9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,f90183a444aeade1,836b7a75fd253af9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:05.506 INFO 2981023 --- [ocp-async-5,f782ed2baa104c1c,d92296b9fcca7822] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:38:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:38:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:38:05.992 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:38:05.998 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:38:06.002 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:38:06.003 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:38:06.003 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:38:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.005 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:38:06.006 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.006 INFO 2981023 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:38:06.050 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:35:41, start=2024-09-11T18:32:41, end=2024-09-11T18:35:11, updates=288, granule=120000ms, delay=180000ms 2024-09-11 18:38:06.055 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:35:41, target=2024-09-11T18:35:11, granule=120000ms, delay=30000ms) 2024-09-11 18:38:06.058 INFO 2981023 --- [ocp-schedules-7,14a75b5c9e663a56,9f05c1f10b812a71] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:38:06.059 INFO 2981023 --- [ocp-schedules-7,14a75b5c9e663a56,9f05c1f10b812a71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:38:06.108 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:38:06.108 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:38:06.108 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.109 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.112 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:38:06.113 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:38:06.113 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:38:06.116 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.117 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.118 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.119 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.129 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.129 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.131 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.131 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.136 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.136 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:38:06.137 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.146 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.146 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:38:06.146 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:38:06.149 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.149 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.152 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.152 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:38:06.157 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.157 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.158 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:38:06.160 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.168 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.169 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.169 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:38:06.170 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:38:06.170 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.171 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:38:06.171 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:38:06.172 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.173 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:38:06.173 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.174 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:38:06.174 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.174 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.175 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:38:06.175 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:38:06.175 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:38:06.177 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.179 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:38:06.180 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.180 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:38:06.181 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:38:06.181 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.182 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:38:06.182 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.183 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:38:06.183 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:38:06.185 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:38:06.190 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.191 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.198 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:06.205 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.206 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:38:06.207 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:06.208 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.208 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.209 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:38:06.209 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE75C1D6E1-0-0] 2024-09-11 18:38:06.209 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:38:06.209 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.209 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.210 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:38:06.210 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.210 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - source=2024-09-11T18:35:11, start=2024-09-11T18:20:51, end=2024-09-11T18:30:51, updates=380, granule=600000ms, delay=860000ms 2024-09-11 18:38:06.211 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:38:06.214 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:38:06.214 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.215 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:38:06.219 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:38:06.220 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:38:06.220 INFO 2981023 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=1) - not enough cached interval (source=2024-09-11T18:35:11, target=2024-09-11T18:30:51, granule=600000ms, delay=260000ms) 2024-09-11 18:38:06.220 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=250ms, period=120000ms 2024-09-11 18:38:06.221 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:38:06.221 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.221 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:38:06.221 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:38:06.221 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:38:06.221 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.226 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:38:06.227 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:38:06.227 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:38:06.227 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:38:06.237 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.237 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:38:06.258 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.282 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:38:06.290 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:38:06.290 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE75E1D6D0-0-0] 2024-09-11 18:38:06.291 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:38:06.295 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.296 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:38:06.301 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.301 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:38:06.302 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.302 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:38:06.303 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.303 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:38:06.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:38:06.304 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.305 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:38:06.305 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:38:06.307 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:38:06.316 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:06.318 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.318 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:38:06.327 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:38:06.329 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:38:06.332 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:06.332 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:38:06.341 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:38:06.343 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:38:06.343 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:38:06.343 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:38:06.343 INFO 2981023 --- [ocp-schedules-7,14a75b5c9e663a56,9f05c1f10b812a71] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:38:06.353 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:38:06.352902+08:00, lastProcessTime=2024-09-11T18:37:56.354309+08:00 2024-09-11 18:38:06.485 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:38:06.352902+08:00 2024-09-11 18:38:06.550 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:06.550 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:07.506 INFO 2981023 --- [ocp-async-6,5f0dfbe090c4df36,02ccc3fda6e185ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:07.804 INFO 2981023 --- [ocp-async-9,6afd6ab5e8338f50,6779d2768e8cb446] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:38:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,787990826594fe99,3af5712e83fb1eea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,787990826594fe99,3af5712e83fb1eea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:08.393 INFO 2981023 --- [ocp-schedules-14,6516288c795e49ff,db4ae92ceb9687dc] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:38:08.471 INFO 2981023 --- [ocp-schedules-25,04f64d74756e6bdd,5d39eb3d2cfa0124] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:38:08.471+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:38:08.472 INFO 2981023 --- [ocp-schedules-25,04f64d74756e6bdd,5d39eb3d2cfa0124] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:38:08.473 INFO 2981023 --- [ocp-schedules-25,04f64d74756e6bdd,5d39eb3d2cfa0124] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:38:08.473 INFO 2981023 --- [ocp-schedules-25,04f64d74756e6bdd,5d39eb3d2cfa0124] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:38:09.506 INFO 2981023 --- [ocp-async-4,97383f2c28dce28b,ba03957d6895189e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:09.550 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:09.551 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:10.003 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:38:10.003 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:38:10.866 INFO 2981023 --- [ocp-schedules-9,7d6bbbda44589447,e0df9e317d08d086] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:38:10.921 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:38:10.923 WARN 2981023 --- [scanner-13704243-1,7d6bbbda44589447,ec0e4a653545f5ab] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:38:10.925 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:38:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,20ca9a9c63df2c06,3815275a804749de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,20ca9a9c63df2c06,3815275a804749de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:11.506 INFO 2981023 --- [ocp-async-1,3af5a9aea82dcaea,e593b5ceb26b0a8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:11.748 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726051020, nodeCount=480, affectRows=480 2024-09-11 18:38:12.551 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:12.552 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:13.004 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726051020, nodeCount=1000, affectRows=1000 2024-09-11 18:38:13.007 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726051020, nodeCount=1000, validCount=1000 2024-09-11 18:38:13.020 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726051020, nodeCount=1000, affectRows=1000 2024-09-11 18:38:13.021 INFO 2981023 --- [ocp-schedules-19,580dc461ff1a50cd,dbdd9bd1f2b5bb06] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:38:13.022 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726051020, nodeCount=49, affectRows=49 2024-09-11 18:38:13.022 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726051020, nodeCount=47, validCount=47 2024-09-11 18:38:13.023 INFO 2981023 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726051020, nodeCount=999, validCount=999 2024-09-11 18:38:13.506 INFO 2981023 --- [ocp-async-7,7a0e42784edd28ed,fcc2b5cb90e9c172] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:13.718 INFO 2981023 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726051020, nodeCount=480, validCount=480 2024-09-11 18:38:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,6777ba9f541f2f37,f6e29a60ab12e375] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,581c88938445088b,8aa1a8450ed86408] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:38:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,6777ba9f541f2f37,f6e29a60ab12e375] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:15.507 INFO 2981023 --- [ocp-async-10,8e40bc0e2fd7b1a7,eacffbd1ff299ce8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:15.552 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:15.552 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:16.016 INFO 2981023 --- [ocp-schedules-17,94a8dfdce60fa84a,8d47ce7a1fbc147b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:38:16.020 INFO 2981023 --- [host-check-clockdiff1,94a8dfdce60fa84a,6d4e06e902fd47ba] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:38:16.020 INFO 2981023 --- [host-check-clockdiff3,94a8dfdce60fa84a,4e26d97b07dc6094] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:38:16.022 INFO 2981023 --- [host-check-clockdiff2,94a8dfdce60fa84a,e69c5ce26c5c4f4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:38:16.022 INFO 2981023 --- [host-check-clockdiff4,94a8dfdce60fa84a,11c075ca4748d67e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:38:16.023 INFO 2981023 --- [ocp-schedules-14,57c341e0e846c0c7,f46704b06944896c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:38:16.035 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:38:16.037 INFO 2981023 --- [ocp-schedules-12,af659feaf38dada9,d4d478933ec06259] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:38:16.037 INFO 2981023 --- [host-check-agent-status2,af659feaf38dada9,fd4a1d3778bfc4f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:38:16.037 INFO 2981023 --- [host-check-agent-status1,af659feaf38dada9,b2d680e2196ba456] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:38:16.038 INFO 2981023 --- [host-check-agent-status3,af659feaf38dada9,69fdca3553b6f318] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:38:16.038 INFO 2981023 --- [host-check-agent-status4,af659feaf38dada9,b27988c419eea4aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:38:16.038 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:38:16.038 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:38:16.045 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:38:16.067 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:38:16.077 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:38:16.079 INFO 2981023 --- [host-check-clockdiff2,94a8dfdce60fa84a,e69c5ce26c5c4f4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051096 5 5 , error= 2024-09-11 18:38:16.079 INFO 2981023 --- [host-check-agent-status1,af659feaf38dada9,b2d680e2196ba456] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:38:16.079 INFO 2981023 --- [host-check-agent-status3,af659feaf38dada9,69fdca3553b6f318] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:38:16.079 INFO 2981023 --- [host-check-agent-status2,af659feaf38dada9,fd4a1d3778bfc4f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:38:16.081 INFO 2981023 --- [host-check-agent-status3,af659feaf38dada9,69fdca3553b6f318] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:38:16.081 INFO 2981023 --- [host-check-clockdiff1,94a8dfdce60fa84a,6d4e06e902fd47ba] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051096 0 0 , error= 2024-09-11 18:38:16.084 INFO 2981023 --- [host-check-agent-status4,af659feaf38dada9,b27988c419eea4aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:38:16.084 INFO 2981023 --- [host-check-agent-status4,af659feaf38dada9,b27988c419eea4aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:38:16.092 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:38:16.093 INFO 2981023 --- [host-check-agent-status2,af659feaf38dada9,fd4a1d3778bfc4f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:38:16.093 INFO 2981023 --- [host-check-agent-status1,af659feaf38dada9,b2d680e2196ba456] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:38:16.097 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:38:16.111 INFO 2981023 --- [host-check-clockdiff3,94a8dfdce60fa84a,4e26d97b07dc6094] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051096 0 0 , error= 2024-09-11 18:38:16.114 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:38:16.172 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:38:16.192 INFO 2981023 --- [ocp-schedules-11,b4a3c5ec6719aa65,c2728ec081a40d9a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:38:16.197 INFO 2981023 --- [ocp-schedules-19,e4e68c92aa423cab,e0487b5295e73c4e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:38:16.197 INFO 2981023 --- [ocp-schedules-1,cdb2eb1660cc44ac,c7fb3efb6abd7369] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:38:16.197 INFO 2981023 --- [ocp-schedules-22,46e462a5189ee48a,68797cce3afd1ac4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:38:16.205 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:38:16.258 INFO 2981023 --- [ocp-schedules-11,b4a3c5ec6719aa65,c2728ec081a40d9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:38:16.264 INFO 2981023 --- [ocp-schedules-1,cdb2eb1660cc44ac,c7fb3efb6abd7369] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:38:16.264 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051083504000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:38:16.271 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:38:16.271 INFO 2981023 --- [ocp-schedules-19,e4e68c92aa423cab,e0487b5295e73c4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:38:16.318 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:38:16.331 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:38:16.330714057+08:00 2024-09-11 18:38:16.337 INFO 2981023 --- [ocp-schedules-19,e4e68c92aa423cab,e0487b5295e73c4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:38:16.354 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:38:16.331+08:00 2024-09-11 18:38:16.355 INFO 2981023 --- [host-check-agent-status4,af659feaf38dada9,b27988c419eea4aa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:38:16.357 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:38:16.331+08:00 and status AVAILABLE 2024-09-11 18:38:16.359 INFO 2981023 --- [host-check-agent-status4,af659feaf38dada9,b27988c419eea4aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:38:16.359 INFO 2981023 --- [host-check-agent-status4,af659feaf38dada9,b27988c419eea4aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:38:16.359 INFO 2981023 --- [host-check-agent-status4,af659feaf38dada9,b27988c419eea4aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:38:16.360 INFO 2981023 --- [host-check-agent-status2,af659feaf38dada9,fd4a1d3778bfc4f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:38:16.361 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:38:16.363 INFO 2981023 --- [ocp-schedules-27,188471b781e45061,d93846a94d701e7c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:38:16.364 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:38:16.368799632+08:00 2024-09-11 18:38:16.364 INFO 2981023 --- [host-check-agent-status1,af659feaf38dada9,b2d680e2196ba456] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:38:16.364 INFO 2981023 --- [host-check-agent-status2,af659feaf38dada9,fd4a1d3778bfc4f0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:38:16.364 INFO 2981023 --- [host-check-agent-status2,af659feaf38dada9,fd4a1d3778bfc4f0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:38:16.364 INFO 2981023 --- [host-check-agent-status2,af659feaf38dada9,fd4a1d3778bfc4f0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:38:16.367 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:38:16.364+08:00 2024-09-11 18:38:16.368 INFO 2981023 --- [host-check-agent-status1,af659feaf38dada9,b2d680e2196ba456] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:38:16.368 INFO 2981023 --- [host-check-agent-status1,af659feaf38dada9,b2d680e2196ba456] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:38:16.368 INFO 2981023 --- [host-check-agent-status1,af659feaf38dada9,b2d680e2196ba456] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:38:16.370 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:38:16.364+08:00 and status AVAILABLE 2024-09-11 18:38:16.372 INFO 2981023 --- [host-check-agent-status3,af659feaf38dada9,69fdca3553b6f318] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:38:16.372 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:38:16.377 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.377 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:38:16.378 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:38:16.381 INFO 2981023 --- [host-check-agent-status3,af659feaf38dada9,69fdca3553b6f318] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:38:16.381 INFO 2981023 --- [host-check-agent-status3,af659feaf38dada9,69fdca3553b6f318] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:38:16.381 INFO 2981023 --- [host-check-agent-status3,af659feaf38dada9,69fdca3553b6f318] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:38:16.385 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:38:16.385 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:38:16.386 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:38:16.386212995+08:00 2024-09-11 18:38:16.387 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:38:16.386+08:00 2024-09-11 18:38:16.387 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.388 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:38:16.389 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:38:16.386+08:00 and status AVAILABLE 2024-09-11 18:38:16.389 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:38:16.390 INFO 2981023 --- [ocp-schedules-11,b4a3c5ec6719aa65,c2728ec081a40d9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:38:16.391 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.391 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:16.394 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:38:16.395 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:38:16.394926858+08:00 2024-09-11 18:38:16.397 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:38:16.393308+08:00, lastProcessTime=2024-09-11T18:38:06.352902+08:00 2024-09-11 18:38:16.397 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:38:16.395+08:00 2024-09-11 18:38:16.400 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:38:16.395+08:00 and status AVAILABLE 2024-09-11 18:38:16.400 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:38:16.401 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:38:16.404 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:38:16.454 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:38:16.460 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:38:16.462 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:38:16.463 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.463 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:16.468 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.469 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:16.469 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:38:16.470 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:38:16.470 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:38:16.472 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:38:16.479 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:38:16.485 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:38:16.498 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:16.498 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.499 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:38:16.500 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:38:16.502 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:38:16.502 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:38:16.511 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:38:16.527 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:38:16.530 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:38:16.530 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051083504000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:38:16.532 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:38:16.535 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:38:16.553 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:38:16.562 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:38:16.564 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:16.565 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.565 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:38:16.565 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:16.566 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:38:16.567 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.569 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:38:16.570 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:38:16.581 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:38:16.582 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:38:16.588 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:38:16.590 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:38:16.591 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:38:16.591 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:38:16.603 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:38:16.611 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:38:16.612 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:38:16.617 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:38:16.620 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:38:16.624 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:38:16.624 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:38:16.628 INFO 2981023 --- [ocp-schedules-20,0a0a97a674c6a2bf,04133cfb5742163c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:38:16.629 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:38:16.635 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:16.635 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.636 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:38:16.636 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:38:16.638 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:38:16.640 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:38:16.654 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:38:16.658 INFO 2981023 --- [ocp-schedules-20,0a0a97a674c6a2bf,04133cfb5742163c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:38:16.658 INFO 2981023 --- [ocp-schedules-20,0a0a97a674c6a2bf,04133cfb5742163c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:38:16.658 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:38:16.659 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:38:16.659 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:38:16.662 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:38:16.664 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:38:16.670 INFO 2981023 --- [host-check-host-status4,57c341e0e846c0c7,7e818aba0a450c1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:38:16.670 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:38:16.671 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:16.672 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.672 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:38:16.689 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:38:16.690 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:38:16.693 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:38:16.698 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:16.698 INFO 2981023 --- [host-check-host-status1,57c341e0e846c0c7,6d30a80a269ff89d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:38:16.699 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.700 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:38:16.700 INFO 2981023 --- [host-check-host-status3,57c341e0e846c0c7,d3e3c8d5ed16f2d1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:38:16.703 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:16.705 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:38:16.706 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.707 INFO 2981023 --- [host-check-host-status2,57c341e0e846c0c7,08538d4c0a12b009] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:38:16.708 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:38:16.709 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:38:16.729 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:38:16.729 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:38:16.746 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:38:16.757 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:16.758 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.758 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:38:16.759 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:38:16.763 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:38:16.766 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:38:16.766 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:38:16.767 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:38:16.771 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:38:16.393308+08:00 2024-09-11 18:38:16.772 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:16.773 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.773 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:38:16.779 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:38:16.779 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:38:16.780 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:38:16.809 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:38:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:16.815 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:16.816 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:38:16.821 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:38:16.821 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:38:16.822 INFO 2981023 --- [ocp-schedules-1,cdb2eb1660cc44ac,c7fb3efb6abd7369] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:38:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,9f1bea6c2f16df17,9b96106af6742a2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,9f1bea6c2f16df17,9b96106af6742a2d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:17.507 INFO 2981023 --- [ocp-async-8,bfc18224eb780402,53ec11cdab14aaf6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:17.521 INFO 2981023 --- [ocp-schedules-21,3baed2b2d151ab7e,a354c7c41f2aca1a] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:38:17.804 INFO 2981023 --- [ocp-async-7,d624f699605c57ff,a96b1fbb18f70f76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:38:18.552 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:18.553 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:19.507 INFO 2981023 --- [ocp-async-6,93a2e5cbf6745bf8,83d6eb11f5f574c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,c4ed8bd147d82d04,1ae8f26388d04b00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,c4ed8bd147d82d04,1ae8f26388d04b00] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:21.111 INFO 2981023 --- [ocp-schedules-6,5b1ee6a3df865718,9d663696cc972217] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:38:21.184 WARN 2981023 --- [scanner-13705942-1,5b1ee6a3df865718,cf28a035d8be4185] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:38:21.203 INFO 2981023 --- [alarm-detect-7,5b1ee6a3df865718,954751da45b9e646] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646878, data is empty. 2024-09-11 18:38:21.206 INFO 2981023 --- [alarm-detect-7,5b1ee6a3df865718,954751da45b9e646] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646924, data is empty. 2024-09-11 18:38:21.210 INFO 2981023 --- [alarm-detect-7,5b1ee6a3df865718,954751da45b9e646] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646924, data is empty. 2024-09-11 18:38:21.229 INFO 2981023 --- [alarm-detect-20,5b1ee6a3df865718,4398c786275a8508] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646878, data is empty. 2024-09-11 18:38:21.230 INFO 2981023 --- [alarm-detect-2,5b1ee6a3df865718,e860662005e1ed66] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646878, data is empty. 2024-09-11 18:38:21.237 INFO 2981023 --- [alarm-detect-2,5b1ee6a3df865718,e860662005e1ed66] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646924, data is empty. 2024-09-11 18:38:21.237 INFO 2981023 --- [alarm-detect-20,5b1ee6a3df865718,4398c786275a8508] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646924, data is empty. 2024-09-11 18:38:21.244 INFO 2981023 --- [alarm-detect-20,5b1ee6a3df865718,4398c786275a8508] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646924, data is empty. 2024-09-11 18:38:21.245 INFO 2981023 --- [alarm-detect-2,5b1ee6a3df865718,e860662005e1ed66] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646924, data is empty. 2024-09-11 18:38:21.340 INFO 2981023 --- [alarm-detect-14,5b1ee6a3df865718,6b535480b8f85598] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646878, data is empty. 2024-09-11 18:38:21.343 INFO 2981023 --- [alarm-detect-14,5b1ee6a3df865718,6b535480b8f85598] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646924, data is empty. 2024-09-11 18:38:21.348 INFO 2981023 --- [alarm-detect-14,5b1ee6a3df865718,6b535480b8f85598] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646924, data is empty. 2024-09-11 18:38:21.507 INFO 2981023 --- [ocp-async-9,056a21d25b7ad649,74dac571a80deece] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:21.553 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:21.553 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:21.930 INFO 2981023 --- [host-check-clockdiff4,94a8dfdce60fa84a,11c075ca4748d67e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051101 0 0 , error= 2024-09-11 18:38:23.024 INFO 2981023 --- [ocp-schedules-28,23350fb162b81145,98466aa551d958fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:38:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,ffd4ff48e0e6b83b,cd7457d804a906b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,ffd4ff48e0e6b83b,cd7457d804a906b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:23.507 INFO 2981023 --- [ocp-async-5,5fc1029511779b9b,5f996c038de60d7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:24.554 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:24.554 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:24.567 INFO 2981023 --- [ocp-schedules-18,c1e37111a757e18e,3bacc9292ebc298a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:38:24.865 INFO 2981023 --- [ocp-schedules-11,ec59722d24dac86d,3cf6bd0aa5722efb] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-09-11 18:38:24.865 INFO 2981023 --- [ocp-schedules-11,ec59722d24dac86d,3cf6bd0aa5722efb] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-09-11 18:38:24.880 INFO 2981023 --- [pool-621-thread-1,ec59722d24dac86d,0451171ec64f9a57] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:38:24.881 INFO 2981023 --- [pool-621-thread-2,ec59722d24dac86d,115ae27b52954616] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:38:24.881 INFO 2981023 --- [pool-621-thread-3,ec59722d24dac86d,80569e3afa3d1525] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-09-11 18:38:24.902 INFO 2981023 --- [pool-621-thread-1,ec59722d24dac86d,0451171ec64f9a57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:24.903 INFO 2981023 --- [pool-621-thread-3,ec59722d24dac86d,80569e3afa3d1525] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:24.903 INFO 2981023 --- [pool-621-thread-2,ec59722d24dac86d,115ae27b52954616] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:24.905 INFO 2981023 --- [pool-621-thread-1,ec59722d24dac86d,0451171ec64f9a57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726050804000000, 1726051104000000] 2024-09-11 18:38:24.906 INFO 2981023 --- [pool-621-thread-2,ec59722d24dac86d,115ae27b52954616] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726050804000000, 1726051104000000] 2024-09-11 18:38:24.906 INFO 2981023 --- [pool-621-thread-3,ec59722d24dac86d,80569e3afa3d1525] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1726050804000000, 1726051104000000] 2024-09-11 18:38:24.910 INFO 2981023 --- [pool-621-thread-3,ec59722d24dac86d,80569e3afa3d1525] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:38:24.980 INFO 2981023 --- [pool-621-thread-1,ec59722d24dac86d,0451171ec64f9a57] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:38:25.053 INFO 2981023 --- [pool-621-thread-2,ec59722d24dac86d,115ae27b52954616] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-09-11 18:38:25.054 INFO 2981023 --- [ocp-schedules-11,ec59722d24dac86d,3cf6bd0aa5722efb] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-09-11 18:38:25.507 INFO 2981023 --- [ocp-async-3,13c4a5531ae7c39a,411331aa49301a89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,a9e6dfd93935c254,ed3c5b4dbc0485f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,a9e6dfd93935c254,ed3c5b4dbc0485f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:26.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:38:26.338025+08:00, lastProcessTime=2024-09-11T18:38:16.393308+08:00 2024-09-11 18:38:26.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:38:26.338025+08:00 2024-09-11 18:38:27.253 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9684fd04f7ea5584,83625d0e633f53e1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=9684fd04f7ea5584, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 18:38:27.264 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,9684fd04f7ea5584,83625d0e633f53e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.31.33.41, traceId=9684fd04f7ea5584, duration=14 ms] 2024-09-11 18:38:27.337 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,61fdbaca4f2921e9,64251d8f351d708b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.31.33.41, traceId=61fdbaca4f2921e9, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2024-09-11 18:38:27.356 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,50b6dfe7b4df4501,e2a22f99630aead7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.31.33.41, traceId=50b6dfe7b4df4501, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2024-09-11 18:38:27.356 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,86cfc54d9066cfe7,316e0a797bb7a863] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.31.33.41, traceId=86cfc54d9066cfe7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2024-09-11 18:38:27.361 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6099a1a8a5611fb7,4fe78a77b9c0ebd0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.31.33.41, traceId=6099a1a8a5611fb7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2024-09-11 18:38:27.369 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6099a1a8a5611fb7,4fe78a77b9c0ebd0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=6099a1a8a5611fb7, duration=11 ms] 2024-09-11 18:38:27.369 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,50b6dfe7b4df4501,e2a22f99630aead7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=50b6dfe7b4df4501, duration=16 ms] 2024-09-11 18:38:27.369 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,86cfc54d9066cfe7,316e0a797bb7a863] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.31.33.41, traceId=86cfc54d9066cfe7, duration=15 ms] 2024-09-11 18:38:27.374 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,61fdbaca4f2921e9,64251d8f351d708b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.31.33.41, traceId=61fdbaca4f2921e9, duration=39 ms] 2024-09-11 18:38:27.381 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,01eb1a9b4057297a,4fc8a606781dee4e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_ODP&type=TOP&maxPoints=360, client=10.31.33.41, traceId=01eb1a9b4057297a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_ODP,] 2024-09-11 18:38:27.388 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7248813881aebbbf,c013730101d835fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_TENANT&type=TOP&maxPoints=360, client=10.31.33.41, traceId=7248813881aebbbf, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_TENANT,] 2024-09-11 18:38:27.393 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,01eb1a9b4057297a,4fc8a606781dee4e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=01eb1a9b4057297a, duration=15 ms] 2024-09-11 18:38:27.394 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,7248813881aebbbf,c013730101d835fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=7248813881aebbbf, duration=9 ms] 2024-09-11 18:38:27.485 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,451365650bb3eb76,d52adb52d67ba69b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_memory_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=451365650bb3eb76, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_memory_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 18:38:27.488 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,451365650bb3eb76,d52adb52d67ba69b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=451365650bb3eb76, duration=6 ms] 2024-09-11 18:38:27.490 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6ef2b0b4f59da220,18faaa5749e9f996] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_cpu_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=6ef2b0b4f59da220, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_cpu_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 18:38:27.490 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9900b35062c4f16d,68cc1d40dfd7cf90] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_HOST&type=TOP&maxPoints=360, client=10.31.33.41, traceId=9900b35062c4f16d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_HOST,] 2024-09-11 18:38:27.493 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,6ef2b0b4f59da220,18faaa5749e9f996] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=6ef2b0b4f59da220, duration=5 ms] 2024-09-11 18:38:27.494 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b5eace453c76b088,f37a6197db5db64f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_disk_used_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.31.33.41, traceId=b5eace453c76b088, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_disk_used_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2024-09-11 18:38:27.495 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9900b35062c4f16d,68cc1d40dfd7cf90] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.31.33.41, traceId=9900b35062c4f16d, duration=8 ms] 2024-09-11 18:38:27.497 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,b5eace453c76b088,f37a6197db5db64f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.31.33.41, traceId=b5eace453c76b088, duration=5 ms] 2024-09-11 18:38:27.507 INFO 2981023 --- [ocp-async-2,2073aa35dfa21aca,22bb9a59b01f51ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:27.508 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,e055ab2e1342729b,c97159f82e015155] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_memory_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=e055ab2e1342729b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_memory_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 18:38:27.514 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,e055ab2e1342729b,c97159f82e015155] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=e055ab2e1342729b, duration=8 ms] 2024-09-11 18:38:27.530 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b578c6097a61cc1b,8d66358b0e885d7e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_cpu_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=b578c6097a61cc1b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_cpu_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 18:38:27.533 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,b578c6097a61cc1b,8d66358b0e885d7e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=b578c6097a61cc1b, duration=6 ms] 2024-09-11 18:38:27.554 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:27.554 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:27.598 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0c324353369e84b8,59935a8fe6febe42] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_tenant_disk_usage_v4&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.31.33.41, traceId=0c324353369e84b8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_tenant_disk_usage_v4,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2024-09-11 18:38:27.601 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,0c324353369e84b8,59935a8fe6febe42] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.31.33.41, traceId=0c324353369e84b8, duration=5 ms] 2024-09-11 18:38:27.610 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,3290d65ed019e393,e308ead6710286a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?foldStandby=false, client=10.31.33.41, traceId=3290d65ed019e393, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 18:38:27.615 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e1d9110f4d60d625,7b3da77dd9f9ce3b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.31.33.41, traceId=e1d9110f4d60d625, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2024-09-11 18:38:27.617 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6d670ea9abb76b7d,61c8ef76c99d3e0a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.31.33.41, traceId=6d670ea9abb76b7d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2024-09-11 18:38:27.617 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,6d670ea9abb76b7d,61c8ef76c99d3e0a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.31.33.41, traceId=6d670ea9abb76b7d, duration=3 ms] 2024-09-11 18:38:27.619 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,e1d9110f4d60d625,7b3da77dd9f9ce3b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.31.33.41, traceId=e1d9110f4d60d625, duration=6 ms] 2024-09-11 18:38:27.620 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,3290d65ed019e393,e308ead6710286a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=3290d65ed019e393, duration=12 ms] 2024-09-11 18:38:27.629 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f1b5a9e29433d003,4d555b42ccd498ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.31.33.41, traceId=f1b5a9e29433d003, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2024-09-11 18:38:27.629 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,f1b5a9e29433d003,4d555b42ccd498ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.31.33.41, traceId=f1b5a9e29433d003, duration=1 ms] 2024-09-11 18:38:27.667 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,9ff895f8869e4c80,0940ed241c6ca20f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.31.33.41, traceId=9ff895f8869e4c80, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2024-09-11 18:38:27.667 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-2,9ff895f8869e4c80,0940ed241c6ca20f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.31.33.41, traceId=9ff895f8869e4c80, duration=1 ms] 2024-09-11 18:38:27.735 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,c1a5eefa6b07302c,1590e3c9c8efef72] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.31.33.41, traceId=c1a5eefa6b07302c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2024-09-11 18:38:27.742 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,c1a5eefa6b07302c,1590e3c9c8efef72] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.31.33.41, traceId=c1a5eefa6b07302c, duration=10 ms] 2024-09-11 18:38:27.744 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,1ab1323d17e77970,1cf10513b01a44f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.31.33.41, traceId=1ab1323d17e77970, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2024-09-11 18:38:27.745 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-7,1ab1323d17e77970,1cf10513b01a44f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.31.33.41, traceId=1ab1323d17e77970, duration=2 ms] 2024-09-11 18:38:27.746 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4d7ff3d62990e869,8dcc79fbc0cde2e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.31.33.41, traceId=4d7ff3d62990e869, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2024-09-11 18:38:27.805 INFO 2981023 --- [ocp-async-5,2e1d7ad4109ab7b5,1e37b07acf904ff4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:38:27.838 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:38:27.838 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, client=10.31.33.41, traceId=] 2024-09-11 18:38:27.838 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:38:27.841 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.31.33.41, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2024-09-11 18:38:27.841 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, client=10.31.33.41, traceId=] 2024-09-11 18:38:27.841 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.31.33.41, traceId=, duration=0 ms] 2024-09-11 18:38:27.913 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,4d7ff3d62990e869,8dcc79fbc0cde2e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.31.33.41, traceId=4d7ff3d62990e869, duration=171 ms] 2024-09-11 18:38:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,aa15c548ac1fe324,3faf7400bb407380] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,aa15c548ac1fe324,3faf7400bb407380] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:29.451 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.453 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdo 2024-09-11 18:38:29.463 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.463 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:38:29.470 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.478 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.491 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.495 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.495 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:38:29.497 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.500 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.503 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.504 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.507 INFO 2981023 --- [ocp-async-8,33aaacd35cf5c7fd,cecbaf2a3aec96e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:29.511 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.512 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.513 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.514 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.515 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.516 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.517 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.517 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.518 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.523 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.540 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 24 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.586 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 10 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.600 INFO 2981023 --- [ocp-schedules-2,58d951579b26473c,97432b0a8f4eb19f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:38:29.600 INFO 2981023 --- [ocp-schedules-2,58d951579b26473c,97432b0a8f4eb19f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:38:29.601 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:38:29.601 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:38:29.604 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:38:29.605 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:38:29.605 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:38:29.605 INFO 2981023 --- [ocp-schedules-2,58d951579b26473c,97432b0a8f4eb19f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:38:29.606 INFO 2981023 --- [ocp-schedules-2,58d951579b26473c,97432b0a8f4eb19f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:38:29.606 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 164 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.607 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:38:29.607 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:38:29.610 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:38:29.611 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:38:29.630 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:38:29.630 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 1 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.631 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:38:29.634 INFO 2981023 --- [ocp-schedules-2,58d951579b26473c,97432b0a8f4eb19f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:38:29.634 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.634 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2024-09-11 18:38:29.635 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.636 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.637 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:38:29.637 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:32:29.188, end=2024-09-11T18:37:29.447} 2024-09-11 18:38:29.637 INFO 2981023 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 5 dubious sqls 2024-09-11 18:38:29.639 INFO 2981023 --- [ocp-schedules-2,58d951579b26473c,97432b0a8f4eb19f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:38:29.834 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,76a8b1372e52c0a3,6a7b02a727532e90] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.31.33.41, traceId=76a8b1372e52c0a3, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2024-09-11 18:38:29.840 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-6,76a8b1372e52c0a3,6a7b02a727532e90] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.31.33.41, traceId=76a8b1372e52c0a3, duration=10 ms] 2024-09-11 18:38:30.554 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:30.556 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:31.388 INFO 2981023 --- [ocp-schedules-3,1d0080c05415c171,030b092500f12fef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:38:31.434 WARN 2981023 --- [scanner-13706268-1,1d0080c05415c171,f1315fea7b5ce9d7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:38:31.507 INFO 2981023 --- [ocp-async-7,4b3891184096590f,519e0ccf5aab1f30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:31.770 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e14ce2be26155db4,a655b65363d1b199] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.31.33.41, traceId=e14ce2be26155db4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2024-09-11 18:38:31.778 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-3,e14ce2be26155db4,a655b65363d1b199] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.31.33.41, traceId=e14ce2be26155db4, duration=11 ms] 2024-09-11 18:38:31.779 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5264d6f226127dcc,44db47c57926cddf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/obproxy/clusters?null, client=10.31.33.41, traceId=5264d6f226127dcc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObproxyController.listObproxyClusters(Pageable,List,String,List,Boolean), args=Page request [number: 0, size 5000, sort: id: ASC],false,] 2024-09-11 18:38:31.779 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,12efd2394a461019,d9c3788ac65f3f2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?excludedServiceType=BINLOG_SERVICE, client=10.31.33.41, traceId=12efd2394a461019, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2024-09-11 18:38:31.782 WARN 2981023 --- [http-nio-0.0.0.0-8080-exec-5,12efd2394a461019,d9c3788ac65f3f2e] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2024-09-11 18:38:31.789 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-1,5264d6f226127dcc,44db47c57926cddf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/obproxy/clusters client=10.31.33.41, traceId=5264d6f226127dcc, duration=13 ms] 2024-09-11 18:38:31.816 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-5,12efd2394a461019,d9c3788ac65f3f2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.31.33.41, traceId=12efd2394a461019, duration=39 ms] 2024-09-11 18:38:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,ee8e5151ad5594f8,65a8e752f4369841] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:32.317 INFO 2981023 --- [prometheus-minute-schedule-1,ee8e5151ad5594f8,65a8e752f4369841] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:33.027 INFO 2981023 --- [ocp-schedules-10,452d5e50501d7dc9,5bd33d12147b93fa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:38:33.508 INFO 2981023 --- [ocp-async-10,2351f71d766f8724,857eaadffe5c2267] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:33.556 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:33.557 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,1c65533fdab64419,70bebaa90cbbaf8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,1c65533fdab64419,70bebaa90cbbaf8e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:35.508 INFO 2981023 --- [ocp-async-4,0a4666b9d1a78b50,e5354c86cb9ab429] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:36.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:38:36.352186+08:00, lastProcessTime=2024-09-11T18:38:26.338025+08:00 2024-09-11 18:38:36.480 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:38:36.352186+08:00 2024-09-11 18:38:36.557 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:36.557 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:36.661 INFO 2981023 --- [ocp-schedules-6,3be2245ff4bacb93,38347938b497887f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:38:36.669 INFO 2981023 --- [ocp-schedules-6,3be2245ff4bacb93,38347938b497887f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:38:36.669 INFO 2981023 --- [ocp-schedules-6,3be2245ff4bacb93,38347938b497887f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:38:37.314 INFO 2981023 --- [prometheus-cache-monitor1,9c828b6e4625056e,c677f47d9f87ef05] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:38:37.315 INFO 2981023 --- [prometheus-cache-monitor1,9c828b6e4625056e,c677f47d9f87ef05] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:38:37.508 INFO 2981023 --- [ocp-async-1,c4a95224ecab0ada,bba708eba094321a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:37.805 INFO 2981023 --- [ocp-async-2,d446ec1861ed1d78,2a48752afa6c70dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:38:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,c2b1f4e1ba44d56c,a8af8dfed9b294af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,c2b1f4e1ba44d56c,a8af8dfed9b294af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:39.508 INFO 2981023 --- [ocp-async-6,9d71e72dfa88e050,d057e446ea6639c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:39.557 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:39.558 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:40.003 INFO 2981023 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:38:40.003 INFO 2981023 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:38:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,a0650d32fc59a289,e37e1584a99cbaf8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,a0650d32fc59a289,e37e1584a99cbaf8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:41.508 INFO 2981023 --- [ocp-async-9,a156f7b460bbc789,97776c297734b0f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:41.595 INFO 2981023 --- [ocp-schedules-21,0d31ee7f9d000392,0345f4e6e2096fcd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:38:41.612 WARN 2981023 --- [scanner-13706467-1,0d31ee7f9d000392,116c7ebc96f8bfb8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:38:41.660 INFO 2981023 --- [alarm-detect-20,0d31ee7f9d000392,75c59067c9dc1651] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646924, data is empty. 2024-09-11 18:38:41.666 INFO 2981023 --- [alarm-detect-20,0d31ee7f9d000392,75c59067c9dc1651] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646924, data is empty. 2024-09-11 18:38:41.669 INFO 2981023 --- [alarm-detect-20,0d31ee7f9d000392,75c59067c9dc1651] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646924, data is empty. 2024-09-11 18:38:41.693 INFO 2981023 --- [alarm-detect-10,0d31ee7f9d000392,ee62ce50a5ea80ea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646924, data is empty. 2024-09-11 18:38:41.693 INFO 2981023 --- [alarm-detect-2,0d31ee7f9d000392,f91690e218ec313f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646924, data is empty. 2024-09-11 18:38:41.696 INFO 2981023 --- [alarm-detect-10,0d31ee7f9d000392,ee62ce50a5ea80ea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646924, data is empty. 2024-09-11 18:38:41.697 INFO 2981023 --- [alarm-detect-2,0d31ee7f9d000392,f91690e218ec313f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646924, data is empty. 2024-09-11 18:38:41.700 INFO 2981023 --- [alarm-detect-10,0d31ee7f9d000392,ee62ce50a5ea80ea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646924, data is empty. 2024-09-11 18:38:41.701 INFO 2981023 --- [alarm-detect-2,0d31ee7f9d000392,f91690e218ec313f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646924, data is empty. 2024-09-11 18:38:41.802 INFO 2981023 --- [alarm-detect-19,0d31ee7f9d000392,24fa6d4bf2e666b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646924, data is empty. 2024-09-11 18:38:41.805 INFO 2981023 --- [alarm-detect-19,0d31ee7f9d000392,24fa6d4bf2e666b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646924, data is empty. 2024-09-11 18:38:41.808 INFO 2981023 --- [alarm-detect-19,0d31ee7f9d000392,24fa6d4bf2e666b8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646924, data is empty. 2024-09-11 18:38:42.558 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:42.559 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:43.030 INFO 2981023 --- [ocp-schedules-22,69bbc2fa11345c95,08e79d9a566dab50] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:38:43.508 INFO 2981023 --- [ocp-async-8,79317c80a8bf3eb1,906e3c66da889104] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,155396faaa56bf20,e9345bf9b8babfc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,155396faaa56bf20,e9345bf9b8babfc5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:45.101 INFO 2981023 --- [prometheus-second-schedule-1,d44298fbf3299a99,4d538bba74cc5fe1] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:38:45.508 INFO 2981023 --- [ocp-async-7,3461b1d4f7634e29,b1f895c1d68e3811] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:45.559 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:45.560 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:46.015 INFO 2981023 --- [ocp-schedules-16,26d771071d640cfd,678be8b631ee82fc] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:38:46.015 INFO 2981023 --- [host-check-clockdiff1,26d771071d640cfd,b3f12e81e840f196] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:38:46.016 INFO 2981023 --- [host-check-clockdiff2,26d771071d640cfd,fa1fedc5785716e3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:38:46.019 INFO 2981023 --- [host-check-clockdiff3,26d771071d640cfd,d3ae641b0f7aace3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:38:46.023 INFO 2981023 --- [host-check-clockdiff4,26d771071d640cfd,eac1a79b51ccaa7a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:38:46.025 INFO 2981023 --- [ocp-schedules-21,5f480b67e0a3650b,6a0a7a17d7ca0ad5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:38:46.027 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:38:46.029 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:38:46.039 INFO 2981023 --- [ocp-schedules-20,63b7985547bea688,47876c8b0d3322ea] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:38:46.040 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:38:46.041 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:38:46.042 INFO 2981023 --- [host-check-agent-status1,63b7985547bea688,84f9c36653606b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:38:46.049 INFO 2981023 --- [host-check-agent-status2,63b7985547bea688,385e06953a24cfc2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:38:46.050 INFO 2981023 --- [host-check-agent-status3,63b7985547bea688,49fecd192432c92b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:38:46.050 INFO 2981023 --- [host-check-agent-status4,63b7985547bea688,a66e0016330d0a36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:38:46.052 INFO 2981023 --- [host-check-clockdiff2,26d771071d640cfd,fa1fedc5785716e3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051126 5 5 , error= 2024-09-11 18:38:46.052 INFO 2981023 --- [host-check-clockdiff3,26d771071d640cfd,d3ae641b0f7aace3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051126 0 0 , error= 2024-09-11 18:38:46.061 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:38:46.062 INFO 2981023 --- [host-check-agent-status2,63b7985547bea688,385e06953a24cfc2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:38:46.063 INFO 2981023 --- [host-check-agent-status3,63b7985547bea688,49fecd192432c92b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:38:46.065 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:38:46.065 INFO 2981023 --- [host-check-agent-status3,63b7985547bea688,49fecd192432c92b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:38:46.069 INFO 2981023 --- [host-check-agent-status2,63b7985547bea688,385e06953a24cfc2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:38:46.070 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:38:46.070 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:38:46.073 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:38:46.073 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:38:46.177 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:38:46.195 INFO 2981023 --- [ocp-schedules-24,e67b56f7d20ba0f9,116eb05eda42c9a9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:38:46.196 INFO 2981023 --- [ocp-schedules-1,386b72561be229e8,4b99c2b1449b15a3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:38:46.198 INFO 2981023 --- [ocp-schedules-2,69d57139bc16843e,97a1097fb39ee67f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:38:46.199 INFO 2981023 --- [ocp-schedules-27,42abeef27d5bec00,77db99d450e7b07b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:38:46.213 INFO 2981023 --- [host-check-agent-status4,63b7985547bea688,a66e0016330d0a36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:38:46.214 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:38:46.214 INFO 2981023 --- [ocp-schedules-2,69d57139bc16843e,97a1097fb39ee67f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:38:46.264 INFO 2981023 --- [ocp-schedules-5,232808be35f0fa23,02ced82bf787ba31] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:38:46.264 INFO 2981023 --- [host-check-agent-status4,63b7985547bea688,a66e0016330d0a36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:38:46.265 INFO 2981023 --- [ocp-schedules-1,386b72561be229e8,4b99c2b1449b15a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:38:46.269 INFO 2981023 --- [ocp-schedules-24,e67b56f7d20ba0f9,116eb05eda42c9a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:38:46.269 INFO 2981023 --- [host-check-agent-status1,63b7985547bea688,84f9c36653606b8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:38:46.272 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:38:46.277 INFO 2981023 --- [ocp-schedules-1,386b72561be229e8,4b99c2b1449b15a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:38:46.287 INFO 2981023 --- [host-check-agent-status1,63b7985547bea688,84f9c36653606b8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:38:46.310 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051083504000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:38:46.400 INFO 2981023 --- [ocp-schedules-5,232808be35f0fa23,02ced82bf787ba31] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:38:46.403 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:38:46.415 INFO 2981023 --- [ocp-schedules-29,b7c689d0cb2d4d1f,eb7ef0af78c356c8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:38:46.418 INFO 2981023 --- [host-check-agent-status3,63b7985547bea688,49fecd192432c92b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:38:46.420 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:38:46.415875+08:00, lastProcessTime=2024-09-11T18:38:36.352186+08:00 2024-09-11 18:38:46.422 INFO 2981023 --- [host-check-agent-status3,63b7985547bea688,49fecd192432c92b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:38:46.422 INFO 2981023 --- [host-check-agent-status3,63b7985547bea688,49fecd192432c92b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:38:46.422 INFO 2981023 --- [host-check-agent-status3,63b7985547bea688,49fecd192432c92b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:38:46.458 INFO 2981023 --- [host-check-agent-status4,63b7985547bea688,a66e0016330d0a36] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:38:46.464 INFO 2981023 --- [host-check-agent-status4,63b7985547bea688,a66e0016330d0a36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:38:46.464 INFO 2981023 --- [host-check-agent-status4,63b7985547bea688,a66e0016330d0a36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:38:46.464 INFO 2981023 --- [host-check-agent-status4,63b7985547bea688,a66e0016330d0a36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:38:46.464 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:38:46.465 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:38:46.464738854+08:00 2024-09-11 18:38:46.466 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:38:46.467 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:38:46.465+08:00 2024-09-11 18:38:46.467 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:38:46.466957326+08:00 2024-09-11 18:38:46.469 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:38:46.465+08:00 and status AVAILABLE 2024-09-11 18:38:46.471 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:38:46.467+08:00 2024-09-11 18:38:46.474 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:38:46.467+08:00 and status AVAILABLE 2024-09-11 18:38:46.479 INFO 2981023 --- [host-check-agent-status2,63b7985547bea688,385e06953a24cfc2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:38:46.479 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:38:46.483 INFO 2981023 --- [host-check-agent-status2,63b7985547bea688,385e06953a24cfc2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:38:46.483 INFO 2981023 --- [host-check-agent-status2,63b7985547bea688,385e06953a24cfc2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:38:46.483 INFO 2981023 --- [host-check-agent-status2,63b7985547bea688,385e06953a24cfc2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:38:46.483 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:38:46.485 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:38:46.485072859+08:00 2024-09-11 18:38:46.485 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.485 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:38:46.486 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:38:46.485+08:00 2024-09-11 18:38:46.486 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:38:46.487 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:38:46.492 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:38:46.485+08:00 and status AVAILABLE 2024-09-11 18:38:46.493 INFO 2981023 --- [host-check-agent-status1,63b7985547bea688,84f9c36653606b8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:38:46.495 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.496 INFO 2981023 --- [host-check-agent-status1,63b7985547bea688,84f9c36653606b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:38:46.496 INFO 2981023 --- [host-check-agent-status1,63b7985547bea688,84f9c36653606b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:38:46.496 INFO 2981023 --- [host-check-agent-status1,63b7985547bea688,84f9c36653606b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:38:46.496 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:38:46.497 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:38:46.497 INFO 2981023 --- [ocp-schedules-24,e67b56f7d20ba0f9,116eb05eda42c9a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:38:46.497 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:38:46.503 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:38:46.504 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:38:46.509497653+08:00 2024-09-11 18:38:46.506 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:38:46.504+08:00 2024-09-11 18:38:46.510 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:38:46.504+08:00 and status AVAILABLE 2024-09-11 18:38:46.515 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:38:46.516 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:38:46.516 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.517 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:46.554 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:38:46.555 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:38:46.558 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:38:46.558 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.558 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:46.574 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.575 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:46.577 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:38:46.580 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:38:46.584 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:38:46.586 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.587 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:38:46.587 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:38:46.588 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.589 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:38:46.590 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:38:46.591 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:38:46.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.602 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:38:46.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:38:46.610 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:38:46.610 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:38:46.627 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:38:46.637 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:38:46.660 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:38:46.668 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:38:46.677 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:38:46.686 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:38:46.689 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.689 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.689 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.690 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:46.690 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.690 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:38:46.690 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:38:46.691 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.693 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:38:46.694 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:38:46.694 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.694 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:38:46.699 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:38:46.699 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051083504000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:38:46.703 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:38:46.713 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:38:46.725 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:38:46.771 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:38:46.771 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:38:46.771 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:38:46.773 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:38:46.782 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:38:46.789 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:38:46.790 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.792 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:38:46.804 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.806 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.808 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:38:46.808 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:38:46.808 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.809 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:38:46.809 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:38:46.810 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:38:46.810 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:38:46.811 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.813 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:38:46.816 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:38:46.817 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:38:46.820 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.820 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.820 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:38:46.824 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:38:46.838 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:38:46.859 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.860 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.861 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:38:46.863 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:38:46.868 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:38:46.870 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:38:46.870 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:38:46.876 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:38:46.877 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:38:46.881 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:38:46.884 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:38:46.895 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.896 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.897 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:38:46.897 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:38:46.898 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.900 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:38:46.901 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:38:46.901 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:38:46.902 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:38:46.904 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:38:46.905 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:38:46.906 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:38:46.929 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:38:46.931 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:38:46.968 INFO 2981023 --- [host-check-host-status3,5f480b67e0a3650b,675f591780d30a00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:38:46.968 INFO 2981023 --- [host-check-host-status4,5f480b67e0a3650b,32a64e39dfa0d4cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:38:46.970 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:38:46.976 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:38:46.976 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:38:46.980 INFO 2981023 --- [host-check-clockdiff1,26d771071d640cfd,b3f12e81e840f196] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051126 0 0 , error= 2024-09-11 18:38:46.980 INFO 2981023 --- [host-check-host-status1,5f480b67e0a3650b,83d0227400ccb5cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:38:46.983 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.984 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.985 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:38:46.987 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:38:46.992 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:46.993 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.994 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:38:46.994 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:38:46.994 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:46.996 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:38:46.996 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:38:47.002 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:38:47.010 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:47.010 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:38:47.011 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:47.011 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:38:47.016 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:38:47.016 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:38:47.020 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:38:47.025 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:38:47.026 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:38:47.029 INFO 2981023 --- [host-check-host-status2,5f480b67e0a3650b,0946332572e80979] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:38:47.045 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:38:47.057 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:38:47.059 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:38:46.415875+08:00 2024-09-11 18:38:47.068 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:47.069 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:47.069 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:38:47.070 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:38:47.070 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:47.076 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:38:47.082 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:38:47.082 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:47.083 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:38:47.088 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:38:47.088 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:38:47.095 INFO 2981023 --- [ocp-schedules-2,69d57139bc16843e,97a1097fb39ee67f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:38:47.125 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:38:47.156 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:38:47.163 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:38:47.165 INFO 2981023 --- [ocp-schedules-12,44401e9e8fc96616,e88b128b38f28a3b] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:38:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,71d222bde1e86213,67eda7ad606935d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,71d222bde1e86213,67eda7ad606935d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:47.508 INFO 2981023 --- [ocp-async-4,d1af36422596c4ad,a2e21a5c811d5183] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:47.805 INFO 2981023 --- [ocp-async-7,1c5fb9644e8b4b88,666d8406d9600b26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:38:48.560 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:48.560 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:49.508 INFO 2981023 --- [ocp-async-8,3483ced12838afc8,c43acb8798e71676] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:38:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:38:50.054 INFO 2981023 --- [ocp-schedules-5,e89e30c6385667b6,2cc728833d39d60e] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:38:50.076 WARN 2981023 --- [ocp-schedules-5,e89e30c6385667b6,2cc728833d39d60e] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:38:50.313 INFO 2981023 --- [prometheus-minute-schedule-1,2ce9243a7cebb763,0e8c0ff4c5821fc9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:50.317 INFO 2981023 --- [prometheus-minute-schedule-1,2ce9243a7cebb763,0e8c0ff4c5821fc9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:50.393 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-09-11 18:38:50.397 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/check_file, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/format.obbak, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/pieces, statCount=131), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p21, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p22, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p23, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p24, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001r1p25, statCount=24), AlarmTargetStat(alarmType=backup_storage_capacity_retry_times_exceeded, target=alarm_template_id=0:ob_cluster=rtdob-1718689631:storage_url=/obbackup/rtdob/1718689631/tenant_incarnation_1/1004/clog/piece_d1001 2024-09-11 18:38:50.397 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.397 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.397 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.397 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.397 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.398 INFO 2981023 --- [ocp-schedules-1,39c70ddf631de0cc,4e071c9673af0e06] c.o.o.a.service.OcpAlarmSearchService : rtdob-1718689631 2024-09-11 18:38:50.410 INFO 2981023 --- [ocp-schedules-9,3c6394b2f9e2b91f,52af1a72e405d202] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:38:51.509 INFO 2981023 --- [ocp-async-2,3f4f12b5c0c33a5f,53c5eb053522c924] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:51.560 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:51.562 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:51.666 INFO 2981023 --- [ocp-schedules-18,83cbdc5e8eb1c80d,d04046e993f08b80] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:38:51.666 INFO 2981023 --- [ocp-schedules-18,83cbdc5e8eb1c80d,d04046e993f08b80] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:38:51.667 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:38:51.667 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:38:51.667 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:38:51.667 INFO 2981023 --- [ocp-schedules-18,83cbdc5e8eb1c80d,d04046e993f08b80] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:38:51.857 INFO 2981023 --- [ocp-schedules-29,2eea873e9ffa33b2,d39cb5c4246b7ab0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:38:51.896 WARN 2981023 --- [scanner-13706781-1,2eea873e9ffa33b2,bd8fabd5f8c97ee0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:38:52.061 INFO 2981023 --- [host-check-clockdiff4,26d771071d640cfd,eac1a79b51ccaa7a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051132 0 0 , error= 2024-09-11 18:38:53.032 INFO 2981023 --- [ocp-schedules-30,cadb04abbffbe660,00c1d3ee5d209f75] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:38:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,9fd4d722e3a7f21e,7c883e1d34f212ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:53.316 INFO 2981023 --- [prometheus-minute-schedule-1,9fd4d722e3a7f21e,7c883e1d34f212ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:53.452 INFO 2981023 --- [ocp-schedules-1,43172b0ba97d12b8,1a6e9e481a0be885] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:38:53.452 INFO 2981023 --- [ocp-schedules-1,43172b0ba97d12b8,1a6e9e481a0be885] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:38:53.509 INFO 2981023 --- [ocp-async-9,751a2a50b3cf7db5,201a35f4c0cd6c8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:54.562 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:54.563 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:54.568 INFO 2981023 --- [ocp-schedules-5,33c980916fc57b54,529c0eb0e6116460] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:38:55.071 INFO 2981023 --- [ocp-schedules-1,17ff6df1f3dd37d7,a0f6b976775898d6] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-09-11T18:38:55.071+08:00, tenantSize=6 2024-09-11 18:38:55.074 INFO 2981023 --- [ocp-schedules-1,17ff6df1f3dd37d7,a0f6b976775898d6] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1726051135071000 2024-09-11 18:38:55.074 INFO 2981023 --- [ocp-schedules-1,17ff6df1f3dd37d7,a0f6b976775898d6] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1726050835029000 2024-09-11 18:38:55.105 INFO 2981023 --- [ocp-schedules-1,17ff6df1f3dd37d7,a0f6b976775898d6] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2024-09-11 18:38:55.509 INFO 2981023 --- [ocp-async-3,fae8ffac03a1006d,a91edb5f0b204ccb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:55.984 INFO 2981023 --- [ocp-schedules-18,aac9875601144803,cafab1aa9a6549c5] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:38:55.984 INFO 2981023 --- [ocp-schedules-18,aac9875601144803,cafab1aa9a6549c5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:38:55.985 INFO 2981023 --- [ocp-schedules-18,aac9875601144803,cafab1aa9a6549c5] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:38:55.985 INFO 2981023 --- [ocp-schedules-18,aac9875601144803,cafab1aa9a6549c5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:38:55.985 INFO 2981023 --- [ocp-schedules-18,aac9875601144803,cafab1aa9a6549c5] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:38:55.985 INFO 2981023 --- [ocp-schedules-18,aac9875601144803,cafab1aa9a6549c5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:38:56.075 INFO 2981023 --- [ocp-schedules-6,d7bd7ef5f9d4110b,3df642c9b51bdceb] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:38:56.094 WARN 2981023 --- [ocp-schedules-6,d7bd7ef5f9d4110b,3df642c9b51bdceb] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:38:56.226 INFO 2981023 --- [ocp-schedules-25,d0df85d00fea866f,4b05cfa0c9776c2c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:38:56.233 INFO 2981023 --- [ocp-schedules-25,d0df85d00fea866f,4b05cfa0c9776c2c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:38:56.233 INFO 2981023 --- [ocp-schedules-25,d0df85d00fea866f,4b05cfa0c9776c2c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:38:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,0669381022d9de6e,e1b2ae7fb921ee73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:56.317 INFO 2981023 --- [prometheus-minute-schedule-1,0669381022d9de6e,e1b2ae7fb921ee73] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:56.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:38:56.352009+08:00, lastProcessTime=2024-09-11T18:38:46.415875+08:00 2024-09-11 18:38:56.490 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:38:56.352009+08:00 2024-09-11 18:38:56.672 INFO 2981023 --- [ocp-schedules-29,c718bf8f1bd97398,227cdd8fbe252aa7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:38:56.681 INFO 2981023 --- [ocp-schedules-29,c718bf8f1bd97398,227cdd8fbe252aa7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:38:56.681 INFO 2981023 --- [ocp-schedules-29,c718bf8f1bd97398,227cdd8fbe252aa7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:38:57.509 INFO 2981023 --- [ocp-async-10,4d43add3071ed7dd,d0f814f12a33fb3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:57.563 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:38:57.565 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:38:57.805 INFO 2981023 --- [ocp-async-4,01d1aab02c380184,a87073ea05110938] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:38:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,68f652f3db28a5d4,8ed1e11d8579c9c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051140, interval=60 2024-09-11 18:38:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,68f652f3db28a5d4,8ed1e11d8579c9c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:38:59.509 INFO 2981023 --- [ocp-async-9,1ca0441a0dab759f,a709f8817fdc8a10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:38:59.611 INFO 2981023 --- [ocp-schedules-28,c80c7ce57a12e041,0948021d57cb472f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:38:59.611 INFO 2981023 --- [ocp-schedules-28,c80c7ce57a12e041,0948021d57cb472f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:38:59.628 INFO 2981023 --- [Collect all tenants major compactions1,c80c7ce57a12e041,29b5b53397219e6e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:38:59.628 INFO 2981023 --- [Collect all tenants major compactions2,c80c7ce57a12e041,1953f546d05aac61] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:38:59.629 INFO 2981023 --- [Collect all tenants major compactions4,c80c7ce57a12e041,73a84457408b3de9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:38:59.631 INFO 2981023 --- [Collect all tenants major compactions6,c80c7ce57a12e041,b67ea1c0ba0ffe07] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:38:59.635 INFO 2981023 --- [Collect all tenants major compactions5,c80c7ce57a12e041,aa52767b8586ccc6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:38:59.638 INFO 2981023 --- [Collect all tenants major compactions3,c80c7ce57a12e041,0fcf6b01d1e3d7e1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:38:59.653 INFO 2981023 --- [Collect all tenants major compactions1,c80c7ce57a12e041,29b5b53397219e6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:59.654 INFO 2981023 --- [Collect all tenants major compactions1,c80c7ce57a12e041,29b5b53397219e6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:38:59.654 INFO 2981023 --- [Collect all tenants major compactions6,c80c7ce57a12e041,b67ea1c0ba0ffe07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:59.654 INFO 2981023 --- [Collect all tenants major compactions3,c80c7ce57a12e041,0fcf6b01d1e3d7e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:59.655 INFO 2981023 --- [Collect all tenants major compactions6,c80c7ce57a12e041,b67ea1c0ba0ffe07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:38:59.655 INFO 2981023 --- [Collect all tenants major compactions4,c80c7ce57a12e041,73a84457408b3de9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:59.655 INFO 2981023 --- [Collect all tenants major compactions2,c80c7ce57a12e041,1953f546d05aac61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:59.655 INFO 2981023 --- [Collect all tenants major compactions5,c80c7ce57a12e041,aa52767b8586ccc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:38:59.656 INFO 2981023 --- [Collect all tenants major compactions3,c80c7ce57a12e041,0fcf6b01d1e3d7e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:38:59.656 INFO 2981023 --- [Collect all tenants major compactions4,c80c7ce57a12e041,73a84457408b3de9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:38:59.657 INFO 2981023 --- [Collect all tenants major compactions5,c80c7ce57a12e041,aa52767b8586ccc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:38:59.657 INFO 2981023 --- [Collect all tenants major compactions1,c80c7ce57a12e041,29b5b53397219e6e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:38:59.657 INFO 2981023 --- [Collect all tenants major compactions1,c80c7ce57a12e041,29b5b53397219e6e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:38:59.657 INFO 2981023 --- [Collect all tenants major compactions2,c80c7ce57a12e041,1953f546d05aac61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:38:59.658 INFO 2981023 --- [Collect all tenants major compactions3,c80c7ce57a12e041,0fcf6b01d1e3d7e1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:38:59.658 INFO 2981023 --- [Collect all tenants major compactions6,c80c7ce57a12e041,b67ea1c0ba0ffe07] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:38:59.659 INFO 2981023 --- [Collect all tenants major compactions6,c80c7ce57a12e041,b67ea1c0ba0ffe07] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:38:59.659 INFO 2981023 --- [Collect all tenants major compactions3,c80c7ce57a12e041,0fcf6b01d1e3d7e1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:38:59.659 INFO 2981023 --- [Collect all tenants major compactions4,c80c7ce57a12e041,73a84457408b3de9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:38:59.659 INFO 2981023 --- [Collect all tenants major compactions4,c80c7ce57a12e041,73a84457408b3de9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:38:59.659 INFO 2981023 --- [Collect all tenants major compactions5,c80c7ce57a12e041,aa52767b8586ccc6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:38:59.660 INFO 2981023 --- [Collect all tenants major compactions5,c80c7ce57a12e041,aa52767b8586ccc6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:38:59.660 INFO 2981023 --- [Collect all tenants major compactions2,c80c7ce57a12e041,1953f546d05aac61] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:38:59.660 INFO 2981023 --- [Collect all tenants major compactions2,c80c7ce57a12e041,1953f546d05aac61] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:38:59.661 INFO 2981023 --- [ocp-schedules-28,c80c7ce57a12e041,0948021d57cb472f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:39:00.565 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:00.565 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:01.509 INFO 2981023 --- [ocp-async-3,0ad75b3748397ddc,7df44f76f63d8610] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:02.073 INFO 2981023 --- [ocp-schedules-15,68e20340827aa1df,13612ac511cf00ab] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:39:02.127 WARN 2981023 --- [scanner-13707031-1,68e20340827aa1df,b9af2be4ba6a20ba] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:39:02.157 INFO 2981023 --- [alarm-detect-12,68e20340827aa1df,3a49394fd9b8c19e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646924, data is empty. 2024-09-11 18:39:02.166 INFO 2981023 --- [alarm-detect-12,68e20340827aa1df,3a49394fd9b8c19e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646924, data is empty. 2024-09-11 18:39:02.173 INFO 2981023 --- [alarm-detect-12,68e20340827aa1df,3a49394fd9b8c19e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646924, data is empty. 2024-09-11 18:39:02.174 INFO 2981023 --- [alarm-detect-17,68e20340827aa1df,cb69e07c504aeb3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646924, data is empty. 2024-09-11 18:39:02.175 INFO 2981023 --- [alarm-detect-2,68e20340827aa1df,efaf175bd1694573] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646924, data is empty. 2024-09-11 18:39:02.178 INFO 2981023 --- [alarm-detect-17,68e20340827aa1df,cb69e07c504aeb3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646924, data is empty. 2024-09-11 18:39:02.181 INFO 2981023 --- [alarm-detect-2,68e20340827aa1df,efaf175bd1694573] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646924, data is empty. 2024-09-11 18:39:02.183 INFO 2981023 --- [alarm-detect-17,68e20340827aa1df,cb69e07c504aeb3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646924, data is empty. 2024-09-11 18:39:02.186 INFO 2981023 --- [alarm-detect-2,68e20340827aa1df,efaf175bd1694573] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646924, data is empty. 2024-09-11 18:39:02.300 INFO 2981023 --- [alarm-detect-20,68e20340827aa1df,90a1a7860cfe2b5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646924, data is empty. 2024-09-11 18:39:02.304 INFO 2981023 --- [alarm-detect-20,68e20340827aa1df,90a1a7860cfe2b5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646924, data is empty. 2024-09-11 18:39:02.308 INFO 2981023 --- [alarm-detect-20,68e20340827aa1df,90a1a7860cfe2b5f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646924, data is empty. 2024-09-11 18:39:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,5c9dced7ac6bb7d6,fd4892e76f74b183] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:02.320 INFO 2981023 --- [prometheus-minute-schedule-1,5c9dced7ac6bb7d6,fd4892e76f74b183] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:02.627 INFO 2981023 --- [ocp-schedules-14,854ffd79a7b6827a,081a60ad13382d36] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:39:02.627+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:39:02.629 INFO 2981023 --- [ocp-schedules-14,854ffd79a7b6827a,081a60ad13382d36] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:39:02.664 INFO 2981023 --- [ocp-schedules-14,854ffd79a7b6827a,081a60ad13382d36] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 39 to SqlAggregateData 2024-09-11 18:39:02.666 INFO 2981023 --- [ocp-schedules-14,854ffd79a7b6827a,081a60ad13382d36] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:39:03.035 INFO 2981023 --- [ocp-schedules-23,5498c1f0ea7f5ed2,9254e5b621b43237] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:39:03.509 INFO 2981023 --- [ocp-async-5,43b08a7f6daa4e51,831f06b2068d653e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:03.566 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:03.566 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,7715863e4929cab4,b6d6a8dee8975ce6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:05.316 INFO 2981023 --- [prometheus-minute-schedule-1,7715863e4929cab4,b6d6a8dee8975ce6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:05.509 INFO 2981023 --- [ocp-async-1,1527076de909e125,58d5c275fe944fce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:06.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:39:06.337659+08:00, lastProcessTime=2024-09-11T18:38:56.352009+08:00 2024-09-11 18:39:06.343 INFO 2981023 --- [ocp-schedules-5,f95c8c4524f22736,e78bca060a50eaa0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:39:06.344 INFO 2981023 --- [ocp-schedules-5,f95c8c4524f22736,e78bca060a50eaa0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:39:06.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:39:06.379 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.381 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:39:06.381 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.381 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:39:06.381 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.385 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:39:06.386 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:39:06.388 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:39:06.389 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.389 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.390 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.390 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.395 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.395 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.401 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.402 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.402 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.402 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.402 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.403 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:39:06.413 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.414 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:39:06.417 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.418 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.420 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.420 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:39:06.421 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.421 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.428 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.429 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.429 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:39:06.429 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:39:06.429 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.430 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:39:06.436 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.436 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.436 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.437 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:39:06.437 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:39:06.437 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE76F1D6D6-0-0] 2024-09-11 18:39:06.437 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:39:06.437 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:39:06.438 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.438 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:39:06.442 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:39:06.445 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.446 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:39:06.447 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.447 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:39:06.448 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.448 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:39:06.449 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.450 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:39:06.450 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.451 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:39:06.451 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.452 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:39:06.452 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.453 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:39:06.454 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.454 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.454 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:39:06.455 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.456 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:39:06.456 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:39:06.458 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:39:06.460 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:06.462 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:39:06.463 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.463 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.470 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:06.472 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.472 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:39:06.477 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:39:06.478 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:39:06.480 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:39:06.480 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:39:06.480 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:39:06.481 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:39:06.481 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.481 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:39:06.482 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.483 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:39:06.483 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:39:06.483 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE76F1D6DF-0-0] 2024-09-11 18:39:06.484 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:39:06.487 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:39:06.489 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:39:06.489 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.489 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.490 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.490 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:39:06.496 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:39:06.498 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:39:06.498 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:39:06.498 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:39:06.501 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.505 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:39:06.514 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.518 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:39:06.528 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.534 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:39:06.537 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:39:06.337659+08:00 2024-09-11 18:39:06.538 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.539 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:39:06.539 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:39:06.540 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:39:06.542 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:39:06.551 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:06.559 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.562 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:39:06.566 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:06.567 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:06.572 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:39:06.573 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:39:06.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:06.574 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:39:06.583 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:39:06.584 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:39:06.584 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:39:06.584 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:39:06.584 INFO 2981023 --- [ocp-schedules-5,f95c8c4524f22736,e78bca060a50eaa0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:39:07.509 INFO 2981023 --- [ocp-async-7,5ea230cc1541df53,554f2f4a209dfa99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:07.805 INFO 2981023 --- [ocp-async-5,29151ca4eef0c531,7763e1f1a6ab880f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:39:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,4f8fcd3f200165d8,f07d31f26cd68db4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,4f8fcd3f200165d8,f07d31f26cd68db4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:08.409 INFO 2981023 --- [ocp-schedules-29,a3bd8758b24ada55,d7e05b41ce991310] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:39:08.488 INFO 2981023 --- [ocp-schedules-26,f1f2a9e22f993400,e71d461869161631] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:39:08.488+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:39:08.489 INFO 2981023 --- [ocp-schedules-26,f1f2a9e22f993400,e71d461869161631] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:39:08.490 INFO 2981023 --- [ocp-schedules-26,f1f2a9e22f993400,e71d461869161631] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:39:08.490 INFO 2981023 --- [ocp-schedules-26,f1f2a9e22f993400,e71d461869161631] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:39:09.510 INFO 2981023 --- [ocp-async-10,7d1fb50ac0f90ed8,46067a92fe53e322] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:09.567 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:09.568 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:10.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:39:10.003 INFO 2981023 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:39:10.916 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:39:10.919 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:39:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,531e51cbde88f903,fcabdb911966c176] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:11.316 INFO 2981023 --- [prometheus-minute-schedule-1,531e51cbde88f903,fcabdb911966c176] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:11.510 INFO 2981023 --- [ocp-async-4,4801e4919c0fa0ca,0781690a95704cf2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:12.351 INFO 2981023 --- [ocp-schedules-12,e583bea151140412,2cf965c978637d81] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:39:12.395 WARN 2981023 --- [scanner-13708876-1,e583bea151140412,60262d116d51c0a2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:39:12.568 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:12.568 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:13.037 INFO 2981023 --- [ocp-schedules-21,1e556b6549f76384,1e135170b0addd22] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:39:13.078 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726051080, nodeCount=1000, affectRows=1000 2024-09-11 18:39:13.080 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726051080, nodeCount=999, validCount=999 2024-09-11 18:39:13.082 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726051080, nodeCount=1000, affectRows=1000 2024-09-11 18:39:13.085 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726051080, nodeCount=1000, validCount=1000 2024-09-11 18:39:13.085 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726051080, nodeCount=48, affectRows=48 2024-09-11 18:39:13.085 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726051080, nodeCount=46, validCount=46 2024-09-11 18:39:13.510 INFO 2981023 --- [ocp-async-8,c2e0953aa1e65a02,d32037d330cf44b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:13.750 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726051080, nodeCount=480, affectRows=480 2024-09-11 18:39:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,5773fc2df9f0483d,c91666d5a45bfeeb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,eb2be0c433bf1029,aeff1e4e68633f99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:39:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,5773fc2df9f0483d,c91666d5a45bfeeb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:15.335 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2024-09-11T18:35:14.817, end=2024-09-11T18:37:15.329} 2024-09-11 18:39:15.342 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, 2024-09-11 18:39:15.342 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:39:15.347 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:35:14.817, end=2024-09-11T18:37:15.329},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:39:15.347 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=sys, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:39:15.347 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:39:15.371 INFO 2981023 --- [plan_change_detector24,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:rtdob-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:39:15.389 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755389000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=500002, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=4, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.389Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=4, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.391 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2024-09-11 18:39:15.391 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.396 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.396 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.397 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.397 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.402 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.402 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1, obServerId=null, obDbId=500002, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.404 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2024-09-11 18:39:15.404 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.389, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=4, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=ocs, obDbId=500002, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.404 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:39:15.406 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:35:14.817, end=2024-09-11T18:37:15.329},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:39:15.411 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1000006, name=sys, clusterName=linkrtd, obClusterId=1719994443, clusterId=1000006, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=OPERATING, syncStatus=VALID, mode=MYSQL, createTime=2024-09-11T15:21:32+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.4.0, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:39:15.411 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:39:15.423 INFO 2981023 --- [plan_change_detector8,,] i.PlanChangePerfDegradationDetectService : Can't find SqlText:linkrtd-sys-201001-F5E53FCE0C3170EA60825DDEC1139E4E 2024-09-11 18:39:15.442 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755442000, endTimeUs=1726051035329000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, sqlId=DDA5C566D6A13F14B7BE8B6D084693F9, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=1000006, obClusterId=1719994443, clusterName=linkrtd, tenantId=1000006, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.2.4.0, startTime=2024-09-10T10:39:15.442Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[null, 1], serverMap={null=10.31.35.7:2882, 1=10.31.35.6:2882}, serverIpMap={10.31.35.7=null, 10.31.35.6=1}, cluster=ObClusterEntity{id=1000006, name=linkrtd, obClusterId=1719994443, status=OPERATING}, tenant=ObTenantEntity{id=1000006, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=1000024, ip=10.31.35.6, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=4.2.4.0_100000082024070810-556a8f594436d32a23ee92289717913cf503184b(Jul 8 2024 11:05:25), withRootserver=true, hostId=2000007, status=RUNNING, innerStatus=ACTIVE, startTime=2024-09-11T15:21:26+08:00, stopTime=1970-01-01T08:00+08:00}, ObServerEntity{id=1000029, ip=10.31.35.7, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=admin, version=null, withRootserver=false, hostId=2000010, status=CREATING, innerStatus=null, startTime=null, stopTime=null}], server=null) 2024-09-11 18:39:15.444 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2024-09-11 18:39:15.444 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.451 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.451 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=DDA5C566D6A13F14B7BE8B6D084693F9, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.452 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.452 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.459 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.459 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1719994443, clusterName=linkrtd, obTenantId=1, obServerId=null, obDbId=201001, sqlId=DDA5C566D6A13F14B7BE8B6D084693F9, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.461 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.463 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.442, end=2024-09-11T18:37:15.329}, clusterId=1000006, tenantId=1000006, sqlId=DDA5C566D6A13F14B7BE8B6D084693F9, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.464 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:39:15.465 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:35:14.817, end=2024-09-11T18:37:15.329},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:39:15.465 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:39:15.465 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:39:15.466 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:39:15.466 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:35:14.817, end=2024-09-11T18:37:15.329},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:39:15.466 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:39:15.466 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:39:15.468 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:39:15.469 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:35:14.817, end=2024-09-11T18:37:15.329},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:39:15.469 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=linkreit, clusterName=rtdob, obClusterId=1718689631, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-04T10:24:04+08:00, primaryZone=zone1, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.1.0, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:39:15.469 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:39:15.472 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726051080, nodeCount=480, validCount=480 2024-09-11 18:39:15.510 INFO 2981023 --- [ocp-async-6,25a913155d7a53c9,8f0071b90e990a29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:15.510 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755510000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.510Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.512 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755512000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.512Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.514 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755514000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.514Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.518 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755518000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.518Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.522 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755522000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=3DF6D92BA61DE793DB65085C7736D50E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.522Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.524 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755524000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.524Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.527 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755527000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.527Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.531 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755531000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.531Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.565 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.565 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.567 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.567 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.568 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:15.572 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 72 2024-09-11 18:39:15.572 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.573 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:15.577 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.577 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.580 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.580 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.582 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.582 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.584 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.584 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.587 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:39:15.587 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.590 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:39:15.590 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.590 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.590 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.591 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.591 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.592 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.592 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.593 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2024-09-11 18:39:15.593 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.593 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.593 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.597 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.597 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.601 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.601 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.601 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.601 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.603 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.603 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.604 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.604 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.604 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.605 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.605 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.607 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.607 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.608 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.608 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.608 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.610 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.610 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.611 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.611 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.611 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.611 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.613 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.613 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=3DF6D92BA61DE793DB65085C7736D50E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.614 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.614 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.616 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.616 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.616 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.616 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.622 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.622 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.625 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.629 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.512, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=0D624A93C94C9432C3BF57F7A93E32EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.629 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.630 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.630 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.633 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.633 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.634 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.634 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.51, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=B0F9C8DFC05A6B25D78D760FF5F19AAF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.644 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.645 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.646 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.522, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=3DF6D92BA61DE793DB65085C7736D50E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.646 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.652 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.524, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=FC9C2BCECA13CCBBC569D42C4FE1F7A7, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.652 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.531, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=EDF6D01062C61663E7781BD2B6B7C86E, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.654 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.514, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=AFFEB73F78E95F8C48EBC87881D08A47, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.655 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.518, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=EBA2D4807660A8A8C93E1EB6233854EC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.658 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755658000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.658Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.667 INFO 2981023 --- [plan_change_detector11,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.527, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=FBFDF4CB3E0E4576FD202762D3E2ED78, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.671 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.671 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.685 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755685000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=D974920610DD12A7243DCB93DC5CF3A1, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.685Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.686 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.686 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.691 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755691000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=02CA7A7B032ED8F45814F8819833FE6A, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.691Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.691 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755691000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=84040970D5542715C52F7DC0925136DD, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.691Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.692 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755692000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.692Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.700 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.700 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.700 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.700 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.703 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.703 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.707 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.707 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.709 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.709 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.710 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755710000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=1ED31157F7096F47941EA2233AA4101C, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.710Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.715 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1725964755715000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, sqlId=B8D29A629AD48EB08D4324E33D63F650, obServerId=null, obDbId=504413, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1718689631, clusterName=rtdob, tenantId=6, obTenantId=1004, tenantName=linkreit, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.1.0, startTime=2024-09-10T10:39:15.715Z, endTime=2024-09-11T10:37:15.329Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.31.35.5:2882}, serverIpMap={10.31.35.5=1}, cluster=ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING}, tenant=ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL}, servers=[ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2024-09-11 18:39:15.717 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.717 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.720 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.720 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.721 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.721 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.721 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.722 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.724 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.724 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.724 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.724 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.724 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.727 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.727 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.728 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.728 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.730 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 262 2024-09-11 18:39:15.730 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2024-09-11 18:39:15.730 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.730 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.730 INFO 2981023 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.658, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=06C09AF2DC0E20BC61C857AA1AF999DF, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.732 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.732 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.735 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.735 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=D974920610DD12A7243DCB93DC5CF3A1, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.738 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.738 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.738 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.738 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.738 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=02CA7A7B032ED8F45814F8819833FE6A, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.741 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.741 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=84040970D5542715C52F7DC0925136DD, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.741 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.741 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.742 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.742 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.742 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.743 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.743 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.743 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.744 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.744 INFO 2981023 --- [plan_change_detector24,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.685, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=D974920610DD12A7243DCB93DC5CF3A1, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.746 INFO 2981023 --- [plan_change_detector15,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.691, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=02CA7A7B032ED8F45814F8819833FE6A, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.747 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2024-09-11 18:39:15.747 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2024-09-11 18:39:15.747 INFO 2981023 --- [plan_change_detector13,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.692, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=38F01F7A22E292CEA95DFDDE037D5BBC, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.748 INFO 2981023 --- [plan_change_detector20,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.691, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=84040970D5542715C52F7DC0925136DD, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.751 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.751 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=1ED31157F7096F47941EA2233AA4101C, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.753 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.753 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2024-09-11 18:39:15.753 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1725926400000000, endTimeUs=1726051035329000, obClusterId=1718689631, clusterName=rtdob, obTenantId=1004, obServerId=null, obDbId=504413, sqlId=B8D29A629AD48EB08D4324E33D63F650, sqlIdList=null, dynamicSql=false) 2024-09-11 18:39:15.754 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2024-09-11 18:39:15.756 INFO 2981023 --- [plan_change_detector19,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.71, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=1ED31157F7096F47941EA2233AA4101C, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.757 INFO 2981023 --- [plan_change_detector18,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2024-09-10T18:39:15.715, end=2024-09-11T18:37:15.329}, clusterId=2, tenantId=6, sqlId=B8D29A629AD48EB08D4324E33D63F650, serverId=null, dbName=yongcepro, obDbId=504413, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2024-09-11 18:39:15.758 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2024-09-11 18:39:15.758 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null), Interval{start=2024-09-11T18:35:14.817, end=2024-09-11T18:37:15.329},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2024-09-11 18:39:15.758 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=linkprdocp, obClusterId=1719994437, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2024-07-03T16:27:33+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.2.1.2, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null), right=null) 2024-09-11 18:39:15.758 INFO 2981023 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2024-09-11 18:39:15.759 INFO 2981023 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2024-09-11 18:39:16.012 INFO 2981023 --- [ocp-schedules-22,9a217eb14990cdbc,4853604d487699ee] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:39:16.013 INFO 2981023 --- [host-check-clockdiff1,9a217eb14990cdbc,a1316aadf784e8cd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:39:16.013 INFO 2981023 --- [host-check-clockdiff2,9a217eb14990cdbc,2238e5e51692ea63] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:39:16.014 INFO 2981023 --- [host-check-clockdiff3,9a217eb14990cdbc,01b3977bc7120c83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:39:16.014 INFO 2981023 --- [host-check-clockdiff4,9a217eb14990cdbc,0a9954745d3faccc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:39:16.023 INFO 2981023 --- [host-check-clockdiff3,9a217eb14990cdbc,01b3977bc7120c83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051156 0 0 , error= 2024-09-11 18:39:16.029 INFO 2981023 --- [ocp-schedules-30,de3d1eb90bce0a33,9379b03165ac17ab] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:39:16.029 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:39:16.029 INFO 2981023 --- [ocp-schedules-5,ebe18ad35096104e,ab81e3e1028bb017] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:39:16.029 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:39:16.029 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:39:16.036 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:39:16.036 INFO 2981023 --- [host-check-agent-status1,ebe18ad35096104e,582383ed41033a00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:39:16.037 INFO 2981023 --- [host-check-agent-status2,ebe18ad35096104e,c610dc7b07572af1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:39:16.038 INFO 2981023 --- [host-check-agent-status3,ebe18ad35096104e,183640dfe3322acc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:39:16.038 INFO 2981023 --- [host-check-agent-status4,ebe18ad35096104e,69ae1ff4ec8f0e1d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:39:16.059 INFO 2981023 --- [host-check-clockdiff2,9a217eb14990cdbc,2238e5e51692ea63] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051156 5 5 , error= 2024-09-11 18:39:16.066 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:39:16.067 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:39:16.069 INFO 2981023 --- [host-check-agent-status2,ebe18ad35096104e,c610dc7b07572af1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:39:16.070 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:39:16.070 INFO 2981023 --- [host-check-agent-status2,ebe18ad35096104e,c610dc7b07572af1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:39:16.071 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:39:16.075 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:39:16.075 INFO 2981023 --- [host-check-agent-status1,ebe18ad35096104e,582383ed41033a00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:39:16.076 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:39:16.077 INFO 2981023 --- [host-check-agent-status1,ebe18ad35096104e,582383ed41033a00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:39:16.080 INFO 2981023 --- [host-check-agent-status4,ebe18ad35096104e,69ae1ff4ec8f0e1d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:39:16.109 INFO 2981023 --- [host-check-agent-status4,ebe18ad35096104e,69ae1ff4ec8f0e1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:39:16.109 INFO 2981023 --- [host-check-agent-status3,ebe18ad35096104e,183640dfe3322acc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:39:16.171 INFO 2981023 --- [host-check-agent-status3,ebe18ad35096104e,183640dfe3322acc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:39:16.192 INFO 2981023 --- [ocp-schedules-14,c9e528a4ea6fc85d,a1e16f8e1d85a31d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:39:16.192 INFO 2981023 --- [ocp-schedules-24,3035805ced3182db,de8ce50dec2899f3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:39:16.192 INFO 2981023 --- [ocp-schedules-15,b5def32ada0dd40f,f36c8b44e3f11d6b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:39:16.192 INFO 2981023 --- [ocp-schedules-25,583a87c7f89c101a,b8c4f772441a821b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:39:16.211 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:39:16.257 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:39:16.268 INFO 2981023 --- [ocp-schedules-24,3035805ced3182db,de8ce50dec2899f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:39:16.268 INFO 2981023 --- [ocp-schedules-15,b5def32ada0dd40f,f36c8b44e3f11d6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:39:16.272 INFO 2981023 --- [ocp-schedules-24,3035805ced3182db,de8ce50dec2899f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:39:16.274 INFO 2981023 --- [ocp-schedules-14,c9e528a4ea6fc85d,a1e16f8e1d85a31d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:39:16.295 INFO 2981023 --- [host-check-agent-status3,ebe18ad35096104e,183640dfe3322acc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:39:16.308 INFO 2981023 --- [ocp-schedules-13,e211edf5846d15ae,81e391ec0aa59023] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:39:16.317 INFO 2981023 --- [host-check-agent-status3,ebe18ad35096104e,183640dfe3322acc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:39:16.317 INFO 2981023 --- [host-check-agent-status3,ebe18ad35096104e,183640dfe3322acc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:39:16.317 INFO 2981023 --- [host-check-agent-status3,ebe18ad35096104e,183640dfe3322acc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:39:16.319 INFO 2981023 --- [host-check-agent-status1,ebe18ad35096104e,582383ed41033a00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:39:16.320 INFO 2981023 --- [host-check-agent-status2,ebe18ad35096104e,c610dc7b07572af1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:39:16.321 INFO 2981023 --- [host-check-agent-status4,ebe18ad35096104e,69ae1ff4ec8f0e1d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:39:16.322 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:39:16.322 INFO 2981023 --- [host-check-agent-status1,ebe18ad35096104e,582383ed41033a00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:39:16.322 INFO 2981023 --- [host-check-agent-status1,ebe18ad35096104e,582383ed41033a00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:39:16.322 INFO 2981023 --- [host-check-agent-status1,ebe18ad35096104e,582383ed41033a00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:39:16.326 INFO 2981023 --- [host-check-agent-status4,ebe18ad35096104e,69ae1ff4ec8f0e1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:39:16.326 INFO 2981023 --- [host-check-agent-status4,ebe18ad35096104e,69ae1ff4ec8f0e1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:39:16.326 INFO 2981023 --- [host-check-agent-status4,ebe18ad35096104e,69ae1ff4ec8f0e1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:39:16.327 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051083504000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:39:16.328 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:39:16.328111837+08:00 2024-09-11 18:39:16.329 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:39:16.330 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:39:16.328+08:00 2024-09-11 18:39:16.330 INFO 2981023 --- [host-check-agent-status2,ebe18ad35096104e,c610dc7b07572af1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:39:16.330 INFO 2981023 --- [host-check-agent-status2,ebe18ad35096104e,c610dc7b07572af1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:39:16.330 INFO 2981023 --- [host-check-agent-status2,ebe18ad35096104e,c610dc7b07572af1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:39:16.331 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:39:16.336319083+08:00 2024-09-11 18:39:16.365 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:39:16.331+08:00 2024-09-11 18:39:16.365 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:39:16.328+08:00 and status AVAILABLE 2024-09-11 18:39:16.369 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:39:16.369 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:39:16.331+08:00 and status AVAILABLE 2024-09-11 18:39:16.369 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:39:16.369585909+08:00 2024-09-11 18:39:16.370 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:39:16.371 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:39:16.371001163+08:00 2024-09-11 18:39:16.372 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:39:16.369+08:00 2024-09-11 18:39:16.372 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:39:16.371+08:00 2024-09-11 18:39:16.374 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:39:16.369+08:00 and status AVAILABLE 2024-09-11 18:39:16.374 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:39:16.371+08:00 and status AVAILABLE 2024-09-11 18:39:16.376 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:39:16.385 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.385 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:16.386 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.386 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:39:16.387 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:39:16.394 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:39:16.398 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:39:16.398 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:39:16.399 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.399 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:39:16.399157+08:00, lastProcessTime=2024-09-11T18:39:06.337659+08:00 2024-09-11 18:39:16.399 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:39:16.400 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:39:16.400 INFO 2981023 --- [ocp-schedules-14,c9e528a4ea6fc85d,a1e16f8e1d85a31d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:39:16.403 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.403 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:16.404 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:39:16.411 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:39:16.412 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:39:16.413 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:39:16.414 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:39:16.415 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:39:16.416 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:39:16.418 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:39:16.462 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:16.463 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.463 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.463 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:39:16.463 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:39:16.464 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:16.470 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:39:16.473 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:39:16.473 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:39:16.490 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:39:16.494 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:39:16.500 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:39:16.510 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:39:16.512 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:39:16.512 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:16.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.513 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:16.514 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:39:16.514 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.515 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:39:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:39:16.517 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:39:16.517 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:39:16.524 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:39:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:39:16.557 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:39:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:39:16.566 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051083504000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:39:16.567 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:39:16.573 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:39:16.573 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:39:16.573 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:39:16.585 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:39:16.589 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:39:16.593 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:16.594 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.595 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:39:16.596 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:39:16.598 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:39:16.599 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:39:16.602 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:16.603 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:39:16.604 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:39:16.607 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:39:16.608 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:39:16.608 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:39:16.608 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:39:16.616 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:39:16.618 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:39:16.618 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:39:16.619 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:16.620 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.620 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:16.620 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:39:16.621 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.622 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:39:16.624 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:39:16.624 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:39:16.627 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:39:16.633 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:39:16.639 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:39:16.639 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:39:16.643 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:39:16.645 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:39:16.651 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:39:16.653 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:39:16.656 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:39:16.661 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:16.662 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.662 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:39:16.663 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:39:16.668 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:39:16.673 INFO 2981023 --- [host-check-host-status4,de3d1eb90bce0a33,2afc73d638c220cd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:39:16.673 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:39:16.674 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:39:16.675 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:39:16.676 INFO 2981023 --- [host-check-host-status1,de3d1eb90bce0a33,0548491eec51f063] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:39:16.685 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:39:16.686 INFO 2981023 --- [ocp-schedules-6,19bcddc2e75d8e04,f2f4b2c16e1605ed] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:39:16.696 INFO 2981023 --- [host-check-host-status3,de3d1eb90bce0a33,dce75f6e14f2394a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:39:16.700 INFO 2981023 --- [host-check-host-status2,de3d1eb90bce0a33,c3c94e62762a6471] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:39:16.701 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:39:16.708 INFO 2981023 --- [ocp-schedules-6,19bcddc2e75d8e04,f2f4b2c16e1605ed] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:39:16.708 INFO 2981023 --- [ocp-schedules-6,19bcddc2e75d8e04,f2f4b2c16e1605ed] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:39:16.712 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:39:16.399157+08:00 2024-09-11 18:39:16.720 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:39:16.726 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:16.726 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.727 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:39:16.731 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:39:16.731 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:39:16.732 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:39:16.737 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:16.738 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:16.738 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:39:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:39:16.743 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:39:16.743 INFO 2981023 --- [ocp-schedules-15,b5def32ada0dd40f,f36c8b44e3f11d6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:39:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,4279c6d1c28a3f92,16e06c5e1514952c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:17.317 INFO 2981023 --- [prometheus-minute-schedule-1,4279c6d1c28a3f92,16e06c5e1514952c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:17.510 INFO 2981023 --- [ocp-async-2,d80819a2a0728ba2,6d38af37c474ff89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:17.806 INFO 2981023 --- [ocp-async-8,7b8f6ca66c21006c,45a6f97689253b12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:39:17.806 INFO 2981023 --- [host-check-clockdiff1,9a217eb14990cdbc,a1316aadf784e8cd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051157 0 0 , error= 2024-09-11 18:39:17.976 INFO 2981023 --- [ocp-schedules-15,a8c94c63bb5b1b29,ebdc98fe24268897] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:39:18.573 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:18.574 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:19.510 INFO 2981023 --- [ocp-async-7,b396d859b5151b19,1f190b3ac861f3d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,62d365cced9ccd18,8f2819b736554daa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:20.316 INFO 2981023 --- [prometheus-minute-schedule-1,62d365cced9ccd18,8f2819b736554daa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:21.510 INFO 2981023 --- [ocp-async-5,f5cd407d34ca385c,baf3346638826b9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:21.574 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:21.575 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:21.983 INFO 2981023 --- [host-check-clockdiff4,9a217eb14990cdbc,0a9954745d3faccc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051161 0 0 , error= 2024-09-11 18:39:22.557 INFO 2981023 --- [ocp-schedules-25,7bba05bf2eb8a0e5,5f621c87ff8cf41d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:39:22.604 WARN 2981023 --- [scanner-13709065-1,7bba05bf2eb8a0e5,7be5cec391f0b164] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:39:22.618 INFO 2981023 --- [alarm-detect-9,7bba05bf2eb8a0e5,bb984f21fff48749] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646924, data is empty. 2024-09-11 18:39:22.625 INFO 2981023 --- [alarm-detect-9,7bba05bf2eb8a0e5,bb984f21fff48749] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646963, data is empty. 2024-09-11 18:39:22.631 INFO 2981023 --- [alarm-detect-9,7bba05bf2eb8a0e5,bb984f21fff48749] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646963, data is empty. 2024-09-11 18:39:22.642 INFO 2981023 --- [alarm-detect-9,7bba05bf2eb8a0e5,6364c124ddd78aa4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646924, data is empty. 2024-09-11 18:39:22.645 INFO 2981023 --- [alarm-detect-7,7bba05bf2eb8a0e5,8042d4c84cbe4fdd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646924, data is empty. 2024-09-11 18:39:22.649 INFO 2981023 --- [alarm-detect-9,7bba05bf2eb8a0e5,6364c124ddd78aa4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646963, data is empty. 2024-09-11 18:39:22.652 INFO 2981023 --- [alarm-detect-7,7bba05bf2eb8a0e5,8042d4c84cbe4fdd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646963, data is empty. 2024-09-11 18:39:22.656 INFO 2981023 --- [alarm-detect-9,7bba05bf2eb8a0e5,6364c124ddd78aa4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646963, data is empty. 2024-09-11 18:39:22.659 INFO 2981023 --- [alarm-detect-7,7bba05bf2eb8a0e5,8042d4c84cbe4fdd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646963, data is empty. 2024-09-11 18:39:22.751 INFO 2981023 --- [alarm-detect-6,7bba05bf2eb8a0e5,9e02bbd774ff98cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646924, data is empty. 2024-09-11 18:39:22.760 INFO 2981023 --- [alarm-detect-6,7bba05bf2eb8a0e5,9e02bbd774ff98cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646963, data is empty. 2024-09-11 18:39:22.763 INFO 2981023 --- [alarm-detect-6,7bba05bf2eb8a0e5,9e02bbd774ff98cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646963, data is empty. 2024-09-11 18:39:23.040 INFO 2981023 --- [ocp-schedules-21,8b4015146a3a708f,2724326d0031b57c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:39:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,1b733f03a1f454e3,5310716e0ace7b57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,1b733f03a1f454e3,5310716e0ace7b57] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:23.510 INFO 2981023 --- [ocp-async-1,d7e998f582314e25,fc29a753d74f33e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:24.571 INFO 2981023 --- [ocp-schedules-26,faf96bfb281db008,83a6f63ba3e4f727] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:39:24.575 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:24.575 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:25.510 INFO 2981023 --- [ocp-async-9,49c54e33231f7cf4,b2b514c1363dfc1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,85c4c458dd4c6b02,bb4d7872f8c06a94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,85c4c458dd4c6b02,bb4d7872f8c06a94] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:26.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:39:26.337349+08:00, lastProcessTime=2024-09-11T18:39:16.399157+08:00 2024-09-11 18:39:26.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:39:26.337349+08:00 2024-09-11 18:39:27.511 INFO 2981023 --- [ocp-async-3,cc5dfa13e37b0f4c,60eaf68c4c4745fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:27.575 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:27.576 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:27.806 INFO 2981023 --- [ocp-async-1,2d645035417f516b,faea7191c3c3b5da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:39:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,696c6ec66aa9760a,71a0aca0ddba6b00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,696c6ec66aa9760a,71a0aca0ddba6b00] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:29.511 INFO 2981023 --- [ocp-async-2,45f86bdab868f3c4,14653356ba494d59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:29.643 INFO 2981023 --- [ocp-schedules-14,16969fb155a2ad72,7d5411951faa81c7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:39:29.643 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:39:29.643 INFO 2981023 --- [ocp-schedules-14,16969fb155a2ad72,7d5411951faa81c7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:39:29.643 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:39:29.643 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:39:29.648 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:39:29.648 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:39:29.648 INFO 2981023 --- [ocp-schedules-14,16969fb155a2ad72,7d5411951faa81c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:39:29.648 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:39:29.649 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:39:29.649 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:39:29.650 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:39:29.650 INFO 2981023 --- [ocp-schedules-14,16969fb155a2ad72,7d5411951faa81c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:39:29.666 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:39:29.669 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:39:29.670 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:39:29.670 INFO 2981023 --- [ocp-schedules-14,16969fb155a2ad72,7d5411951faa81c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:39:29.672 INFO 2981023 --- [ocp-schedules-14,16969fb155a2ad72,7d5411951faa81c7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:39:30.576 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:30.576 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:31.511 INFO 2981023 --- [ocp-async-8,612aac568a3f173a,d59bcb52afdea645] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,7ad6212289cc837b,a5d1f1eab3d341e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:32.316 INFO 2981023 --- [prometheus-minute-schedule-1,7ad6212289cc837b,a5d1f1eab3d341e7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:32.800 INFO 2981023 --- [ocp-schedules-14,2fd95121395b75aa,e1c3ba86421dcfb7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:39:32.822 WARN 2981023 --- [scanner-13709369-1,2fd95121395b75aa,84e43acef1b614f4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:39:33.043 INFO 2981023 --- [ocp-schedules-9,d898724761aa26ac,ca15373c462f99f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:39:33.511 INFO 2981023 --- [ocp-async-10,6396785912b537f3,8ee80a1c79eec898] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:33.577 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:33.577 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,9bd3b1bdfdd21d7e,38d5037e174c722a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,9bd3b1bdfdd21d7e,38d5037e174c722a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:35.511 INFO 2981023 --- [ocp-async-6,e84434dc91cb223e,bb68c4aa0da03a76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:36.338 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:39:36.337679+08:00, lastProcessTime=2024-09-11T18:39:26.337349+08:00 2024-09-11 18:39:36.462 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:39:36.337679+08:00 2024-09-11 18:39:36.577 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:36.578 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:36.710 INFO 2981023 --- [ocp-schedules-13,8352c009ca8f9028,4ec214bfdec93eec] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:39:36.715 INFO 2981023 --- [ocp-schedules-13,8352c009ca8f9028,4ec214bfdec93eec] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:39:36.715 INFO 2981023 --- [ocp-schedules-13,8352c009ca8f9028,4ec214bfdec93eec] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:39:37.314 INFO 2981023 --- [prometheus-cache-monitor1,89f952c14368bf6e,abe8e81afc09bf0e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:39:37.315 INFO 2981023 --- [prometheus-cache-monitor1,89f952c14368bf6e,abe8e81afc09bf0e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:39:37.517 INFO 2981023 --- [ocp-async-4,c8400db77298aa37,25dfb53b59d25d57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:37.806 INFO 2981023 --- [ocp-async-3,0c91769827e75fd1,6d797220762a8984] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:39:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,e87d6b9dd2bda411,e9884001e57e3fcf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,e87d6b9dd2bda411,e9884001e57e3fcf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:39.517 INFO 2981023 --- [ocp-async-10,10f0a28ceade1676,87429438bc39a53d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:39.578 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:39.578 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:40.002 INFO 2981023 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:39:40.002 INFO 2981023 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:39:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,442873ebfcd6a56c,cd0ec247f5e12350] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,442873ebfcd6a56c,cd0ec247f5e12350] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:41.517 INFO 2981023 --- [ocp-async-1,950a57ca12e88c9d,c0441e40bc3a753f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:42.578 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:42.579 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:42.992 INFO 2981023 --- [ocp-schedules-9,db62243754bdee1b,97c8c4ec28bf5bf3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:39:43.043 WARN 2981023 --- [scanner-13709625-1,db62243754bdee1b,3612f6e7618ffe7b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:39:43.047 INFO 2981023 --- [ocp-schedules-6,24af5ba4624e7da3,e1f93f912cc8006b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:39:43.051 INFO 2981023 --- [alarm-detect-18,db62243754bdee1b,de72ad9dea6e8d31] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646963, data is empty. 2024-09-11 18:39:43.057 INFO 2981023 --- [alarm-detect-18,db62243754bdee1b,de72ad9dea6e8d31] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646963, data is empty. 2024-09-11 18:39:43.063 INFO 2981023 --- [alarm-detect-18,db62243754bdee1b,de72ad9dea6e8d31] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646963, data is empty. 2024-09-11 18:39:43.085 INFO 2981023 --- [alarm-detect-9,db62243754bdee1b,0bffc18ce5f77136] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646963, data is empty. 2024-09-11 18:39:43.090 INFO 2981023 --- [alarm-detect-11,db62243754bdee1b,af442f9b23f591de] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646963, data is empty. 2024-09-11 18:39:43.094 INFO 2981023 --- [alarm-detect-9,db62243754bdee1b,0bffc18ce5f77136] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646963, data is empty. 2024-09-11 18:39:43.095 INFO 2981023 --- [alarm-detect-11,db62243754bdee1b,af442f9b23f591de] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646963, data is empty. 2024-09-11 18:39:43.101 INFO 2981023 --- [alarm-detect-11,db62243754bdee1b,af442f9b23f591de] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646963, data is empty. 2024-09-11 18:39:43.101 INFO 2981023 --- [alarm-detect-9,db62243754bdee1b,0bffc18ce5f77136] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646963, data is empty. 2024-09-11 18:39:43.199 INFO 2981023 --- [alarm-detect-17,db62243754bdee1b,da007b1ed0b899af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646963, data is empty. 2024-09-11 18:39:43.203 INFO 2981023 --- [alarm-detect-17,db62243754bdee1b,da007b1ed0b899af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646963, data is empty. 2024-09-11 18:39:43.209 INFO 2981023 --- [alarm-detect-17,db62243754bdee1b,da007b1ed0b899af] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646963, data is empty. 2024-09-11 18:39:43.517 INFO 2981023 --- [ocp-async-2,54f803582bd517c4,a30868b9acc4fa70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:44.313 INFO 2981023 --- [prometheus-minute-schedule-1,f866a7fb7ac5f668,af88c8039c828dd1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:44.316 INFO 2981023 --- [prometheus-minute-schedule-1,f866a7fb7ac5f668,af88c8039c828dd1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:45.101 INFO 2981023 --- [prometheus-second-schedule-1,b475cf403f68b227,fe14ce9a79d790b2] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=8, collectTaskCount=8 2024-09-11 18:39:45.517 INFO 2981023 --- [ocp-async-8,dd6a9ed3c1127bc6,081219d71eb2fa84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:45.579 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:45.579 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:45.997 INFO 2981023 --- [ocp-schedules-3,98a32060ae240513,a6b98e72183cbdf5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:39:45.998 INFO 2981023 --- [host-check-clockdiff1,98a32060ae240513,54ca356bde92b039] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:39:45.998 INFO 2981023 --- [host-check-clockdiff2,98a32060ae240513,f29a355d5a29f4b2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:39:45.998 INFO 2981023 --- [host-check-clockdiff4,98a32060ae240513,34512c72f0020f5e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:39:45.999 INFO 2981023 --- [host-check-clockdiff3,98a32060ae240513,a8b7834d2716047e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:39:46.007 INFO 2981023 --- [ocp-schedules-1,3a2105953d9ecd2d,e6860d2c71a58504] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:39:46.007 INFO 2981023 --- [ocp-schedules-4,4cf1270535fc8540,9ec1cf51f664496f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:39:46.007 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:39:46.007 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:39:46.008 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:39:46.008 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:39:46.011 INFO 2981023 --- [host-check-clockdiff3,98a32060ae240513,a8b7834d2716047e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051186 0 0 , error= 2024-09-11 18:39:46.012 INFO 2981023 --- [host-check-agent-status1,4cf1270535fc8540,3753fe78c79a6b0b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:39:46.012 INFO 2981023 --- [host-check-agent-status2,4cf1270535fc8540,997d57f7b3502e59] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:39:46.012 INFO 2981023 --- [host-check-agent-status3,4cf1270535fc8540,2d52d51dec826ae5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:39:46.012 INFO 2981023 --- [host-check-agent-status4,4cf1270535fc8540,afbf887ed03df6bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:39:46.013 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:39:46.014 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:39:46.016 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:39:46.016 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:39:46.017 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:39:46.017 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:39:46.018 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:39:46.019 INFO 2981023 --- [host-check-agent-status1,4cf1270535fc8540,3753fe78c79a6b0b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:39:46.020 INFO 2981023 --- [host-check-agent-status1,4cf1270535fc8540,3753fe78c79a6b0b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:39:46.020 INFO 2981023 --- [host-check-agent-status2,4cf1270535fc8540,997d57f7b3502e59] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:39:46.021 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:39:46.021 INFO 2981023 --- [host-check-agent-status4,4cf1270535fc8540,afbf887ed03df6bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:39:46.021 INFO 2981023 --- [host-check-agent-status2,4cf1270535fc8540,997d57f7b3502e59] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:39:46.022 INFO 2981023 --- [host-check-agent-status3,4cf1270535fc8540,2d52d51dec826ae5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:39:46.023 INFO 2981023 --- [host-check-agent-status4,4cf1270535fc8540,afbf887ed03df6bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:39:46.027 INFO 2981023 --- [host-check-agent-status3,4cf1270535fc8540,2d52d51dec826ae5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:39:46.045 INFO 2981023 --- [host-check-agent-status1,4cf1270535fc8540,3753fe78c79a6b0b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:39:46.045 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:39:46.046 INFO 2981023 --- [host-check-agent-status2,4cf1270535fc8540,997d57f7b3502e59] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:39:46.048 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:39:46.048272602+08:00 2024-09-11 18:39:46.048 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:39:46.048 INFO 2981023 --- [host-check-agent-status1,4cf1270535fc8540,3753fe78c79a6b0b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:39:46.048 INFO 2981023 --- [host-check-agent-status1,4cf1270535fc8540,3753fe78c79a6b0b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:39:46.048 INFO 2981023 --- [host-check-agent-status1,4cf1270535fc8540,3753fe78c79a6b0b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:39:46.048 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:39:46.049 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:39:46.049114279+08:00 2024-09-11 18:39:46.049 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:39:46.049+08:00 2024-09-11 18:39:46.049 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:39:46.048+08:00 2024-09-11 18:39:46.050 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:39:46.052 INFO 2981023 --- [host-check-agent-status2,4cf1270535fc8540,997d57f7b3502e59] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:39:46.052 INFO 2981023 --- [host-check-agent-status2,4cf1270535fc8540,997d57f7b3502e59] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:39:46.052 INFO 2981023 --- [host-check-agent-status2,4cf1270535fc8540,997d57f7b3502e59] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:39:46.053 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:39:46.048+08:00 and status AVAILABLE 2024-09-11 18:39:46.053 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:39:46.049+08:00 and status AVAILABLE 2024-09-11 18:39:46.053 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:39:46.055688696+08:00 2024-09-11 18:39:46.054 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:39:46.053+08:00 2024-09-11 18:39:46.055 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:39:46.054085783+08:00 2024-09-11 18:39:46.056 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:39:46.053+08:00 and status AVAILABLE 2024-09-11 18:39:46.056 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:39:46.055+08:00 2024-09-11 18:39:46.056 INFO 2981023 --- [host-check-agent-status4,4cf1270535fc8540,afbf887ed03df6bd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:39:46.057 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:39:46.055+08:00 and status AVAILABLE 2024-09-11 18:39:46.058 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:39:46.058 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:39:46.059 INFO 2981023 --- [host-check-agent-status3,4cf1270535fc8540,2d52d51dec826ae5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:39:46.060 INFO 2981023 --- [host-check-agent-status4,4cf1270535fc8540,afbf887ed03df6bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:39:46.060 INFO 2981023 --- [host-check-agent-status4,4cf1270535fc8540,afbf887ed03df6bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:39:46.060 INFO 2981023 --- [host-check-agent-status4,4cf1270535fc8540,afbf887ed03df6bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:39:46.061 INFO 2981023 --- [host-check-agent-status3,4cf1270535fc8540,2d52d51dec826ae5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:39:46.061 INFO 2981023 --- [host-check-agent-status3,4cf1270535fc8540,2d52d51dec826ae5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:39:46.061 INFO 2981023 --- [host-check-agent-status3,4cf1270535fc8540,2d52d51dec826ae5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:39:46.062 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:39:46.065 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:39:46.065 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:39:46.065 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:39:46.066 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:39:46.066 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:39:46.070 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:39:46.072 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:39:46.073 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:39:46.076 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:39:46.093 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:39:46.094 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:39:46.096 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:39:46.110 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:39:46.118 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:39:46.126 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:39:46.132 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:39:46.134 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:39:46.141 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:39:46.143 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:39:46.150 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:39:46.157 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:39:46.157 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-09-11 18:39:46.160 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:39:46.164 INFO 2981023 --- [ocp-schedules-8,01c2a718707f5dd8,71519c0f4ed26dea] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-09-11 18:39:46.173 INFO 2981023 --- [ocp-schedules-8,01c2a718707f5dd8,71519c0f4ed26dea] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-09-11 18:39:46.177 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-09-11 18:39:46.187 INFO 2981023 --- [ocp-schedules-15,4a1d8642be6cb68f,7f1e5793f6dd1b62] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:39:46.187 INFO 2981023 --- [ocp-schedules-18,daf529ea9f0bf1c5,098c1e9251937ba9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:39:46.188 INFO 2981023 --- [ocp-schedules-21,71260ee78dcd411d,a97dcea03ec0e174] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:39:46.189 INFO 2981023 --- [ocp-schedules-18,daf529ea9f0bf1c5,098c1e9251937ba9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:39:46.191 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:39:46.193 INFO 2981023 --- [ocp-schedules-24,a8695e8c15c1ad64,5aedc5d9e680645c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:39:46.193 INFO 2981023 --- [ocp-schedules-15,4a1d8642be6cb68f,7f1e5793f6dd1b62] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:39:46.195 INFO 2981023 --- [ocp-schedules-21,71260ee78dcd411d,a97dcea03ec0e174] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:39:46.209 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:39:46.209 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:39:46.210 INFO 2981023 --- [ocp-schedules-18,daf529ea9f0bf1c5,098c1e9251937ba9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:39:46.210 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051083504000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:39:46.217 INFO 2981023 --- [ocp-schedules-14,6db171f11bbd04b0,22bfd04d756b8c2a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:39:46.221 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:39:46.238 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:39:46.253 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:39:46.263 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.264 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:39:46.265 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:39:46.265 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.275 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:39:46.282 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.283 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [1] 2024-09-11 18:39:46.283 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.286 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:39:46.287 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:46.287 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:39:46.288 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:39:46.290 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.291 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:39:46.291 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:39:46.292 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:39:46.292 INFO 2981023 --- [ocp-schedules-15,4a1d8642be6cb68f,7f1e5793f6dd1b62] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:39:46.295 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:39:46.297 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:39:46.298 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.299 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:46.306 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.307 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:46.309 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:39:46.316 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:39:46.337 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:39:46.351 INFO 2981023 --- [host-check-host-status2,3a2105953d9ecd2d,5e8314201d75818e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:39:46.353 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:39:46.353 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051083504000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:39:46.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:39:46.353721+08:00, lastProcessTime=2024-09-11T18:39:36.337679+08:00 2024-09-11 18:39:46.357 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.359 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.359 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:39:46.364 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:39:46.370 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.377 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:39:46.377 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.378 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:39:46.379 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:39:46.380 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:39:46.380 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.380 INFO 2981023 --- [host-check-host-status1,3a2105953d9ecd2d,0cb66bf68c488828] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:39:46.399 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.404 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:39:46.404 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.407 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-09-11 18:39:46.407 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:39:46.416 INFO 2981023 --- [host-check-host-status4,3a2105953d9ecd2d,8eb35b2786a92387] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:39:46.418 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=1, failedTasks=[] 2024-09-11 18:39:46.420 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.421 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.421 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:46.422 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:39:46.432 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:39:46.433 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.433 INFO 2981023 --- [host-check-host-status3,3a2105953d9ecd2d,957521694aee67e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:39:46.435 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:39:46.438 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:39:46.444 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.445 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.445 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.445 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:39:46.446 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:39:46.446 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.449 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:39:46.450 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:39:46.450 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:39:46.450 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:39:46.455 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.455 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.456 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-09-11 18:39:46.456 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-09-11 18:39:46.457 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.457 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:39:46.457 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:39:46.458 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.459 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:39:46.459 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:39:46.464 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-09-11 18:39:46.467 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:39:46.470 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:39:46.479 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.479 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.479 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.480 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:39:46.480 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:39:46.483 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:39:46.484 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.487 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:39:46.487 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-09-11 18:39:46.488 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:39:46.491 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:39:46.492 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:39:46.497 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:39:46.503 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:39:46.504 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:39:46.505 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:39:46.506 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.507 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.508 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:39:46.508 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:39:46.508 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.519 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:39:46.522 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-09-11 18:39:46.535 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:39:46.545 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-09-11 18:39:46.547 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.547 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.548 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:39:46.555 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:39:46.556 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050612665000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:39:46.562 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.562 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.563 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-09-11 18:39:46.563 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +00:00 2024-09-11 18:39:46.563 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.593 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:39:46.601 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:39:46.604 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:46.605 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:39:46.611 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:39:46.611 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726050589296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:39:46.612 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [1] 2024-09-11 18:39:46.612 INFO 2981023 --- [ocp-schedules-21,71260ee78dcd411d,a97dcea03ec0e174] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:39:46.616 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:39:46.353721+08:00 2024-09-11 18:39:46.627 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=1, tooOldTasks=[] 2024-09-11 18:39:46.633 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.b.i.alarm.BackupAlarmService : cluster=rtdob does not enable secondary backup 2024-09-11 18:39:46.636 INFO 2981023 --- [ocp-schedules-16,6c496b77320d166a,d16aad9eef24ae28] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///obbackup/], metricValues size:[0]. 2024-09-11 18:39:46.850 INFO 2981023 --- [ocp-schedules-6,12f9f2b63a4577b4,96ae5540a62e0bc1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:39:46.850 INFO 2981023 --- [ocp-schedules-6,12f9f2b63a4577b4,96ae5540a62e0bc1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:39:46.944 INFO 2981023 --- [host-check-clockdiff2,98a32060ae240513,f29a355d5a29f4b2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051186 5 5 , error= 2024-09-11 18:39:47.313 INFO 2981023 --- [prometheus-minute-schedule-1,8218d1f9c4066464,fff92fd599fb2126] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:47.316 INFO 2981023 --- [prometheus-minute-schedule-1,8218d1f9c4066464,fff92fd599fb2126] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:47.517 INFO 2981023 --- [ocp-async-6,56401fbbc3ae794c,b650d0ad7eaecdc9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:47.806 INFO 2981023 --- [ocp-async-7,c40e0380e5f827f8,fc669eef2ad12dc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:39:48.579 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:48.580 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:49.295 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9ed97aa220eb13b2,818201af81391df5] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=linkprdocp&version=2&ObClusterId=1719994437, client=10.31.35.10, traceId=9ed97aa220eb13b2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=linkprdocp,1719994437,2,{"ObRegion":"linkprdocp","ObCluster":"linkprdocp","ObRegionId":1719994437,"ObClusterId":1719994437,"Type":"PRIMARY","timestamp":1726051189294484,"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:39:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9ed97aa220eb13b2,818201af81391df5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=linkprdocp, obClusterId=1719994437, rootServerJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051189296000}, groupName=linkprdocp, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:39:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9ed97aa220eb13b2,818201af81391df5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:39:49.303 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9ed97aa220eb13b2,818201af81391df5] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=linkprdocp, clusterId=1719994437, affectRows=1 2024-09-11 18:39:49.304 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-9,9ed97aa220eb13b2,818201af81391df5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.10, traceId=9ed97aa220eb13b2, duration=10 ms] 2024-09-11 18:39:49.517 INFO 2981023 --- [ocp-async-2,8a7cb863ce52b44c,a7692fe533b79708] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:49.905 INFO 2981023 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:39:49.910 INFO 2981023 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-09-11 18:39:50.150 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-09-11 18:39:50.150 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-09-11 18:39:50.153 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:39:50.153 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:39:50.153 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:39:50.153 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:39:50.157 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-09-11 18:39:50.157 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : init spring 2024-09-11 18:39:50.157 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-09-11 18:39:50.157 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : starting 2024-09-11 18:39:50.205 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:39:50.212 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:39:50.212 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:39:50.212 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:39:50.212 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:39:50.212 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:39:50.218 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:39:50.251 INFO 2981023 --- [ocp-schedules-30,41c1b75603e0a4ea,519e6525f3314d28] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:39:50.281 WARN 2981023 --- [ocp-schedules-30,41c1b75603e0a4ea,519e6525f3314d28] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:39:50.332 INFO 2981023 --- [prometheus-minute-schedule-1,c0ac5fbfaae8a481,e46d2a2abd4060b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:50.336 INFO 2981023 --- [prometheus-minute-schedule-1,c0ac5fbfaae8a481,e46d2a2abd4060b6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:50.413 INFO 2981023 --- [ocp-schedules-24,d8a3c5dd9dc61015,b6190bec98ab4f69] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-09-11 18:39:50.519 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-09-11 18:39:50.781 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.624407987S 2024-09-11 18:39:50.781 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-09-11 18:39:50.833 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-09-11 18:39:50.833 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-09-11 18:39:50.848 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-09-11 18:39:50.878 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-09-11 18:39:50.878 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-09-11 18:39:50.878 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-09-11 18:39:50.878 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.1 2024-09-11 18:39:50.878 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.1 2024-09-11 18:39:50.878 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-09-11 18:39:50.884 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-09-11 18:39:50.961 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] o.s.boot.SpringApplication : Started application in 0.808 seconds (JVM running for 86990.834) 2024-09-11 18:39:50.961 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.808751157S 2024-09-11 18:39:51.178 INFO 2981023 --- [ocp-schedules-1,d839be4f46413377,aa451881b7592c87] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-09-11 18:39:51.287 INFO 2981023 --- [host-check-clockdiff1,98a32060ae240513,54ca356bde92b039] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051191 0 0 , error= 2024-09-11 18:39:51.518 INFO 2981023 --- [ocp-async-8,0849e16d0f27e06a,008c5b93c626557d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:51.580 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:51.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:51.667 INFO 2981023 --- [ocp-schedules-12,0077b5ae64457036,6a14bf6dadccbc4a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-09-11 18:39:51.668 INFO 2981023 --- [ocp-schedules-12,0077b5ae64457036,6a14bf6dadccbc4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:39:51.668 INFO 2981023 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:39:51.669 INFO 2981023 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:39:51.669 INFO 2981023 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-09-11 18:39:51.669 INFO 2981023 --- [ocp-schedules-12,0077b5ae64457036,6a14bf6dadccbc4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:39:51.987 INFO 2981023 --- [host-check-clockdiff4,98a32060ae240513,34512c72f0020f5e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051191 0 0 , error= 2024-09-11 18:39:53.056 INFO 2981023 --- [ocp-schedules-4,64b6b36668df3f7d,2665c96de8e7d987] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:39:53.245 INFO 2981023 --- [ocp-schedules-21,9fe73f7c16d2ae71,47b9ce768d64a1b7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:39:53.299 WARN 2981023 --- [scanner-13709934-1,9fe73f7c16d2ae71,eec7bc084b24b6c0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:39:53.313 INFO 2981023 --- [prometheus-minute-schedule-1,f130a5e0a857c0ef,0d21aa60f8b985f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:53.320 INFO 2981023 --- [prometheus-minute-schedule-1,f130a5e0a857c0ef,0d21aa60f8b985f2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:53.459 INFO 2981023 --- [ocp-schedules-3,73a79c7c6809735a,01c69bf86f8054a1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-09-11 18:39:53.459 INFO 2981023 --- [ocp-schedules-3,73a79c7c6809735a,01c69bf86f8054a1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-09-11 18:39:53.518 INFO 2981023 --- [ocp-async-1,e784d84b1b4c356c,5a6dab924ab3fce3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:54.573 INFO 2981023 --- [ocp-schedules-26,13ecaf6d6c1d8847,e0e917ccf7b228c7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:39:54.581 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:54.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:55.247 INFO 2981023 --- [ocp-schedules-23,53184a4e3fd54dde,67f7390e5a8cf994] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-09-11 18:39:55.247 INFO 2981023 --- [ocp-schedules-23,53184a4e3fd54dde,67f7390e5a8cf994] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-09-11 18:39:55.251 INFO 2981023 --- [ocp-schedules-23,53184a4e3fd54dde,67f7390e5a8cf994] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-09-11 18:39:55.251 INFO 2981023 --- [ocp-schedules-23,53184a4e3fd54dde,67f7390e5a8cf994] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-09-11 18:39:55.251 INFO 2981023 --- [ocp-schedules-23,53184a4e3fd54dde,67f7390e5a8cf994] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-09-11 18:39:55.254 INFO 2981023 --- [ocp-schedules-23,53184a4e3fd54dde,67f7390e5a8cf994] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-09-11 18:39:55.254 INFO 2981023 --- [ocp-schedules-23,53184a4e3fd54dde,67f7390e5a8cf994] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-09-11 18:39:55.254 INFO 2981023 --- [ocp-schedules-23,53184a4e3fd54dde,67f7390e5a8cf994] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-09-11 18:39:55.265 INFO 2981023 --- [ocp-schedules-23,53184a4e3fd54dde,67f7390e5a8cf994] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-09-11 18:39:55.518 INFO 2981023 --- [ocp-async-9,1e8607b116d5805f,f43770eebc2a3822] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:55.985 INFO 2981023 --- [ocp-schedules-1,caf49a2a6318f0d8,f49f47a07ecb1719] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-09-11 18:39:55.985 INFO 2981023 --- [ocp-schedules-1,caf49a2a6318f0d8,f49f47a07ecb1719] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690422503666118656 2024-09-11 18:39:55.988 INFO 2981023 --- [ocp-schedules-1,caf49a2a6318f0d8,f49f47a07ecb1719] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-09-11 18:39:55.988 INFO 2981023 --- [ocp-schedules-1,caf49a2a6318f0d8,f49f47a07ecb1719] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690381659793813504 2024-09-11 18:39:55.990 INFO 2981023 --- [ocp-schedules-1,caf49a2a6318f0d8,f49f47a07ecb1719] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-09-11 18:39:55.990 INFO 2981023 --- [ocp-schedules-1,caf49a2a6318f0d8,f49f47a07ecb1719] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=690403629801435136 2024-09-11 18:39:56.121 INFO 2981023 --- [ocp-schedules-26,d1b8c0de5e1022c4,401f60ad03002a63] c.o.o.a.s.OcpAlarmDetectRuleService : alarmType binlog_node_status_abnormal detect rule does not effect, due to empty targets. 2024-09-11 18:39:56.139 WARN 2981023 --- [ocp-schedules-26,d1b8c0de5e1022c4,401f60ad03002a63] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class java.util.HashMap, json:{"name_en_us":Binlog Service Disk Quota Usage","name_zh_cn":"Binlog 服务磁盘配额使用率","name_zh_tw":"Binlog 服務磁盤配額使用率"} 2024-09-11 18:39:56.233 INFO 2981023 --- [ocp-schedules-21,4470eabac615fed2,833253ef97cf8de8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-09-11 18:39:56.239 INFO 2981023 --- [ocp-schedules-21,4470eabac615fed2,833253ef97cf8de8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-09-11 18:39:56.239 INFO 2981023 --- [ocp-schedules-21,4470eabac615fed2,833253ef97cf8de8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-09-11 18:39:56.313 INFO 2981023 --- [prometheus-minute-schedule-1,fdb80943c81b7ef0,7d7dc95900329225] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:56.316 INFO 2981023 --- [prometheus-minute-schedule-1,fdb80943c81b7ef0,7d7dc95900329225] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:56.351 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:39:56.350557+08:00, lastProcessTime=2024-09-11T18:39:46.353721+08:00 2024-09-11 18:39:56.483 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:39:56.350557+08:00 2024-09-11 18:39:56.719 INFO 2981023 --- [ocp-schedules-5,f6bd9596df9bd26f,b91f7606c2795395] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:39:56.728 INFO 2981023 --- [ocp-schedules-5,f6bd9596df9bd26f,b91f7606c2795395] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:39:56.728 INFO 2981023 --- [ocp-schedules-5,f6bd9596df9bd26f,b91f7606c2795395] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:39:57.518 INFO 2981023 --- [ocp-async-10,5f832f89a1456700,e5584cec9cc3acf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:57.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:39:57.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:39:57.806 INFO 2981023 --- [ocp-async-4,211715c035d275fb,f0990e03860c224a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:39:59.313 INFO 2981023 --- [prometheus-minute-schedule-1,6a0e8123f8096169,e494d80fc464465d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051200, interval=60 2024-09-11 18:39:59.316 INFO 2981023 --- [prometheus-minute-schedule-1,6a0e8123f8096169,e494d80fc464465d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:39:59.518 INFO 2981023 --- [ocp-async-1,11cf262cdff05b24,7c4592d14b70a24b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:39:59.661 INFO 2981023 --- [ocp-schedules-11,d4afd982b9665634,13395522b7b131cb] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-09-11 18:39:59.661 INFO 2981023 --- [ocp-schedules-11,d4afd982b9665634,13395522b7b131cb] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-09-11 18:39:59.683 INFO 2981023 --- [Collect all tenants major compactions1,d4afd982b9665634,af52ad797f6d1233] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:39:59.683 INFO 2981023 --- [Collect all tenants major compactions4,d4afd982b9665634,a6a609d3d68ac636] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:39:59.684 INFO 2981023 --- [Collect all tenants major compactions3,d4afd982b9665634,53e452c7eb9088cc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:39:59.685 INFO 2981023 --- [Collect all tenants major compactions2,d4afd982b9665634,b59fe02413a1a5c2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:39:59.685 INFO 2981023 --- [Collect all tenants major compactions5,d4afd982b9665634,de8e240154adbdc6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:39:59.689 INFO 2981023 --- [Collect all tenants major compactions6,d4afd982b9665634,225894ec98f19036] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:39:59.705 INFO 2981023 --- [Collect all tenants major compactions1,d4afd982b9665634,af52ad797f6d1233] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:59.706 INFO 2981023 --- [Collect all tenants major compactions1,d4afd982b9665634,af52ad797f6d1233] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2024-09-11 18:39:59.706 INFO 2981023 --- [Collect all tenants major compactions3,d4afd982b9665634,53e452c7eb9088cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:59.707 INFO 2981023 --- [Collect all tenants major compactions4,d4afd982b9665634,a6a609d3d68ac636] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:59.708 INFO 2981023 --- [Collect all tenants major compactions5,d4afd982b9665634,de8e240154adbdc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:59.708 INFO 2981023 --- [Collect all tenants major compactions3,d4afd982b9665634,53e452c7eb9088cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:39:59.708 INFO 2981023 --- [Collect all tenants major compactions4,d4afd982b9665634,a6a609d3d68ac636] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:39:59.709 INFO 2981023 --- [Collect all tenants major compactions6,d4afd982b9665634,225894ec98f19036] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:59.710 INFO 2981023 --- [Collect all tenants major compactions2,d4afd982b9665634,b59fe02413a1a5c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:39:59.711 INFO 2981023 --- [Collect all tenants major compactions6,d4afd982b9665634,225894ec98f19036] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:39:59.711 INFO 2981023 --- [Collect all tenants major compactions2,d4afd982b9665634,b59fe02413a1a5c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-09-11 18:39:59.711 INFO 2981023 --- [Collect all tenants major compactions5,d4afd982b9665634,de8e240154adbdc6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-09-11 18:39:59.712 INFO 2981023 --- [Collect all tenants major compactions3,d4afd982b9665634,53e452c7eb9088cc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991203275667531, broadcast scn:1725991203275667531, saved frozen scn:1725991203275667531 2024-09-11 18:39:59.712 INFO 2981023 --- [Collect all tenants major compactions1,d4afd982b9665634,af52ad797f6d1233] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991200511587037, broadcast scn:1725991200511587037, saved frozen scn:1725991200511587037 2024-09-11 18:39:59.712 INFO 2981023 --- [Collect all tenants major compactions4,d4afd982b9665634,a6a609d3d68ac636] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020000831647000, broadcast scn:1726020000831647000, saved frozen scn:1726020000831647000 2024-09-11 18:39:59.713 INFO 2981023 --- [Collect all tenants major compactions6,d4afd982b9665634,225894ec98f19036] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726039307729952000, broadcast scn:1726039307729952000, saved frozen scn:1726039307729952000 2024-09-11 18:39:59.714 INFO 2981023 --- [Collect all tenants major compactions2,d4afd982b9665634,b59fe02413a1a5c2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1725991201274736911, broadcast scn:1725991201274736911, saved frozen scn:1725991201274736911 2024-09-11 18:39:59.715 INFO 2981023 --- [Collect all tenants major compactions5,d4afd982b9665634,de8e240154adbdc6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1726020001489757000, broadcast scn:1726020001489757000, saved frozen scn:1726020001489757000 2024-09-11 18:39:59.716 INFO 2981023 --- [Collect all tenants major compactions4,d4afd982b9665634,a6a609d3d68ac636] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:linkreit 2024-09-11 18:39:59.718 INFO 2981023 --- [Collect all tenants major compactions3,d4afd982b9665634,53e452c7eb9088cc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:sys 2024-09-11 18:39:59.719 INFO 2981023 --- [Collect all tenants major compactions1,d4afd982b9665634,af52ad797f6d1233] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_meta 2024-09-11 18:39:59.720 INFO 2981023 --- [Collect all tenants major compactions5,d4afd982b9665634,de8e240154adbdc6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:rtdob, tenant:sys 2024-09-11 18:39:59.722 INFO 2981023 --- [Collect all tenants major compactions2,d4afd982b9665634,b59fe02413a1a5c2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkprdocp, tenant:ocp_monitor 2024-09-11 18:39:59.723 INFO 2981023 --- [Collect all tenants major compactions6,d4afd982b9665634,225894ec98f19036] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:linkrtd, tenant:sys 2024-09-11 18:39:59.723 INFO 2981023 --- [ocp-schedules-11,d4afd982b9665634,13395522b7b131cb] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-09-11 18:40:00.290 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:35:01.0, nextRun=2024-09-11 18:40:00.0, currentDate=Wed Sep 11 18:40:00 HKT 2024 2024-09-11 18:40:00.290 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:35:01.0, nextRunTime=2024-09-11 18:40:00.0} 2024-09-11 18:40:00.293 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-09-11 18:40:00.293 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:40:00.293 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:40:00.293+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:40:00.307 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:35:01.0, nextRun=2024-09-11 18:40:00.0, currentDate=Wed Sep 11 18:40:00 HKT 2024 2024-09-11 18:40:00.307 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-09-11 18:35:01.0, nextRunTime=2024-09-11 18:40:00.0} 2024-09-11 18:40:00.319 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2024-09-11 18:40:00.341 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2024-09-11 18:40:00.341 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2024-09-11 18:40:00.341 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:40:00.341 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,89fe332d18997e44] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2024-09-11T18:40:00.341+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:40:00.582 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:00.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:01.245 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,628805e86df645e9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048258, operation:EXECUTE, state: RUNNING 2024-09-11 18:40:01.359 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,628805e86df645e9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048259, operation:EXECUTE, state: RUNNING 2024-09-11 18:40:01.417 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048258, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048228, task_operation=execute, latest_execution_start_time=2024-09-11T18:40:01.227+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=4048258}, listMap={}}, executor=10.31.35.10 2024-09-11 18:40:01.421 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048259, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=4048229, task_operation=execute, latest_execution_start_time=2024-09-11T18:40:01.357+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=4048259}, listMap={cluster_ids=[[1,2]]}}, executor=10.31.35.10 2024-09-11 18:40:01.423 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2024-09-11 18:40:01.436 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:40:01.448 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1002,1726012800000000,2000,30000000 2024-09-11 18:40:01.450 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2024-09-11 18:40:01.463 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1719994437,linkprdocp,1004,1726012800000000,2000,30000000 2024-09-11 18:40:01.464 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2024-09-11 18:40:01.464 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2024-09-11 18:40:01.477 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:40:01.487 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1718689631,rtdob,1004,1726012800000000,2000,30000000 2024-09-11 18:40:01.506 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=linkreit, obTenantId=1004, status=NORMAL} 2024-09-11 18:40:01.518 INFO 2981023 --- [ocp-async-9,bb690b8e54a18618,cb5b25f28f9cfa7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:01.523 INFO 2981023 --- [subtask-executor31,ddd72be6e49ae268,49b45ddcab51f087] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048259, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:40:01.578 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:40:01.597 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:40:01.621 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:40:01.633 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for linkprdocp 2024-09-11 18:40:01.633 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1261.2542153744503], b:[-2.1723439609063435E12] 2024-09-11 18:40:01.639 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:40:01.742 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:40:01.793 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkprdocp 2024-09-11 18:40:01.813 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:40:01.831 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:40:01.888 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:40:01.900 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:40:01.915 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:40:01.922 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for rtdob 2024-09-11 18:40:01.922 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.ocp.diagnosis.model.NeuralNetwork : W:[3395.459579986], b:[-5.807309276841022E12] 2024-09-11 18:40:01.927 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:40:01.989 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:40:02.030 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048229, status:SUCCESS 2024-09-11 18:40:02.031 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:rtdob 2024-09-11 18:40:02.033 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048229, status:SUCCESS 2024-09-11 18:40:02.045 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:40:02.052 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:40:02.062 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:40:02.070 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:40:02.084 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:40:02.086 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for linkrtd 2024-09-11 18:40:02.087 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:40:02.102 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:40:02.112 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:linkrtd 2024-09-11 18:40:02.124 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,c0731c6c18f1339a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048258, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:40:02.313 INFO 2981023 --- [prometheus-minute-schedule-1,10fd7f1eebb197a1,a493ad717635f66c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:02.316 INFO 2981023 --- [prometheus-minute-schedule-1,10fd7f1eebb197a1,a493ad717635f66c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:02.680 INFO 2981023 --- [ocp-schedules-14,0c86c40b9ab2024f,7b9ec443c1e8f9e2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-09-11T18:40:02.680+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:40:02.683 INFO 2981023 --- [ocp-schedules-14,0c86c40b9ab2024f,7b9ec443c1e8f9e2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkprdocp for slowSqlStat, sql count is 0. 2024-09-11 18:40:02.795 INFO 2981023 --- [ocp-schedules-14,0c86c40b9ab2024f,7b9ec443c1e8f9e2] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for rtdob, save 50 to SqlAggregateData 2024-09-11 18:40:02.796 INFO 2981023 --- [ocp-schedules-14,0c86c40b9ab2024f,7b9ec443c1e8f9e2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster linkrtd for slowSqlStat, sql count is 0. 2024-09-11 18:40:03.044 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048228, status:SUCCESS 2024-09-11 18:40:03.046 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048228, status:SUCCESS 2024-09-11 18:40:03.059 INFO 2981023 --- [ocp-schedules-28,77b4d8e10356b955,4399a147f1b977f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:40:03.449 INFO 2981023 --- [ocp-schedules-3,5e5f00153676339d,89bed1c7417796c0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:40:03.515 INFO 2981023 --- [ocp-schedules-6,c2a5071de8225613,da1713822215c2a9] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-09-11 18:40:03.516 INFO 2981023 --- [ocp-schedules-6,c2a5071de8225613,da1713822215c2a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:40:03.518 INFO 2981023 --- [ocp-async-8,b58f9985e4aea2e3,7922a0d48a6cc9ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:03.522 WARN 2981023 --- [scanner-13710345-1,5e5f00153676339d,9edcdd6dcb51d5a3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:40:03.553 INFO 2981023 --- [alarm-detect-15,5e5f00153676339d,6de336e1aca39d7e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8646963, data is empty. 2024-09-11 18:40:03.558 INFO 2981023 --- [alarm-detect-15,5e5f00153676339d,6de336e1aca39d7e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8647013, data is empty. 2024-09-11 18:40:03.566 INFO 2981023 --- [alarm-detect-15,5e5f00153676339d,6de336e1aca39d7e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8647013, data is empty. 2024-09-11 18:40:03.583 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:03.584 INFO 2981023 --- [pool-12-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:03.589 INFO 2981023 --- [alarm-detect-7,5e5f00153676339d,d06f069a1730fabb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8646963, data is empty. 2024-09-11 18:40:03.589 INFO 2981023 --- [alarm-detect-16,5e5f00153676339d,a7baa8cda8184bce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8646963, data is empty. 2024-09-11 18:40:03.593 INFO 2981023 --- [alarm-detect-16,5e5f00153676339d,a7baa8cda8184bce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8647013, data is empty. 2024-09-11 18:40:03.594 INFO 2981023 --- [alarm-detect-7,5e5f00153676339d,d06f069a1730fabb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8647013, data is empty. 2024-09-11 18:40:03.599 INFO 2981023 --- [alarm-detect-16,5e5f00153676339d,a7baa8cda8184bce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8647013, data is empty. 2024-09-11 18:40:03.599 INFO 2981023 --- [alarm-detect-7,5e5f00153676339d,d06f069a1730fabb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8647013, data is empty. 2024-09-11 18:40:03.615 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:03.616 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-09-11 18:40:03.624 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:03.626 INFO 2981023 --- [Refresh root service list3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.31.35.6, 2882] 2024-09-11 18:40:03.627 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : assembled rootserver_json={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051203627000} 2024-09-11 18:40:03.636 INFO 2981023 --- [Refresh root service list3,,] c.o.o.s.t.b.cluster.RefreshRsListTask : update rootserver_json success, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:40:03.643 INFO 2981023 --- [Refresh root service list3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by 2c6527982f, eventId=1000244, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:40:03.669 INFO 2981023 --- [ocp-schedules-6,c2a5071de8225613,da1713822215c2a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, tryCount=3, successCount=3 2024-09-11 18:40:03.718 INFO 2981023 --- [alarm-detect-2,5e5f00153676339d,ad1aa6f0a945b854] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8646963, data is empty. 2024-09-11 18:40:03.723 INFO 2981023 --- [alarm-detect-2,5e5f00153676339d,ad1aa6f0a945b854] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8647013, data is empty. 2024-09-11 18:40:03.728 INFO 2981023 --- [alarm-detect-2,5e5f00153676339d,ad1aa6f0a945b854] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8647013, data is empty. 2024-09-11 18:40:03.827 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [linkrtd] cache 2024-09-11 18:40:03.827 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2024-09-11 18:40:03.827 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db 2024-09-11 18:40:03.832 INFO 2981023 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster linkrtd cache info from db success. 2024-09-11 18:40:03.833 INFO 2981023 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by 2c6527982f, eventId=1000244, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["linkrtd"]} 2024-09-11 18:40:04.495 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,dfaa7044973649f2] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-09-11 18:30:05.0, nextRun=2024-09-11 18:40:04.0, currentDate=Wed Sep 11 18:40:04 HKT 2024 2024-09-11 18:40:04.495 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,dfaa7044973649f2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-09-11 18:30:05.0, nextRunTime=2024-09-11 18:40:04.0} 2024-09-11 18:40:04.497 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,dfaa7044973649f2] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-09-11 18:40:04.497 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,dfaa7044973649f2] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: null, current state: PENDING, new state: READY 2024-09-11 18:40:04.497 INFO 2981023 --- [task-coordinator2,ddd72be6e49ae268,dfaa7044973649f2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.31.35.10', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-09-11T18:40:04.497+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='ddd72be6e49ae268', operation=EXECUTE} 2024-09-11 18:40:04.554 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000244 2024-09-11 18:40:05.313 INFO 2981023 --- [prometheus-minute-schedule-1,43187608063d267e,c6c4a2bca2b6efca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:05.317 INFO 2981023 --- [prometheus-minute-schedule-1,43187608063d267e,c6c4a2bca2b6efca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:05.426 INFO 2981023 --- [subtask-coordinator2,ddd72be6e49ae268,efbeb71d16d1d65b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048260, operation:EXECUTE, state: RUNNING 2024-09-11 18:40:05.447 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=4048260, context=Context{parallelIdx=-1, stringMap={task_instance_id=4048230, task_operation=execute, latest_execution_start_time=2024-09-11T18:40:05.424+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=4048260}, listMap={}}, executor=10.31.35.10 2024-09-11 18:40:05.455 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-09-11 18:40:05.455 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-09-11 18:40:05.456 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.456 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:40:05.457 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.457 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.458 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.458 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-09-11 18:40:05.468 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.469 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.469 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.470 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-09-11 18:40:05.478 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.478 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.479 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.479 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-09-11 18:40:05.488 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.488 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.489 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.489 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-09-11 18:40:05.498 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.498 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.499 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.499 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-09-11 18:40:05.502 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.502 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.503 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.503 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-09-11 18:40:05.506 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.506 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.507 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.507 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-09-11 18:40:05.511 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.511 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.512 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.512 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-09-11 18:40:05.516 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.517 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.517 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.518 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-09-11 18:40:05.519 INFO 2981023 --- [ocp-async-6,20ffaaea0942c3d6,e8028596855e958f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:05.522 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.522 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.523 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.523 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-09-11 18:40:05.528 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.529 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.529 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.529 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-09-11 18:40:05.536 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.537 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.537 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.538 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-09-11 18:40:05.547 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.547 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.548 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.548 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-09-11 18:40:05.551 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.552 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.552 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.553 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-09-11 18:40:05.558 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.558 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.558 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.559 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-09-11 18:40:05.563 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.563 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.564 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.564 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-09-11 18:40:05.567 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.567 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.568 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.568 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-09-11 18:40:05.571 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.571 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.572 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.572 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-09-11 18:40:05.575 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.575 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.575 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.576 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-09-11 18:40:05.579 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.579 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.580 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.580 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-09-11 18:40:05.582 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.583 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.583 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.584 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-09-11 18:40:05.586 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.587 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.587 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.587 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-09-11 18:40:05.590 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.590 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.591 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.591 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-09-11 18:40:05.594 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.594 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.594 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.595 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-09-11 18:40:05.597 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.598 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.598 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.598 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-09-11 18:40:05.601 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.601 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.602 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.602 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-09-11 18:40:05.605 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.605 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.606 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.606 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-09-11 18:40:05.609 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.609 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.610 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.610 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-09-11 18:40:05.613 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.613 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.614 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.614 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-09-11 18:40:05.617 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.617 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.617 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.618 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-09-11 18:40:05.620 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.621 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.621 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.621 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-09-11 18:40:05.624 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.624 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.625 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.625 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-09-11 18:40:05.628 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.628 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.629 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.629 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-09-11 18:40:05.632 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.632 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.633 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.633 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-09-11 18:40:05.637 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.638 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.638 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.638 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-09-11 18:40:05.642 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.643 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.643 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.643 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-09-11 18:40:05.646 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.647 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.647 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.647 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-09-11 18:40:05.650 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.651 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.651 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.651 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-09-11 18:40:05.653 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.654 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.654 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.654 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-09-11 18:40:05.657 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.657 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.657 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.658 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-09-11 18:40:05.660 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.660 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.661 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.661 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-09-11 18:40:05.663 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.663 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.664 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.664 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-09-11 18:40:05.671 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.671 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.672 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.672 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-09-11 18:40:05.687 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.687 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.687 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.688 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-09-11 18:40:05.692 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.692 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.692 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.693 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-09-11 18:40:05.696 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.697 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.697 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.697 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-09-11 18:40:05.701 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.702 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.702 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.702 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-09-11 18:40:05.705 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.705 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.706 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.707 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-09-11 18:40:05.709 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.710 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.710 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.710 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-09-11 18:40:05.713 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-09-11 18:40:05.713 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-09-11 18:40:05.717 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.718 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:40:05.719 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.719 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.720 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.720 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-09-11 18:40:05.724 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.724 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:05.725 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.725 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-09-11 18:40:05.727 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-09-11 18:40:05.749 INFO 2981023 --- [subtask-executor30,ddd72be6e49ae268,987ae9d13df81691] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 4048260, operation:EXECUTE, state: SUCCESSFUL 2024-09-11 18:40:05.968 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:40:05.969 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:05.970 INFO 2981023 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:40:05.970 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2024-09-11 18:40:05.996 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=linkprdocp, obClusterId=1719994437, status=RUNNING} 2024-09-11 18:40:06.001 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=rtdob, obClusterId=1718689631, status=RUNNING} 2024-09-11 18:40:06.006 WARN 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster linkrtd(1000006), status=OPERATING 2024-09-11 18:40:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (linkprdocp:1719994437-1) 2024-09-11 18:40:06.006 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (rtdob:1718689631-1) 2024-09-11 18:40:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.009 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-09-11 18:40:06.010 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.010 INFO 2981023 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-09-11 18:40:06.049 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=rtdob:1718689631, server=1, level=0->1, c=0) - source=2024-09-11T18:37:41, start=2024-09-11T18:35:11, end=2024-09-11T18:37:41, updates=297, granule=120000ms, delay=150000ms 2024-09-11 18:40:06.055 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=0->1, c=1) - not enough cached interval (source=2024-09-11T18:37:41, target=2024-09-11T18:37:41, granule=120000ms, delay=0ms) 2024-09-11 18:40:06.055 INFO 2981023 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=rtdob:1718689631, server=1, level=1->2, c=0) - not enough cached interval (source=2024-09-11T18:37:41, target=2024-09-11T18:30:51, granule=600000ms, delay=410000ms) 2024-09-11 18:40:06.055 INFO 2981023 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=85ms, period=120000ms 2024-09-11 18:40:06.068 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4048230, status:SUCCESS 2024-09-11 18:40:06.071 INFO 2981023 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4048230, status:SUCCESS 2024-09-11 18:40:06.354 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:40:06.353210+08:00, lastProcessTime=2024-09-11T18:39:56.350557+08:00 2024-09-11 18:40:06.487 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:40:06.353210+08:00 2024-09-11 18:40:06.584 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:06.584 INFO 2981023 --- [ocp-schedules-4,9a6033c7db987c24,15bbeece7c47a8d1] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-09-11 18:40:06.586 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:06.586 INFO 2981023 --- [ocp-schedules-4,9a6033c7db987c24,15bbeece7c47a8d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:40:06.627 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:40:06.627 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.629 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:40:06.630 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:40:06.631 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:40:06.632 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.635 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:40:06.637 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.638 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.639 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.640 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.641 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.644 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.644 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.644 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-09-11 18:40:06.649 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.649 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:40:06.651 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.651 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.651 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.651 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.655 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.655 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:40:06.662 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.663 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:40:06.665 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.665 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.668 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.668 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:40:06.675 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.676 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.676 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.677 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:40:06.678 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.678 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:40:06.679 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.679 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:40:06.680 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000006 2024-09-11 18:40:06.681 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.682 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:40:06.682 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:40:06.683 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.684 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:40:06.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.685 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-09-11 18:40:06.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.686 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:40:06.687 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:40:06.687 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.687 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:40:06.688 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.689 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:40:06.689 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.690 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:40:06.690 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.691 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:40:06.691 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-09-11 18:40:06.693 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:40:06.697 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.697 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.698 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:40:06.698 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE7831D6D3-0-0] 2024-09-11 18:40:06.698 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:40:06.700 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.701 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:40:06.702 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkrtd, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:06.703 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.703 INFO 2981023 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.704 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:06.705 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.706 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-09-11 18:40:06.711 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:40:06.713 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-09-11 18:40:06.714 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.714 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:40:06.716 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000006, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:40:06.717 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000006, userWhitelist=% 2024-09-11 18:40:06.717 INFO 2981023 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000006 2024-09-11 18:40:06.717 INFO 2981023 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000006, clusterName=linkrtd, obClusterId=1719994443 2024-09-11 18:40:06.719 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:40:06.720 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-09-11 18:40:06.720 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.720 INFO 2981023 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.723 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.724 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:40:06.725 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:40:06.726 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-09-11 18:40:06.726 INFO 2981023 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-09-11 18:40:06.726 INFO 2981023 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:40:06.734 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.734 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:40:06.734 WARN 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] query failed, sql:[SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?], error message:[PreparedStatementCallback; bad SQL grammar [SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?]; nested exception is java.sql.SQLSyntaxErrorException: (conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist] 2024-09-11 18:40:06.735 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : Last Trace Info:[YB420A1F2305-00061CCE7831D6DD-0-0] 2024-09-11 18:40:06.735 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.o.tenant.MysqlTenantOperator : exception when append ob tenant, message=(conn=3221558883) Table 'oceanbase.CDB_OB_SERVICES' doesn't exist 2024-09-11 18:40:06.747 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.749 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-09-11 18:40:06.750 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.750 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:40:06.751 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.751 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:40:06.752 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.752 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-09-11 18:40:06.753 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.753 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-09-11 18:40:06.754 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.754 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-09-11 18:40:06.754 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2024-09-11 18:40:06.756 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.31.35.10 2024-09-11 18:40:06.765 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:06.766 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.766 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-09-11 18:40:06.775 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:40:06.777 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2024-09-11 18:40:06.780 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:06.812 INFO 2981023 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-09-11 18:40:06.824 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.31.35.10, userWhitelist=% 2024-09-11 18:40:06.825 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2024-09-11 18:40:06.825 INFO 2981023 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2024-09-11 18:40:06.825 INFO 2981023 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:40:06.825 INFO 2981023 --- [ocp-schedules-4,9a6033c7db987c24,15bbeece7c47a8d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, tryCount=3, successCount=3 2024-09-11 18:40:07.519 INFO 2981023 --- [ocp-async-7,b54b6cd1087b7057,aaba7db687329dc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:07.806 INFO 2981023 --- [ocp-async-8,9c6e594a4541a832,aa7f36ccd14bcbf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:40:08.313 INFO 2981023 --- [prometheus-minute-schedule-1,3cc8678c84797580,02c084ec97327e12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:08.316 INFO 2981023 --- [prometheus-minute-schedule-1,3cc8678c84797580,02c084ec97327e12] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:08.424 INFO 2981023 --- [ocp-schedules-25,58faaa533bd0d099,574148110da4fa1e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-09-11 18:40:08.504 INFO 2981023 --- [ocp-schedules-13,fa3654f7066514a9,d22a9c7b02279e26] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-09-11T18:40:08.504+08:00, clusters=[1, 2, 1000006] 2024-09-11 18:40:08.505 INFO 2981023 --- [ocp-schedules-13,fa3654f7066514a9,d22a9c7b02279e26] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkprdocp, trans stat result is empty. 2024-09-11 18:40:08.506 INFO 2981023 --- [ocp-schedules-13,fa3654f7066514a9,d22a9c7b02279e26] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster rtdob, trans stat result is empty. 2024-09-11 18:40:08.507 INFO 2981023 --- [ocp-schedules-13,fa3654f7066514a9,d22a9c7b02279e26] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster linkrtd, trans stat result is empty. 2024-09-11 18:40:09.519 INFO 2981023 --- [ocp-async-10,c1720635f9127aaf,6f8addf5d58cffe9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:09.586 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:09.586 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:10.003 INFO 2981023 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:40:10.004 INFO 2981023 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:40:10.917 INFO 2981023 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 9, ruleSize = 9. 2024-09-11 18:40:10.917 INFO 2981023 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 16, ruleSize = 16. 2024-09-11 18:40:11.313 INFO 2981023 --- [prometheus-minute-schedule-1,149983b8a2986cff,5d4f78d37b9ba63d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:11.317 INFO 2981023 --- [prometheus-minute-schedule-1,149983b8a2986cff,5d4f78d37b9ba63d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:11.519 INFO 2981023 --- [ocp-async-4,58c5b9cefe578a14,454fbcc9d94d3856] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:12.587 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:12.588 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:12.661 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1baed062960f550b,d6875f1e926650f0] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=rtdob&version=2&ObClusterId=1718689631, client=10.31.35.5, traceId=1baed062960f550b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=rtdob,1718689631,2,{"ObRegion":"rtdob","ObCluster":"rtdob","ObRegionId":1718689631,"ObClusterId":1718689631,"Type":"PRIMARY","timestamp":1726051212659353,"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-09-11 18:40:12.669 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1baed062960f550b,d6875f1e926650f0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=rtdob, obClusterId=1718689631, rootServerJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051212662000}, groupName=rtdob, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-09-11 18:40:12.669 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1baed062960f550b,d6875f1e926650f0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-09-11 18:40:12.669 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1baed062960f550b,d6875f1e926650f0] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=rtdob, clusterId=1718689631, affectRows=1 2024-09-11 18:40:12.670 INFO 2981023 --- [http-nio-0.0.0.0-8080-exec-4,1baed062960f550b,d6875f1e926650f0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.31.35.5, traceId=1baed062960f550b, duration=10 ms] 2024-09-11 18:40:13.061 INFO 2981023 --- [ocp-schedules-7,b3dfac4c0690d6d1,1803a28f2ee1d209] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:40:13.154 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726051140, nodeCount=1000, affectRows=1000 2024-09-11 18:40:13.157 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726051140, nodeCount=999, validCount=999 2024-09-11 18:40:13.174 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726051140, nodeCount=1000, affectRows=1000 2024-09-11 18:40:13.177 INFO 2981023 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726051140, nodeCount=1000, validCount=1000 2024-09-11 18:40:13.183 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1726051140, nodeCount=48, affectRows=48 2024-09-11 18:40:13.183 INFO 2981023 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1726051140, nodeCount=46, validCount=46 2024-09-11 18:40:13.519 INFO 2981023 --- [ocp-async-3,a935511ec9bc6e77,15d69ed3c5e98424] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:13.768 INFO 2981023 --- [ocp-schedules-29,857f137e4f41a295,9ff05eb61dc95d32] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:40:13.825 WARN 2981023 --- [scanner-13712168-1,857f137e4f41a295,766308b58ad50a5f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:40:14.313 INFO 2981023 --- [prometheus-minute-schedule-1,ddebc42b101f2982,7fbad5955253cd63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:14.314 INFO 2981023 --- [prometheus-hour-schedule-1,96652dbe0ce19e63,d480e6a94a41f70f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726052400, interval=3600 2024-09-11 18:40:14.316 INFO 2981023 --- [prometheus-minute-schedule-1,ddebc42b101f2982,7fbad5955253cd63] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:15.519 INFO 2981023 --- [ocp-async-2,634149ce17b364e9,7e7b3f6eb1725ba9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:15.575 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1726051140, nodeCount=480, affectRows=480 2024-09-11 18:40:15.588 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:15.589 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:16.021 INFO 2981023 --- [ocp-schedules-2,a38ea32908bc39ad,2622b220d39da947] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:40:16.023 INFO 2981023 --- [ocp-schedules-23,80dd5302444aad9f,8fe9886b148fd702] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:40:16.024 INFO 2981023 --- [host-check-clockdiff1,a38ea32908bc39ad,66b0b0c81bb95fb9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.5], timeoutMillis=20000 2024-09-11 18:40:16.024 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:40:16.030 INFO 2981023 --- [ocp-schedules-28,40ce3e22e8feb0f4,f4d754252d15d8b1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:40:16.030 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:40:16.031 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:40:16.031 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:40:16.032 INFO 2981023 --- [host-check-clockdiff2,a38ea32908bc39ad,376d83c0e4067a7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.6], timeoutMillis=20000 2024-09-11 18:40:16.033 INFO 2981023 --- [host-check-clockdiff3,a38ea32908bc39ad,2909ec2bbe779dc5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.10], timeoutMillis=20000 2024-09-11 18:40:16.033 INFO 2981023 --- [host-check-clockdiff4,a38ea32908bc39ad,c75dda1000eeae17] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.31.35.7], timeoutMillis=20000 2024-09-11 18:40:16.046 INFO 2981023 --- [host-check-agent-status1,40ce3e22e8feb0f4,c12e0840c433a6d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.31.35.5, originalStatus=ONLINE 2024-09-11 18:40:16.047 INFO 2981023 --- [host-check-agent-status2,40ce3e22e8feb0f4,19d6c35f579b1423] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000007, ip=10.31.35.6, originalStatus=ONLINE 2024-09-11 18:40:16.047 INFO 2981023 --- [host-check-agent-status3,40ce3e22e8feb0f4,17364da75c26b15d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000009, ip=10.31.35.10, originalStatus=ONLINE 2024-09-11 18:40:16.047 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:40:16.047 INFO 2981023 --- [host-check-agent-status4,40ce3e22e8feb0f4,fa60ba0cfac4ead3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2000010, ip=10.31.35.7, originalStatus=ONLINE 2024-09-11 18:40:16.056 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:40:16.061 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:40:16.064 INFO 2981023 --- [host-check-clockdiff3,a38ea32908bc39ad,2909ec2bbe779dc5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051216 0 0 , error= 2024-09-11 18:40:16.064 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:40:16.066 INFO 2981023 --- [host-check-agent-status1,40ce3e22e8feb0f4,c12e0840c433a6d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:40:16.068 INFO 2981023 --- [host-check-agent-status4,40ce3e22e8feb0f4,fa60ba0cfac4ead3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:40:16.069 INFO 2981023 --- [host-check-agent-status4,40ce3e22e8feb0f4,fa60ba0cfac4ead3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:40:16.069 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:40:16.070 INFO 2981023 --- [host-check-agent-status2,40ce3e22e8feb0f4,19d6c35f579b1423] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:40:16.074 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:40:16.074 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:40:16.075 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:40:16.076 INFO 2981023 --- [host-check-agent-status2,40ce3e22e8feb0f4,19d6c35f579b1423] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:40:16.172 INFO 2981023 --- [host-check-agent-status3,40ce3e22e8feb0f4,17364da75c26b15d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:40:16.172 INFO 2981023 --- [host-check-agent-status1,40ce3e22e8feb0f4,c12e0840c433a6d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:40:16.183 INFO 2981023 --- [host-check-agent-status3,40ce3e22e8feb0f4,17364da75c26b15d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:40:16.194 INFO 2981023 --- [ocp-schedules-5,5b6965040bae41d9,3302c14ad0b93a4e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-09-11 18:40:16.194 INFO 2981023 --- [ocp-schedules-1,c294c85e5d0e939c,e48def6d155d142b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-09-11 18:40:16.198 INFO 2981023 --- [ocp-schedules-22,170d62437e7cf52f,9e0320e0a13ab046] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-09-11 18:40:16.204 INFO 2981023 --- [ocp-schedules-9,e5556db4348bb908,57514683169d1211] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-09-11 18:40:16.278 INFO 2981023 --- [ocp-schedules-5,5b6965040bae41d9,3302c14ad0b93a4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:40:16.278 INFO 2981023 --- [ocp-schedules-1,c294c85e5d0e939c,e48def6d155d142b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:40:16.284 INFO 2981023 --- [ocp-schedules-22,170d62437e7cf52f,9e0320e0a13ab046] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2, 1000006] 2024-09-11 18:40:16.297 INFO 2981023 --- [ocp-schedules-21,c6b2ca8e6e9162af,72efb05ee3f8ac37] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2024-09-11 18:40:16.313 INFO 2981023 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051203627000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:40:16.325 INFO 2981023 --- [ocp-schedules-1,c294c85e5d0e939c,e48def6d155d142b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, tryCount=3, successCount=3 2024-09-11 18:40:16.362 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/time, params:null 2024-09-11 18:40:16.364 INFO 2981023 --- [host-check-agent-status2,40ce3e22e8feb0f4,19d6c35f579b1423] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/agent/status, params:null 2024-09-11 18:40:16.366 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000010, ip=10.31.35.7, timestamp=2024-09-11T18:40:16.363874015+08:00 2024-09-11 18:40:16.367 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000010, lastAvailableTime=2024-09-11T18:40:16.366+08:00 2024-09-11 18:40:16.369 INFO 2981023 --- [host-check-agent-status2,40ce3e22e8feb0f4,19d6c35f579b1423] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.6 2024-09-11 18:40:16.369 INFO 2981023 --- [host-check-agent-status2,40ce3e22e8feb0f4,19d6c35f579b1423] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.6 2024-09-11 18:40:16.369 INFO 2981023 --- [host-check-agent-status2,40ce3e22e8feb0f4,19d6c35f579b1423] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.6 2024-09-11 18:40:16.373 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000010 last available time 2024-09-11T18:40:16.366+08:00 and status AVAILABLE 2024-09-11 18:40:16.379 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/time, params:null 2024-09-11 18:40:16.381 INFO 2981023 --- [host-check-agent-status4,40ce3e22e8feb0f4,fa60ba0cfac4ead3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62888/api/v1/agent/status, params:null 2024-09-11 18:40:16.381 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.31.35.5, timestamp=2024-09-11T18:40:16.381481538+08:00 2024-09-11 18:40:16.382 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2024-09-11T18:40:16.381+08:00 2024-09-11 18:40:16.384 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-09-11T18:40:16.381+08:00 and status AVAILABLE 2024-09-11 18:40:16.385 INFO 2981023 --- [host-check-agent-status4,40ce3e22e8feb0f4,fa60ba0cfac4ead3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.7 2024-09-11 18:40:16.385 INFO 2981023 --- [host-check-agent-status4,40ce3e22e8feb0f4,fa60ba0cfac4ead3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.7 2024-09-11 18:40:16.385 INFO 2981023 --- [host-check-agent-status4,40ce3e22e8feb0f4,fa60ba0cfac4ead3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.7 2024-09-11 18:40:16.386 INFO 2981023 --- [host-check-agent-status3,40ce3e22e8feb0f4,17364da75c26b15d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/agent/status, params:null 2024-09-11 18:40:16.387 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62888/api/v1/time, params:null 2024-09-11 18:40:16.387 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000007, ip=10.31.35.6, timestamp=2024-09-11T18:40:16.392711302+08:00 2024-09-11 18:40:16.389 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000010 2024-09-11 18:40:16.389 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000007, lastAvailableTime=2024-09-11T18:40:16.387+08:00 2024-09-11 18:40:16.393 INFO 2981023 --- [host-check-agent-status1,40ce3e22e8feb0f4,c12e0840c433a6d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62888/api/v1/agent/status, params:null 2024-09-11 18:40:16.393 INFO 2981023 --- [host-check-agent-status3,40ce3e22e8feb0f4,17364da75c26b15d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.10 2024-09-11 18:40:16.393 INFO 2981023 --- [host-check-agent-status3,40ce3e22e8feb0f4,17364da75c26b15d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.10 2024-09-11 18:40:16.393 INFO 2981023 --- [host-check-agent-status3,40ce3e22e8feb0f4,17364da75c26b15d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.10 2024-09-11 18:40:16.395 INFO 2981023 --- [host-check-agent-status1,40ce3e22e8feb0f4,c12e0840c433a6d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.31.35.5 2024-09-11 18:40:16.395 INFO 2981023 --- [host-check-agent-status1,40ce3e22e8feb0f4,c12e0840c433a6d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.31.35.5 2024-09-11 18:40:16.395 INFO 2981023 --- [host-check-agent-status1,40ce3e22e8feb0f4,c12e0840c433a6d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.31.35.5 2024-09-11 18:40:16.396 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000007 last available time 2024-09-11T18:40:16.387+08:00 and status AVAILABLE 2024-09-11 18:40:16.398 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2024-09-11 18:40:16.409 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.410 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62888/api/v1/time, params:null 2024-09-11 18:40:16.410 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2000009, ip=10.31.35.10, timestamp=2024-09-11T18:40:16.410713346+08:00 2024-09-11 18:40:16.412 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:40:16.411636+08:00, lastProcessTime=2024-09-11T18:40:06.353210+08:00 2024-09-11 18:40:16.412 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:40:16.413 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2000009, lastAvailableTime=2024-09-11T18:40:16.411+08:00 2024-09-11 18:40:16.417 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2000009 last available time 2024-09-11T18:40:16.411+08:00 and status AVAILABLE 2024-09-11 18:40:16.454 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:40:16.454 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000007 2024-09-11 18:40:16.455 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:40:16.460 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.461 INFO 2981023 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:40:16.463 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2000009 2024-09-11 18:40:16.464 INFO 2981023 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.2 2024-09-11 18:40:16.465 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:40:16.467 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.467 INFO 2981023 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:40:16.467 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:40:16.468 INFO 2981023 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.1.0 2024-09-11 18:40:16.470 INFO 2981023 --- [ocp-schedules-5,5b6965040bae41d9,3302c14ad0b93a4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:40:16.473 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:40:16.474 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:40:16.475 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:40:16.475 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:40:16.478 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.479 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:40:16.486 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:40:16.488 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.489 INFO 2981023 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:40:16.502 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:40:16.506 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:40:16.507 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000058 2024-09-11 18:40:16.508 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-09-11 18:40:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:16.521 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.521 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000058 2024-09-11 18:40:16.523 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:40:16.528 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000052 2024-09-11 18:40:16.533 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000055 2024-09-11 18:40:16.539 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000049 2024-09-11 18:40:16.555 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000052 2024-09-11 18:40:16.565 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000049 2024-09-11 18:40:16.567 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:40:16.569 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:16.570 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.570 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:40:16.571 WARN 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:40:16.571 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:40:16.574 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:40:16.575 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:40:16.578 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000055 2024-09-11 18:40:16.588 INFO 2981023 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=1000006, status=OPERATING 2024-09-11 18:40:16.588 INFO 2981023 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000006, name=linkrtd, obClusterId=1719994443, vpcId=1, vpcName=system-default, type=PRIMARY, status=OPERATING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"linkrtd","ObClusterId":1719994443,"ObRegion":"linkrtd","ObRegionId":1719994443,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.6:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051203627000}, createTime=2024-09-11T15:15:31+08:00, creator=admin, obVersion=4.2.4.0, tenantCount=1) 2024-09-11 18:40:16.588 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000059 2024-09-11 18:40:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:40:16.589 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:40:16.599 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000059 2024-09-11 18:40:16.602 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000053 2024-09-11 18:40:16.603 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:40:16.609 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-09-11 18:40:16.616 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000053 2024-09-11 18:40:16.620 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000027, ip=10.31.35.10, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.2_102010022023121415-f6752332b5001069034f96c6dbb620926dc666f0(Dec 14 2023 15:39:00), withRootserver=true, hostId=2000009, status=RUNNING, innerStatus=ACTIVE, startTime=2024-07-11T11:19:55.246806+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:40:16.625 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.7:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:40:16.631 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:16.632 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.633 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:40:16.634 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:16.634 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster linkprdocp 2024-09-11 18:40:16.635 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.636 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-09-11 18:40:16.637 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 2024-09-11 18:40:16.638 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000050 2024-09-11 18:40:16.638 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000056 2024-09-11 18:40:16.645 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:40:16.646 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000060 2024-09-11 18:40:16.647 INFO 2981023 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:40:16.648 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000050 2024-09-11 18:40:16.651 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000056 2024-09-11 18:40:16.652 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:40:16.656 INFO 2981023 --- [host-check-host-status4,80dd5302444aad9f,c47704a7f66fd363] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000060 2024-09-11 18:40:16.665 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:16.665 INFO 2981023 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:40:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.666 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-09-11 18:40:16.667 WARN 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-09-11 18:40:16.667 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.670 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-09-11 18:40:16.672 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.6:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:40:16.674 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-09-11 18:40:16.676 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000054 2024-09-11 18:40:16.684 INFO 2981023 --- [host-check-host-status1,80dd5302444aad9f,18c84b56331bf627] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000054 2024-09-11 18:40:16.686 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.10:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-09-11 18:40:16.688 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:40:16.688 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00} 2024-09-11 18:40:16.699 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000051 2024-09-11 18:40:16.708 INFO 2981023 --- [host-check-host-status2,80dd5302444aad9f,830dad430a3d3806] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000051 2024-09-11 18:40:16.711 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.31.35.5, port=2882, sqlPort=2881, installPath=/data/rtdob/oceanbase, runPath=/data/rtdob/oceanbase, runUser=root, version=4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 15:42:26), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2024-06-18T13:49:20.377309+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-09-11 18:40:16.722 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2000057 2024-09-11 18:40:16.726 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.731 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-09-11 18:40:16.732 INFO 2981023 --- [ocp-schedules-26,1d8857f024441e0a,4dbeb32f1b03f948] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:40:16.740 INFO 2981023 --- [host-check-host-status3,80dd5302444aad9f,9c03d3007dcc42d4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2000057 2024-09-11 18:40:16.744 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster rtdob 2024-09-11 18:40:16.745 INFO 2981023 --- [ocp-schedules-26,1d8857f024441e0a,4dbeb32f1b03f948] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:40:16.745 INFO 2981023 --- [ocp-schedules-26,1d8857f024441e0a,4dbeb32f1b03f948] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:40:16.749 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2024-09-11 18:40:16.753 INFO 2981023 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2000009, sqlPort: 2881 got ObProcessInfoResult(pid=10066, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-09-11 18:40:16.755 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:40:16.756 INFO 2981023 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:40:16.761 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=linkprdocp, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:16.764 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.764 INFO 2981023 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:40:16.771 INFO 2981023 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.31.35.5:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-09-11 18:40:16.775 INFO 2981023 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=linkprdocp, obClusterId=1719994437 2024-09-11 18:40:16.775 INFO 2981023 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=linkprdocp, obClusterId=1719994437, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"linkprdocp","ObClusterId":1719994437,"ObRegion":"linkprdocp","ObRegionId":1719994437,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.10:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051189296000}, createTime=2024-07-03T16:25:58+08:00, creator=admin, obVersion=4.2.1.2, tenantCount=3) 2024-09-11 18:40:16.782 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:40:16.411636+08:00 2024-09-11 18:40:16.804 INFO 2981023 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=85317, runPath=/data/rtdob/oceanbase, runUser=root, execute=/data/rtdob/oceanbase/bin/observer, installPath=/data/rtdob/oceanbase) 2024-09-11 18:40:16.810 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=rtdob, tenantName=sys, dbUser=ocp_monitor 2024-09-11 18:40:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-09-11 18:40:16.811 INFO 2981023 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-09-11 18:40:16.817 INFO 2981023 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=rtdob, obClusterId=1718689631 2024-09-11 18:40:16.817 INFO 2981023 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=rtdob, obClusterId=1718689631, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"rtdob","ObClusterId":1718689631,"ObRegion":"rtdob","ObRegionId":1718689631,"ReadonlyRsList":[],"RsList":[{"address":"10.31.35.5:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1726051212662000}, createTime=2024-07-03T16:57:29+08:00, creator=admin, obVersion=4.3.1.0, tenantCount=2) 2024-09-11 18:40:16.817 INFO 2981023 --- [ocp-schedules-22,170d62437e7cf52f,9e0320e0a13ab046] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, tryCount=3, successCount=3 2024-09-11 18:40:16.975 INFO 2981023 --- [host-check-clockdiff1,a38ea32908bc39ad,66b0b0c81bb95fb9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051216 0 0 , error= 2024-09-11 18:40:17.313 INFO 2981023 --- [prometheus-minute-schedule-1,18d26791c74473be,0707851cc0c37ea0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:17.316 INFO 2981023 --- [prometheus-minute-schedule-1,18d26791c74473be,0707851cc0c37ea0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:17.519 INFO 2981023 --- [ocp-async-5,052c50425f5e5606,686a057c865a710a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:17.806 INFO 2981023 --- [ocp-async-7,96234f434adb60cd,b8b00180786becae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:40:18.427 INFO 2981023 --- [ocp-schedules-19,8a8ee3eab2a71216,cbedca9c02a79db3] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 1000 2024-09-11 18:40:18.589 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:18.590 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:18.866 INFO 2981023 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1726051140, nodeCount=480, validCount=480 2024-09-11 18:40:19.519 INFO 2981023 --- [ocp-async-3,9c1819ccbd607189,0ff7d0b5179c6993] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:20.313 INFO 2981023 --- [prometheus-minute-schedule-1,2325de541cde0a24,319bd4aa07474725] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:20.317 INFO 2981023 --- [prometheus-minute-schedule-1,2325de541cde0a24,319bd4aa07474725] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:20.547 INFO 2981023 --- [host-check-clockdiff2,a38ea32908bc39ad,376d83c0e4067a7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051220 5 5 , error= 2024-09-11 18:40:21.519 INFO 2981023 --- [ocp-async-8,fa70337b57a72f3d,11a6391b3956b9bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:21.590 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:21.592 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:22.025 INFO 2981023 --- [host-check-clockdiff4,a38ea32908bc39ad,c75dda1000eeae17] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1726051222 0 0 , error= 2024-09-11 18:40:23.064 INFO 2981023 --- [ocp-schedules-8,20af7be8ff56eb6c,e11f000ab7de3e8e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:40:23.313 INFO 2981023 --- [prometheus-minute-schedule-1,8ee7055180130fa4,95b19666c95f51ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:23.316 INFO 2981023 --- [prometheus-minute-schedule-1,8ee7055180130fa4,95b19666c95f51ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:23.519 INFO 2981023 --- [ocp-async-6,a4d359aa894d5b9d,44ec30587c8dc2f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:23.977 INFO 2981023 --- [ocp-schedules-18,a3ba647859c1d96e,52764921cb5ca945] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:40:24.034 WARN 2981023 --- [scanner-13712327-1,a3ba647859c1d96e,be9257996aa65611] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:40:24.064 INFO 2981023 --- [alarm-detect-8,a3ba647859c1d96e,cc9a6486c389927b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8647013, data is empty. 2024-09-11 18:40:24.071 INFO 2981023 --- [alarm-detect-8,a3ba647859c1d96e,cc9a6486c389927b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8647013, data is empty. 2024-09-11 18:40:24.076 INFO 2981023 --- [alarm-detect-8,a3ba647859c1d96e,cc9a6486c389927b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 8647013, data is empty. 2024-09-11 18:40:24.118 INFO 2981023 --- [alarm-detect-10,a3ba647859c1d96e,4adb337002b99e19] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8647013, data is empty. 2024-09-11 18:40:24.119 INFO 2981023 --- [alarm-detect-14,a3ba647859c1d96e,aff09be3efb380ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8647013, data is empty. 2024-09-11 18:40:24.124 INFO 2981023 --- [alarm-detect-10,a3ba647859c1d96e,4adb337002b99e19] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8647013, data is empty. 2024-09-11 18:40:24.128 INFO 2981023 --- [alarm-detect-14,a3ba647859c1d96e,aff09be3efb380ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8647013, data is empty. 2024-09-11 18:40:24.131 INFO 2981023 --- [alarm-detect-10,a3ba647859c1d96e,4adb337002b99e19] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 8647013, data is empty. 2024-09-11 18:40:24.134 INFO 2981023 --- [alarm-detect-14,a3ba647859c1d96e,aff09be3efb380ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 8647013, data is empty. 2024-09-11 18:40:24.205 INFO 2981023 --- [alarm-detect-20,a3ba647859c1d96e,c9733c55e702c0a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8647013, data is empty. 2024-09-11 18:40:24.211 INFO 2981023 --- [alarm-detect-20,a3ba647859c1d96e,c9733c55e702c0a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8647013, data is empty. 2024-09-11 18:40:24.215 INFO 2981023 --- [alarm-detect-20,a3ba647859c1d96e,c9733c55e702c0a1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 8647013, data is empty. 2024-09-11 18:40:24.576 INFO 2981023 --- [ocp-schedules-4,1737c740acec4ec9,e470cd514a042a1a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-09-11 18:40:24.592 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:24.592 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:25.520 INFO 2981023 --- [ocp-async-4,82b3567c0ab40eec,0790cd5dd26c75b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:26.313 INFO 2981023 --- [prometheus-minute-schedule-1,078c9322029da4ea,2978a1e39dc5b643] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:26.316 INFO 2981023 --- [prometheus-minute-schedule-1,078c9322029da4ea,2978a1e39dc5b643] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:26.352 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:40:26.351713+08:00, lastProcessTime=2024-09-11T18:40:16.411636+08:00 2024-09-11 18:40:26.489 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:40:26.351713+08:00 2024-09-11 18:40:27.520 INFO 2981023 --- [ocp-async-3,f89e7d716bd10174,578b2632ba564f10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:27.592 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:27.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:27.806 INFO 2981023 --- [ocp-async-5,e463f16ebb5a2a8c,a52f290a34dd6f51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:40:29.313 INFO 2981023 --- [prometheus-minute-schedule-1,3c8f33db7d558746,46228f4048c4279c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:29.316 INFO 2981023 --- [prometheus-minute-schedule-1,3c8f33db7d558746,46228f4048c4279c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:29.520 INFO 2981023 --- [ocp-async-10,d50d4d0bec5e83a9,a9d968a178716327] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:29.675 INFO 2981023 --- [ocp-schedules-26,896784f3b480f0b3,e35032346696fe6c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.31.35.5, 10.31.35.6, 10.31.35.10, 10.31.35.7] 2024-09-11 18:40:29.676 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000007 2024-09-11 18:40:29.676 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2024-09-11 18:40:29.676 INFO 2981023 --- [ocp-schedules-26,896784f3b480f0b3,e35032346696fe6c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000009 2024-09-11 18:40:29.676 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2000010 2024-09-11 18:40:29.700 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000007 2024-09-11 18:40:29.703 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000007 2024-09-11 18:40:29.703 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-09-11 18:40:29.704 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-09-11 18:40:29.705 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000010 2024-09-11 18:40:29.707 INFO 2981023 --- [ocp-schedules-26,896784f3b480f0b3,e35032346696fe6c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2000009 2024-09-11 18:40:29.707 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000010 2024-09-11 18:40:29.711 INFO 2981023 --- [ocp-schedules-26,896784f3b480f0b3,e35032346696fe6c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2000009 2024-09-11 18:40:29.965 INFO 2981023 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.6:62889/api/v1/log/alarms, params:null 2024-09-11 18:40:29.966 INFO 2981023 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.7:62889/api/v1/log/alarms, params:null 2024-09-11 18:40:29.966 INFO 2981023 --- [ocp-schedules-26,896784f3b480f0b3,e35032346696fe6c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.10:62889/api/v1/log/alarms, params:null 2024-09-11 18:40:29.967 INFO 2981023 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.31.35.5:62889/api/v1/log/alarms, params:null 2024-09-11 18:40:29.968 INFO 2981023 --- [ocp-schedules-26,896784f3b480f0b3,e35032346696fe6c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-09-11 18:40:30.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:30.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:31.520 INFO 2981023 --- [ocp-async-1,59c6ae8506aa2b17,596c45e06188b869] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:32.313 INFO 2981023 --- [prometheus-minute-schedule-1,7ee66737d994e26b,d1a0f7403e07b1c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:32.315 INFO 2981023 --- [prometheus-minute-schedule-1,7ee66737d994e26b,d1a0f7403e07b1c8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:33.067 INFO 2981023 --- [ocp-schedules-4,28403738c9838c3a,c9e591df3d61203e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-09-11 18:40:33.520 INFO 2981023 --- [ocp-async-2,9f93f361c4417cfe,c83a5cfa88455399] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:33.593 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:33.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:34.254 INFO 2981023 --- [ocp-schedules-29,c26434adc6b30105,e3f183f058619be7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-09-11 18:40:34.294 WARN 2981023 --- [scanner-13712660-1,c26434adc6b30105,74fcb4584cf515e7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 32 2024-09-11 18:40:35.313 INFO 2981023 --- [prometheus-minute-schedule-1,d60db84e545a5dd9,2736356aa54707bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:35.316 INFO 2981023 --- [prometheus-minute-schedule-1,d60db84e545a5dd9,2736356aa54707bc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:35.520 INFO 2981023 --- [ocp-async-8,45da3759f242ea69,e40e4b1def0bd167] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:36.337 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2024-09-11T18:40:36.336712+08:00, lastProcessTime=2024-09-11T18:40:26.351713+08:00 2024-09-11 18:40:36.492 INFO 2981023 --- [alarm-task-1,4fdb9ad9399f4082,69616936f4f5fa75] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2024-09-11T18:40:36.336712+08:00 2024-09-11 18:40:36.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:36.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:36.748 INFO 2981023 --- [ocp-schedules-27,50e7fda650f793c0,7cfa9de5399b23ae] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-09-11 18:40:36.756 INFO 2981023 --- [ocp-schedules-27,50e7fda650f793c0,7cfa9de5399b23ae] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:40:36.756 INFO 2981023 --- [ocp-schedules-27,50e7fda650f793c0,7cfa9de5399b23ae] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[15e56163-77af-4ad5-8387-c46f238ccb39] 2024-09-11 18:40:37.314 INFO 2981023 --- [prometheus-cache-monitor1,4e1674a3362658e2,a63728c45d788aff] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":8193,"minuteWriteQueue":0,"minuteReadCacheSize":5022,"secondWriteCacheSize":2048,"seriesIdKeySize":10273,"secondWriteQueue":0,"minuteWriteCacheSize":5150} 2024-09-11 18:40:37.315 INFO 2981023 --- [prometheus-cache-monitor1,4e1674a3362658e2,a63728c45d788aff] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sql_event_in_plsql_compilation=8, ob_sysstat_40012=32, ob_server_resource_memory_assigned_bytes=24, ob_sysstat_40011=32, node_memory_sreclaimable_bytes=31, ob_sysstat_40010=32, ob_unit_num=22, tcp_retrans=29, ob_database_disk_required_size=58, go_memstats_mspan_sys_bytes=21, ob_sysstat_30213=26, go_memstats_stack_sys_bytes=21, ob_sysstat_30212=26, exporter_export_metrics_bytes_total=30, ob_sysstat_30217=26, node_filesystem_size_bytes=94, ob_sysstat_30216=26, http_server_requests_seconds_count=166, ob_tenant_request_queue_total_size=37, jvm_gc_memory_allocated_bytes_total=2, ob_sysstat_200013=26, ob_compaction_frozen_time=29, ob_system_event_total_waits=42, ob_sysstat_200010=26, monagent_input_sql_duration_seconds_sum=70, ob_tenant_disk_v4_log_disk_size=29, log_stream_degraded_count=31, ob_sysstat_40019=34, ob_sysstat_40018=34, processor_process_metrics_total=27, ob_plan_cache_hit_total=31, jvm_gc_max_data_size_bytes=2, node_filesystem_readonly=94, go_memstats_frees_total=21, io_await=81, host_available=22, ob_sysstat_30220=26, ob_sysstat_30222=26, ob_sql_event_in_sequence_load=16, ob_sysstat_30221=26, go_memstats_mallocs_total=21, mysql_output_write_sql_failed_count=17, ob_unit_config_max_memory_bytes=54, ob_sysstat_200006=32, ob_database_disk_data_size=55, io_util=81, ob_sysstat_200005=32, ob_sysstat_200002=32, ob_sysstat_200001=32, ob_tenant_memstore_freeze_cnt=23, ob_tenant_disk_v4_data_size=29, ob_tenant_unit_min_memory=44, ob_sysstat_60087=32, ob_memstore_total_bytes=37, metric_cardinality_count=25, jvm_threads_states_threads=8, process_resident_memory_bytes=21, ob_cache_size_bytes=49, ob_tenant_assigned_cpu_total=31, ob_tenant_unit_max_memory=44, input_collect_metrics_total=27, ob_sysstat_50039=32, ob_sysstat_60083=32, ob_sysstat_50038=32, node_boot_time_seconds=28, node_disk_io_time_weighted_seconds_total=47, node_disk_write_time_seconds_total=47, cpu_count=29, output_write_metrics_total=19, ob_cluster_sync=20, process_open_fds=21, ob_sysstat_50037=32, process_memory_rss_bytes=34, htt 2024-09-11 18:40:37.520 INFO 2981023 --- [ocp-async-10,4bbe553b5080508b,2ee3a3b5b24e0a74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:37.806 INFO 2981023 --- [ocp-async-3,ffd6b21b41ab9e40,c37127768bffcd17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-09-11 18:40:38.313 INFO 2981023 --- [prometheus-minute-schedule-1,b41f8b1506b0ae94,31bcf7f316358f13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:38.316 INFO 2981023 --- [prometheus-minute-schedule-1,b41f8b1506b0ae94,31bcf7f316358f13] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:39.521 INFO 2981023 --- [ocp-async-9,cad1463b1d7abdda,9a8e75c60080356a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-09-11 18:40:39.594 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.31.35.10:8080 refreshes policies... 2024-09-11 18:40:39.595 INFO 2981023 --- [pool-12-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2024-09-11 18:40:40.003 INFO 2981023 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2024-09-11 18:40:40.003 INFO 2981023 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2024-09-11 18:40:41.313 INFO 2981023 --- [prometheus-minute-schedule-1,aa9b77c3fcaec1d5,b58cc973fc103f74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1726051260, interval=60 2024-09-11 18:40:41.316 INFO 2981023 --- [prometheus-minute-schedule-1,aa9b77c3fcaec1d5,b58cc973fc103f74] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=10, collectTaskCount=10 2024-09-11 18:40:41.521 INFO 2981023 --- [ocp-async-6,73527e8bbf4c598b,03f287f9104698f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents